Limitations at scale?

Come here for help or to post comments on Sphider
Post Reply
kraisor
Posts: 4
Joined: Wed Dec 16, 2020 4:03 pm

Limitations at scale?

Post by kraisor »

Hey again,

I'm looking at using this again (very much appreciate the fact that you've done just a great job keeping it updated!) and wanted to get your thoughts on the upper limitations.

My plan involves crawling and indexing multiple domains with a few million IRLs, do you feel that the software would be able to keep up with that, including vreacans where content gets updated frequently?

Thanks
kraisor
Posts: 4
Joined: Wed Dec 16, 2020 4:03 pm

Re: Limitations at scale?

Post by kraisor »

I forgot to ask, how is Spider doing currently with JavaScript rendering? Some of the sites I'd like to index were built with JavaScript.
User avatar
captquirk
Site Admin
Posts: 299
Joined: Sun Apr 09, 2017 8:49 pm
Location: Arizona, USA
Contact:

Re: Limitations at scale?

Post by captquirk »

First, sorry for the long delay at responding. I was having some health issues.

Sphider does not do very well at indexing JavaScript generated content.

As to scalability, Sphider's main intent is for indexing personal sites. HOWEVER, you can index many websites. I have no idea what the upper limits may be, but will most likely depending on the machine setup... available disk space, MySQL settings regarding memory, swap file sizes, etc. I have successfully indexed as many as 25 sites (of varying sizes - tiny to OMG!) and stayed functioning. You get really big and you may notice it functions slower, especially if you perform a maintenance function. Backups were a concern at one time, but I THINK I have that straightened out. (One can hope, anyway.)
Post Reply