Page 2 of 2

Re: .com search at Sphider 1.42

Posted: Thu Jan 31, 2019 3:20 am
by captquirk
I mean to tell Sphider he has to stop when fetch 40 or 50 links from a domain.
In PDO 2.2.0, in spider.php, line 1009:

Code: Select all

while ($count < count($links)) {
You can try:

Code: Select all

while ($count < count($links) && $num < 50) {
I HAVE NOT TESTED THIS! NO GUARANTEE! It SEEMS like a good idea though. :lol:

Re: .com search at Sphider 1.42

Posted: Thu Jan 31, 2019 5:57 pm
by exploro
As far as eliminating advance options for an image search
I make the modifications as per your instructions and It's working. The Image Search tab looks exactly just like the web Search tab!
I HAVE NOT TESTED THIS! NO GUARANTEE! It SEEMS like a good idea though.
Your code for limiting the number of links Sphider fetch from each domain works just fine! Still have both versions running. Sphider 1.42 and Sphider PDO 2.2.0 with two separate databases. At the end I guess I will keep the PDO version only. I might come up with more requests for mods :)
Thanks for all your help in getting things running. I will look forward for the next update of the PDO version.

Re: .com search at Sphider 1.42

Posted: Thu Jan 31, 2019 10:47 pm
by captquirk
Your code for limiting the number of links Sphider fetch from each domain works just fine!
That's good. It has occurred to me that you may want to keep an eye on the 'temp' and 'pending' tables. Since indexing may stop before ALL the links are processed, these tables may contain "leftovers". "Clear temp table" on the Clean tables tab will clean up the 'temp' table. The 'pending' table is another matter. On the Database tab, you will be able to see if there is anything in the 'pending' table. If there is, you can 'check' the box next to 'pending', then "Truncate (empty) selected tables" button near the bottom. Be careful you are checkmarking ONLY the 'pending' table! Truncate can be a dangerous tool if you don't have a backup.

Re: .com search at Sphider 1.42

Posted: Fri Feb 01, 2019 9:28 pm
by exploro
You are right about the 'temp' and 'pending' tables. It happened to me lately and was wondering why indexing was stopped. Your tips and advice are very helpful for both novice and advanced users of Sphider.