.com search at Sphider 1.42

Come here for help or to post comments on Sphider
User avatar
captquirk
Site Admin
Posts: 113
Joined: Sun Apr 09, 2017 8:49 pm
Location: Arizona, USA
Contact:

Re: .com search at Sphider 1.42

Post by captquirk » Thu Jan 31, 2019 3:20 am

I mean to tell Sphider he has to stop when fetch 40 or 50 links from a domain.
In PDO 2.2.0, in spider.php, line 1009:

Code: Select all

while ($count < count($links)) {
You can try:

Code: Select all

while ($count < count($links) && $num < 50) {
I HAVE NOT TESTED THIS! NO GUARANTEE! It SEEMS like a good idea though. :lol:

exploro
Posts: 6
Joined: Sun Jan 27, 2019 2:10 pm

Re: .com search at Sphider 1.42

Post by exploro » Thu Jan 31, 2019 5:57 pm

As far as eliminating advance options for an image search
I make the modifications as per your instructions and It's working. The Image Search tab looks exactly just like the web Search tab!
I HAVE NOT TESTED THIS! NO GUARANTEE! It SEEMS like a good idea though.
Your code for limiting the number of links Sphider fetch from each domain works just fine! Still have both versions running. Sphider 1.42 and Sphider PDO 2.2.0 with two separate databases. At the end I guess I will keep the PDO version only. I might come up with more requests for mods :)
Thanks for all your help in getting things running. I will look forward for the next update of the PDO version.

User avatar
captquirk
Site Admin
Posts: 113
Joined: Sun Apr 09, 2017 8:49 pm
Location: Arizona, USA
Contact:

Re: .com search at Sphider 1.42

Post by captquirk » Thu Jan 31, 2019 10:47 pm

Your code for limiting the number of links Sphider fetch from each domain works just fine!
That's good. It has occurred to me that you may want to keep an eye on the 'temp' and 'pending' tables. Since indexing may stop before ALL the links are processed, these tables may contain "leftovers". "Clear temp table" on the Clean tables tab will clean up the 'temp' table. The 'pending' table is another matter. On the Database tab, you will be able to see if there is anything in the 'pending' table. If there is, you can 'check' the box next to 'pending', then "Truncate (empty) selected tables" button near the bottom. Be careful you are checkmarking ONLY the 'pending' table! Truncate can be a dangerous tool if you don't have a backup.

exploro
Posts: 6
Joined: Sun Jan 27, 2019 2:10 pm

Re: .com search at Sphider 1.42

Post by exploro » Fri Feb 01, 2019 9:28 pm

You are right about the 'temp' and 'pending' tables. It happened to me lately and was wondering why indexing was stopped. Your tips and advice are very helpful for both novice and advanced users of Sphider.

Post Reply