Sphider Help Forum

Many times problems with indexing a site can be prevented... at least when indexing sites over which you have control. Here are three tips to improve the odds of proper indexing. Be consistent with the use of "http" and "https" in your links. While browsers are much more forgivin...

Thank you!

Trying to do anything to a individual table (mainly deletes) is difficult to impossible. Sphider tables are linked to each other using keys. For example, the sites table is linked to the links table by way of a key. The links table is attached to the link-keyword tables in the same way. Only through...

Is it possible on the Site's "Edit" page that either "Sphider can leave domain" or "Index foreign images" is checked? Also, on the "Clean tables" tab, run "Clean domains". This will remove any domains which are not in active use. This will not clean ...

I understand the issue. The database is properly formatted on your local machine. Unfortunately, during the export/import operation, Unicode characters get corrupted. That is why your remote site gives improper results. I have yet to find a definitive solution to this issue. This has happened to me ...

For anyone curious as to the outcome, an issue was found in the PHP configuration. "Allow_url_fopen" was set to "Off". Changing this value to "On" resolved many issues.
This can be done either by editing the php.ini file, or through CPanel.

Presuming the main issue is 500 errors stalling the indexing, the command line should do the trick. Then it is just a matter of getting the right stuff in the sitemap.xml.

Commandline from sphider/admin:
php spider.php -u https://harrymccracken.com/blog/ -f -s

I tried indexing the your blog using your sitemap. Results were partial success. By this I mean that I made two runs and both ran to completion. No stalls. So in this regards I suspect you are encountering the dreaded 500 error! The best work around for this is to index from a command prompt. BUT --...

Except for "id" and "language", are all of the other settings the same? Especially "stem_language". The minor changes you made shouldn't be causing this. However, if "stem_language" varies, that MIGHT be an issue. The settings snippet you provided does not sho...

The language selected ONLY affects the text on the search page. Therefore, no matter what the search language, Sphider SHOULD return the exact same results for any search term, regardless of the search page language. Obviously, it does not! I even confirmed that myself! I'm not sure what the devil i...