Sphider Help Forum

Sphider is fully Unicode capable. Example: If you index a Russian web site, then want to search for "грузовик", are you actually searching for "грузовик", or some numeric Unicode equivalent? Also, check your database. Is the default character set UTF8_mb4, or just UTF8? MySql def...

I tried you mod out and there are issues. Using the 5.3.0 checkRobotsTxt(): Disallowed files and directories in robots.txt: https://sphider.worldspaceflight.com/contact/ https://sphider.worldspaceflight.com/include/ https://sphider.worldspaceflight.com/download/ https://sphider.worldspaceflight.com/...

You are correct. Looking closer you can see the existing function ONLY looks for "disallows" and not allows". There is definitely room for improvement here. Also, thanks for your proposed improvements. I will look it over and may incorporate it, or some version of it, in a future (nex...

This is crazy. I tried the url and got that page, and NO LINKS FOUND! By email, lets see the full log (looks like it started but then hung...?), the setting screen, and the site advanced edit screen. Also see if there is anything relevant in the error logs. (As shipped, Sphider has logging turned on...

First off, Sphider will index https sites. It follows robots.txt. To allow Sphider, but disallow all other bots, try something like this: User-agent: * Disallow: / User-agent: Sphider (sphidersearch.com) Allow: / You could also go into the Settings tab and change the User Agent string to something y...

Permanent, automatic fix in Sphider 5.3.0, SphiderLite 2.4.0.

No more user code tweaking necessary!

The newest releases now REQUIRE the mbstring PHP extension. The mbstring functions are no longer emulated if missing. Jquery has been updated, code has been cleaned up to PSR-2 standards (except some special modules). On the advanced search screen, references to categories has been removed when cate...

I tried various things to isolate the issue, such as stemming off, stemming on, various settings changes... to no avail. Then I upgraded the CentOS Sphider from 5.2.1 to 5.3.0.... Here is the log file: Spidering http://localhost/ 1. Retrieving: http://localhost/ at 03:44:36. Size of page: 10.78kb. S...

Okay, I got Centos & working with Sphider 5.2.1. Log file: Spidering http://localhost/ 1. Retrieving: http://localhost/ at 01:24:59. Size of page: 10.78kb. Starting indexing at 01:24:59. Indexed Links found: 5. New links: 5 2. Retrieving: http://localhost/ at 01:25:00. already in database 3. Ret...

Thanks. Give me a bit... My VM of CentOS 7 had a dated version of PHP (now at least a reasonable 8.0), but my database version is also old and I am struggling to get it updated. I am used to Ubuntu, so CentOS is trying to intimidate me! I won't let it. Once that is done I can better simulate your se...