Sphider Help Forum

Your database does not seem to be the issue. With web pages, it is possible for a Unicode character to appear correctly in a browser window, BUT the source code behind that page is not Unicode, but a replacement. For example: Browser displays: Д Source code is: Д Another possible issue is ...

This issue has been resolved. It turns out that Sphider uses the PHP exec() function to run the pdftotext converter Some Linux installations, for security reasons, block some functions such as exec(). A slight change to php.ini allowed exec() to function and pdf's were then indexed. As I said, the d...

Check this: viewtopic.php?p=547#p547 While not a FINAL solution, user-agent: Sphider Allow: / should now allow access, disregarding all the disallows in user-agent: *. Any desired disallows need to be added to user_agent: Sphider, even if they are duplicates of some disallows in user-agent: *. Feedb...

This is a partial solution to fixing checkRobotsTxt() function. It reads the robots.txt file, considering both the * user-agent and the Sphider user-agent, both allows and disallows. It produces a master array of denys and allows which are compiled based on the rules mentioned in a previous post. Th...

It is possible for a particular database to have a character set and collation DIFFERENT than defaults. Before giving up completely, let's be sure that character set really is the issue. Both the "install.php" and manual "tables.sql" provided make every effort to be utf8mb4! Go t...

The checkRobotsTxt() function in Sphider is deficient. It is not case sensitive, but that is a minor problem easily corrected. Of more major cancern is the lack of support for the Allow directive. I have gathered some thoughts on what needs to be done and would appreciate any comment or suggestion a...

Yes, Sphider can index PDF files. However, the translation of PDF to text is not native to Sphider. That is done by a utility, pdftotext. This is pretty much a standard executable in Linux based systems, typically residing at /usr/bin/pdftotext. On a Windows system, pdftotext.exe is NOT present by d...

Yes. the utf8mb4 is the correct encoding. As for collation, I tend to use utf8mb4_general_ci, BUT --- it really doesn't matter that much. Different collations may present different sorting, but the IMPORTANT thing is that it be utf8mb4!!! Why MySQL doesn't have full 4 byte UTF8 encoding as a default...

Sphider is fully Unicode capable. Example: If you index a Russian web site, then want to search for "грузовик", are you actually searching for "грузовик", or some numeric Unicode equivalent? Also, check your database. Is the default character set UTF8_mb4, or just UTF8? MySql def...

I tried you mod out and there are issues. Using the 5.3.0 checkRobotsTxt(): Disallowed files and directories in robots.txt: https://sphider.worldspaceflight.com/contact/ https://sphider.worldspaceflight.com/include/ https://sphider.worldspaceflight.com/download/ https://sphider.worldspaceflight.com/...