Search found 302 matches
- Fri Sep 08, 2023 12:09 am
- Forum: Sphider Help
- Topic: Robots.txt - would allow support be useful for many
- Replies: 6
- Views: 4481
Re: Robots.txt - would allow support be useful for many
You are correct. Looking closer you can see the existing function ONLY looks for "disallows" and not allows". There is definitely room for improvement here. Also, thanks for your proposed improvements. I will look it over and may incorporate it, or some version of it, in a future (nex...
- Thu Sep 07, 2023 3:45 pm
- Forum: Sphider Help
- Topic: An issue with PDF indexing.
- Replies: 16
- Views: 8539
Re: An issue with PDF indexing.
This is crazy. I tried the url and got that page, and NO LINKS FOUND! By email, lets see the full log (looks like it started but then hung...?), the setting screen, and the site advanced edit screen. Also see if there is anything relevant in the error logs. (As shipped, Sphider has logging turned on...
- Thu Sep 07, 2023 2:43 am
- Forum: Sphider Help
- Topic: Robots.txt - would allow support be useful for many
- Replies: 6
- Views: 4481
Re: Robots.txt - would allow support be useful for many
First off, Sphider will index https sites. It follows robots.txt. To allow Sphider, but disallow all other bots, try something like this: User-agent: * Disallow: / User-agent: Sphider (sphidersearch.com) Allow: / You could also go into the Settings tab and change the User Agent string to something y...
- Mon Sep 04, 2023 6:56 pm
- Forum: Sphider Help
- Topic: Sphider 5.2.0 and SphiderLite 2.3.0 fail after install
- Replies: 1
- Views: 3287
Re: Sphider 5.2.0 and SphiderLite 2.3.0 fail after install
Permanent, automatic fix in Sphider 5.3.0, SphiderLite 2.4.0.
No more user code tweaking necessary!
No more user code tweaking necessary!
- Mon Sep 04, 2023 4:22 pm
- Forum: Announcements
- Topic: Sphider 5.3.0 and SphiderLite 2.4.0 released
- Replies: 0
- Views: 6661
Sphider 5.3.0 and SphiderLite 2.4.0 released
The newest releases now REQUIRE the mbstring PHP extension. The mbstring functions are no longer emulated if missing. Jquery has been updated, code has been cleaned up to PSR-2 standards (except some special modules). On the advanced search screen, references to categories has been removed when cate...
- Sun Sep 03, 2023 3:56 am
- Forum: Sphider Help
- Topic: An issue with PDF indexing.
- Replies: 16
- Views: 8539
Re: An issue with PDF indexing.
I tried various things to isolate the issue, such as stemming off, stemming on, various settings changes... to no avail. Then I upgraded the CentOS Sphider from 5.2.1 to 5.3.0.... Here is the log file: Spidering http://localhost/ 1. Retrieving: http://localhost/ at 03:44:36. Size of page: 10.78kb. S...
- Sun Sep 03, 2023 1:47 am
- Forum: Sphider Help
- Topic: An issue with PDF indexing.
- Replies: 16
- Views: 8539
Re: An issue with PDF indexing.
Okay, I got Centos & working with Sphider 5.2.1. Log file: Spidering http://localhost/ 1. Retrieving: http://localhost/ at 01:24:59. Size of page: 10.78kb. Starting indexing at 01:24:59. Indexed Links found: 5. New links: 5 2. Retrieving: http://localhost/ at 01:25:00. already in database 3. Ret...
- Sat Sep 02, 2023 10:56 pm
- Forum: Sphider Help
- Topic: An issue with PDF indexing.
- Replies: 16
- Views: 8539
Re: An issue with PDF indexing.
Thanks. Give me a bit... My VM of CentOS 7 had a dated version of PHP (now at least a reasonable 8.0), but my database version is also old and I am struggling to get it updated. I am used to Ubuntu, so CentOS is trying to intimidate me! I won't let it. Once that is done I can better simulate your se...
- Sat Sep 02, 2023 7:03 pm
- Forum: Sphider Help
- Topic: An issue with PDF indexing.
- Replies: 16
- Views: 8539
Re: An issue with PDF indexing.
I tried indexing the pdf's locally. 1. Retrieving: http://localhost/ at 18:51:01. Size of page: 10.78kb. Starting indexing at 18:51:01. Indexed Image 1. Retrieving: http://localhost/icons/ubuntu-logo.png at 18:51:01. Indexed Links found: 5. New links: 5 2. Retrieving: http://localhost/ at 18:51:01. ...
- Sat Sep 02, 2023 3:26 pm
- Forum: Sphider Help
- Topic: An issue with PDF indexing.
- Replies: 16
- Views: 8539
Re: An issue with PDF indexing.
We have a mystery! We don't need to blab the content of your pages to the world, so email me say two of the PDF files and I will try them locally to see what kind of results I get. It seems you have things configured correctly. Is there somethi8ng different with these files? I can't imagine there wo...