Search found 302 matches

by captquirk
Fri Sep 08, 2023 12:09 am
Forum: Sphider Help
Topic: Robots.txt - would allow support be useful for many
Replies: 6
Views: 3091

Re: Robots.txt - would allow support be useful for many

You are correct. Looking closer you can see the existing function ONLY looks for "disallows" and not allows". There is definitely room for improvement here. Also, thanks for your proposed improvements. I will look it over and may incorporate it, or some version of it, in a future (nex...
by captquirk
Thu Sep 07, 2023 3:45 pm
Forum: Sphider Help
Topic: An issue with PDF indexing.
Replies: 16
Views: 5502

Re: An issue with PDF indexing.

This is crazy. I tried the url and got that page, and NO LINKS FOUND! By email, lets see the full log (looks like it started but then hung...?), the setting screen, and the site advanced edit screen. Also see if there is anything relevant in the error logs. (As shipped, Sphider has logging turned on...
by captquirk
Thu Sep 07, 2023 2:43 am
Forum: Sphider Help
Topic: Robots.txt - would allow support be useful for many
Replies: 6
Views: 3091

Re: Robots.txt - would allow support be useful for many

First off, Sphider will index https sites. It follows robots.txt. To allow Sphider, but disallow all other bots, try something like this: User-agent: * Disallow: / User-agent: Sphider (sphidersearch.com) Allow: / You could also go into the Settings tab and change the User Agent string to something y...
by captquirk
Mon Sep 04, 2023 6:56 pm
Forum: Sphider Help
Topic: Sphider 5.2.0 and SphiderLite 2.3.0 fail after install
Replies: 1
Views: 2328

Re: Sphider 5.2.0 and SphiderLite 2.3.0 fail after install

Permanent, automatic fix in Sphider 5.3.0, SphiderLite 2.4.0.

No more user code tweaking necessary!
by captquirk
Mon Sep 04, 2023 4:22 pm
Forum: Announcements
Topic: Sphider 5.3.0 and SphiderLite 2.4.0 released
Replies: 0
Views: 4449

Sphider 5.3.0 and SphiderLite 2.4.0 released

The newest releases now REQUIRE the mbstring PHP extension. The mbstring functions are no longer emulated if missing. Jquery has been updated, code has been cleaned up to PSR-2 standards (except some special modules). On the advanced search screen, references to categories has been removed when cate...
by captquirk
Sun Sep 03, 2023 3:56 am
Forum: Sphider Help
Topic: An issue with PDF indexing.
Replies: 16
Views: 5502

Re: An issue with PDF indexing.

I tried various things to isolate the issue, such as stemming off, stemming on, various settings changes... to no avail. Then I upgraded the CentOS Sphider from 5.2.1 to 5.3.0.... Here is the log file: Spidering http://localhost/ 1. Retrieving: http://localhost/ at 03:44:36. Size of page: 10.78kb. S...
by captquirk
Sun Sep 03, 2023 1:47 am
Forum: Sphider Help
Topic: An issue with PDF indexing.
Replies: 16
Views: 5502

Re: An issue with PDF indexing.

Okay, I got Centos & working with Sphider 5.2.1. Log file: Spidering http://localhost/ 1. Retrieving: http://localhost/ at 01:24:59. Size of page: 10.78kb. Starting indexing at 01:24:59. Indexed Links found: 5. New links: 5 2. Retrieving: http://localhost/ at 01:25:00. already in database 3. Ret...
by captquirk
Sat Sep 02, 2023 10:56 pm
Forum: Sphider Help
Topic: An issue with PDF indexing.
Replies: 16
Views: 5502

Re: An issue with PDF indexing.

Thanks. Give me a bit... My VM of CentOS 7 had a dated version of PHP (now at least a reasonable 8.0), but my database version is also old and I am struggling to get it updated. I am used to Ubuntu, so CentOS is trying to intimidate me! I won't let it. Once that is done I can better simulate your se...
by captquirk
Sat Sep 02, 2023 7:03 pm
Forum: Sphider Help
Topic: An issue with PDF indexing.
Replies: 16
Views: 5502

Re: An issue with PDF indexing.

I tried indexing the pdf's locally. 1. Retrieving: http://localhost/ at 18:51:01. Size of page: 10.78kb. Starting indexing at 18:51:01. Indexed Image 1. Retrieving: http://localhost/icons/ubuntu-logo.png at 18:51:01. Indexed Links found: 5. New links: 5 2. Retrieving: http://localhost/ at 18:51:01. ...
by captquirk
Sat Sep 02, 2023 3:26 pm
Forum: Sphider Help
Topic: An issue with PDF indexing.
Replies: 16
Views: 5502

Re: An issue with PDF indexing.

We have a mystery! We don't need to blab the content of your pages to the world, so email me say two of the PDF files and I will try them locally to see what kind of results I get. It seems you have things configured correctly. Is there somethi8ng different with these files? I can't imagine there wo...