Search found 189 matches

by captquirk
Tue Apr 19, 2022 9:18 pm
Forum: Sphider MODS
Topic: Hack to restart an interrupted re-index run
Replies: 0
Views: 31

Hack to restart an interrupted re-index run

When an indexing run gets interrupted, Sphider has always had the ability to pick up where it left off. Re-indexing, however, is a different process. If a re-index gets interrupted, the only option has been to start over. Sphider 4.0.0, 4.0.1, and 4.0.2 (SphiderLite 2.0.0, 2.0.1, and 2.0.2) introduc...
by captquirk
Thu Apr 14, 2022 2:47 am
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 16
Views: 1206

Re: Relocation: http 301 error in terminal

Sorry for the long delay since the last post, but here is what I have found... Many times, the 301 error which has been our nemesis, is "real" in that IS what is reported to Sphider. I can duplicate this by other methods. HOWEVER, in some cases there IS NOT relocation! Wild guess... webmas...
by captquirk
Thu Mar 17, 2022 4:06 pm
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 16
Views: 1206

Re: Relocation: http 301 error in terminal

Hopefully the sitemap.xml mod will help in indexing sites. If testing doesn't show any serious problems, the mod will be incorporated into the next release. What I HAVE noticed, however, then even when indexing with a sitemap, the very first (landing) page has to be able to be indexed, If that ends ...
by captquirk
Tue Mar 01, 2022 2:32 am
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 16
Views: 1206

Re: Relocation: http 301 error in terminal

In the case of https://yourstory.com/ , the Sphider nod found in the MODS section of this forum and titled "Index from sitemaps when sitemap is a list of sitemaps" will allow you to index using their sitemap. BUT be forewarned: the sitemap references 759 other sitemaps with a total of 61,3...
by captquirk
Mon Feb 28, 2022 10:33 pm
Forum: Sphider MODS
Topic: Index from sitemaps when sitemap is a list of sitemaps
Replies: 0
Views: 107

Index from sitemaps when sitemap is a list of sitemaps

Sphider can index from sitemaps if they are simple sitemaps. If the initial sitemap is a list of links to additional sitemaps (popular with larger websites), it doesn't work. This mod shows promise to correct that. There may be situations which interfere in this working, but only testing will identi...
by captquirk
Sat Feb 26, 2022 4:17 pm
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 16
Views: 1206

Re: Relocation: http 301 error in terminal

I have been looking into this. Basically, web pages are evolving in a manner which Sphider is incapable of understanding. I have a blog post addressing this and the ramifications.

https://www.blog.worldspaceflight.com/2 ... -obsolete/
by captquirk
Sat Jan 29, 2022 5:02 pm
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 16
Views: 1206

Re: Relocation: http 301 error in terminal

The method of having a sitemap.xml consist of a list of additional sitemaps is perfectly valid. However, is is used primarily by very large sites in which a single sitemap would be HUGE! The maps are reduced to a manageable size, then referenced by a single master. I MAY look into a procedure to rea...
by captquirk
Sat Jan 15, 2022 4:11 pm
Forum: Sphider Help
Topic: Page contains less than 10 words. error in terminal
Replies: 6
Views: 739

Re: Page contains less than 10 words. error in terminal

This page has content which is made up of primarily JavaScript and references to other content. Sphider does not index JavaScript and the no-follow flag prevents following the references. I will check further as to whether there is any content Sphider could (or should) index. EDIT/UPDATE: I do get t...
by captquirk
Sat Jan 15, 2022 4:03 pm
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 16
Views: 1206

Re: Relocation: http 301 error in terminal

I took a quick look at the source code for https://yourstory.com/, and ... WOW! Definitely not your traditional HTML! I'll have to look deeper, but at this moment I'm not sure Sphider is sophisticated enough to digest it. I'll check deeper and elaborate. EDIT/UPDATE: This site is proving difficult t...
by captquirk
Tue Dec 21, 2021 5:27 pm
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 16
Views: 1206

Re: Relocation: http 301 error in terminal

I have determined that sugermint.com is built using WordPress. I also know Sphider has difficulties with WordPress sites. A lot has to do with how robots.txt is set up. Sphider CAN index some WordPress sites, but it is an iffy proposition. (From experience using Sphider on my own WordPress blog.) In...