Sphider Help Forum

When an indexing run gets interrupted, Sphider has always had the ability to pick up where it left off. Re-indexing, however, is a different process. If a re-index gets interrupted, the only option has been to start over. Sphider 4.0.0, 4.0.1, and 4.0.2 (SphiderLite 2.0.0, 2.0.1, and 2.0.2) introduc...

Sorry for the long delay since the last post, but here is what I have found... Many times, the 301 error which has been our nemesis, is "real" in that IS what is reported to Sphider. I can duplicate this by other methods. HOWEVER, in some cases there IS NOT relocation! Wild guess... webmas...

Hopefully the sitemap.xml mod will help in indexing sites. If testing doesn't show any serious problems, the mod will be incorporated into the next release. What I HAVE noticed, however, then even when indexing with a sitemap, the very first (landing) page has to be able to be indexed, If that ends ...

In the case of https://yourstory.com/ , the Sphider nod found in the MODS section of this forum and titled "Index from sitemaps when sitemap is a list of sitemaps" will allow you to index using their sitemap. BUT be forewarned: the sitemap references 759 other sitemaps with a total of 61,3...

Sphider can index from sitemaps if they are simple sitemaps. If the initial sitemap is a list of links to additional sitemaps (popular with larger websites), it doesn't work. This mod shows promise to correct that. There may be situations which interfere in this working, but only testing will identi...

I have been looking into this. Basically, web pages are evolving in a manner which Sphider is incapable of understanding. I have a blog post addressing this and the ramifications.

https://www.blog.worldspaceflight.com/2 ... -obsolete/

The method of having a sitemap.xml consist of a list of additional sitemaps is perfectly valid. However, is is used primarily by very large sites in which a single sitemap would be HUGE! The maps are reduced to a manageable size, then referenced by a single master. I MAY look into a procedure to rea...

This page has content which is made up of primarily JavaScript and references to other content. Sphider does not index JavaScript and the no-follow flag prevents following the references. I will check further as to whether there is any content Sphider could (or should) index. EDIT/UPDATE: I do get t...

I took a quick look at the source code for https://yourstory.com/, and ... WOW! Definitely not your traditional HTML! I'll have to look deeper, but at this moment I'm not sure Sphider is sophisticated enough to digest it. I'll check deeper and elaborate. EDIT/UPDATE: This site is proving difficult t...

I have determined that sugermint.com is built using WordPress. I also know Sphider has difficulties with WordPress sites. A lot has to do with how robots.txt is set up. Sphider CAN index some WordPress sites, but it is an iffy proposition. (From experience using Sphider on my own WordPress blog.) In...