Search found 308 matches
- Thu Apr 14, 2022 2:47 am
- Forum: Sphider Help
- Topic: Relocation: http 301 error in terminal
- Replies: 19
- Views: 22567
Re: Relocation: http 301 error in terminal
Sorry for the long delay since the last post, but here is what I have found... Many times, the 301 error which has been our nemesis, is "real" in that IS what is reported to Sphider. I can duplicate this by other methods. HOWEVER, in some cases there IS NOT relocation! Wild guess... webmas...
- Thu Mar 17, 2022 4:06 pm
- Forum: Sphider Help
- Topic: Relocation: http 301 error in terminal
- Replies: 19
- Views: 22567
Re: Relocation: http 301 error in terminal
Hopefully the sitemap.xml mod will help in indexing sites. If testing doesn't show any serious problems, the mod will be incorporated into the next release. What I HAVE noticed, however, then even when indexing with a sitemap, the very first (landing) page has to be able to be indexed, If that ends ...
- Tue Mar 01, 2022 2:32 am
- Forum: Sphider Help
- Topic: Relocation: http 301 error in terminal
- Replies: 19
- Views: 22567
Re: Relocation: http 301 error in terminal
In the case of https://yourstory.com/ , the Sphider nod found in the MODS section of this forum and titled "Index from sitemaps when sitemap is a list of sitemaps" will allow you to index using their sitemap. BUT be forewarned: the sitemap references 759 other sitemaps with a total of 61,3...
- Mon Feb 28, 2022 10:33 pm
- Forum: Sphider MODS
- Topic: Index from sitemaps when sitemap is a list of sitemaps
- Replies: 0
- Views: 26328
Index from sitemaps when sitemap is a list of sitemaps
Sphider can index from sitemaps if they are simple sitemaps. If the initial sitemap is a list of links to additional sitemaps (popular with larger websites), it doesn't work. This mod shows promise to correct that. There may be situations which interfere in this working, but only testing will identi...
- Sat Feb 26, 2022 4:17 pm
- Forum: Sphider Help
- Topic: Relocation: http 301 error in terminal
- Replies: 19
- Views: 22567
Re: Relocation: http 301 error in terminal
I have been looking into this. Basically, web pages are evolving in a manner which Sphider is incapable of understanding. I have a blog post addressing this and the ramifications.
https://www.blog.worldspaceflight.com/2 ... -obsolete/
https://www.blog.worldspaceflight.com/2 ... -obsolete/
- Sat Jan 29, 2022 5:02 pm
- Forum: Sphider Help
- Topic: Relocation: http 301 error in terminal
- Replies: 19
- Views: 22567
Re: Relocation: http 301 error in terminal
The method of having a sitemap.xml consist of a list of additional sitemaps is perfectly valid. However, is is used primarily by very large sites in which a single sitemap would be HUGE! The maps are reduced to a manageable size, then referenced by a single master. I MAY look into a procedure to rea...
- Sat Jan 15, 2022 4:11 pm
- Forum: Sphider Help
- Topic: Page contains less than 10 words. error in terminal
- Replies: 6
- Views: 7897
Re: Page contains less than 10 words. error in terminal
This page has content which is made up of primarily JavaScript and references to other content. Sphider does not index JavaScript and the no-follow flag prevents following the references. I will check further as to whether there is any content Sphider could (or should) index. EDIT/UPDATE: I do get t...
- Sat Jan 15, 2022 4:03 pm
- Forum: Sphider Help
- Topic: Relocation: http 301 error in terminal
- Replies: 19
- Views: 22567
Re: Relocation: http 301 error in terminal
I took a quick look at the source code for https://yourstory.com/, and ... WOW! Definitely not your traditional HTML! I'll have to look deeper, but at this moment I'm not sure Sphider is sophisticated enough to digest it. I'll check deeper and elaborate. EDIT/UPDATE: This site is proving difficult t...
- Tue Dec 21, 2021 5:27 pm
- Forum: Sphider Help
- Topic: Relocation: http 301 error in terminal
- Replies: 19
- Views: 22567
Re: Relocation: http 301 error in terminal
I have determined that sugermint.com is built using WordPress. I also know Sphider has difficulties with WordPress sites. A lot has to do with how robots.txt is set up. Sphider CAN index some WordPress sites, but it is an iffy proposition. (From experience using Sphider on my own WordPress blog.) In...
- Mon Dec 20, 2021 12:07 am
- Forum: Sphider Help
- Topic: Page contains less than 10 words. error in terminal
- Replies: 6
- Views: 7897
Re: Page contains less than 10 words. error in terminal
Seems your setting are correct. That you CAN index SOME sites also tells me your installation is valid.
List a couple more URLs that are giving you the too few words message. I'll keep playing around on my end and maybe I'll finally see a clue.
List a couple more URLs that are giving you the too few words message. I'll keep playing around on my end and maybe I'll finally see a clue.