Search found 11 matches

by kas
Wed Sep 07, 2022 4:02 am
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 19
Views: 13733

Re: Relocation: http 301 error in terminal

So what did you mention in user Agent box in the settings at backend, Should I leave it blank or type something etc or as xyzbot .

Yeah some news media sites like sifted.eu not taking up if so they show sitemaps in results not the actual articles
by kas
Fri May 06, 2022 1:55 am
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 19
Views: 13733

Re: Relocation: http 301 error in terminal

https://sifted.eu/ gonna check this site again and few others news nd blogs sites to check if it crawls or stops up. Okay let me check with and play around also with pixabay.com random sites Also there are category created in admin but what is there a way to user to add the Title, website, summary b...
by kas
Thu Mar 17, 2022 6:35 am
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 19
Views: 13733

Re: Relocation: http 301 error in terminal

Trying to make a landing page and results page saperate so that some clean design as a project. Sphider is capable of crawling ebooks and list in gallery view? What I can experiment is that crawling news website is failing as sitemaps exists and sub-sitemaps with several thousands articles urls in t...
by kas
Tue Feb 22, 2022 10:40 am
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 19
Views: 13733

Re: Relocation: http 301 error in terminal

Yeah some sites are crawling and some are not but some websites show called but won't show up on search results what could be the reason is the indexing not properly crawled???
by kas
Fri Jan 28, 2022 1:07 pm
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 19
Views: 13733

Re: Relocation: http 301 error in terminal

Yeah similar thing faced by another website https://e27.co/ also similar pattern has sitemaps but not crawling webpages search results show as text links of sitemap.html etc not actual title or url or description of article.
by kas
Sun Jan 09, 2022 3:42 pm
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 19
Views: 13733

Re: Relocation: http 301 error in terminal

https://yourstory.com/ This is another website of this same 301 error, also this is crawling only sitemaps.xml and not really indexing the webpages under url. How do we crawl XML sitemap index which has several other sitemaps in it.
by kas
Thu Jan 06, 2022 8:10 pm
Forum: Sphider Help
Topic: Page contains less than 10 words. error in terminal
Replies: 6
Views: 5542

Re: Page contains less than 10 words. error in terminal

Here is an simple another link tried to index via admin interface with max depth: 2 throws an Error again Spidering https://www.startmeup.hk/ 1. Retrieving: https://www.startmeup.hk/ at 14:04:24. Size of page: 161.00kb. Starting indexing at 14:04:26. No-follow flag set. Page contains less than 10 wo...
by kas
Sun Dec 19, 2021 8:53 pm
Forum: Sphider Help
Topic: Page contains less than 10 words. error in terminal
Replies: 6
Views: 5542

Re: Page contains less than 10 words. error in terminal

Hey My spider settings has 10 for Number of words for index and 5 for Minimum words to index as it is default untouched. Don't know where to check in for such errors almost out of 10 weblinks only 3 get index rest shows same error .
by kas
Sun Dec 19, 2021 8:47 pm
Forum: Sphider Help
Topic: Relocation: http 301 error in terminal
Replies: 19
Views: 13733

Re: Relocation: http 301 error in terminal

Yeah me too figuring out on various channels most of the websites I tried get this error ex. https://yourstory.com eventually sphider is not able to handle the redirect or curious like how is google handling this kind of issue still the website gets crawled
by kas
Fri Dec 17, 2021 7:53 pm
Forum: Sphider Help
Topic: Page contains less than 10 words. error in terminal
Replies: 6
Views: 5542

Page contains less than 10 words. error in terminal

Searching through all the forums on the web and tried changing in spider settings to 0, still same error occurs infact it says 'Page contains less than 0 words' even after changing the settings in admin almost all websites same error showing up. Sphider intalled 5 times fresh completely still error:...