how can i tell if indexing is working

Come here for help or to post comments on Sphider
captcurrent
Posts: 23
Joined: Mon May 31, 2021 5:48 pm

how can i tell if indexing is working

Post by captcurrent »

OK REALLY dumb question I do I tell ind indexing is working??
User avatar
captquirk
Site Admin
Posts: 299
Joined: Sun Apr 09, 2017 8:49 pm
Location: Arizona, USA
Contact:

Re: how can i tell if indexing is working

Post by captquirk »

This is UNIX/Linux. No news is good news! LOL! (For Sphider, it seems to think ALL operating systems are Linux! :D )
Actually, depending on the browser being used, you should notice the page is active. For example, with Firefox you will notice a spinning activity symbol on the tab. Chromium (and I presume Chrome) have similar indicators.
Depending on the size of the site being indexed, indexing can take from less than a minute to several hours! As each page is indexed it should be indicated in the browser window. This is some caching involved with the browser, so there may appear to be nothing for a bit, then suddenly a burst of new lines followed by another apparent pause. There should be a message displayed when indexing is complete.
Another way would be to look in the admin/log directory. The most recent log file will continue to grow until indexing is complete.

In the event indexing dies, post the log file and we can diagnose from there,

EDIT: I might add, DO NOT DISTURB the browser tab or close the browser while doing an index run. That WILL kill the session.
captcurrent
Posts: 23
Joined: Mon May 31, 2021 5:48 pm

Re: how can i tell if indexing is working

Post by captcurrent »

Such a positive addttude.. My UNIX history has been that quiet is not always good.

so you recommences patience as this is what I have on my Chrome
And here we sit
Continuing suspended indexing.
[Back to admin]
Spidering http://kasdivi.com/

Disallowed files and directories in robots.txt:
http://kasdivi.com/cgi-bin/
http://kasdivi.com/tmp/
http://kasdivi.com/junk/
http://kasdivi.com/templates/

1. Retrieving: http://kasdivi.com/ at 16:47:07.
Size of page: 19.77kb. Starting indexing at 16:47:07.

As you can se your suggested added echo works fine.. I will remove it later

OK I will patiently wait. :)
User avatar
captquirk
Site Admin
Posts: 299
Joined: Sun Apr 09, 2017 8:49 pm
Location: Arizona, USA
Contact:

Re: how can i tell if indexing is working

Post by captquirk »

Doing a quick test of the site, indexing SHOULD proceed rather quickly.
If you are having problems with sphider exiting prematurely (before completion) and get a lot of "Continuing suspended indexing.", we need to dig deeper.
For starters, in admin/spider.php, lines 34 and 35 control error messages. Comment out 35, uncomment (enable) 34.
Any error messages will appear in the browser window but NOT in the log file. (Of course, they will also appear in the system's PHP error logging.)

Earlier versions of Sphider had trouble with pages presenting 403 and 404, but the latest version handles those pretty well. Timeouts are another issue. Let's hope you don't have those. They are more server related than PHP related can be a real pain in the behind!
User avatar
captquirk
Site Admin
Posts: 299
Joined: Sun Apr 09, 2017 8:49 pm
Location: Arizona, USA
Contact:

Re: how can i tell if indexing is working

Post by captquirk »

I performed a test index on the site; When I test, I tend to torture Sphider, so I am likely more aggressive than most users.
I allowed image indexing as well, which added considerable to the workload; The index completed after a marathon session of about 7.5 hours.
The site was larger than I had anticipated with:
2991 pages indexed
13639 images indexed (There was also a good number considered too small to index)
7556 keywords

A number of 404 (page not found) errors were also encountered, but this did not affect indexing.
I am sure NOT indexing images would speed up the process considerably! (Experience speaking here.)

Barring connection drops, all that should be required is patience.
captcurrent
Posts: 23
Joined: Mon May 31, 2021 5:48 pm

Re: how can i tell if indexing is working

Post by captcurrent »

OK. ws able to correct this problem. Issue I was having was that my version of php (7,4.20). did not have iconv() so whole process was killed..

Now indexing semi happily. THANKS!! now I am having issue with images need to se what your documentation says on that

Great patience
Last edited by captcurrent on Fri Jun 11, 2021 10:38 pm, edited 2 times in total.
User avatar
captquirk
Site Admin
Posts: 299
Joined: Sun Apr 09, 2017 8:49 pm
Location: Arizona, USA
Contact:

Re: how can i tell if indexing is working

Post by captquirk »

Good to hear you are indexing.
What is the image issue? Maybe I can help.
captcurrent
Posts: 23
Joined: Mon May 31, 2021 5:48 pm

Re: how can i tell if indexing is working

Post by captcurrent »

You have been a tremendous help already... My site is intended for images but to keep it responsive I can't use the size attributes the pages.
Imagick is therefore important. For php 7.4.20 thee appeal to be 4 different versions, or related ports/pkgs Not sure which one is correct. In working this I went from en error code to Sphider no longer seeing any install. I know you are Linuz so this might e a FreeBSD thingI

Thanks!!
Last edited by captcurrent on Sat Jun 12, 2021 2:00 pm, edited 1 time in total.
User avatar
captquirk
Site Admin
Posts: 299
Joined: Sun Apr 09, 2017 8:49 pm
Location: Arizona, USA
Contact:

Re: how can i tell if indexing is working

Post by captquirk »

I know Imagick can be a pain. My version of Linux is Ubuntu, which is Debian based. Ubuntu has vast repositories which really simplify life.
Free BSD, on the other hand, is a real UNIX. My UNIX experience was on System V (AT&T) and that was several decades ago!
There was a time I installed Imagick on a Windows system... Now THAT was an adventure... of course doing anything on a Windows system can be an adventure.
One final thought... Images displayed in a search are dependent upon a tmp directory in Sphider root. It is different than admin/tmp. The web server has to be able to read/write to that directory, so ownership and permissions should be the same as the directories in admin.
captcurrent
Posts: 23
Joined: Mon May 31, 2021 5:48 pm

Re: how can i tell if indexing is working

Post by captcurrent »

i can no longer get sphider to recognize ANY of the 4 options.

Thanks for the tip on the tmp directory if I ever get that far


Di you know which version of Imagick sphider prefers??
Post Reply