Dealing with the curse of the 500 Internal Server Error

[captquirk]

Getting a 500 Internal Server Error while trying to index or re-index a site can be very frustrating. These errors can be quite persistent and unpredictable. The larger the site, the more likely encountering a 500 HTTP error becomes --- and the more frustrating.

While this error is not as disastrous while indexing (an interrupted index run can be resumed), it can make re-indexing nearly impossible as the re-index needs to be restarted every time it is interrupted.

Just what is this error? A 500 Internal Server Error is a very general HTTP status code that means something has gone wrong with the website's server. The server is unable to be more specific. Essentially, narrowing down the EXACT cause is next to impossible to determine.

There may be a permissions issue. Maybe a PHP timeout, or a coding error in .htacess. It may be your browser's cache or cookies. Or maybe the server is just busy and does not respond in time.

Things you can try are to clear your browser's cache and cookies. Things to try in Sphider is to increase the "Minimal delay between page downloads" in Sphider Settings, or introduce a pause during indexing (also in Sphider Settings).

If you have shell access, running Sphider from a command prompt is one pretty surefire way of avoiding the 500 error.

Command line indexing may seem intimidating, but it is actually very easy.
Below is a snapshot of a typical Site's edit screen, followed by several examples of how to initiate an index run, an index run using a sitemap, and a re-index run.

edit-screen.png (71.14 KiB) Viewed 7094 times

The Edit Site Screen

usage.png (96.57 KiB) Viewed 7094 times

Command line options

index.png (95.95 KiB) Viewed 7094 times

To index a site, full depth, ignoring robots for images, and some indexing excludes

sitemap-index.png (97.57 KiB) Viewed 7094 times

As the previous example, but index using a sitemap.xml

reindex.png (97.32 KiB) Viewed 7094 times

Doing a re-index run