Wild guess... webmaster is sending a 301 to stop applications like Sphider.
I found a work around. First off, this work around basically IGNORES 301 errors, EVEN IF IT IS LEGITIMATE. So I definitely would not want to make this a part of a published version of Sphider.
Now to the "fix" --- more of a HACK, actually!
In Sphider 4.2.0 and 4.2.1, lines 334 and 335, or Sphiderlite 2.2.0 and 2.2.1, lines 311 and 312, find:
Remove the "//" to uncomment the lines and save.// $url_status['state'] = "ok";
// $url_status['content'] = "text";
If there really is content on those (formerly) 301 pages, it will now index.
As I said, I wouldn't recommend this to be a formal part of Sphider, just a hacker's approach to index the un-indexable.
Remember to restore the "//" to comment those lines out again when you are finished! There IS such a thing as a genuine 301 error.