Okay. This has been an adventure!
I finally got CentOS 7 set up and working in virtualbox. Took a few tries to get the configuration right, but got it running. Many years ago, I was a system administrator on a couple UNIX System V boxes. More recently, I have become comfortable using Ubuntu. CentOS is still Linux, but there are differences. I had a heck of a time getting the right permissions setup so Sphider could write to log files, do backups, etc. When I FINALLY realized that I wasn't just dealing with Linux, but SELinux, I was able to get things working there as well. Apparently, w2hen you install CentOS, YOU GET SELinux!
So then once Sphider installed an operational, I set up a site, https://www.okanaganpud.org
. Spidering was successful, with the process following the robots.txt.
I guess I'm going to have to give Drupal a whirl to see if that makes a difference. I would really like to know what is causing the problems you have had.
UPDATE: Nov. 6. I have Drupal installed although really haven't used it yet. But given that I now have a much better idea of just what Drupal is (a CMS), and I have to believe that while it may affect Sphider search operations, it should have ZERO impact on Sphider's spidering (building the database to be searched) operations. But I will play around and see what happens...