Fulltxt field
Posted: Fri Jul 23, 2021 3:46 pm
And just a suggestion. This is what the fulltext field of one of the DMOZ pages looked like in the Sphiderlinks table :
\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n\r\n \r\n\r\n\r\n \r\n\r\n \r\n\r\n \r\n\r\n \r\n \r\n \r\n \r\n DMOZ \r\n \r\n\r\n #OrganizeTheWeb \r\n \r\n \r\n \r\n Important Notice \r\n Welcome to our archive of dmoz.org. \r\n Visit resource-zone \r\n to stay in touch with the community.\r\n \r\n \r\n\r\n \r\n \r\n\r\n \r\n\r\n Follow @dmoz \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n About \r\n Become an Editor \r\n Suggest a Site \r\n ...
To save space in my crawler, I trim all that plus double spaces out of the fulltxt field.
\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n\r\n \r\n\r\n\r\n \r\n\r\n \r\n\r\n \r\n\r\n \r\n \r\n \r\n \r\n DMOZ \r\n \r\n\r\n #OrganizeTheWeb \r\n \r\n \r\n \r\n Important Notice \r\n Welcome to our archive of dmoz.org. \r\n Visit resource-zone \r\n to stay in touch with the community.\r\n \r\n \r\n\r\n \r\n \r\n\r\n \r\n\r\n Follow @dmoz \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n \r\n About \r\n Become an Editor \r\n Suggest a Site \r\n ...
To save space in my crawler, I trim all that plus double spaces out of the fulltxt field.