Newzcrawler annoyance
So, for a while there, it looked as though CavLec offered comment feeds. Since CavLec doesn’t offer comments, that seemed a wee bit strange. So I got rid of the links to them.
But I didn’t want them 404ing, because I check my 404 logs and try to make disappearing things reappear. For a while I 410ed them, but Newzcrawler still checked, with enough frequency and insistence that it was sucking more of my bandwidth than I really cared to donate to nonexistent files. So I changed the 410 to 403, because 410 returns information and 403 just tells the user agent to take a hike, no bandwidth exchanged.
Guess what? Newzcrawler is STILL CHECKING. What’s more, it never stops checking for nonexistent comment feeds it couldn’t previously download. So every time a Newzcrawler user hits CavLec, s/he tries for more and more nonexistent feeds.
Bug, guys. Fix it, please and thank you. I would think it slows down your UI, actually; you’ll be doing your users a favor too.