

1·
1 year agoThe internet being what it is, I’d be more surprised if there wasn’t already a website set up somewhere with a malicious robots.txt file to screw over ANY crawler regardless of providence.


The internet being what it is, I’d be more surprised if there wasn’t already a website set up somewhere with a malicious robots.txt file to screw over ANY crawler regardless of providence.
Within reason. Libel, incitement to violence, hate speech, etc, should be illegal for obvious reasons