- cross-posted to:
- hackernews@lemmy.bestiver.se
- cross-posted to:
- hackernews@lemmy.bestiver.se
A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what’s working now.
A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what’s working now.
I thought Cloudflare’s anti-AI defenses were meant to dynamically catch all of this. That’s a lot of work to do to keep bot traffic away.
FWIW, I tried loading it with a VPN, it showed the content, but not all the images. Turned off VPN, it fully loaded. Maybe the blog has different rules.