Hacker News new | past | comments | ask | show | jobs | submit
bobpaw | on: Cloudflare crawl endpoint
TIL about the Crawl-delay directive. Although it seems that most honest bots move slower and dishonest bots will learn to.