Hacker News new | past | comments | ask | show | jobs | submit
skybrian | on: Cloudflare crawl endpoint
If two customers crawl the same website and it uses crawl-delay, how does it handle that? Are they independent, or does each one run half as fast?
PeterStuer | parent
You put a governor on the domain, and you return from the cache instead.