Javascript is not enabled. This site can still works but it'll be more interactive when javascript is enabled.
loading...
Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
Macha
7 hours ago
|
on: Cloudflare crawl endpoint
It is cloudflare who made the claim that they are well behaved unlike those other bots and that their behaviour can be controlled by robots.txt
If I need to treat cloudflare bots the same as malicious bots, that undermines their claim.
reply