Hacker News new | past | comments | ask | show | jobs | submit
It is cloudflare who made the claim that they are well behaved unlike those other bots and that their behaviour can be controlled by robots.txt

If I need to treat cloudflare bots the same as malicious bots, that undermines their claim.