Hacker News new | past | comments | ask | show | jobs | submit
charcircuit | on: Cloudflare crawl endpoint
>Honors robots.txt

Is it possible to ignore robot.txt in the case the crawl was triggered by a human?