Hacker News new | past | comments | ask | show | jobs | submit
chickensong | on: Ask HN: What Are You Working On? (July 2026)
Do your crawlers respect robots.txt and have a fixed IP pool that can be blocked?