Hacker News new | past | comments | ask | show | jobs | submit
nerdo | on: Google Search is now less than half of Alphabet’s revenue
You can't legally make a crawler to compete with Google anymore, it's explicitly stated in a lot of robots.txt's.