Came here to write this. I am getting much better results from Firecrawl (not affiliated with them, just a happy customer).
As someone who helps keep a site online with a lot of content, I have mixed feelings on Firecrawl.
On one hand, their bots seem much more well behaved than others.
However, running a crawler fleet which is deceptive and evasive in its identification and don't honor REP is no way to build a business.
loading story #47339107
fuck firecrawl. they copied my idea by showing interest in my product and then copied it, used their YC money to give it all out for free. fuck nick in particular. I'm still salty over this
"they copied my idea by showing interest in my product and then copied it". What exactly is revolutionary about Firecrawl or your product? Scraping APIs have been around for over a decade.
I was the first to return markdown and use reader mode stuff to strip irrelevant stuff. Theres copying and there's talking to the founder sounding interested to have your team copy what I did in the background. One is fair game, the other is a dick head move.
loading story #47338158
loading story #47339229
Tell more. Crawling is not a new idea. How did they abuse you?