Only three metrics went up: LOC, token spend and presumably release speed/frequency.
What's the markup (profit/COGS)? is it up?
Customer satisfaction? Is NPS moving one way or another?
i will say, at least this article says "we merge 5x more and our revert rate has stayed the same." so thats at least a metric i can start to respect
I don't think it makes sense to A/B test "AI-native" vs other regimes of development unless you are a large, decentralized company like an F5 company or a Big Tech. How will a team of 25 (the size of the company if you read the thread) have enough output to even A/B test the difference?
What can you possibly be doing as a product that justifies you releasing 60 times in a working day?
It's certainly better than the old models of releasing on a quarterly cadence with the test team and all that, but it feels like the metric has become the goal when you are releasing that often.