I have also done the agentic thing and built a full CLI tool via back-and-forth engagement with Claude and that worked great - I didn't write a single line of code. Because the CLI tool was calling an API, I could ask Claude to run the requests it was generating and adjust based on the result - errors, bad requests etc, and it would fairly rapidly fix and coalesce on a working solution.
After I was done though, I reckon that if instead of this I had just done the work myself I would have had a much smaller, more reliable project. Less error handling, no unit tests, no documentation sure, but it would have worked and worked better - I wouldn't need to iterate off the API responses because I would have started with a better contract-based approach. But all of that would have been hard, would have required more 'slow thinking'. So... I didn't really draw a clean conclusion from the effort.
Continuing to experiment, not giving up on anything yet.