I've seen some newer testing frameworks where you prompt for the outcome you want under a specific scenario. The first run is a LLM agent that figures out the steps but those steps are then used for recurring runs. If something fails, it goes back to the agent loop to see if it's an actual error.
This works well in e2e type tests where the test implementation might change but the customer experience hasn't.
So maybe there is a hybrid approach in our future.
Yeah, that's an interesting case study. It's pretty common practice to do something like llm-as-a-judge if your programming agentic systems.
I think those are different from pure "natural langyage programming" though because there's normally code you've written that's calling out to the LLM, so it's within the context of written code.
I think this, and lots of other comments are so interesting. There are a lot of different increasingly complex ways LLM outputs and coding are being tied together by people, that I hadn't really considered.