Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?
At least for my workflow, which is fairly similar, the result of the first few stages is a refined single prompt, which I then run and the result is a PR. so I, at least, would include that final single prompt over the prior intermediate results.
I also think that might not be everything you need to really vet/recreate those PRs since there's a lot of strategic and operational stuff that goes into the pre-composition runs (sometimes I actually throw everything away and restart), so something a little more sophisticated would be interesting.
Good question, and I think that this is a problem that needs to be solved. Just like a commit repository, it would be lit to record the chain of thoughts of the agents you're using.
Would have to include the full conversion history. But who is gonna review that it's probably as big as some of these big PRs are in the first place.
Yep true, I'm still thinking about this as of today so my thoughts on this are not really mature and there are definitely a lot of caveats in that approach I didn't see.
However, having a rigorous spec process which ends up giving lots of details to the agent seems to me the right way to build stuff with LLMs. Rather than iterating and piling changes after changes.