Hacker News new | past | comments | ask | show | jobs | submit
courtcircuits | on: No AI Slop: A Manifesto
> Improving the system over patching the output

This really connect with the latest conversations I had with coworkers regarding how can we review gigantic PR generated by AI. One way could be to ask the submitter to give their prompt as part of their contribution. This way it's easier for the reviewer to understand what was the thought process of the contributor and if this very process is sound. Not saying that this way to review is a silver bullet, but at least it removes some weight.

grey-area | parent | next
Just ban large prs. They can always be split up, and usually as well as helping review it makes them less risky and easier to find bugs and roll back if there was a problem.

The discipline of splitting them up helps in almost every dimension.

photios | root | parent | next
Fully agree with a twist - ban dumb large PRs.

To add to my openspec shilling above, I've been known to "fix" large PRs by reverse engineering specs from them and building each spec out in a separate PR.

Last week me and a coworker turned a 15k line PR into 10 x 1.5k line ones that went in smoothly.

grey-area | root | parent
Yes I do like to make a big pr first sometimes then it's usually pretty easy to split it into small ones and it's a helpful process in many ways. You can even present both together so people can see it either way, but I think in most systems it's better to have smaller prs merged one at a time.

For a rewrite, I guess sometimes it's better to have a spec then write to that, and PRs might be larger.

tracerbulletx | root | parent
So what do you think of Discord rewriting in rust, and AWS shipping an emulator that was ai generated? Do you think those were done with small prs? Do you think that it's bad they did those things?
forbiddenvoid | root | parent | next
Please add your own thoughts to the conversation rather than aggressively asking rhetorical questions. "Counterexamples exist" is hardly a valid criticism of another human's perspective.
tracerbulletx | root | parent
If someone says PRs need to be small, and I say here are some examples of huge PRs that I think are good, do you think these are still bad? That is not remotely invalid.
loading story #50489478
loading story #50489530
yjftsjthsd-h | root | parent | next
I, personally, would certainly argue that those should be done in small PRs. If you have a working application and want to rewrite in rust, I would expect it to be done by replacing individual components one at a time while making sure relevant tests keep passing. If you're making a new application, I would expect it to be built incrementally by adding parts incrementally.
grey-area | root | parent
I think they could do those things with small PRs if they chose. If their agents are not intelligent enough to make small prs, I guess they can YOLO and assume everything is just fine with what the agents did. I do think small PRs would be better for agent review as well though, because they avoid any problems with context being overwhelmed and things being missed/forgotten.

What are the PRs you're thinking of though. If you mean a full clean rewrite, they'd probably proceed with spec first then just write it in bits to the spec. No reason it has to be large prs or smaller that's a design choice.

Often an exercise I find useful is do a quick big PR, then split it into small chunks that make sense independently for review. Or you can chain prs. Both those things can be done.

You're welcome to do large PRs, I'm not a fan of huge PRs though and would not pretend to review them (as pretending is what it would be), nor would I like to work somewhere where this is the culture.

thibsrn | parent | next
> One way could be to ask the submitter to give their prompt as part of their contribution

Zed's delta (https://delta.dev/) seems to be solving this problem by extending the version control to the conversation layer. I don't know if I'd want all my conversation to be tracked, some stuff belongs to the draft sheet.

photios | parent | next
Look into [openspec](https://openspec.dev). My team is routinely passing 1-2-5k line PRs through review, with the following caveats:

1. PRs come with "the prompt". In openspec parlance this is a set of docs: a proposal, design, specs, task list.

2. I have a tiny skill to slice and dice PRs, so I can review each part of the system separately by rewriting commits in order of reviewal: specs/docs, production code, tests, evidence. I sometimes pull DB queries to a separate commit for focused review.

3. The evidence bit is built by another skill: go over all added spec scenarios and produce a markdown document with links to the code implementing the scenario and the tests proving the implementation works. Emacs helps me jump back and forth via those links and read the parts I'm interested in.

My next goal is to extend openspec schemas with custom workflow steps to make PRs land in my preferred review structure and contain the evidence report, so I don't have to build it.

courtcircuits | root | parent | next
Your workflow looks really neat, I'll definitely give it a try.
estetlinus | root | parent
So now I get to review the prompts and a slopped PR-description? Oh man, that sounds nice.
photios | root | parent | next
Happy to hear how you do it. We're all figuring that out.
eden-u4 | root | parent
Up until you notice that in the spec an agent wrote "All operations should be atomic". But, the you observe that it keeps modifying the same value across multiple threads because there's another spec that states "No perf regression, pls".
viklove | parent | next
Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?
jaggederest | root | parent | next
At least for my workflow, which is fairly similar, the result of the first few stages is a refined single prompt, which I then run and the result is a PR. so I, at least, would include that final single prompt over the prior intermediate results.

I also think that might not be everything you need to really vet/recreate those PRs since there's a lot of strategic and operational stuff that goes into the pre-composition runs (sometimes I actually throw everything away and restart), so something a little more sophisticated would be interesting.

courtcircuits | root | parent | next
Good question, and I think that this is a problem that needs to be solved. Just like a commit repository, it would be lit to record the chain of thoughts of the agents you're using.
fuzzzerd | root | parent
Would have to include the full conversion history. But who is gonna review that it's probably as big as some of these big PRs are in the first place.
courtcircuits | root | parent
Yep true, I'm still thinking about this as of today so my thoughts on this are not really mature and there are definitely a lot of caveats in that approach I didn't see.

However, having a rigorous spec process which ends up giving lots of details to the agent seems to me the right way to build stuff with LLMs. Rather than iterating and piling changes after changes.

jurgenburgen | parent | next
How does reviewing the long discussion the author had with the LLM prove the code is correct?
courtcircuits | root | parent
Because, unless you are changing some code by hand. Your prompt and the context your giving to your agent is the seed for the outcome of the agent itself. No agent as far as I know can generate content ex-nihilo. So having a look at what people are telling their agent to do can definitely help understanding the result of the same agent in my opinion.
dude250711 | parent
If a PR is gigantic it means the team lead screwed up slicing the work.
TexanFeller | root | parent | next
That is patently false in the real world we live in currently. I constantly see lazy coworkers post 2000 line PRs for a trivial change that their LLM generated when 500 lines might accomplish the same in a better way. It's almost always a lazy human accepting what Claude did by default without any manual review or asking the LLM to do anything differently.

Claude absolutely loves to reimplement the wheel locally instead of just calling a method in a library we already depend on. Every doc comment and README.md update is bloated by 2-5x what's needed to communicate intent or to actually be read by a human later. It makes random drive-by edits that aren't necessary and weren't asked for in the prompt. It's a well oiled machine for producing astronomical bloat by default if you don't supervise and rein it in!

courtcircuits | root | parent | next
Yeah true, actually my phrasing was poor. I came to that idea through debate with other coworkers. This is not a solution for gigantic PRs but rather a method on how people should be reviewing AI changes.
camgunz | root | parent | next
It also means the engineer didn't check in with the lead as their PR became unwieldy.
photios | root | parent
Why, yes, blame someone else, and not the person that shat out the PR :)

I had a recent case where I, being the lead, sliced up the work real good into an epic and what not and a guy on the team just did a "hurr durr, Claude, go wild with all that" and threw a huge PR over the wall linked to 5+ stories.

Well, we got to have "the conversation" after that and learned how to fix these kinds of PRs together...