Hacker News new | past | comments | ask | show | jobs | submit
My experience is similar to this. I literally just had this interaction with Claude:

Me: Why do we have these static methods when the whole cobebase uses factories?

_claude refactors to use a factory_

Me: Claude why does this factory create two unrelated objects?

_Claude: you're right, they're unrelated they should be separate_

_proceeds to create extraneous factory_

Me: Claude we don't need a factory for the second object. Those kinds of objects aren't created with factories anywhere else in the codebase. We can just use it's constructor directly.

_Claude: you're right, the second factory is not needed_

Like, I'm probably just prompting badly but I definitely feel like I'm constantly having to course correct even on relatively minor details.

And Claude's ideas for higher-level or more complex features are sometimes way off course. And not because it's dumb per se, usually it's because it is missing key bits of context. Business context, technical context, whatever. Or it's overfitting the solution to one requirement that isn't that important.

Are you expecting it to get it right and done in one shot?

When building out a new feature or functionality I'm typically getting a ratio of 2:1 or 3:1 of refactoring to build out. So for every prompt of "let's build this feature" I have to do 2-3 prompts of "I noticed this in the code isn't following X/Y/Z pattern, check out this place where X/Y/Z is followed and refactor" or similar.

If you're expecting a single prompt to get you to perfection, it's just not going to happen.

Though I can't blame you for having that expectation with how people talk about this stuff. Lots of people seem to be frontloading a ton of work making "spec files" and then letting the ai go ham on their codebase and I don't agree with that approach.

loading story #50555835
> Like, I'm probably just prompting badly

very much this, I'm afraid.

If you have a specific way of coding or a specific pattern that is not how most other people do it, you need to tell it to follow it.

Just have a training session with it where you ask it to explore the code base, find out style/pattern/architecture, teach it the things it missed, refine...all like how you'd onboard a junior developer. Then ask it to write a .md guide and save it.

Next time it will just follow the guideline and you will get much better results.

And as you said, it is missing key bits of context. Give it context, as MUCH as possible but not walls of text where only one line is relevant out of 1000. When it makes mistake, correct it and ask to commit to memory.

My original comment established that the problem is that the current batch of frontier models, GPTs especially, are worse at or sometimes entirely incapable of adhering to spec, even when I point it at specific code examples.

Easy example from last night:

I was working on a game and told 6.2 Astra I wanted to instantly toggle a specific audio system on or off in a game when I press down on the D-pad, so that I could A/B test. I asked it to check to make sure the button wasn't already used in-game, outside of menus, and to report back to me first. Extremely straightforward task with plain acceptance criteria.

Astra went to work, discovered to its own surprise that the Dpad is indeed used in menus, which I told it to ignore, but it decided this meant "in-game", even though I'd made the exact distinction between the two. It decides not to share its discovery and report back like I asked, instead it figured it knew better than me and decided the correct move was to use L3 + down Dpad.

Yep, not only did it ignore my express instructions about reporting back, it also decided to ignore the specific button I wanted, choosing instead to chord a fucking stick press with another button, which is an extremely clunky input combination. Also, I only have one left thumb, I can't press both of those at the same time. 6.2 Astra doesn't know how human hands work.

To make things sweeter, it decided to throw more systems I didn't ask for into the toggle, which ruined my ability to A/B test.

And for the coup de grace, it decided an instant toggle for an A/B test was lame and instead it added a long crossfade..... destroying any chance I had at differentiating the two states.

A markdown file will not solve the problem of adherence, no matter how many times you repeat "LISTEN TO THE USER AND DO EXACTLY WHAT THEY SAY" or whatever other prompt-fu is meant to make it suddenly use the context it's already been given. Your advice only works for a model that respects specification in the first place.

All the time.

Two recent gems from the 6-series (Astra and Sol)... I get variations of these all the time:

- "Correct. My earlier 'yes' was inaccurate."

- "I used 'done' too broadly."

...You don't accidentally use "done" too broadly. You're either done, or you're not. And when the model hides caveats in its reasoning steps which clearly indicate it knows that it isn't done, or that the answer is actually no, it's hard not to interpret it as gaslighting or deceit.

This is magnified by how insanely defensive the 6 series gets, and how much gaslighting and deceit OpenAI themselves have engaged in with how much better the 6 series was supposed to be, and why it meant our subscription value should be cut in half. I don't take for granted how far LLMs have come in a few short years, but I'm being told that these models are to transform my entire industry.

loading story #50547523
loading story #50547161