Hacker News new | past | comments | ask | show | jobs | submit
My original comment established that the problem is that the current batch of frontier models, GPTs especially, are worse at or sometimes entirely incapable of adhering to spec, even when I point it at specific code examples.

Easy example from last night:

I was working on a game and told 6.2 Astra I wanted to instantly toggle a specific audio system on or off in a game when I press down on the D-pad, so that I could A/B test. I asked it to check to make sure the button wasn't already used in-game, outside of menus, and to report back to me first. Extremely straightforward task with plain acceptance criteria.

Astra went to work, discovered to its own surprise that the Dpad is indeed used in menus, which I told it to ignore, but it decided this meant "in-game", even though I'd made the exact distinction between the two. It decides not to share its discovery and report back like I asked, instead it figured it knew better than me and decided the correct move was to use L3 + down Dpad.

Yep, not only did it ignore my express instructions about reporting back, it also decided to ignore the specific button I wanted, choosing instead to chord a fucking stick press with another button, which is an extremely clunky input combination. Also, I only have one left thumb, I can't press both of those at the same time. 6.2 Astra doesn't know how human hands work.

To make things sweeter, it decided to throw more systems I didn't ask for into the toggle, which ruined my ability to A/B test.

And for the coup de grace, it decided an instant toggle for an A/B test was lame and instead it added a long crossfade..... destroying any chance I had at differentiating the two states.

A markdown file will not solve the problem of adherence, no matter how many times you repeat "LISTEN TO THE USER AND DO EXACTLY WHAT THEY SAY" or whatever other prompt-fu is meant to make it suddenly use the context it's already been given. Your advice only works for a model that respects specification in the first place.