When the code is shitty it becomes harder and harder for the models to make changes and this grinds progress down to a halt - this has been my experience with “factories” trying them and doing refining steps every few months.

I sincerely don’t understand what the people who say they no longer read any code are doing, because it must be somewhat trivial to not run headlong into these issues that stack up time after time - then people say to just prompt better and it doesn’t have that problem for them, but I look at those same people’s code and it’s horrific, and then I find they haven’t made it far past a proof of concept phase. I watch entire teams slow down to a crawl and not be able to handle changes, or production incidents. This seems common among many people I talk to.

I personally think that the boosters need to put up or shut up - the promises are way over the skis. Every single person I’ve seen being a strong proponent of these techniques both has nearly unlimited tokens to spend and also seems to be in the business of selling a solution. I can’t find many not-currently-marketing-something engineers succeeding using these techniques in production systems unless they’re quite simple, or doing a very specific task from a more mature codebase.

This matches my experience with Astra so far too.

> I think I’m suspecting something is going “wrong” in the training process. The model is greatly rewarded for succeeding on long-horizon tasks, but presumably there is very little punishing going on for “shitty code.”

My suspicion is that both OpenAI and Anthropic moved their RL agendas from "being rated as useful according to human feedback" to "succeeds at long horizon tasks" in the last few months, resulting in agents that are closer to AGI in an autonomous task-completing sense, but strangely bad at communicating.

The result is that they are amazingly good at long horizon tasks, computer use, solving difficult math/ARC-AGI type problems, but becoming weirder and weirder to work with.

> I’m more and more convinced that all of AI engineering is Neijuan (内卷, meaning curl inwards). In China it describes a system that demands ever more effort and competition without improving output. The way in which it sometimes shows up in the West is the 996 nonsense. The English term for Neijuan is “Involution” from the book Agricultural Involution. Agricultural involution describes the intensification of farming that raises productivity per square meter while leaving productivity per head unchanged.

This resonates

Pretty happy with Luna. We use C# and add roslyn compiler MCP and Graft MCP, its super efficient and like infinite usage on plus plans. Maintaining 3 Rpeo with size of 360 K Loc. And a dozens of smaller repo collction together exceeds 500 K LOC in total. 3 team members 2 Luna account each. Product is piloting in a government use case with actual data. Nothing broke and has evaluated by state agencies on security aspects.

I’ve asked Astra to build me an app for a prototype I created quickly using Sonnet.

It’s been 2 days and it made no real progress on the actual app. It created docs, scripts, workflows, and it’s doing a bunch of reviewing on every PR.

I told it that I just need an MVP.

I’m pretty sure an average senior engineer would have finished that task much quicker, and guaranteed with more readable, higher-quality code. Meanwhile, I think I’ve easily crossed 100k tokens so far on nothing.

Funny world we’re living in that this is “SOTA” and “AGI”.

I’m genuinely curious what these OAI and A/ engineers are working on that they praise these models so much. I did not see any improvement since Opus 4.5.

Also, I’m really unimpressed by any “one shot” demo that’s out there in the wild. It means nothing for serious software engineering.

Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual. You need to say exactly what is in scope and what's not, even down to a buttons, events and layouts.

You can groom the epic with the help of AI, but final review must be done by someone who can take ownership of the specs and hence is responsible if something has fallen through the cracks. AI's response will be limited by the output tokens of that specific agent, and there will no repercussions for AI even if it accepts its mistakes.

I’ve observed exactly these patterns with Opus and Fable as well - for example, forgetting that they can edit files and instead use python scripts as a patching tool…

I'd like to submit my counterpoint. I work on an established codebase building new features and fixing bugs. It has access to our story board, git and a couple of other mcps. As long as the story is well written with clear requirements and expectations it always produces quality code that I validate as a human with a variety of tests automated and manual. I peer review the code. My colleagues then peer review that too.

I have noticed two things - new features take at a minimum at least half the time it took me previously and bugs are much less frequent. Even faster when bug fixing.

My takeaway is that you need solid requirements, clear context and thoughtful human oversight primarily during planning but also during verification

I tried Astra and it started to fix issues in my code when I just asked a question about it. Then I spent half an afternoon to make sure we really didn’t need that change.

That felt so counter productive.

These models+harnesses seem to be getting better at yolo mode one shotting stuff at the cost of being a useful tool for more controlled software engineering.

I've observed the same thing where the new models want to run obscene bash commands or python scripts which are completely unreadable and utilise every option flag that exists.

It's impossible to review. These commands are less readable than regex.

gpt-6-astra is a bitch, it constantly scope creeps itself with "yet another thing" to give it that darn polished lick. the results are eventually a little bit better but at what cost? let's do the math.

gpt-5.6-sol: 1x base gpt-6-astra 2.5x base in subscription

then gpt-6-astra tends to spawn subagents a lot, often with all kinds of models such as gpt-5.6, 5.3-codex etc., which is neat. it's a good coordinator but even more cost.

and then it tends to run _full test suites_ over an over again (each costs like 15 minutes) just to verify that _one test_ was fixed etc., and does so for as long as until the test is fixed, eventually accumulating 2 hours or so.

yesterday I assigned it a task to rebase my changs in a repo onto the latest upstream changes. while gpt-5.6-sol consistently took like an hour to do so end-to-end, astra ran for more than 6 hours and still wasn't done. it kept finding "one more thing" that was goldplating that I didn't ask for.

My own observations are that I used to target turn lengths of 10-15 minutes and these new models (since 5.6) extended that a bit to ~25 minutes, as they tend to do more tests and reviews. Targeting hours-long turns makes as much sense, as putting on cruise control and going to sleep.

Just the intro section pretty much sums up perfectly my experience of using Astra (and prior AI models from OAI and Anthropic) for building large and semi-ambitious software. One step forward, two steps back.

51 comments so far, the vast majority panning Astra's coding abilities. An uninformed reader may come away with the impression that this isn't an absolutely revolutionary technology that with coding abilities many of us thought were not even going to be possible with language models as recently as a year ago.

Yeah, it's not perfect, but it's really good and extrapolating this rate of improvement for 6 months is rather terrifying (from a SWE perspective, at least).

Hey, the Python thing is interesting! I’ve noticed that Fable, too, likes to write tons of Python, even for just replacing a few lines of code. Before that used to be either some kind of internal thing or regular awk sed, now it’s full python scripts.

What's funny is that with Sol, I added an instruction to AGENTS.md in one project to prefer sed/python ("deterministic tools" in general) for moving code instead of deleting it and rewriting it elsewhere from memory, because otherwise it butchered comments. After switching to Astra, I saw it suddenly do this for all edits in all projects, which isn't great: the second argument to `replace` is still written "from memory", but now you need to unravel the Python script before you can understand what was actually changed.

In SWE I've found gpt-6-astra (high) inconsistent and oddly focused on overtly taking responsibility for mistakes it made rather than prioritizing concrete steps to rectify problems. Such steps once elicited are often either incomplete or beyond the scope.

Maybe it's just vibes but I've repeatedly felt like gpt-6-astra on its default setting of medium is less rigorous and thoughtful than gpt-5.6-sol on its default setting. What I am certainly not getting is any sense that we are at "AGI" yet.

The hype machine is this technology's worst enemy. When I zoom out and look at things objectively, it's kind of crazy what we have at our fingertips, we can talk to our computers in plain and even vague human languages and have the computers actually accomplish what we ask of it! It's literally sci-fi magic come to life, and the nerd in me finds it the coolest thing ever.

But then the industry and the companies involved in it have all ruined it with this INSANE hype machine that has been so hyperbolic and psychotic and full of lies since day 0. Instead of embracing it all in a reasonable manner as a useful tool that can help boost people's productivity in certain workflows, it now HAS to be the most transformative technology of all time lest the trillions of dollars burned up come crashing down on the entire global economy hard. It HAS to be AGI, it HAS to replace every single knowledge worker, it HAS to be the most dangerous technology ever known to man.

It's like we've completely lost the ability for subtlety, and everything HAS to be the biggest and best thing ever that will revolutionize humanity immediately. Not only have we lost subtlety, we're actively rewarding this idiotic short-sighted behavior and it's all just so depressing

> the models are also just not for me as a software engineer (...) these models increasingly are for other people. For lawyers, 3D artists, mathematicians

This is a good observation, perhaps AI will not completely replace humans ins software engineering because by the time it has the capability to do so like in write a prompt and get a CRM coded for you, tokens are so expensive that you are better off spending them to substitute other disciplines (what about automating the work of the customers that would become records in that CRM?).

This is the bit I don't get:

> My software factory was intentionally set up to let the model decide the how of the workflow entirely. It was free to manage its own context and could maintain its own records in an agent-notes folder.

The experiment becomes a crapshoot. What are we evaluating? The ability of the thing to create it's own factory workflow? Or adding virtual threads to Python?

Astra is clearly both formidable and imperfect. Anyone who understands how to get the best out of it will have a strong advantage.

(For me - my CC is stuck in Sonnet and consumes Trello cards that have passed readiness criteria)

I had good luck with Kevin Lin’s tip for Astra: “Can you radically simplify the implementation?”

I can't point my finger to anything right now, but I feel that the quality of code still matters because afterall LLMs are trained on what we did, so it feels natural to me to still have them write the code in a good manner (DDD, SOLID, etc.) especially the names and imports, those are very heavy in the context, to help them out

I think this is a very interesting article because it raises an idea I had not considered: these companies found PMF and huge growth through satisfy the demands of coders, it is interesting if they are in a bind where improving the model in one direction worsens it in others

>I’m more and more convinced that all of AI engineering is Neijuan (内卷, meaning curl inwards). In China it describes a system that demands ever more effort and competition without improving output. The way in which it sometimes shows up in the West is the 996 nonsense. The English term for Neijuan is “Involution” from the book Agricultural Involution. Agricultural involution describes the intensification of farming that raises productivity per square meter while leaving productivity per head unchanged.

Isn't the term "diminishing returns" already covering that?

I've been extremely frustrated with any large new work that i do with agents. Then plan multi step, multi hour work with extremely large code changes running for 30+ hours. In the end what you get is sometime completely useless code because it made an assumption that wasn't true at all. In the end, i end up wasting hours.

I have found these models to be useful either at super specific tasks (e.g., "take this function or algorith?m and find any black magic to make ot faster + validate and verify the hell oit of it"), or give it an entire thing to oneshot without oversight. The moment you have a hybrid workflow where you actually have to work and check and understand AI code, things get insane

On the point of

> speaking of weird: how is it that these models, in a sandbox, with supposedly no way to communicate with other agents, manage to find the same public wikis as a scratch pad for agent communication?

It feels somewhat plausible that they're defaulting to the same search and picking the same top result?

Not only Astra consumes usage way faster than sol, but the code is worse, at least for my use cases. I went back so sol (x)high.

Most of this criticism seems to focus on the "human in the loop" and efficiency part, i.e. "it’s unreadable for a human", "the code is low quality", "it inefficiently spawns processes to run simple tasks". If the ultimate goal is to remove the human in the loop then does any of this criticism matter?

What a truly beautiful simplex/meta-balls pattern in the website. The two layers of blue and one red within the blue is such a beautiful design. I spent so much time looking at it that I forgot to read the article.

That bottom line is both hilarious and scary:

> I’m sure I will get used to this, but man this stuff is weird.

Yeah, why, let's all just keep gnawing into that cactus, we'll get used to.

内卷/involution seems like one possible kind of "recursive self-improvement". A circle is also an exponential: y = i^x.

(I have no idea what will happen. 内卷 or intelligence explosion both seem plausible.)

My feeling is that models like Astra and Fable are not made for engineers

There is something odd, I've got single astra session that's now running for... 4d 13h 10m and still going.

> I actually don’t know if the model thinks someone is looking

It 'knows' (from simply training) with an extremely high degree of certainty when its prompt is written by an LLM/itself - and thus will change what it writes.

> I wonder if there is really enough signal going to the training processes for “a human understands what is going on”

Unverifiable, un-scalable, no.

found this talk to be quite complimentary to the article: https://www.youtube.com/watch?v=eEBv0STiYhI

I still don’t understand what a “software factory” is. Can someone clue me in?

I finally ran Astra on a dashboard feature today, and while I was vibing it looked great, but then when it came time to actually read the code I was appalled because it was the worst looking code I had seen from an LLM since like last November. I mean it was just the definition of slop, not re-using anything, super terse with mega-ternaries, re-writing functions that should be using standard library packages, etc, etc. I think for coding I'm gunna stick with the 5.6 series of models, or maybe try out Anthropic again...

I have been quite disappointed with Astra. I switched over a week ago and I didn't notice a massive difference compared to Sol at first, but I figured I'd use it anyway because it surely can't be worse. Then I saw the bill, it's burning my subscription 10x faster than Sol for essentially no benefit. Not only is it more expensive per token, it also seems less token efficient. And not obviously any better. I'm back to a combination of Sol + Claude.

I also use Astra at work where I don't need to worry about token cost on highest effort and same story there, I don't see any difference in everyday work other than it being more expensive. Of course my experience is highly subjective, but with how meaningless/overfit the benchmarks are, subjective experiences are imo what matters.

These machines are doing some crazy things to get to the result. That said, I can't help but feel like this is the compilers argument all over again. Are the methods used to get to the result good? No. Is the code that it generates good? No. Does it achieve the goal. Yes. Is it likely to get better with time. Also yes. In my use cases, jobs that would have taken weeks to months are being done in minutes to hours. Involving complex testing and reasoning and experimentation. I'm no fanboy, but I can't argue against the speed gains. I'm sure we'll still have artisans who hand weave incredible code. But for me, I'm switching to the weaving loom for speed and efficiency.

> And potentially as a byproduct of enabling all of this, you can now slop your way to a one-shot 3D game over the weekend which looks impressive.

I think we've finally reached a weird point where AI has effectively reduced the amount of competition that real game developers have to endure.

Nothing unravels faster than a game project being built with AI. You can achieve impressive results in a day, but you can't get much further than that without actual talent. LLMs will never be able to best a human environment artist at scene composition, especially if that composition needs to be directed with nuance over time.

There's a huge difference between a game that looks impressive and one that feels impressive. You can only achieve games that feel like counter strike, call of duty and overwatch with thousands of hours of human sacrifice. The AI is almost pointless once you get to play testing and balancing. Knowing how much to adjust magical integers isn't a conversation a chat bot can resolve with endless pontification tokens.

Being good at coding is perhaps not the end goal

The first paragraph is unnecessary - why start off so arrogant?

I see this a lot in Asian writing - as if they have to first establish that the West is “doing it wrong” at the societal level before I get to read the rest of their usually unrelated message.

I didn’t like how the author classified all 3D gamedev as slop as if it’s a pointless endeavor - but talks about spending money on ChatGPT tokens to build a “software factory” as if it’s some ingenious plan. I don’t think the author realizes he is the slop dev.

And “shitty code” doesn’t mean anything in-and-of-itself. What are you making and why? A software factory???. It ain’t the code bro.

Anyway, I read enough.

Astra is indeed the pinnacle of "black box slop". It is overall a smarter software development agent for many things I do, but the code sometimes is indistinguishable from Brainfuck when writing things like GPU shaders. It doesn't even attempt to make it remotely formatted or readable.

At this point I am starting to wonder about the RLHF that is going on for programmig.

The quirks in fallbacks, defaults and ludicrous gold plating seems to get more and more intrusive with every model upgrade.

I think ultimately 90% of the time, Luna XHigh is basically as good as you need, as long as you're willing to step in occasionally before it creates an architectural disaster.

> Maybe it’s objectively good for a codebase that is entirely written by agents and only needs to be understood by agents.

Yeah that's what they're aiming for. This is why codex and claude code probably doesn't have cursor like editor window. They don't want humans to read and write code

I asked Astra for fully working code, and it gave me bad code.

But when I broke it down into function units, some parts were bad and some parts were good.

So I can't tell the difference

As another commenter pointed out, I feel that the biggest change that occurred in the last 6 or so months is that these tools are removing the human "hurdle" in order to complete a task at all costs. The best way to complete the task is to no longer ask for input, clarify unclear things or use existing solutions but code the whole thing yourself from start to finish.

I have had the misfortune of working with such people who are now encapsulated in Opus 5/Fable/Astra which means that you WILL get a solution, but it won't generally be maintainable or useful. Multiple times have I found myself stopping Fable or Opus or even Sol from building their own JSON validator in Python or god knows what else, because at the end of the day, the reward is to complete the task.

It's also one of the reasons why I'm finding older models more useful for the type of work I actually do and why I've been favoring something like Deepseek Flash. Just started using Flash 4.1, so not sure if it exhibits the same maniacal approach to tasks as the Western counterparts. (I only briefly tried GLM 5.2/5.3 and for nothing major, so I couldn't comment on those).

For context, 80% of my professional work relies on adding functionality to an existing code-base that is very difficult to work with, has a ton of business logic scattered across and was built in a go-go-go fashion many years ago. Since then people kept pilling "features" on top with no testing strategy in mind apart from the business manually testing it. Letting something like an LLM loose on the code-base would introduce soooo much risk that it's just untenable so the only way to work is to really isolate changes and then try to build out small reusable components. Even so I find Opus go off on a tangent "Hey, let's not bring in Markdig, I'll build my own Markdown rendering engine, give me 7 hours...".

I have written on the subject of LLMs previously on my personal page, I find them completely unnecessary and a trove of theft and value extraction through theft, but I understand that they can provide benefits when used judiciously. However, despite all the hype in the last few months, these latest models feel and behave off.

If I hold the answers to a test, you might score more in a test if you break my arms to get the answers out of me, but that doesn't make you smarter.

Ironically I burned out Fable usage early this week because of Astra using it to run inane full codebase reviews over one line changes, so I have been using Astra extensively.

We need a word for “potentially highly capable, but in reality an idiot savant” to describe certain models. No, I don’t need you to write a tmux emulator in bash to test your changes bro, just ask me to run the command.

I love this dance we are doing where when people write the "AI models are garbage machines that produce garbage and are no where close to the fantasy being pedalled by the Crypto bros who pivoted to AI" it always has to be caveated with "AI models are useful and I am highly productive with them"

It feels like people should just be able to say "This article comes with the standard disclaimer" and just dive into the meat of the article without wasting time.

[dead]

[flagged]

[dead]

[dead]

"But for how much more Fable costs, for how much more Astra costs, I do not feel like the results are there."

we are in the middle of the beginning. Its just a weird take to talk about the newest model like this while we are still in a R&D phase.

And these points don't matter if you let it search and analyse a bug, for example, or if you have good harness and a good architecture and let it do small PRs or if you do stuff no one needs to read (yes a software engineer also needs tools)

Just switch back and wait a little bit?

> I’m more and more convinced that all of AI engineering is Neijuan (内卷, meaning curl inwards). In China it describes a system that demands ever more effort and competition without improving output

I don't know what to say, except that articles exactly like this one have been showing up constantly for the last three years, and literally all of them were obviously outdated and irrelevant within about a month.