- Python interpreter in Rust (fail): https://github.com/BlueBlazin/pyrs
- Low-poly Google Earth like engine (success): https://github.com/BlueBlazin/lowpolyearth
- Codebase -> 2d infinite-zoom visualization (success): https://github.com/BlueBlazin/blockviz
I have never understood or liked all the obsession with one-shot builds.
Publishing screenshots and the README on GitHub but not the code is sneaky at best, deceiving at worst.
It is owned/run by Microsoft. The mission is the big tech mission: usage of any kind regardless of the impact on anything else, and profit.
I took it to 39% complete, as a language, over 8-weeks. Then, I spent just 50 minutes describing all the missing parts and it finished the language in 1-shot. I didn’t think it would work. It did.
I used Opus 5.5 on a $20 plan and I didn’t even hit my limits.
But we've also spend a lot of effort over the past year and a half to make the codebase and tooling more approachable to LLMs so they know how we like to do things and follow our standards.
Anything more you can say about this? We desperately need to restructure and standardize our codebase right now.
To get a toy version that can run most things takes under 2 hrs. Half of that time is collecting the tests you want to pass, how it should evaluate the code, what your criteria are, etc.
Reinforcement learning doesn’t work by training an LLM on a series of inputs like ‘rewrite python in befunge’ paired with a prepared model implementation. It doesn’t matter if an open source implementation was in the dataset that trained the language model. All it got from that was an appreciation for syntax and language idiom; when it is generating code it’s doing so according to the approaches and patterns that it’s been reinforced into following in the context of a coding prompt - regurgitating existing codebases is undesirable so it is a behavior it will have been reinforced against.
I see it this way in layman sense:
- you feed the model with data;
- you run the query;
- you activate a network and it gives you most probable token, refeeds it back and generates next and so on;
- nobody knows exactly why and how this next token comes and can tell strictly which input data gives which output data without running the exact query with 0 temperature.
So yes, it's not an exact copy, but you also can't say that a particular repo is not reflected anyhow. And I mean that the weights are reflected by the particular logic, not a syntax.
Otherwise why all the hassle if we can just feed the language reference doc to 100mb model and it should be able to produce everything just fine if only the syntax connections are needed and everything else is freshly derivated and not based on the actual implementations.
‘It’s in the dataset’ sort of implies ‘the model is just reproducing output it was trained on’.
In this case you’re saying ‘the model is repeating tasks it was trained to do’
That’s the difference between accusing someone of just regurgitating something they read, vs praising them for having prepared well for an exam.
Sounds like it's you who is behind the times. With things like REA we know that LLMs can reverse engineer the behaviour of binaries and basically treat them as a form of specification to rebuild an application.
This means that old python interpreter acts like a specification for what kind of features must be included inside the project.
LLMs are amazing at input to output transformations and that hasn't changed at all.
I think it's you who has to update his mental model away from "stochastic parrot era" accusations.
PS: if you're trying to accuse me of being a "stochastic parroter", that's exactly what I mean.
Taking an input and transforming it to an output is the exact opposite of repeating something.
https://app.preview.craftspec.com/app/city/
This whole thing is just a proof of concept, and needs a mouse to interact with.
It’s incredible what AI has made possible with experimental projects that would require months of coding.
Icon it created is same purple as yours :) Background color of the page? Yup, same. Map of a codebase? Yup, there, although very different take on it. I added a 3d walking simulator within - is cool for a minute but useless later.
https://github.com/bishonen/public/tree/main/images/code-atl...
Main use case is understanding a personal life operating system with 300k LOC a bit more. It has todo's, habits, calendars (google), trips and more baked in, with the quality (UI wise) of paid SaaS (if I say so myself). While I tried reviewing the code at the beginning, I now ask for features which would normally take me days/weeks and they're done in a few minutes by claude.
https://blueblazin.github.io/blockviz/
I tried it on some of my code and it's pretty neat, though not fully intuitive to follow more complex code structures for me.
There's so many youtube channels getting hundreds of thousands of views on EVERY new major model release just doing essentially zero effort one-shots, and people lap it up as if building things that already exist is somehow the best their imagination allows for.
I do love being able to test a model for 3+ weeks (>1 month is too long personally for my budget) and actually learning how to build scalable software and actually develop your "engineering" skills.
Same, and I have found some of the failure modes in the realm of eyes free coding and limited auditing. I often see it more as pushing the process than the model, as I tend to bounce around models for the benefits diversity provides.
Have you also found there is a point where they hit code complexity and start churning without progress? (at which point you have to step in to help the bot step back and think big picture, which usually takes a few sessions)
I was motivated to make the codebase visualizer linked above to help me understand how the various pieces of the system that is being vibe coded connect with each other.
Now I'm thinking about how I can build something similar for agents. How can we teach "systems thinking" to a model?
I've been mostly working on very large modeling projects (think detailed architeture over 1000 acres) or animation because this really was the "new" capability area that Opus 5.5 "unlocked".
I have been surprised how well it has worked at scale with many sub-agents. However the code base is limited to the rendering engine mostly.
Certainly in the older models they hit a complexity limit, but I haven't really hit it with Opus 5.5. I have had it make data flow designs that I didn't like because they were more complicated than it should have been or the perenial problem of trying to make everything backwards compatible when working on greenfield refactors, but I haven't seen it choke on a code base yet.
To my experience, it's a simple complexity curve where the first bits go fast and the speed goes down very fast once you reach a certain level of complexity. Size doesn't matter as much as complexity: with the right foundations, it keeps zooming through it. Now I do wonder how much my inexperience in the complicated project is a brake on progress vs. it just being a hard to tackle problem.
On the other hand, I have a TS only project core is sub-20k LoC (NES harness), where it is struggling (for now, see other comments)
It helps us keep track of things, why not AI, too?
I've been working on process more than model. Similar to having a PR review team, you need multiple teams or processes that run over the code, markdown, assumptions. I still don't think this is enough.
I'm working on an NES harness, and the agent was churning with detecting certain state changes like dialogs, menu navigation, screen change. The bot and I got churning and I put it down for a while. A few days later it occured to me to ask the bot "Does the emulator and game state change with frames? Can we just get the state each frame rather than trying to be clever about it?"
It said yes, but added some nonsense about being performance hindered... (flash model). I haven't worked on it again yet, but my intention was to next have it build a small recorder based on this so it can get frame accurate game state change.
This is just a long way of saying, sometimes you have to step back and help them help themselves. They have a really hard time with recognizing ambiguity or how to determine lack of context that could be filled.