> Claude suggested that it would run the fuel pump on its own. Um and I said, "Hey, if you can do that, why not? I don't think you can do that." And it started priming the fuel rail and it kept priming it and kept priming it. The fuel pump was on for about 10 seconds and I started to freak out uh and requested it to stop because that was the first time that Claude had actually done something inside of the computer to have a change.
Although we've seen this play out with handing cloud keys to Claude and it just going off on its own, seeing it poke around and make physical, real-world changes in its clumsy, cavalier way is both awesome and terrifying.
In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff (both diagnostics and repair/modification). I would not suggest even frontier models if someone didn't understand what they were doing.
finnicking... pondering... clauding..
529 please try again later
This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist.
Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time.
So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.
I notice with mine that it only matches my level, by default it's a dump slop generator, then if I start asking, "shouldn't you test this, shouldn't you refactor that" suddenly it 'remembers' all kinds of software engineering knowledge.