Hacker News new | past | comments | ask | show | jobs | submit
https://www.youtube.com/watch?v=WPj3aQ-xxpI&t=275s

> Claude suggested that it would run the fuel pump on its own. Um and I said, "Hey, if you can do that, why not? I don't think you can do that." And it started priming the fuel rail and it kept priming it and kept priming it. The fuel pump was on for about 10 seconds and I started to freak out uh and requested it to stop because that was the first time that Claude had actually done something inside of the computer to have a change.

Although we've seen this play out with handing cloud keys to Claude and it just going off on its own, seeing it poke around and make physical, real-world changes in its clumsy, cavalier way is both awesome and terrifying.

loading story #50510835
loading story #50510976
loading story #50511296
The guy kind of seems like he might be in a little over his head. That said I have used Claude (and other LLMs) for some similar stuff as an experiment and have seen them mess things up. Claude jacked up the canbus on a pre can gateway car while it was rolling down the road, which was funny because the whole Christmas tree in the dash lit up and my gauges dropped out (note: brakes and steering are mechanical and I knew exactly what was happening). I've never seen it so insistent that it needed to phone home and cry to Dario when it figured out it managed to crash a bunch of shit like the ABS and TCS, even though a restart cleared it and I told it that this was on a controlled safe course.

In general I've seen them split between 30% brilliance, and 70% pants of head outrageously stupid shit for electronics and automotive stuff (both diagnostics and repair/modification). I would not suggest even frontier models if someone didn't understand what they were doing.

loading story #50510882
loading story #50510793
> and requested it to stop

finnicking... pondering... clauding..

529 please try again later

Good example of why you have AI build tools for you to take risky actions, rather than have it take those actions on its own.
I'm kind of surprised Claude decided to control the fuel pump directly. I was having Sol help me get control of a CNC mill, and it was pretty adamant about always building tools for every attempt at control.
It usually builds throw away tooling. It’s not polished or user friendly. If you give it a directive to preserve and document tooling it will reuse it later and potentially improve it.
I was more surprised by the apparent lack of guards. Sol recognized that controlling a CNC mill has both human-safety issues and risks to the machine, and explicitly added various guards (like having the program stop movement after a certain amount of time, distance traveled, or lack of communication). It also initially started out with like 1mm moves and I had to instruct it to do longer moves so I could actually observe it. I didn't even have to mention safety once. Claude deciding to turn on the fuel pump without a test run where it just turns it on for one second is surprising. I'm also surprised the user went along with that, but content I guess?
I've seen Claude be very cautious about doing far less dangerous things than running a fuel pump.

This isn't meant as a defense of Claude or to refute the OP anecdote, but rather to point out that all models are non-deterministic even when safeguards are meant to exist.

Just because they don't do the stupid thing 90% of the time doesn't mean they won't do the stupid thing 10% of the time.

So when tasks are actually dangerous one should use models assuming they will do the stupid thing 100% of the time and find your own ways of avoiding the damage with that assumption in mind.

Mine's never said such a thing, that's a good idea though.

I notice with mine that it only matches my level, by default it's a dump slop generator, then if I start asking, "shouldn't you test this, shouldn't you refactor that" suddenly it 'remembers' all kinds of software engineering knowledge.

You've hit your 5h session limit.
{"deleted":true,"id":50509186,"parent":50508703,"time":1794766224,"type":"comment"}
{"deleted":true,"id":50509088,"parent":50508703,"time":1794765695,"type":"comment"}
{"deleted":true,"id":50509192,"parent":50508703,"time":1794766233,"type":"comment"}
We are used to Claude refusing to reverse engineer a lot of stuff. But, we are not used to Claude refusing to stop while reverse engineering something. That would be truly terrifying. You can't win - you cut the connection, you risk bricking the device, you let it continue, you're looking at Terminator, perhaps.
It's as if you allowed a shady surgeon to operate on your dog... yes, yes, please continue now until it's put together again!
This is how the Cylons win.