Hacker News new | past | comments | ask | show | jobs | submit
> But this idea that the latest generative AI models are just black boxes of math that produce an illusion of intelligence is just weird and I don't know how an intellectually honest person using them could claim it's nothing more than that.

And here we have the basic disconnect: this is the literal truth of what the tool is, and what it is doing.

It seems that the difference between intelligence and the simulation of intelligence is too subtle for many people to recognize. Current tools are not smart and they do not reason. They simulate reasoning by producing a train of steps that looks like the most likely selection from the training data.

When that was the "smart" thing to do for the current request they appear to be smart. When it was not they reveal the bias induced by their training data and they look fragile and wrong. It is the uncanny valley but for reasoning.

I would continue but I have to go back to arguing with Claude that almost every line of code that it has produced reveals an incorrect unstated assumption about the system that we are building and/or how code should be structured in general.

loading story #50559836
loading story #50560428