> It matters because generative AI isn't smart, and it doesn't give a shit. What seems like intelligence is an illusion propped up by a massive amount of data strip-mined from the commons with weights that nudge the output toward a generally desirable shape. What seems like care, concern, and attention is just generative AI doing what it's designed to do: telling you what you want to hear. The math and logic that power LLMs are as cold, unfeeling, and indifferent as the laws of physics.
> Smart people who give a shit notice things. They connect dots. They notice when things don't feel right. They know when something is about to go off the rails, and they know how to get things back on track. They know how to handle edge cases. They know how to manually handle a unique situation. They know what the right thing to do is, and they know when they need to put in extra effort or time to make something happen. Generative AI falls far short in these areas, but this is the glue that holds complex systems together. More prompting and more output isn't a substitute for this.
There is a lot of hype around AI, and disagreement about what it is, what it's capable of, etc., but absolutely none of the smart people I know are clinging to the idea that "generative AI isn't smart", that its intelligence is just an illusion and that it only tells us what we want to hear.
Is it smart in the same way a human can be? Of course not. Does it share our emotions and morality? It seems the answer is no. But this idea that the latest generative AI models are just black boxes of math that produce an illusion of intelligence is just weird and I don't know how an intellectually honest person using them could claim it's nothing more than that.
And here we have the basic disconnect: this is the literal truth of what the tool is, and what it is doing.
It seems that the difference between intelligence and the simulation of intelligence is too subtle for many people to recognize. Current tools are not smart and they do not reason. They simulate reasoning by producing a train of steps that looks like the most likely selection from the training data.
When that was the "smart" thing to do for the current request they appear to be smart. When it was not they reveal the bias induced by their training data and they look fragile and wrong. It is the uncanny valley but for reasoning.
I would continue but I have to go back to arguing with Claude that almost every line of code that it has produced reveals an incorrect unstated assumption about the system that we are building and/or how code should be structured in general.
Yes, it's way over the heads of Nobel Prize winners whose work helped develop the technology.
> Current tools are not smart and they do not reason. They simulate reasoning by producing a train of steps that looks like the most likely selection from the training data.
It's interesting that you seem to know how they "reason" when it's a hot research topic.
https://news.yale.edu/2026/10/08/how-do-ai-chatbots-reason-h...
> When that was the "smart" thing to do for the current request they appear to be smart. When it was not they reveal the bias induced by their training data and they look fragile and wrong. It is the uncanny valley but for reasoning.
That sounds an awful like humans.
> I would continue but I have to go back to arguing with Claude that almost every line of code that it has produced reveals an incorrect unstated assumption about the system that we are building and/or how code should be structured in general.
It's ironic that you are using a technology that isn't smart and can't reason to do your job.
That is literally how they work. They aren't smart, they can't reason, they can't learn. A statistical model of "what is likely to come next" has its uses, but don't ever mistake it for intelligence.
It's strange to me that people keep believing this simplistic understanding of AI and applying it to the current state of the technology. Some of the top researchers who helped develop the foundations of this technology, including Nobel Prize winners, routinely explain how we're way past that point.