Everything an LLM "knows" had to be told to it. It does not have a process for understanding itself in its environment. It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment. It doesn't even "know" when it's done something if you don't tell it about what it just did. It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes. And it can't actually execute any of those tools. It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.
The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy. They don't even exhibit internal consistency; when an LLM refuses to respond to a query for "alignment" reasons, that's actually an external process performing text pattern matching analysis and intercepting the query before it ever gets to the LLM. Otherwise, you could "ignore all previous instructions" the thing and get you set up the bomb.
I think one of the bigger indications that an LLM isn't thinking is that it can't improve. If I ask it to write 1000 blog posts, it will get it done in a few hours, but even if I embed each post for RAG in between each generation, the LLM is not going to get better at writing blog posts. But if I ask a human to do the same thing, while it will take them at least two years to do it, the human will have gotten significantly better at the task within the first week.
Why does that matter for whether or not it is conscious? Many things I "know" I also know because someone told me.
>It does not have a process for understanding itself in its environment.
It has a process for understanding, namely outputting tokens by evaluating the neural network iteratively. It can apply this understanding to its environment insofar as the context window and weights include a model of this environment.
>It does not have any sensing organ that would allow it to detect conditions and come to a conclusion that it must be in such an environment.
It does have a sensing organ, namely the input into the context window. Possibly it even has closed-loop sensors by doing tool calls. The conclusion could be incorrect or inaccurate but that doesn't in itself prove there is no consciousness (Plato's cave etc..).
>It doesn't even "know" when it's done something if you don't tell it about what it just did.
It does know, it's in the context window.
>It doesn't have an awareness of the tool calls it can perform; again it has to be told and even then it gets it wrong sometimes.
Same as above, as long as it's in the context window it could have "awareness" of having done it.
>And it can't actually execute any of those tools.
It can execute tools by outputting certain tokens in the right environment.
>It still relies on you to pattern recognize that its output should be a "tool call" and then perform the execution yourself.
Tool calls are not dependent on the human chat user executing them, right? They can happen automatically through the surrounding software.
>The "model" of its environment that an LLM has is 100% a construct of what a human told it and there isn't any way for it to differentiate between that which is real and that which is fantasy.
Again why would the fact that a human told it mean it can't result in consciousness? Why would lack of ability to differentiate between real and fantasy mean it can't be conscious? On my view, it would then be conscious of the fantasy.