> LLMs basically never shower, do the washing up, kick their foot in the dirt while sitting on a park bench with a coffee: the domain of the default mode network. And they never sit frustrated looking at things, guided by their long experience, with the itchy brain feeling that there is a solution before they can even say the words.
I'm sure the training data of LLMs contains a lot of those things, though.
It contains the averaged written ghosts of the outcomes of those things (as well as, I am sure, some descriptions of those experiences, attached to descriptions of the outcomes — "so I was sitting on the bench on the common and…" would be routine in a training set derived from me!)
But they are not the things themselves — they are incomplete representations; just words that approximately describe the application of that intuition to the problem at hand.
Think about how you visualise complex code projects in your mind — how it's not quite diagrams or buildings or words but multiple things in a sort of multi-dimensional, itchy space that is satisfied when a really good implementation appears. Think about how often the code you have built retains that wordless shape long after you wrote it down. Think about how impossible it is to adequately describe it.
loading story #50439732
It may contain writings about these things, not the actual things. That's more like saving a jpg at high compression over and over if you want catchy metaphors.
loading story #50439719