And it seems that, given enough input/outputs/compute, it is possible to train the necessary function.
Details of how the building bricks look like (matmul, electromagnetism or quantum effects) are not that relevant in the broader picture.
What is missing right now, is the fact that the function in question changes over time in biomachines, while our LLMs are static at inference time.
a) The brain might have an entropy source (then it can't be modeled as a function). Trivially to fix, and in some sense, with diffusion models starting from random numbers, AI has done so.
b) The hidden function might be not computable. I would have no idea how that would work, but I think this is what it boils down to if people say "the human brain is more than a machine".