If I did all of these calculations by hand, would it be conscious...
That's a powerful argument I haven't seen stated quite that way before..
I do think it's hard to know when consciousness exists, because we can't really prove it for our neighbor. We just intuitively know that it would be crazy, even immoral, to assume otherwise.
But, It's likely easier to dismiss consciousness, once we understand the mechanism, than it is to prove it.
It's basically https://en.wikipedia.org/wiki/Chinese_room – a classic in the philosophy of consciousness and AI.
It's was a genuine question though. I'm not saying LLM use neurons, are like neurons, or are meant to be like neurons. I'm saying if you can do the math, for a substrate, that doesn't mean much. The simple math that runs it isn't the secret sauce.