The more I think about this, the more I become convinced that consciousness, as understood by humans, is meaningless once you can stop, store and replay the internal machinery that is supposed to produce it.

A human brain in a jar is still human, still conscious - and can still suffer. But if you somehow managed to digitize the whole thing, and run it in a computer it becomes something different entirely. You could record the most pleasurable thing in existence and have the digital brain relive it a million times, and it would be equally meaningless to torturing it a million times.

This is NOT inherently tied to meat vs machine - although it's difficult to imagine how you'd access the information stored in biological neurons, while for silicon chips it's trivial.

Whatever makes experiences, both good and bad, meaningful is tied to their permanence. Memory rooted in linear time, not something you can store, load or replay. Remove that, and whatever you're left with might be intelligent, but not conscious.

I don't think you could build something with LLMs today that would be considered conscious, even if you somehow manged to keep their context window inaccessible and linear in time. The separation of training vs inference probably makes that infeasible, even if you store "memories" in context, once the contents in it become too disjointed and too numerous, the resulting output of the LLM becomes gibberish. But it is certainly something that can change in the future.

Conversely, even the most intelligent and capable artificial intelligence system, far exceeding human capabilities, would not be conscious in a meaningful way, if you could store, load and replay everything it does.