Birds are alive, are conscious, flap their wings, and fly. Planes are not alive, are not conscious, do not flap their wings - and fly.
Similarly, current AIs are not alive, are not conscious - but think.
All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is assuming that because they think, they're alive and conscious as well. Current AIs are neither, and are therefore profoundly and qualitatively different than humans - even though they do think.
I don’t know “what it’s like to be an LLM” but at some point it will be like something and how will we know?
An albatross might be able to go days flying without a single wing flap and no vertical sources of lift by using dynamic soaring in the wind gradient at the surface of the ocean. Perhaps that's something only birds can do. Except the glider pilot Ingo Renner once found an amazing shear layer at 300m altitude and stayed there with dynamic soaring. Remote control gliders use the lee of ridgelines to approach Mach 1 with dynamic soaring.
Perhaps what defines a bird that flies as opposed to a plane is that a bird produces thrust by flapping its wings? Even an Albatross must flap its wings if it has to take-off from water. Maybe we could add that the flapping is driven by animal muscles? But then is the human powered ornithopter Snowbird a bird that flies as opposed to a plane?
Of course this is all ridiculous because everyone knows what you mean when you refer to a bird or plane. We have other ways to definitively identify the difference rather than their mode of flight. It's trickier when I'm asked if an AI is conscious. There is no definitive base-line to fall back on to decide if this is a conscious or conscious-less thinker.