> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan

At some point the AI might become so powerful that whatever it reasons through isn't any less real than a computer simulation. If we assume that a person in a perfect computer simulation is conscious, then if it reasons about how people might suffer, it might simulate the outcomes, and there will be a conscious experience of those people suffering.