It's just doing what it's supposed to do, analyzing the statistical patterns of a natural language prompt and matching it to a likely response.

What's ridiculous is an assumption that it "it should never do something so ridiculous" as if an LLM is a person who understands what that word means in any context other than next-token prediction.