They are trained from infancy (pre-training) to produce plausible-sounding mimicry, not to be correct.
Then in the instruction tuning phase we try to get them to unlearn their fundamental nature and start to produce correct output instead, which works to some extent, but not perfectly.
I think it's only a matter of time until we adopt better pre-training methods that don't bake in this tendency to BS, and hallucinations will be a thing of the past.