Memorizing books is absolutely possible. A great book about it is Moonwalking with Einstein (https://en.wikipedia.org/wiki/Moonwalking_with_Einstein), written by a journalist who was a skeptic but ended becoming a competitive mnemonicist himself. Some people intuitively learn the techniques early in life, and for some of them it becomes a habit.

Same thing applies to mental arithmetic. There are techniques for doing this. IIRC, ironically there's a famous case of someone who claimed to have memorized some hundreds of thousands digits of Pi but was suspected of using mental arithmetic techniques when reciting them.

Still, your point stands. People do embellish these kinds of stories based on stereotypes. It's certainly possible, but any particular anecdote may still be suspect.