Yeah, I don't think we can ignore the small issues anymore.

No matter how unlikely the issues seem to us, an LLM will happily chain dozens of them and do complex timing attacks to eventually gain access.