Why do you think next token prediction didn’t take off in multi modal space? That’s right! Because it’s prohibitively slow. A diffusion model, by contrast, can update an entire latent image in parallel at each denoising step. Imagine decoding images or a 10 second video one token/pixel at a time! Secondly, images are not naturally sequential. They are spatial representations of information m. Diffusion can progressively refine both local and global spatial structures in the image.
I actually think there is room for next token multi modal prediction but more where world models are concerned. Like predicting the next set of futures for a robot. But actually now I would opt for decision models instead.
But in short, your analysis, and doubts, are incorrect as to why generalist LLMs are not used instead of bespoke diffusion image gen models.