However, based on this article, i have the feeling that on the contrary, JEPA is filling a super niche problem, and is pretty much only trying to improve robotic motions in space.
Which of the two ideas is it ? Can JEPA technics be applied to a much more wide range of problems ? (like LLM and transformers)
OF course your decoder model could end up doing a lot of the heavy lifting if your decoder loss was based on text reconstruction methods (which almost all LLMs use for eval purposes)
For certain tasks, you do need pixel/token level reconstruction fidelity to do a lot of things. Like image generation or text generation.
For other tasks, having a coherent representation of the world that can be computed quickly and intuitively is more salient. I suspect the human mind works more like JEPA than like an autoregressive LLM. Human children train on far less data to achieve extreme fidelity on a lot of tasks generative reconstruction loss based AI/ML techniques struggle with.