As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks.
From what I usually see, ML researches just throw everything at the wall and see what sticks. Whatever approach turns out to work better, gets used more often, and sometimes theoretical background is developed for why it works. RoPE works not because it’s good math, just it turned out to work better than some other equally arbitrary approaches at some point, and then other approaches have also been developed that work better than RoPE.
loading story #50510992
loading story #50510786
loading story #50510012
loading story #50511083
loading story #50512016