1. Rotary positional embeddings are representations of the translation group. The n-dimensional translation group ℝⁿ is abelian, so its irreducible representations are all one-dimensional ρ: ℝⁿ → ℂ, specified by a momentum vector m: ρₘ(x) = exp(i⟨m, x⟩). You can get a multi-dimensional representation by choosing several m, which are your learnable parameters.
2. If you wanted RoPE embeddings to also preserve rotation (perhaps you have scenes viewed at arbitrary angles), you are looking for representations of the special Euclidean group. The irreducible representations in four dimensions are your familiar particles: electrons, protons, etc., and in general are parameterized by a momentum vector and a discrete spin vector. You would have to do some kind of annealing process to train the spin vector (or you could just make it constant).
3. If you want to do attention on quadtrees, well, what is the group? Suppose every node in the quadtree has local coordinates scaled to [0, 1]. Moving up/down/to a neighbor in the quadtree requires an affine transformation of the local coordinates, which compose, so we can label every node by its affine transformation to the root (x, d): y → x + 2⁻ᵈy, where x is the node's translation in the image and d is its depth in the tree. The finite-dimensional irreducible representations are parameterized by momentum vectors m as well as a modulus M and basis vectors {e₁, e₂, ..., eₖ} where k = ord₂(M) and look like ρₘ,ₑ(x, d) = DPᵈ where D is the diagonal matrix with entries Dⱼⱼ = exp(2πi⟨2ʲm, x⟩ / M) and P permutes the basis vectors.
From what I usually see, ML researches just throw everything at the wall and see what sticks. Whatever approach turns out to work better, gets used more often, and sometimes theoretical background is developed for why it works. RoPE works not because it’s good math, just it turned out to work better than some other equally arbitrary approaches at some point, and then other approaches have also been developed that work better than RoPE.
Can you explain more what you mean here? What do electrons (elementary particles) have to do with protons (complex composite particles) and with general principles of four-dimensional spatial transformation? The electrical force is not a direct consequence of space-time geometry, and even less so the specific particles we happen to find in our universe.
However would be good to add that the solution can be structural(like in NoPE), then you dont need explicit representations and it just falls out of your architecture.