Beautiful illustrations I find, 'Playing' is just the free and motivated version of 'exploration'.

One thought on your nicely illustrated "key observation [is] that neural networks tend to place features along directions": my guess is that the neural net was TOLD to behave that way by choosing e.g. Cosine Loss?