If you want to dive into CUDA specifically then I recommend following some of the graphics tutorials. Then mess around with it yourself, trying to implement any cool graphic/visualization ideas or remixes on the tutorial material.

You could also try to recreate or modify a shader you like from https://www.shadertoy.com/playlist/featured

You'll inevitably pick up some of the math along the way and probably have fun doing it along the way.