The quality and internet connection stuff will improve over time. The latency problem essentially never will.
Funny enough, I was just recently revisiting some old Giant Bomb podcasts - the Game of the Year shows they would do every year hold up nicely as a map of worthy titles. And in the 2019 ones (iirc), they spent some time talking about Stadia.
Part of the pitch was that because it was in the cloud, and therefore theoretically had access to better hardware than you could afford, you'd be able to play games at higher resolutions and with better quality. That might have been an interesting tradeoff to make against latency, especially for games where millisecond accurate inputs weren't really important. But in practice, you'd deal with stuttering, compression artifacts, and so on - and that meant that the promised "ideal" quality never really materialized.
I agree with your assessment, in short. I can see the quality problem being solved - but latency will always be an issue for some class of games. I guess the potential solution to that might be that, in a world where cloud gaming becomes the norm, "twitchy" titles just don't get made. As a dude who likes platformers, fighters, rhythm games, and so on - I really hope that never comes to fruition.