That sounds cool. For my uni project 10 years ago I chose to use neural networks to detect guitar notes, but not by pitch, but to detect exactly which string/fret was played (because you can play same note on different strings). It worked okish with a very basic network, quite suprised. I made a small game you could play with your guitar.
Since then I wanted to develop further such a game, and now I'm almost ready to release https://ultimidi.com , check it out, it's not really a rhythm game, but my goal is to add more mini-games to it (including rhythm games) and more instruments (now has piano, but planning to add guitar, ukulele too, not sure if drums would work too).
Very nice! Extracting the guitar track would be awesome but it’s so much hardier than drums. Even cymbals are hard. And guitars have chords and distortion and…
Yeah, that's why my idea was to use neural nets trained on manually labeled data, instead of relying solely on pitch detection.
Also, probably AIs nowadays can help a lot with generating training data, and even improving detection.
That being said, even SOTA voice recognition has only like 95% accuracy, so it gets wrong 1 in 20 words. Imagine if transcribing a melody and 1 in 20 notes are wrong, for a fast melody that would mean a wrong note every few bars.