Turns out it was due to a combination of things.
I was using niri (Wayland window compositor) and this input latency was present with or without v-sync turned on. It happened when I was using a 60hz 4k monitor with an NVIDIA GPU.
Then I tried playing the same game on a laptop (same distro and dotfiles) with an AMD GPU and no external monitor. The delay disappeared.
Then I played the same game on that laptop but hooked it up to the 4k monitor and I had the same keyboard input latency only when v-sync was enabled. When I turned off v-sync and capped my FPS then the input latency was reduced by an amount that I could no longer perceive the delay.
Then I put an AMD GPU in the original desktop I was testing and reproduced the same results as the laptop.
However, when I switched to using KDE Plasma with X or Wayland, the keyboard input latency disappeared. This was with the 4k monitor and both NVIDIA and AMD GPUs.
I reported it to niri but it hasn't gotten traction, I just know I can reproduce it on 2 completely different systems with different GPU vendors and hardware when the common ground is having a 4k 60hz monitor hooked up.
Maybe AI coding agents will make the situation better, but because open source maintainers are too dim to understand the complex changes the AI makes, and too poor to have their own AIs to help them, they won't take the changes. I make improvements to open source but am forced to keep them to myself.