I have a mild distrust of some of the cheap IP KVMs. I don't think vendors are malicious, but I don't expect they get it right every time either.

Admittedly, I haven't looked at any open-sourced firmwares either which could have improved things.

I have found the Sispeed USB KVM very useful, the convenience is well worth the $50 it cost me. The UX isn't great but you don't really need it to be. It works (most of the time) via WebUSB for the keyboard mouse.