yes! it's the worst!
I've been using Instant Space Switcher (which got a small callout in tfa) as a targeted fix for this, and it's lifechanging
I've also switched to Instant Space Switcher, it is soo good! Previously I used BetterMouse for only this feature but they made the space switching worse in later versions (slower, on-par with the default macOS speed).
Here's the link if anyone is curious: https://github.com/jurplel/InstantSpaceSwitcher
loading story #48368052