A few things to consider here:
- This is an abnormal case. Most hardware will work with in-tree drivers. Indeed, few vendors provide out-of-tree drivers for Linux.
- Nvidia is an established and reputable source. We aren't talking about some small hardware developer who doesn't have the resources to create secure drivers.
- Most Nvidia cards have in-tree drivers. There is a loss in performance, but the option usually exists.
Those who do, choose to do so and generally take responsibility for their actions.
It's not the same as tainting a kernel and just winging it.