That's the drawback of monolithic Linux and tendency to put everything in kernel.

It is absolutely possible to have very minimal kernel with drivers and many other subsystems running with lowered privileges.

That would greatly reduce privilege escalation possibilities. Right now single vulnerability in any kernel subsystem gives you highest access to the system.

Imagine that some USB device driver runs as "nobody:nobody" process having access only to selected syscalls related to its functions. Even if it has vulnerability, you won't go anywhere from there.

Modularity and very granular access controls are the "secret" to secure system.

And the most privileged part of the kernel could be so small that it could be verified with advanced methods, basically proving its correctness.