I would've thought that you could actually have an incredibly strict seccomp policy for the jailer which would require you to find an LPE in something like open(2), which seems unlikely even in the "agent swarm" era.
That's basically what the Firecracker jailer does. Always my thoughts turn to the waitid(2) vulnerability; for a little while, you could pass kernel addresses through it and whack arbitrary kernel memory, and nobody was ever going to seccomp "waitid" out. But yes: it's a meaningful defense, if not one you'd want to bet your service on.