The secure element rate limiting is integrated in a way that can only add security. It can't reduce the security of key derivation since it's only used to get an extra input for deriving the key. There are various other inputs in addition to the PIN/password too to add various security properties. The most relevant one is the token obtained from the secure element rate limiting system via successful authentication, but there are others.

The secure element comes from the same company making the main processor for both the upcoming Motorola devices (Qualcomm) and Pixels (Google). Why would they put a backdoor in the isolated secure element rather than the CPU? The backdoor argument can be made about any hardware.

A key is derived from the PIN or password and used for multiple purposes by using separate statically keyed hashes. One of those purposes is obtaining a token from the secure element to implement rate limiting and extremely reliable erasure of all the encrypted data for the profile. It's also passed alongside the obtained token into the final key derivation process. Why would you want to remove the secure element integration? That would mean losing the rate limiting with no benefit and solely relying on erasing key derivation material stored on the SSD for wiping data. That would still work due to hardware support for it but the SSD isn't nearly as reliable as the secure element and can be copied at a hardware level before trying any PIN/password.

We want more secure element features including duress PIN/password support as part of the rate limiting.