Thats not enough. This problem pops up everywhere people do CVE analysis. If a tertiary part of some library is hit by some arbitrary CVE, why should that affect the core functionality people actually use? If I use ChaCha20, and that's bundled in a library with md5.do I care that their md5 library has a vulnerability? No, not really.

If you keep driving the "seperate the packages" angle you end up with is-odd, and that not a nice play to be either.

They should be in separate packages so you won't force yourself to patch CVEs in the parts you don't use?