If you control the domain then yes you could. But if I want to put a link on my website to some SVG hosted elsewhere and I want it to be safe for you to open that link in a new tab then there's not really a way for CSP to protect you the user from the host deploying a malicious SVG.

Like opening a PNG in a new tab is harmless but opening an SVG in a new tab is opening a pretty substantial can of worms.