You misunderstood, I'm not saying that compiler bugs don't exist or that they're unrelated to memory safety.
I'm saying that these compiler bugs are extremely low on our list of priorities. If we want to eliminate all vulnerabilities in all of software we have about a century worth of logic bugs that are likely more critical and easily exploitable than compiler bugs.