These are contrived benchmarks at the extreme end of things. In real world usage the difference is drowned-out by the delays of so many other things happening in order to complete a handshake and key exchange. The mildly higher performance of RSA 3072 versus RSA 4096 wasn't even a big bonus during the CPU performances we had 15 years ago.