The benefits are two-fold:
1. Background tasks don't impact your own issue (as you say).
2. Many CDNs can saturate 4 Gbps, so if you download something, the waiting time is much shorter than say 100Mbps.
Whether it's worth it all depends on the price difference.
4Gbit is kind of the sweet spot currently. 2.5Gbit equipment became quite cheap (e.g. Ubiquiti Flex Mini 2.5Gbit managed switches are often around 50 Euro), existing Cat 5e cabling still works reliably, and if you have a router with 10Gbit upstream and multiple 2.5Gbit downstream ports, none of the subnets can completely saturate the upstream bandwidth.
Also, these bandwidths rock for doing an initial backup, downloading a full backup, pulling many Docker images, large and frequent NixOS rebuilds (unstable-small, yay). Downloading GTA 6 (not me though, not a gamer).