I've run into GC pauses, I think in many (most?) cases there is some class of bulky data that you can either move into slices of pointer-free structs (so the GC doesn't scan them) or off-heap entirely. The workload where GC is slow is also likely prone to fragmentation so whatever the language you'll have to deal with it.
Java with its copying GC deals fine with fragmentation albeit at the cost of more upfront memory. And even in Rust one can change the allocator to try to deal with fragmentation. But with Go there is simply no good options besides the rewrite.