I know that Alpine has a package manager but I haven't tried it because I worry that Musl might present problems compiling software (I sometimes like to compile things myself, like Vim or Emacs). Is this worry valid?
Also are there any issues compiling Go and Rust programs on Alpine?
Alpine is a very popular platform for both compiling and hosting Go applications--especially static, no n-CGO binaries--given it's minimal footprint.
A common issue people have with Alpine is musl libc's DNS resolver, which can behave differently (thus confusingly) than glibc's. But that wouldn't effect Go applications unless CGO is enabled and it uses the libc resolver. Go's native resolver behaves differently than glibc's, too, but I don't think that's any more of an issue. And the musl resolver has fewer quirks today than it used to, so fewer surprises than a few years ago.
If you're only building common packages I wouldn't expect trouble. For example, I don't see any patches in the official vim package: https://gitlab.alpinelinux.org/alpine/aports/-/blob/master/c...