I would look at osxcross[0] if you haven't looked into it already. It can at least do the "build stuff targeting macos" part (well, relatively speaking). Swift support kinda breaks down, unfortunately, and actually getting it set up is a bit of a hassle. but it does work.
Would be nice to have a "sanity check the resulting binary" tool though like you can use Wine for Windows builds.