> you'll need a really comprehensive set of evals and moniotoring to ensure stuff doesn't break if/when you change providers

Unless your self-hosting open weight models, you need this even if you don't change providers, since you have no control of what's on the other side of the API.