Hacker News new | past | comments | ask | show | jobs | submit
Microsoft 365 Status

@MSFT365Status

16m

We've not identified any recent change correlated with the start of impact; however, our investigation indicates the failed health checks may originate from a specific component that's responsible for processing server health. We've taken steps to prevent additional servers from being impacted while we work to return the affected servers to service. More info under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as status.cloud.microsoft if the admin center is not available to you.

Currently (17:22EST):

> Previous status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.

If failures started at some point in time then something must have changed just prior to that. If no changes in traffic were detected (no DDOS) then it must have been a change, be it code or config. Duuuh.

> then something must have changed just prior to that

Not at all, they could have simply hit an unexpected behavior for their infra they hadn't seen, tested, or expected before. Global scale infra gets very, very complex, and tbh this is Microsoft... they're not very good at it.

Unexpected behavior doesn't happen out of the blue, it's not LLM, these are deterministic systems. It's usually config change or code change, not necessarily in the component that's misbehaving, but it is somewhere.

More often than not, it's DNS :)

Is this a case of "who shall healthcheck the health checker?"