Hacker News new | past | comments | ask | show | jobs | submit
> Current status: We’re investigating an issue impacting multiple Microsoft 365 services and we've determined the affected infrastructure is experiencing elevated load across multiple regions, which is likely contributing to the connectivity errors. We're restarting and repairing impacted portions of this infrastructure to restore capacity, and reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load. In parallel, we're evaluating options to redirect traffic to alternate infrastructure to expedite relief.

DDOS?

As someone who worked in infrastructure for a large global CDN, this sort of overload issues happen all the time when trying to bring infrastructure back online after an issue.

First, you have every client retrying connections, whether automated or manual retries. This generates extra load over normal.

In addition, whichever systems come online first often get a ton of traffic sent to them, which can immediately bring them back down if you don't have ways to prevent it. You will get a wave of systems coming back online, a ton of traffic hitting them, and then falling over again.

Now, I am sure Microsoft is used to this and has procedures and systems in place to mitigate the issues (I know we did).

However, sometimes if the right combination of services are failing, even the strategies you had to avoid this sort of failure can themselves fail.

If Outlook can go down because of DDOS then it's a much worse problem.
loading story #50474760
My guess: too many DC resources dedicated to AI, leaving insufficient free capacity to handle traffic spikes
loading story #50473359
isn't Microsoft 365 called Copilot now?
loading story #50474155
loading story #50472885
Nah, just a "Reply-All" chain at NVDA...
>reviewing an unexpected increase in a specific type of service traffic to determine whether it's contributing to the elevated load

This seems to be the smoking gnu, now what would it be ...

loading story #50475296
loading story #50474109
loading story #50474556
loading story #50473453
Probably self inflicted by some code change, as they mention "specific type of service traffic".

EDIT: Seems like now they confirmed that: "We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service."

Microsoft 365 Status

@MSFT365Status

16m

We've not identified any recent change correlated with the start of impact; however, our investigation indicates the failed health checks may originate from a specific component that's responsible for processing server health. We've taken steps to prevent additional servers from being impacted while we work to return the affected servers to service. More info under EX1550543 and MO1550821 in the Microsoft 365 admin center, as well as status.cloud.microsoft if the admin center is not available to you.

Currently (17:22EST):

> Previous status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.

If failures started at some point in time then something must have changed just prior to that. If no changes in traffic were detected (no DDOS) then it must have been a change, be it code or config. Duuuh.

> then something must have changed just prior to that

Not at all, they could have simply hit an unexpected behavior for their infra they hadn't seen, tested, or expected before. Global scale infra gets very, very complex, and tbh this is Microsoft... they're not very good at it.

Unexpected behavior doesn't happen out of the blue, it's not LLM, these are deterministic systems. It's usually config change or code change, not necessarily in the component that's misbehaving, but it is somewhere.

More often than not, it's DNS :)

Is this a case of "who shall healthcheck the health checker?"
{"deleted":true,"id":50472326,"parent":50472046,"time":1794509791,"type":"comment"}
Too many AI agents connecting at once?
loading story #50472331
loading story #50472554