Hacker News new | past | comments | ask | show | jobs | submit
Microsoft is unable to run a reliable hosted email service, given infinite resources. What people don't realise is that the former is in part a consequence of the latter. Without a forcing function, complexity multiplies, silent fragility accumulates, end-to-end understanding dwindles. There is no incentive to simplify.
Im not so sure about the infinite resources part, more like a self inflicted austerity of resources - its a technology problem emergent from a organizational problem, emergent from a certain late stage mindset
~220K staff, ~$330B revenue, ~$200B EBITDA. How they direct those resources (wisely or otherwise, timely or otherwise) is a management decision. I feel I should also point out that when talking about people, "resource" is not the same as "talent".
Are those numbers accurate now that 60k were offered buyouts?

Anyone want to guess if any of those afflicted _could have_ caught an issue that may have prevented this disaster?

loading story #50482922
loading story #50482505
I have a feeling it's like weight gain. If you run lean your whole life you are fine but if you let the pounds pile on then you are in for a whole lot of pain when you eventually try and lose it.
Does having 400M users and 1B emails per day factor into your forgiveness?

I’m no fan Outlook and what they’ve done for the world of email, but if they’re Rolls Royce and this outage is an engine failure then that’s an expected level of service?

Or they’re Boeing and this a 787 Max. I’m on the fence.

loading story #50483039
>Does having 400M users and 1B emails per day factor into your forgiveness?

No.

Because of the infinite resources premise and because there exist email services of that scale that are reliable. Email is not an unsolved problem.

The rest of your comment does not make much sense to me ...

When Microsoft has an outage it tends to be global. They don’t seem to learn institutionally how fault tolerance should work across a planet. All else- I wait for the postmortem to critiq
I don't have very much forgiveness, perhaps only for those junior engineers tending the beast they grew, and which now has indigestion. No-doubt there are all the usual machinations of large corporations at work. Tech companies are no better (in some ways worse) than "normal" companies, because of the invisible complexity of software. Brooks, Ousterhout, and Stroustrup were not fools. But MS run a service, they get freedom to choose how it is engineered internally so long as it continues to present a service interface to the users/client apps/other internal and external services. This is an incredible level of freedom to choose what you engineer for. MS always had the ambition to serve the world with this service. It doesn't seem like they engineered accordingly. It's easy to knock them, but I think they must have made some significant wrong choices.

I'll spare readers the full anecdote, but I worked for the company running the largest mail service in the southern hemisphere, and at the same time a few of the large mail services in the northern hemisphere in the 1990s. We met with the Exchange team who wanted us to adopt their product. Where we could run a given service on 2-3 servers, Exchange sized by MS required 300 of them for just 1 of our countries. Fortunately our cash constraints prevented us going down that road. They offered the license for free and exerted pressure on the C-level, as you can imagine there were heated conversations and some choice language between mgmt and engineering on this topic.

loading story #50482620
No, and I've been discussing this with a friend today.

Internal software goes through rigorous change control, change windows, change freezes around key business dates, etc to minimize risk.

Then you sign up for some SaaS and they blow out email (which aside from security needs 0 updates lets be honest) midweek, mid business day without warning or reason.

>1B emails per day

Jesus, make sure you stay away from signal processing if this trickle of data scares you

Just for a sense of scale, on a single CPU core that is something like 1M instructions per email. ONE CPU CORE.

They could've split it into smaller e-mail services in stead of designing one monolith for the planet.

I mean it's e-mail.

loading story #50481503
loading story #50481685
loading story #50481581
Great insight. I would add that forcing function comes from the self, not from outside. The force is essentially the communication between the parts. When the communication breaks down, things fall apart, establishing a new kind of communication across the pieces to achieve equilibrium. The physical structures also have the same rule. They will crumble into a pile of lego bricks to achieve a new state of equilibrium and a simplified network of communication across the pile.
Today is sounding like one hell of an incentive.
Why?

Do they lose any significant number of customers? Do anybody in leadership get fired?

Do reading the twitter status updates give you the impression of an organization that takes responsibility seriously?

Or do it sound more like an organization that is optimized for evaporating responsibility?

I think this will change nothing.

loading story #50480896
loading story #50480893
Seriously, who will change provider because of this? Our institution is so brainwashed and hidebound to Microsoft it would take a lot more than this. It seems like an abusive relationship: the bad times just increase fear generally, so make change less likely.
loading story #50480763
loading story #50480772
loading story #50481554