Hacker News new | past | comments | ask | show | jobs | submit
karlkloss | on: Microsoft Outlook/365 still down after 12 hours
Why would anyone introduce a single point of failure out of their control into their company?

We're doing anything to avoid single points of failure in our products, but our IT is doing the opposite.

baliex | parent | next
Outside of tech and disaster recovery, people _really_ don't understand that some redundancy--in this context--is a good thing.

The idea of paying for a backup system, or even having one on standby for situations like this is just not on people's minds, or if it is, it's simply not affordable because it would cost you too much compared to your competition.

Razor thin margins and generally getting away with it is what makes this work. Until it doesn't.

rcxdude | root | parent
Or, they get the bad idea that they can just pay one supplier and that supplier will deal with the redundancy for them. When the redundancy needs to be at supplier level.
srdjanr | parent | next
I don't see a reasonable alternative here. Pay for another email/office suite provider, and duplicate all mails and files? Twice the cost and exposure surface just to cover 0.01% of downtime.

Sure, it makes sense for mission critical companies (utilities, emergency services), but I don't think it does for an average company. Especially that you don't take much blame if everyone is down.

themaninthedark | parent | next
Email in a large part is just a tool for communication(and mostly low priority at that). Where I work, we have backups: Teams( same) provider, Cellphone ( different provider ) , Landline( different provider)

We had a situation recently where cellphones were down, we switched to landline.

You don't need a 1 to 1 mode backup for everything, just a failover plan.

Unless your company is infrastructure or medical, I doubt they have a backup/failover for loss of power. Or water.

Arainach | parent
The difference between a distributed system and a monolith is that in a monolith you have some idea what your single point of failure is, but in a distributed system you don't know your dozen single points of failure until they happen in production.
cowlevel | root | parent | next
Example: in 2020, a 6-minute hiccup in a network switch caused asymmetric connectivity, breaking their distributed leader election algorithm, making that system read-only, breaking a different system's heartbeat mechanism causing that system to assume a failure, enact a recovery and degrade in performance, causing 25% of API requests to fail for 6 hours.

https://blog.cloudflare.com/a-byzantine-failure-in-the-real-...

adzm | root | parent
I love the dependency cycles that prevent things from being started up again from scratch.
temikus | root | parent
Google had a secret storage system back in the day, that required special encryption keys to be entered when it got re-initialised. The keys were in a safe. Guess where the combination to the safe was stored? ;)