By building pain into the system. If your hands dealt with injury directly without sending pain signals up to your brain, you'd never change the behaviour that led to that harm or reconsider your priorities. Like it or not, sometimes the best thing for an organisation isn't to just fix every problem and prevent it from bubbling up; it needs to be treated like a learning opportunity for org leadership, which means sending the pain signals upward before just repairing it.

Building the right incentives around that can be tricky, those incentives need to ensure the highest levels of management aren't themselves disincentivising their directs & their departments from surfacing pain & problems - but it's also pretty common for people to mask those signals purely out of a well-intentioned desire to help. It's important to coach people on the idea that in large group sizes, it's more efficient to let certain kinds of problems play out and not be so reactive to them.

Too many companies ground their performance incentives & processes around oversimplified ideas that don't match the reality of human behaviour

In my 35+ years in IT, the "hero attitude" was the one in the top three I most hated traits in a person working with or for me. And talking about traits, I considered crucial to always have in my teams a "saboteur" engineer - the one who thoght, found, come up with all the way we could break a design, service, infra components, app, etc., when all the others were designing or operating for perfect or normal conditions.

At a previous job the CEO/owner had the idea that you'd get some percentage of any cost saving your could find as a bonus. Something like 20% of the savings for the first year.

My colleague in the IT department had one idea, replace our commercial certificates with Let's Encrypt and drop the EV requirement. In total he'd stand to get a bonus of a little over €2000. He never got the money, because things like that was part of his job apparently.

I think a good place to start is tracking all the proactive things being done and reporting them. At least then maybe someone will see why it’s quiet, because you’ve anticipated the problems and stopped them before they start.

When things come up with other teams, you’ll have a catalog of tasks that were done to show why you didn’t have the same issue. The work was done, just at a better time to avoid downtime.

Isn’t this a universal problem though, not just software industry? Even at home, if one kid just does his thing quietly but another kid is difficult, what do we say? “John has his problems but he is trying, we gotta encourage him”. While his brother gets no praise or attention for just doing his thing quietly without fuss.

When things run smoothly, very few people notice. When things break, everyone notices

Econometricians can solve it, bc we can create rigorous models that map causal inputs to output.

It’s extremely advanced technology, though, and most CEOs would rather rent seek / camp than give up some decision-making power (and very few are even aware it’s possible).

> I'm not sure how we solve this, other than having management come from engineering.

I disagree with the implied idea here that "engineers are better managers". The solution is to have good management, not to assume that "engineers are better managers". I have seen good and bad managers, and in both groups there were engineers and non-engineers.

The tragedy is that “nothing broke” looks like “nothing was done” to people far enough away from the system.

> I've been in those companies where "struggling departments" ended up getting all the praises and raise in budgets the following quarter because of the heroic saves they did, and raising awareness on how important they are... For stuff they totally caused on themselves.

This is a very game-able system, and I'd wager a decent amount that any senior engineers on those teams know exactly what they are doing. In a lot of (broken, but aren't they all) management structures, it's better to be seen to swoop in with the save than to quietly fix it ahead of time.

And if your management is structuring rewards like this, it leads to your seniors anticipating a bunch of these failures, lining up 90% of the fix before hand, so that they can jump on the oncall escalation with a 100% "Hail Mary" of a fix...

itemize the problems you are preventing

"Accounted for X situation" "Added gaurdrails to protect against Y"

When working as a business analyst i have to do this sort of thing all hte time or else id get no credit for half my work

Ah, yes - the person who comes in at 7AM and gets shit done by 11 is a slacker, and the one in the office just doing nothing after 6PM is the hard worker. Same thing.

You can't fix this. Out of sight, out of mind. It is hard-wired into us. It's all about the optics, and will always be.

> It's a serious problem in this industry

s/in this industry//

Managers will let you get away with anything if you time your reports correctly. They also don't want to sit in meetings where they are reminded of better outsourcing alternatives and they chose to dogfeed instead.

We've become too comfortable, since actual toil is no longer seen in the company: Manufacturing is overseas, customer support is overseas, logistics is an afterthought with established guarantees. Thus we want the mild weather and smooth meetings. If your engineering team is too smooth, maybe you should already branch out to help other related but "struggling" teams to get your hands dirty and noticed.

This thinking eventually results in The Scream Test. When the screams come as a system fails that is when they act on it.

Alas, for many parts of society there is a large amount of people that would rather be reactive than proactive. It means it is easier today but harder long term.

> It's a serious problem in this industry

It’s not a problem in this industry, it’s a problem everywhere.

> I'm not sure how we solve this, other than having management come from engineering.

You mean the engineers who are causing the chaos you’re complaining about?

Engineers aren’t some magic group of people who know better than others - we’re just as fallible as other people.

I guess the point of view is that if a department is well running, it means it is overressourced. So you reduce the ressources until it's breaking point, just enough for it to not fail. A jaded service manager told me it was part of its official training: if the clients was too satisfied that meant that human ressources were wasted on them, so he had to spin plates between clients. I guess it was economically optimal.

Since it's entirely possible (extremely likely?) the "problem" would never materialize, this is quite reasonable.

I run the media tech at an university solo. Needless to say I am underfunded. But more importantly, the infrastructure was underfunded too. I made it my first policy to also report near misses up the chain with their full implications, e.g. a list of events that we would not have been able to make.

E.g. that time a central media controls power supply broke down which would have made using one of the most prestigious rooms impossible. I fixed it myself by swapping in a spare power supply from a used unit, then went on to remind them twice a year that we are now living on borrowed time and I take no responsibilities if a fault I predict to happen and get no funds to fix will in fact happen. 4 years later I got the funds.

Having stuff costs money. Everybody wants to invest funds once, but nobody wants to keep paying for maintenance.

lol. I hate presentations. I like to run a tight ship. But that does not shine, so they made me do presentations every quarter. If you do some work, you must "take" credit. It is kinda a need when you manage people since you need to build their careers.

I finally moved on to be an IC. Same story, same pressure :) You need to present to directors not because they need to know, but because your managers have a quota of N presentations per quarter, and if you back out, someone else needs to step up.

Needless to say my productivity reduces by half and sometimes to almost zero during the week or fortnight of presentations every quarter.

Which they should. I've been lucky enough to work at places that had great non-technical managers that promoted based on great execution, as well as highly technical managers that also promoted based on great execution.

Now I'm at the other kind of place and it sucks. They'll fire the performative engineers though during layoff season. It's almost like they like playing politics until it really matters.

I feel that disconnect is everywhere, when the suits dont see anything and act on reports

I feel like this is a cultural symptom and something many people are hoping to solve in healthcare. Basically we treat solving problems as amazing rather than preventing problems. You get rewarded if you treat a sickness instead of keeping a healthy person healthy.

This is the same thing. We need to reward things never going wrong as a society since this is pervasive.

Car industry best practices

>>I'm not sure how we solve this, other than having management come from engineering.

Given the whole point of management is to work to ensure their own survival and growth, it would in their interest to kill genuine competition when its coming up.

Who wants to raise their new competition and lose to them, no one!

Track leading indicators, pricing them if possible.

could an AI product solve this?

I believe it's a problem in most industries and even humanity in general. I don't believe it's a business problem at all.

Heroes are lauded even if they solve problems they themselves are the cause of - which is conveniently either forgotten or denied - or they are solving non-issues that are deemed important by the ignorami-class. Politics, for example, is utterly dominated by this dynamic.

It's the first instinct: let the expert run the show. However, one of the (many) ways to let a complex project fall apart completely is to hand over full control to engineers. I'm one myself, but I know what I'm good at and what not. Dunning-Kruger is often mentioned in these discussions, but don't forget it also applies to engineers that often lack any management or leadership experience of any appreciable kind. They vastly overestimate their ability to handle management and organization-wide issues and tend to not only miss the forest for the trees but actually miss the trees for the leaves.

"Unix: A History and a Memoir" by Brian Kernighan actually mentions how proper management was crucial to their success. It's a detail that's frequently conveniently forgotten by the engineers who think themselves better than the "suits". For the record, I don't claim engineers are the primary problem, but it's not just management's either. Quotes like "who holds the company standing" and "who understands how to double click" are enormous smells and IMO make quite clear what's happening here.

I don't have ready-made solutions unfortunately, but I do wish we would look further than "it's the suits". It's a systemic, human problem that I believe is a natural result of operating under informational constraints and, very human, cognitive biases on all sides.