If you are a "delivery driven" person who cancels holidays to keep working, I would say to you, from own experience: stop being stupid.

I know, that, specially when you get recognised for the hard work it is hard, but you will regret every minute.

If you work for a company that counts on your holidays and vacation to sell their products, you are helping to create the problematic world with have today. If many do that, more will need to do. If none do it, and act like it is an absurd to request it (and it is) nobody is forced to do it and the company is forced to do real estimates even if that hurts your fav CEO pocket

Any impressionable kids reading... this story playing out like it did is highly company-dependent, as well as luck-dependent.

In a healthy company/organization:

* That outsourcing implementation approach probably wouldn't have happened, because an experienced person could've guessed how that would turn out, before it was started (it's such a cliche).

* Earlier on, people would tell the CTO it wasn't working out, instead of lying and saying the opposite.

* Whenever they needed to do something smarter/creative to rescue the project, they could work with the CTO on that, and maybe even with the customer.

* There wouldn't be marathon whip-cracking over the holidays, especially not after the team was already burning out.

* A manager/lead would go to bat for the success of the project and the health of the team, and insist upwards that the team needed a break over the holidays, if that needed to be clarified.

On that last point, in a "half-healthy" organization, a manager might intentionally be very vague, and omit information. That happens, and might or might not be a good idea, depending.

But the way this story is told, it sounds like the manager/lead outright lied repeatedly up the chain of command, which is generally considered very-bad, in both healthy and unhealthy companies.

I'll add that these things are vastly easier to armchair-quarterback. Certainly we're all going to make mistakes, especially when put in difficult positions, and/or when overextended/fatigued. But it helps to look at scenarios, to try to learn from them, and to think from a distance how they might've been handled better, so you're armed with that "experience", the next time you're tossed into a rough situation that has some similarities.

> The vendor product stored every customer transaction as a json record in a giant json document.... Adding a new transaction involved reading the entire json document out of the database, then appending the new record to the end.

Does this sound insane to you? Obviously it should, but in the vein of "insane things" I was helping do tech diligence of a potential investment for a fund. The startup's users table also contained the "tickets/booking" data. Each ticket was a column. So if your most active user had 5 tickets in total history, you needed 5 columns. These guys had 500+ columns by the time I reviewed it, and were looking for an investment to "scale".

Of course, it's a solvable problem, but as you can imagine everything was designed in some backwards, janky way. That was just the most obvious "wtf" moment.

They did not get the investment.

Everything in this story is dysfunctional, including the protagonist's approach. It is completely unacceptable for a team leader to give their people time off and lie about it and he is well in to territory where the company could fire him. I wouldn't be surprised if it was a fire-for-cause situation, in fact. Although upper management sounds so off the rails that he would probably get away with it and get a commendation of some sort; it does seem to be a play to the environment he's in.

A tip for new team leads; there is nothing here to feel proud of and a better play is to kick up a huge stink about people are working overtime and insist that the vendor be held to sane standards or project scopes reconsidered to fit in a normal work week schedule. Trying to make this sort of situation work is going to burn people out for nothing, or get people fired with no potential upside.

Unless I were truly desperate for work to keep my family going, a team lead has a responsibility to protect their team's reasonable work hours from insane requests. That is a hill to get sacked over, but not to lie for.

Saved whose day?

The shitty vendor who delivered crap?

The CTO who surrounded himself with yes men and was clearly completely unaware of anything happening in the company?

The devs who were worked to the bone, but don’t worry I gave them a week off?

The protagonist who lied to everyone to hit an arbitrary date for a company that clearly doesn’t give a shit about them?

This story made me cringe at every turn. I’m a hard worker and I have put in extra hours to make a launch go smoothly for a client here and there but this story is pure insanity. I’m willing to put in the extra time on occasion because of the “relationship” I have with my boss and knowing that I can always tell them the truth. In fact that’s a core concept of having a no-blame culture, you only get that IF everyone is telling the truth.

Lying like crazy to meet a deadline for an idiot CTO is quite literally insane. If you find yourself in a situation like this then GTFO of there and find a better place to work.

Maybe I'm lucky but I've never been fired for telling the truth and the truth is easier to align to. "There's a bug in a third party library that's part of the critical path "We can make it harder to hit the bug but can't fix it until the vendor fixes it", or "user growth has revealed performance issues no one thought we'd encounter this soon, we can spend three times as much money on infrastructure to mitigate it for the 2 months it will take is to fix it or lose customers due to performance" or "our biggest customer didn't know what they wanted until we delivered the first iteration, now they want something that isn't at all what we thought we'd be building. We can build it and be profitable or follow our dreams and die".

Again maybe I've been lucky but honesty has worked well for me.

> I dial in every morning for the mandatory death march status call with the CTO and I lie.

> “The team is working hard. Today we hit milestone integration point #73.”

> “The team made good progress yesterday, we finished another web service.”

> Every day I showed up and told the big boss that we were hard at work on stuff that we had already completed over the previous month.

I think it's a really important detail that he was lying about work that was already done. That looks like "underpromise and overdeliver" from a certain angle.

I'd feel worse about lying about work that's not done yet. Certainly riskier, and maybe not the best feeling for the team when they get back: "Here are your tickets, I told the CTO they were already done, so hurry up."

Developer interactions with management suffer from huge information imbalances, and a lack of trust.

For example, I'm working on a project now to update a code base running on a (very) old compiler. We've been working for a year, and large chunks of the system are done. One (fairly major) part is yet to be completed.

Management doesn't understand the process, or have confidence that the project will ultimately be successful. I don't blame them for being skittish, software projects (especially conversions) have a long history of failure.

We had weekly meetings with them on progress - which are now twice-weekly (because that will speed things up.) Mostly they don't attend directly, a middle manager acts as a go-between.

The developer view is that of course this will work (personally I've never been in doubt on that front) but the time-scale is uncertain because the system is large and old. But we're talking months left, not years. Yes, I'm aware of the 80/20 principle, but we're well into the 20% already.

Of course to management it's a binary state, done or not-done. It's hard for them to guage progress. There's no "trust" in what we say.

Which makes sense to me. We might have done 0 work for a year. If all we did was the meetings -they wouldn't know-. (Its a fixed price job, so it doesn't make sense for us to drag it out.)

Of course the risk is all on them. They've spent a bunch of money and are nervous. They made (good) tech decisions (based on the advice of outside people who understand the tech) but they feel very uncertain. Ultimately if the project fails they'll take a hit (not so much us.)

There's no easy fix here - you can't just argue that managers should be technical (the tech is not their core business), and more "consultants" won't make them feel warm and fuzzy. The best we can do is push through and deliver.

The two things we know about the vendor are unforgivable: the fact that they let a core piece of logic depend on indefinitely augmenting a Mongo record, and the fact that a deliberate effort by 3 somewhat above average people could replace them in about 3 months of work.

The fact that the vendor reached the state of doing business with a Fortune 500 client indicates how helpless organizations similar to this client feel facing even modest SW undertakings. Indeed, the project scope in question indicates it could be accomplished by some folks here as a hobby project.

This also explains to me why Retool is so popular with tech leaderships, while not necessarily so with engineers. I wonder what other product approaches could address the same gap.

Man, I wish folks would stop using AI generated header images. Throws me off right from the get go.

This hits hard:

"Of course by customizing their “product” we cleverly combined all the downsides of vendor software with all the downsides of custom software. We simultaneously achieved the holy grail of bad ideas: an inflexible vendor package that would have to be forced into doing something it wasn’t designed to do but would also be forked from their main product codebase - guaranteeing sooner or later it would be end-of-lifed once the vendor realized how expensive it was to keep maintaining."

Don't ever do this if you can avoid it. Quite apart from the tech angle, the vendor will also rip you off for every change.

If you ever come across a project where you are wading through levels of duct tape and wondering how this technical design ever got out the door you can be sure that when you get to the bottom you will find MongoDB.

This sounds kind of stupid and not rockstar-like at all. Imho a rockstar is someone who has integrity. And the integer (lol) thing to do would have been to tell that CTO to go take a hike. It is not your responsibility to compensate for extraordinary stupidity nor to single handedly lift the weight of all your coworkers. You are making this agonizing system work in the first place.

I would not want to work with anyone at either the Fortune 500 company or the vendor. Extremely dysfunctional and unhealthy work environment. Lying to the boss, secret projects using 360 hours of engineering time- not even including all the nights and weekends mentioned. This is not good. This is not right.

This story is all fake, right? I mean, I've worked at multiple Fortune 500 companies, nothing works like this story says it does.

>. Keep in mind that early in my career,

And the CTO is calling you directly? The CTO of a Fortune 500 company? The 'review process' for a vendor pitch is the CTO asking his immediate staff?

The system that needs 3 people a few months to build is being handled by the CTO???

Your comment about heavy customization reminds me about the days of talking to CRM salespeople. Those sales people that were like, “We can make our CRM do anything you want it to”, whenever we asked questions. What that sales person really meant was our product doesn’t actually work, but if you throw a huge truck load of money at us, we can make our product work.

"Because the CTO had a yearly turnover of his direct reports, every status call about the project took some variation of “great idea, boss” even though literally no one involved thought it was even a good idea."

IMO it is still ones job to voice the concern, even if it gets you in trouble. At least if you want subsequent complaints to be taken seriously.

Personally, if I get in a situation where even voicing issues gets me in shit, that's a red flag to go find another employer.

I think this glamorizes and engenders an unhealthy relationship to work. “Rock star” and “ninja coder” works the same lever Steve Jobs used in the 70s to take advantage of smart people without enough sense of self worth.

And don’t lie to people.

Once someone told me that the only people that will remember that you were working hard and for longer hours will be your spouse and your kids.

Stop working overtime and over weekends. A company demanding working on weekends and cancel holidays is a huge red flag

This is not about lying but being, quite literally, economical with the truth.

Just as you don't spend all your money at once, don't tell your boss everything you've achieved at once. Or not always anyway. I had to write a monthly progress report for one recent job. I kept a document of headlines for this, and put only two or three into each report. I kept some back for rainier months. This also helped me discuss my "objectives" since I often could predict at least one thing I would achieve in the next few months.

> As has been typical in my career, when the vendor said they had a product, what they really meant was they had something vaguely resembling a product that vaguely matched what we needed, and with heavy customization they could torture it into doing what we needed.

How many times have I been accused of wanting to "reinvent the wheel" while facing this exact situation ? I can't count.

The story misses a continuation. What happened after CTO found out that the system was silently replaced behind the covers? IMHO, this is the most interesting part. I bet that the CTO was crushed by his ego.

> A software launch is like performing live theater for introverts.

This should be the title!

>Mind you, most of us had already been working 60-80 hour weeks for the past 6 months

I would change the workplace by that time.

Reminiscent of the scene in one of the later episodes of Band of Brothers: In the last weeks of the European war, Major Dick Winters is ordered to send a patrol to cross the Rhine, in rubber boats, during the night — for the second night in a row. The patrol's quite-dangerous mission would be a repeat of what the same soldiers had done the previous night: to capture German prisoners and bring them back for interrogation. Winters concludes that the repeat mission would be a pointless risk of his men's lives. So he disobeys his orders: he tells his people to get a good night's sleep and report to him in the morning that they did the repeat mission but were unable to find any additional German soldiers to capture. Winters's soldiers were visibly relieved.

This is the worst of the technology. It's not sustainable will ruin lives and relationships. I appreciate the authors pride from overdelivering, but I think that speaks to some other need that's not a healthy thing to want from a job - not to mention all that time was lost instead of spending it with friends and family and creating real memories. Finally, to be completely objective about it, the team donated their time free of charge, literally throwing their value away to correct someone elses mistake behind their back. Not a good move, especially if something goes very south.

It's time for our industry to do better and not champion attitudes like this, especially in an era of consolidation where megacorp gets away it to this day and somehow sells it as a positive.

We have to do better.

Once upon a time where I worked...

We had a terrible release management system. (For IBM iSeries folks, it was Turnover.) And we used it for everything. Java code included. In particular, we used it for database migration scripts. And you know how database migration goes. You keep adding files and you never remove any. And there gets to be quite a few very quickly at the beginning of a project. And the folks in charge of setting up the release management system for our database migration scripts did so in such a way that we had to list out each one individually. Every release, we had to type like 50 database migration script filenames one-by-one. With no tab completion. (Not to mention the Java .war files we were deploying as well.) And by "release", I mean every release to the dev environment.

(Now some of you who know of Turnover might know of a janky Eclipse fork that would let you create Turnover forms in a drag-and-drop way rather than through the green screen, but there were reasons we didn't want to go that direction. Long story.)

As soon as I figured out how much of a PITA this would be every single day, I decided to do something about it. I wouldn't ask for permission. I'd ask for forgiveness (if it came to that.) So I set about writing a way to automate this. That way involved running the dumb terminal emulator (the "green screen" app tn5250j) in Xvfb and simulating key strokes to it. God was it a nasty hack, but it worked reliably and God was less painful to deal with than what we had before.

Part of why I didn't ask permission was because I wasn't entirely sure I could make it work and I didn't know if the boss would ok a project that may pay off given our tight deadlines. But once it did work, I didn't keep it a secret and the boss praised me for it. Still don't know he'd have let me undertake that project had I explained and asked permission ahead of time, but everything very much worked out with that project in the end.

Here's to lying to your boss. Don't ever feel bad for saving your boss from themself.

And if that had gone slightly wrong, this person would have gotten fired and also a bad reputation in an industry that is smaller than people think.

That was interesting and correct me if I misunderstood.

So, their CTO asked the team(of 3 members) to work through the Christmas week even when they had been working like crazy for weeks and are on the verge of burnout. The author asks them to take the week off while lying to the CTO that the team has been hitting milestone xyz. The dev team came back fresh from holidays and hit the milestones.

There are some management lessons in there on what to do, what not to do, how/why developers get insane deadlines and how they end up with burnout.

I was expecting the lie to be about a technical or design decision since "CTO" was mentioned and how that saved the day.

Why do companies keep on treating employees like they do ? Just read this article.

If you’re seriously considering lying to the CTO you should instead just get a different job at a different company. That’s a really bad working environment if you think lying is acceptable or something to be proud of.

These days you get to work on a single Jira ticket that has been sliced so fine you barely need to write a single line of code, with morning standups in which you are expected to describe every moment of your day the previous day and every moment that you anticipate in your upcoming day with four people tugging their beards and harrumphing if you describe the slightest challenge.

Doing something "different" would get you a warning, doing something major without full work breakdown, buy in, analysis, tickets and micromanagement would get you fired.

They say that at the time they were an “Enthusiastic Young Dev” but worked for a Fortune 500 company reporting directly to the CTO and interacting directly with the customer and leading a team.

Something doesn’t add up or is it just me?

So, you launched your complete rewrite, without any testings and still make ~100% compatibility with an existing vendor solution. Call me skeptics.

It seems like everyone lied to that CTO. If you are, or become a CTO, be the CTO that no one feels the need to lie to you.

Am I understanding the story correctly? a year plus, no doubt horribly expensive engagement with an external vendor was replaced with a few weeks skunk works project buy an internal team member who executed it by lying up the chain of command and there were no repercussions.

I'm genuinely puzzled why the lie was needed. They do some shadow work and build an in-house solution to replace a vendor. I get that. But why lie after it's done? Why not "Uhh, hey boss, we're already done."

Or did I misread something?

Off topic, I'm distracted by the accompanying photo, the model's expression and the code being displayed at the back of the monitor. Looks like scroll-stopping AI stock photos are the new "YouTube faces".

> The database they chose was MongoDB and at the time Mongo had a record limit of 16MB per document.

I loathe working in MongoDB (have been for the past 3 years at work), but this is one situation where it's just absolutely bad schema design and not necessarily Mongo itself.

I may have missed it, but why let the vendor even proceed with this behavior? I get there's timelines with the client but can't you sue or get a refund from the vendor for their screwup?

> Adding a new transaction involved reading the entire json document out of the database, then appending the new record to the end.

This... doesn't seem right. I understand the issue is with the vendor, but there are lots of MongoDB controls to avoid this. Even in nested documents this should not be necessary.

Build up to the lie is better than the lie. I would never brag about these kind of lies to be honest

Is anyone surprised that part of the problem was MongoDB? (Or that replacing it wasn’t hard?)

Just think what kind of psychopath you'd have to be to demand everyone cancel their Christmas holidays. I honestly think at that point they should have all just quit, and let the project fail to shine a light on the utter failure of the CTO.

I'm curious if the three guys who got the holidays off knew that the author was lying to the CEO. Seems like it would be best not to tell them, so they can't get in trouble. But I imagine they found out after the fact!

Nice funny joke for European people, when it comes about extra time and holidays.

Stories like those always get big response on havker news coz for all the smart ppl we have here there are 10x of not smart ppl not here.

And we all work with them on various levels in the organization.

Remember - being smart is both a gift and a curse.

Great story, nicely written. The goal is to show off ;) But really nice to read.

Definitely, this is a bad example and precedent to build toxic and non-cooperative environments. And instead of solving the issue it make it worth.

Everything that is wrong with contemporary software development, in one blog post. At least the author was not in an adversarial relationship with their devs, so not everything I guess.

This is either a parody, a self-unaware remembering or an outright lie. It's got all the elements: stupid CTO (how'd he even get this high? sucks at programming but good at politics), Lying vendor (everyone there is so stupid and terrible!), sycophant employees (they just tell the CTO what he wants to hear!), heroic dev team and of course our savior, the author.

I questioned the "young and foolish" / "grumpy veteran" position when they started talking about json and mongo (maybe replace that with a text file and proprietary binary format), and yes, this sort of shit happens to varying degrees all the time, but I'm pretty sick of the everyone but developers, and more specifically MY team - or even just me, is an idiot - "I'm the only one who cares and does the right thing" - which this was definitely not. Question for the audience: most of you work for big, non-startup and likely non-software companies; when you look around can you honestly say "nobody else cares, just me"?

Don’t roll the dicey on your own career and reputation (lying through your teeth) to work around some shitty organisational dysfunction.

Sure it could work out OK but that’s just bad strategy.

In my first adult job out of Uni I was working for a startup building a suite of tools for usability tracking/testing and alike. One of our main offerings was gonna be something similar to what HotJar is nowadays. We were busy with other stuff and I considered the requirements for the tool somewhat impossible ;) So it got outsourced.

The outsourcing company was a tiny local software house. They delivered on every single requirement in under 3 months. Despite my better judgement I was impressed.

Until we gave ot to an actual customer. After some back and forth with initial errors (relatable) it started running on a small portion of traffic. A week of running the tool on their website was 100GB of file storage and 100GB of storage behind postgress db.

Utterly unsustainable.

A few weeks later we had a "do what you want" sprint. I mean the engineering team decided (I know, different story tho) every 10 sprints we get one to do whatever we think makes sense for the product.

So a brilliant new intern and I got to come up with new requirements to fulfill the same usecase but without the need to store colossal amount of data per visit.

We wrote a new thing in 2 weeks and had a working demo. We used the next month to productize it.

My initial.design was presenting user behavior as scenes, sort of like a comic book, instead of animating stuff to pretend it's a video. Over time product got that too tho.

When the startup folded (yet another story) the technology behind that tool was the thing that got sold.

Moral of the story? Whether you lie or not, bottom-up decision making can be pivotal to software products if you're lucky.

Impressive, I believe this is so true in every scenario. You cannot just force people to work, a break between hard problems works very well.

> In September we encountered a show stopper bug. The vendor product stored every customer transaction as a json record in a giant json document. So as test data accumulated, performance of the product got slower and slower.

For the love of god, how could they not have thought of using a database?

Forgive me, I struggle with reading (attention) so I tend to skim… but from what I picked up, this vaguely sounds similar to a situation that lost Title Source a $700M lawsuit over intellectual property. Became intimately familiar with the vendor product, and used the same team to re-write a competing (replacement) product that was then sold to the client? Is that right? I wonder what kind of stipulations for use / re-work existed in that contract.

Build up to the lie is better than the lie. I would never brag about these kind of lies to be honeest

Nice funny joke for European people.

What bothers me about MongoDB limits is clearly this vendor was using MongoDB incorrectly.

I _REALLY_ enjoyed reading this. Great story. Please write more stuff GrumpyOldDev!

Sounds like everybody was lying to the Emperor about their new clothes.

All of that could have been avoided from the start if someone had the balls to just tell this brain dead cto how stupid the original idea was and pitch the in house development. Plan accordingly and deliver in a relaxed state with no deception or unethical business practices.

But since this is an f500, cto prob has an ego despite not having touched a terminal in years or decades. So it’s necessary to stroke the ego or become a yes-person; otherwise get 86’d.

Collectively, we need to stop bending to the will of C-level executives. Let their stupid ideas fail so they get replaced.

Halfway decent engineers will always find new work. A majority of the time workers are not impacted by failures of leadership. In past jobs for F50 companies, VP/Director/SVP positions changed hands frequently, almost on a quarterly basis. C-level positions changed less frequently but it happened a couple of times with enough bad quarters

> Many of us who really enjoy software development feel a bit like rock stars.

For a bunch of supposedly smart people, I am always disappointed at how easy it is to exploit developers.

The devs did the deathmarch, and the vendor and CTO gain the accolades, enabling their bullshit.

For any dev who thinks they are a "rockstar", they should have been privy to some conversations I have overheard among product managers and senior management. They would have been disabused of this "rockstar" notion really fast.

I was quite disappointed to see the author only has one blog post. I was kinda hoping for more war stories.

Fuck this work environment.

I can relate to the feelings; if not the actions.

The biggest issue with this approach, is that it absolutely requires the local team to be really good. Not Dunning-Kruger good, but actually good.

Those teams are very rare; especially these days. I have found that most companies deliberately avoid hiring these types of folks. They can be tough to manage.

I was incredibly fortunate to have worked in such a team, but as I have left that silo for a number of years, I’ve also come to realize what a rare privilege it’s been.

If you don’t have one of these teams, then this kind of thing would be incredibly reckless. Even with a good team, it’s still a huge gamble. I would not have done this, myself. It would have been too risky. I also had very technically competent managers, and would not have gotten away with it.

Brings to mind people like Nick Leeson, of Barings Bank fame (Google him). I’m sure he was saying “I’ve got this!”, right up to the end.

[dead]

[dead]

[dead]

[dead]