The really hard part of this does in my opinion not so much lie in the aspects that you describe, but rather in doing this without leaving scorched earth with most/all of the stakeholders involved.
In other words:
Doing what you described is in my opinion something that can be learned, and in my opinion a central reason why many programmers consider this to be difficult is that they never learned it, and/or (related to this) were never given the opportunity to be responsible for all of this, so they lack experience.
On the other hand, navigating the whole office politics, and running the political gauntlet that is collateral to it is hell on earth. The only way to survive this is to give a big "fuck you" to everyone, which I more politely described with "leaving scorched earth with most/all the stakeholders involved" above.