They did a paper on this a long time ago[1], I'm sure they have altered their techniques some, but I this was the earlier signal that they really treated the AI as a separate entity and trained it as such. And focus not just on it's capability, but it's *behavior*, when to push back on the user, etc... Imo. it's more about the philosophy of the constitution in this paper than the specific RLAIHF, I'm sure this idea of "when to follow instructions and when to trust yourself is in their system prompts for all their RL environments, and even their rewards.
This idea of an objective function vs what the model should do during human interaction is a huge question and a very subtle one to get right, but it's why they were winning in real world application even before they were winning in raw benchmarks.
Especially in the era of RL environments dominating more and more compute it's vey easy to overfit a model to be very good at accomplishing a narrow task exactly as specified. Much more complicated to have it respond appropriately on turn 5 of a nuanced conversation with a user about their private codebase.