The problem is in the rewards, but it's more how they average across domains and contexts, so AI misapplies them. Actually, in human preference evaluations, most of Gen Z in particular extremely adverse to the common look of AI images.
So it's not the annotators involved in RLHF, it's the models failing to map to human preferences at the task level. The failure rate is also uneven across domains.
Where there are challenges with human annotation, it's not the ability to gauge what's good but rather how to correct what's wrong. Most people can tell when something is off but struggle to articulate why. That's where more domain expertise is required.
For posterity, the memetic ugly food can be controlled and mostly filtered out with prompt engineering. Here's a quick ad hoc test example which (hopefully) uses the new model: https://chatgpt.com/share/6aebf030-9d38-83e8-a103-3509883905...
First image is fine, but attempts to make it more wacky turn it to the AI food slop. So then followup prompts deslopify it, mostly successfully.
I think the way they'll solve it in the end is to create an ultra-high-fidelity simulation of the world including modelling taste and other senses on the level of physics, and then incorporate all of this into the training rewards.
I think this is how the machines know what tasty wheat tasted like.