I wonder how well this would translate with a much larger model. They ran the experiments with comparatively tiny models. You would presumably need to churn a lot more tokens to move the needle of a multi-trillion-parameter model's weights. It might just be that it's not that hard to improve Llama-3B and Llama 3-8B because they're not very good to begin with. They compare it to other methods - in-context learning and TTT - but I'd be curious to see it compared to a "control group" of RLHF.

As an aside, I get that this is a preprint, but it's very odd to see a piece of writing with obvious typos and grammar issues (e.g. page 13: "We attributes this results to," "In this setting (see Table 4), Sleep achieves a 80% success rate, higher then the other methods.") in an era with LLMs. It sort of locks in that this was written by human beings, and I'm all for that, but it jumps out at me more than it used to.