yeah, the tricky thing about the experimental loop is that: 1. its very difficult to do it in a reproducible manner (the same experiment done twice often gives different results due to small undocumented changes) 2. its expensive to do at scale.

Both of these properties make it hard to hill climb on experiment. What's worked for us so far is precisely what you said - having human experts review and provide feedback. we distil their reviews into rubrics, and have LLMs act as proxy experts using these rubrics. We expect the models will hill climb using this approach, and will reach (close to) human expert level by doing this.