I think the main problem is surfacing two topics:
- Understanding how effort and class interact (Is a high effort terra above a low effort Sol? Are larger models mostly always better regardless of effort?)
- understanding when (if ever) a new model tends to be better than an older but higher class one. Is an old astra better than a new terra?
If increasing effort and increasing model class produce improved results in a different way, I miss making it explicit in which way. In other words: What kind of issue warrants going up in effort? what kind of issue warrants going up in model class? What kind of issue is unlikely to be solved by either?
OpenAI itself cannot answer these questions. A prospective price-sensitive (or speed-sensitive) user has to experiment to find the cheapest (or fastest) model that is sufficient for each task.
Right. Because they don’t know. They’re just pumping out crap desperately trying to stop hemorrhaging billions.
>effort / performance curves by model
Oh, easy:
Artificial Analysis has their own curves which are pretty similar.
That's good, but they haven't updated it for a month. It is missing all of the current frontier models.
Seems to be missing all of the 6.0 and 6.1 models?
> Is a high effort terra above a low effort Sol
You can literally see this in the charts from the article