Hacker News new | past | comments | ask | show | jobs | submit
WhitneyLand | on: Muse Code and Muse Spark 1.2
They chose to compare against Open AI’s mid tier model Terra instead of Sol and still lost some benchmark against it.

They left Opus in and got beat in all but one benchmark.

Nothing wrong with trying to improve, but why the marketing games?

Instead of trying to say in the post you’re “closer” to frontier, first set a clear goal to beat the Chinese labs on price or performance and demonstrate it convincingly.

Then when your ready, come back and talk frontier without playing hide the model.

spmurrayzzz | parent | next
Given the current throughput figures on OpenRouter (~180 tk/s), its likely a much smaller param count on the order of something like Luna. I think the better, more timely comparison (re: your point on Chinese labs) would be to DeepSeek-V4-Flash-0731.

It's definitely confusing from a presentation perspective, but they are somewhat coherent comparisons if you account for the inference heuristics involved.

(They could in theory be gaming the decode speeds with much larger than normal batch sizes given the TTFT is pretty high at around 8s)

sheepscreek | parent | next
It could be that they’re pitching Meta Muse 1.2 against Terra and Opus level models. They probably consider Sol to be a level above, along with Fable.
penetrarthur | parent | next
I wouldn't call Terra a mid tier model. Terra xhigh is the only model I use for coding and it is good for literally everything. The difference between Terra and Sol is negligible, but it is way more limit hungry and much much slower.
dd8601fn | parent | next
It’s pretty clear they’re really not attempting to compete that way. They’re using a profitable ad business to be able to undercut and buy some business to stay relevant.
ac29 | parent | next
> They chose to compare against Open AI’s mid tier model Terra instead of Sol and still lost some benchmark against it.

If you scroll very slightly farther there is a benchmark that includes Sol, showing it outperforming Terra (as expected) and Spark 1.2

krm01 | parent | next
We can throw benchmarks in the bin by now. Each one I've seen is heavily biased and skewed. It holds very little reliable data points (unfortunately)
lacker | root | parent | next
My conclusion is the opposite. If benchmarks were meaningless, surely Meta would be able to find some benchmark that shows they are better than Sol and Fable. The fact that they can't do that tells me that benchmarks still do mean something.
villish | root | parent | next
Muse 1.1 performed relatively well according to benchmarks, putting it within spitting distance of the premier models. However, based on the results I got from it and the review videos I watched, it wasn’t even close.

Opus 5 is incredible at making games. Almost like a generation better than other models from my experience. You won't see that if you just look at the popular benchmarks..

You have to test each model on your actual use case to see how well it really performs.

lacker | root | parent | next
Yeah, totally. But... the problem is that I don't have time to test every single model that comes out. So I rely on reports like this to decide, should I even bother testing out Muse?
dd8601fn | root | parent
> Opus 5 is incredible at making games.

This is a bit vague. What sort of games with what technology?

Jumpstylish | root | parent | next
My son was gifted an old Mac from his grandparents. It only supports OSX 10.13. I’ve been able to make several games that he genuinely likes (7 yo) Opus built them on my workstation and then pushed them to his computer and tested them over SSH. It handled all the asset creation or collection from CC0 licensed sources. I believe everything is built on the Godot engine. It’s really amazing to me. I don’t know what it would cost me to get someone to build custom games on a long deprecated computer architecture, but I paid Anthropic $20.
tudelo | root | parent
I don't think it is vague in the slightest. Take the most simple examples, how many LLM's have you tested making them? There are stylistic choices pertaining to games that is well beyond a 0/1 reward. Even something as basic as breakout or flappy bird can have wildly different quality between models. Yeah, you could call this animal on a bike benchmarking, but I don't think it is. IMO the problem space occupies an interesting area where you can ignore the pass/fail and focus on the actual level of the model to do something beyond that.

I doubt the OP meant something like creating the whole tech stack for WOW.

dd8601fn | root | parent
You seem to think I was disagreeing somehow.

I was just asking what kinds of games and with which technology.

Neither is stated in the original comment, and the answer obviously isn’t “every kind with every technology”.

nrub | root | parent
Or they spent time optimizing their model to real world problems they're facing and didn't waste time trying to game a benchmark.
brokencode | root | parent
Or they did try to game the benchmarks and just didn’t do it well enough.

Benchmarks are one data point, not the only one, but the easiest one to compare.

nrub | root | parent
Right, but the point is that you can't conclude that a model is necessarily bad because it's not hitting the same scores on benchmarks. I just don't agree with lacker's conclusion, because their logic doesn't seem to consider that. Scoring lower on a benchmark doesn't strictly mean they have a bad model, but it may be the case. Like you said it's one data point, but being the easiest, and obviously most gamed, means you should probably weigh them less heavily.
deepsquirrelnet | root | parent | next
If you look at papers on benchmarks, they're usually created to expose gaps in how models are trained. It should be no surprise that models get better on them over time, because you can't get better at what you don't measure.

Cherry picking the benchmarks you present is where the falsehoods lie.

tudelo | root | parent
Another thing that sort of puzzles me about benchmarks is that LLMs are not deterministic and do not always complete a problem. So what are the results actually representing? The best run? The average? It is all in some ways a falsehood
llm_nerd | root | parent
>We can throw benchmarks in the bin by now.

No, you really can't. This rhetoric on here is so profoundly boring and tired by now. People have been saying this noise about benchmarks for time eternal, usually because their pet didn't win.

Meta knows their models aren't as good -- demonstrated by their benchmark performance -- and their value proposition is a much lower price.

jjice | parent
While I won't take their limited benchmarks with much salt, if it actually is this close to opus, but at a third the cost, that's pretty solid. Now, Terra is pretty damn affordable too and you're right that it's suspicious that they don't put Sol in there at all.