Hacker News new | past | comments | ask | show | jobs | submit
ariwilson | on: Claude Fable 5.2 and Mythos 5.2
I at least didn't see this. I had a baseline evaluation from before Nov 3 on Opus 5.5 medium, so I re-ran it this morning.

Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.

20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.

Caveats: one run per task today, medium effort only.

marshalla | parent | next
Personally, I've found myself getting lazier recently, trying to get a single session to handle tasks that should be broken into multiple sessions, etc. I've suspected claims of nerfing might come from changes in user behavior. I'm glad more disciplined checks are being run.
loading story #50434714
loading story #50432366
janalsncm | parent | next
This is an example of LLM text that doesn’t make sense.

Paragraph 1 says there were 30 bug fixes.

Paragraph 2 first says they tested it on 20 of the “tasks”. But why is the denominator 40? Where did that come from? It should be 30.

It goes on to mention another test, this time with 20 tasks. Again, it’s unclear where this is even coming from. Based on the first paragraph it should be 30 not 20.

loading story #50434498
loading story #50434563
loading story #50433985
loading story #50433712
drakythe | parent | next
If I read your results correct it solved 5% more problems and used up between 3-5% fewer tokens, with a faster response time?
loading story #50431143
loading story #50431441
risyachka | parent | next
The spending limit definitely goes down much faster as of last week or so.
| parent
{"deleted":true,"id":50431491,"parent":50430746,"time":1794255940,"type":"comment"}