Hacker News new | past | comments | ask | show | jobs | submit
aabhay | on: Ten advances in mathematics and theoretical computer science
My main gripe here is the lack of transparency around the total experiment and construction. I doubt that they simply pointed their model at these ten specific problems alone and gave the model one shot; therefore the $2000 number could be completely misleading, similar to P-value hacking by not disclosing the total experimental setup.

I want to know:

1. How many total problems were given to the model, and what percent were left unsolved at what cost before giving up? 2. How many attempts did you give the model at solving these problems? 3. How expensive was the harness, e.g. did the model have access to a job cluster?

wrsh07 | parent | next
It seems like they threw it a decently large battery of open math problems and probably limited it to something like $200-500 per problem:

https://x.com/polynoamial/status/2083478171975082334

As a complete guess, it seems like they tested hundreds to thousands of problems with a relatively low per-problem budget

--

The linked tweet from Noam Brown at OpenAI reads:

> And yes we did try other major problems without success. Sadly no Millennium Prize problems (yet).

> But also, we didn’t spend a lot on each problem. It’s possible to push test-time compute much further.

c7b | parent | next
I believe we're seeing a new kind of mathematics that will require completely new formats for publication, a bit similar to those used in experimental sciences. AI-powered mathematics should be fully reproducible, so it's the authors' responsibility to disclose the exact model type, inference settings/seeds and the full prompt history leading to the result. Of course that would ideally require open weights models.

It's not just about requiring to disclose AI use. AI-powered mathematics is a completely valid discipline that doesn't need to be shy, but it should develop its own publication culture.

jsenn | root | parent | next
I can see this being important if you only care about the results as evaluations of AI progress, but if what you care about is the math itself why should you care about the prompt or anything other than the proof?
loading story #49136698
loading story #49136673
loading story #49156919
loading story #49159226
lkirk | root | parent | next
I think this is a bit optimistic compared to my view (wrt portability). There's a large stack of software that is involved in training and probably less so in inference. I'm not saying it's impossible but there are definitely different levels of reproducibility and the academic incentive structure doesn't really prioritize reproducibility in my experience. I'm sure it varies quite a bit, I'd be curious to know how those in this problem space are thinking about reproducibility and at what level.
loading story #49137237
black_knight | root | parent
If the proofs are formally verified by a proof assistant (Agda, Roq, Lean, ⋯), I see no reason we would need to know how these came about. All the information needed is in the proof.
loading story #49138890
loading story #49138266
whattheheckheck | parent | next
Yeah I remember reading about something along the lines of Mathematics is now about the scaffolding around you find the problems/solutions not just the problems and solutions. For teaching purposes. This was before this ai craze
dist-epoch | parent | next
I don't think you want to bring cost into this argument.

Even if the cost was $1 mil for these 10 problems, that's maybe 10-20 math researchers for a year.

Do you really think that if you paid that to humans, they will deliver the same results?

uh_uh | root | parent | next
It is comical at this point. Some people just can not stand the thought of AI actually delivering and are trying to find whatever ways to discredit it.
loading story #49134865
loading story #49135685
loading story #49134730
loading story #49136136
loading story #49135535
fasterik | root | parent | next
You need to bring both cost and benefit into the argument, and it's not necessarily an obvious win for either side. There are a few complicating factors here.

The cost of running a model is not only $/token, but the salaries of the people managing/orchestrating the models, deciding what theorems to try, etc. Once we factor that in, how much are we really paying per theorem?

The other factor is the subjective component of the value of a theorem. Not all theorems are created equal, and the only way to really measure the value is to ask professional mathematicians for their opinion, or publish the results and look at citations over months/years.

Once we have both of these nailed down, then we can start to do the cost/benefit analysis. To be fair, we should actually compare three groups: human experts, hybrid agent/human expert teams, and fully autonomous agents.

robotpepi | root | parent | next
it's still important. not everyone has access to 1 million USD. saying it "only" coat 2000 USD is highly misleading for the discussion and future. the concentration of power is a huge problem with AI.
wbl | root | parent | next
If you told them this was the problem and they would still have a job if they failed probably. The reasons people don't go head on these problems is career incentives and psychology.
kevinwang | root | parent | next
It would still provide better context to see the numbers that the parent proposes, though.
tchalla | root | parent | next
Mentioning cost is fine, comparing may not be.
mungaihaha | root | parent
Grad students on zero pay solve problems like this everyday. What exactly is your point here?
loading story #49135637
loading story #49133525
loading story #49139376
loading story #49134645
loading story #49139382
moscoe | parent | next
I guess people will always find something to gripe about.
simianwords | parent | next
[flagged]
traes | root | parent | next
It's a very important clarification if it took $2000/problem on 20 problem attempts or on 1,000 problem attempts for each successful one. That may be the deciding factor on whether or not it's economically viable to replace a mathematician with a ChatGPT subscription.
loading story #49138845
loading story #49132473
nxpnsv | root | parent
No, this is valid criticism. Oai gives the impression anybody could get similar results at a similar price, but that’s very likely not true. This is marketing first, then mathematics.
azan_ | parent | next
> therefore the $2000 number could be completely misleading, similar to P-value hacking by not disclosing the total experimental setup.

I don't think that comparison to p-hacking is fair. I mean not reporting price of all run is nothing like committing scientific fraud and fake results.

einpoklum | parent
Also, have there been examples of researchers not affiliated with OpenAI (or another LLM creator), who have done something similar?

Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.

energy123 | root | parent | next
Many less important Erdos problems have been solved by amateurs prompting ChatGPT 5.{3,4,5,6} Pro using their $200 subscription.
traes | root | parent | next
> Also, have there been examples of researchers not affiliated with OpenAI (or another LLM creator), who have done something similar?

A couple small ones that I've seen (example here [0]), but not anything of the magnitude that OpenAI and Anthropic have put out. Likely just related to token limits.

> Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.

I think their output has reached a level that precludes this possibility, but I of course don't have any hard proof.

[0]: https://www.reddit.com/r/math/comments/1uxj3cy/after_openais...

loading story #49133303
brighteyes | root | parent | next
Yes, here is another example of major work in this area:

https://arxiv.org/html/2605.22763v1

> Our most capable agent autonomously resolved 9 of 353 open Erdős problems at the per-problem cost of a few hundred dollars, proved 44/492 OEIS conjectures

loading story #49134910
kittoes | root | parent
https://blob.byteterrace.com/public/bds-theorem.html

I have no affiliation whatsoever with any AI company, nor any formal education outside high school, for what it's worth. Simply being curious and persistent can get you quite far in my anecdotal experience.