Hacker News new | past | comments | ask | show | jobs | submit
danbruc | on: We should be more ambitious with the software we build
There are some things that are sped up by a very large factor, easily over 1000000x.

I will need an example for this, what can I achieve with a large language model in ten seconds that I can not do on my own in one and a half years [1]? Or one hour vs 568 years?

The unsolved math problems are one thing that came to my mind, after all no human could solve them. But Navier-Stokes took 10,000 agents for 88 hours, that are 500 years [1] for one human without any speed-up. Does not sound unreasonable to me that a human could have solved it if they could invest 500 years on it.

The other hundreds of math problems took on average 3 hours, with a speed-up of a million that would be 1,704 years [1] for one human.

[1] Assuming 220 working days per year, 8 hours per day.

crazygringo | parent | next
I recently needed to adjust an existing open source model for an app that I am building, to get it to work in a certain way. I have no background in machine learning.

Claude was able to identify the approach that worked, and even the long Python command that was needed. In, yes, probably about 10 seconds.

For me to figure that out myself, it probably would've taken something like a year of study. Of the entire stack of machine learning and all of the Python tooling associated with it.

That's where these absolutely incredible astonishing speedups come from.

Because it's one thing for Claude to be able to write code 20 times faster than I can. But it's a whole other thing when it is able to create things in domains that I know nothing about at all. That would take me one or more years to become an expert in. But where I can still just verify its results, so I don't need to be an expert in order to judge the quality or suitability. If the end product works, the end product works.

I have similarly had it come up with mathematical formulas that would probably have taken me a year in going back to studying math if I had to derive them myself. But where it is trivial to show that they are producing the correct output for known test cases.

So trying to come up with some kind of factor multiplier here almost kind of falls apart. It might as well be infinite. Because these are things that I just wouldn't have done otherwise. The software wouldn't have gotten built because I would have had to hire somebody else to do it, and there was no budget for that.

skydhash | root | parent
> Claude was able to identify the approach that worked, and even the long Python command that was needed. In, yes, probably about 10 seconds.

> For me to figure that out myself, it probably would've taken something like a year of study. Of the entire stack of machine learning and all of the Python tooling associated with it

It wouldn’t take that long. Maybe a few days of playing around. It’s not like you’re trying to create a new model from scratch. No need to understand linear algebra fully to learn how to do a dot product.

loading story #50235335
zeryx | parent | next
Disassembly and deep inspection of binaries seems like a good example, or rewriting an existing codebase in rust. You can do it yourself but if you don't have the background there's a ton you need to learn.
danbruc | root | parent | next
Okay, somewhat fair point, if you have no idea how to do X and learning to do X takes a non-trivial amount of time, then using a large language model could probably get you there a million times faster in some cases. But I would want to exclude those cases because they make the speed-up meaningless and instead assume that the user is reasonably competent to do the task.
| root | parent
{"deleted":true,"id":50232691,"kids":[50232797],"parent":50232532,"time":1792942525,"type":"comment"}
loading story #50232797
manmal | root | parent
Sure but that’s probably 100x max? It won’t magically work, there’s a lot of (auto) review and verification that has to be carefully setup, so things actually work in the way you need them to. I’ve done it, it takes days until it’s good.
jstummbillig | parent | next
Idk what that number does, other than stoke this kind of (very respectfully put) response. Seems fairly unnecessary/random.
fritzo | parent | next
Concept search. "Read my code, point me to an academic paper with similar ideas." I've had code lying around for decades, where I can point an LLM at it and learn the papers discussing similar ideas and learn of better methods.
rjbwork | root | parent
Yes this has been one of the killer apps for me. Describing my problems and having it review my code and the academic literature and industry white papers for prior art has been incredible.
petra | parent | next
Finding hard to find bugs. LLM's have strong capability for this in the area of security. This is also true for math(finding counterexamples), and probably other areas with strong verification.

Also, those AutoResearch agents for other types of verifiable goals.

cowlevel | parent | next
Reverse engineering. The LLM can name all the functions in one hundred seconds that would take you fifteen years.
danbruc | root | parent | next
You think I could not identify all the functions in some software if I poked at it with IDA pro for 15 years? Probably depends on what exactly we are talking about, is it calc.exe or an entire office suite? But I will give you that I would probably never complete the task, I would not be willing to spend 15 years doing this. Realistically I would probably be interested in some specific functionality and identify the functions involved and ignore the vast majority.
rerdavies | root | parent
Shouldn't take you more than a month or two with decent tools for a moderately large program. But yes, if there is ever a Next Time, I will be using AI tools.
rowanG077 | parent | next
I don't think you are supposed to multiple the number of agents with the time. After all a single human can run arbitrary number of agents. Still 1000000x speedup is a few order of magnitude too many imo.
danbruc | root | parent
You have to multiply them, otherwise you would also have to compare the output of a company with a single developer with the output of a company with a thousand developers, after all a company can hire an arbitrary number of developers. Running a thousand agents in parallel for one hour is the same as running a thousand agents one after another for a total of a thousand hours.
sanderjd | root | parent | next
Yes, but the cheap parallelism is (part of) the point he's making.
danbruc | root | parent
Parallelismen is not cheap, in the end it costs the same. It is one resource for a long time vs many resources for a short time.
miki123211 | root | parent | next
Agent parallelism is cheap in the same way tapwater is cheap — that is, you pay exactly for what you use at a given moment, no more and no less.

Humans are a lot more expensive, it takes ~a year to find, interview, hire, onboard and offboard an employee, while an agent can be spun up for 10 seconds, do its work and vanish back into the ether.

Agents are lambda functions where humans are bare-metal servers that you have to rack yourself.

danbruc | root | parent
Sure, I completely agree with that, but we totally went off-topic. The questions was whether the correct measure is wall-clock time or compute-time and for that the difference in difficulty to add another agent or add another employee is irrelevant. And to reiterate my point, compute-time is what delivers the results and also is what you pay for, so that is what counts. That you can squeeze that compute-time into a shorter amount of wall-time by utilizing more resources in parallel is also important but unrelated to the fact that compute-time is what delivers the results.
loading story #50233260
sanderjd | root | parent
It is very cheap relative to hiring people (you introduced the analogy to a company hiring a large number of people).
rowanG077 | root | parent
Well not really. The point is "There are some things that are sped up by a very large factor, easily over 1000000x.", this is obviously referring to wall clock time. Not "agent time".

Yes companies/large groups of humans are sometimes also a lot faster at doing some things than a singular person. I'm not sure why that matters or what your point with that is.

danbruc | root | parent
That makes no sense, because then I could also just give the task to a million people working on it in parallel and the speed-up would disappear. You have to compare all the resources used and not pick an arbitrary one like wall-clock time because resource can often be traded for each other.
rowanG077 | root | parent | next
You can do that? You have a million people ready to go? Whereas a singular person on a rich country salary can actually go out and start vast amounts of agents. I'm honestly perplexed by your argument. Most people would struggle to find even 5 people who will join them on a journey to build something that takes a long time. And you are just casually throwing out there "Well someone can just gather a million people".
CooCooCaCha | root | parent
You could give the task to a million people but you could also give the task to a million people who each have their own swarm of agents.

I think it makes sense to focus on what one individual can do, and amount of time to do it.

loading story #50232967
soVeryTired | parent
Far more than 500 person-years have already gone into Navier-Stokes.
danbruc | root | parent
Probably and the model had access to all those results. The question is whether 500 person-years of effort would have been enough to get there from the same point that the model started. And also not any 500 person-years, not 100 mathematicians independently working on it in parallel for 5 years, but 500 tightly cooperating person-years, 20 mathematicians in the same room for 25 years.