Hacker News new | past | comments | ask | show | jobs | submit
swatcoder | on: AI is forcing us to pay off 40 years of technical debt
Is it forcing us to "pay off" technical debt or is it starting us on a compulsory endless treadmill of feature-neutral fix-work? Collectively, we just won't know for a while.

It totally feels like the former if you watch it find and fix vulnerabilities or other debt-tasks one after another.

But only so many of these "technical debt" issue are little off-by-one errors that get remedied by a surgically narrow patch.

Many enough require some kind of restructuring or redevelopment of their own and it's going to take time to figure out if AI faces the same dilemma as human developers: where each non-trivial change carries the risk of some new issue, ad infinitum.

If that happens to be the case, and AI isn't able to magically produce perfect software in finite time, then what's really happened is that the threat of hostile AI exploiting our technical debt will force us to always be burning and churning code just to remain a moving target.

It's another "we'll see" thing for now, I think.

vjvjvjvjghv | parent | next
"Is it forcing us to "pay off" technical debt or is it starting us on a compulsory endless treadmill of feature-neutral fix-work? Collectively, we just won't know for a while."

Yes, let's fix the foundation. It will make adding features easier once the debt is paid off. And we will probably learn how to do better in the future. It reminds me of settings compiler warning to be treated as errors. Once you do that, you have a ton of work to do. But if you are smart (not all devs are), you learn why the warnings are there and avoid these problems in the future.

khafra | root | parent
> It will make adding features easier once the debt is paid off.

I'm not so sure: DJB-style secure microkernel/microservice architectures are hard to extend, and I have little hope that today's monolithic code bases are securable without retreating to a small, proven, and well-tested TCB--even by ASI.

intrasight | parent | next
The original technical debt, which cannot be paid off, was the decision to use a Von Neumann architectures instead of a Harvard architectures for CPUs.
victorbjorklund | root | parent | next
Explain? (Genuine question because I wanna know)
GTP | root | parent | next
Basically, have an explicit distinction between data and code at the architectural level, to enforce "Write xor Execute" in hardware. In this way, you would prevent e.g. a buffer overflow letting an attacker injecting shellcode that gets executed later by somehow pointing at it. AFAIK the issue would still remain with some programming languages though.
sznio | root | parent | next
don't we already have the NX bit?

and it's not sufficient anyways. even with NX you could have the attacker control a function pointer you call. in any sufficiently complex program you'll have enough widgets to build anything you want just out of data.

code and data are the same thing. whatever inputs your system receives is "code" that your application interprets, just like your application is the code that the CPU interprets. turtles all the way down.

intrasight | root | parent | next
In a Harvard architecture, it's not possible to point the CPU at data.
TeMPOraL | root | parent | next
But that's just irrelevant confusion of terms.

There is no fundamental distinction between code and data. This separation does not exist as category in actual reality (and I vividly remember that being one of the bigger points GEB made, early on). They're fundamentally one and the same, and classifying some bits as data and some as code is a point of view that really depends on what you're doing and why.

As for Harvard architecture - every time you branch on your "data", you're effectively executing it, and your "code" can be seen as emulating a system where your "data" is the actual code.

rootlocus | root | parent | next
So JIT is not possible on a Harvard architecture?
loading story #50373944
mxey | root | parent
What about return oriented programming?
ihavesomeexampl | root | parent | next
But.. alter data that gets executed (interpeter?), I mean, how does this work even in theory? All data is potentially code?
AdamH12113 | root | parent | next
To clarify what intrasight is talking about: A Harvard architecture CPU has two physically separate memory busses for code and data. This makes it possible to build a computer with a program memory that cannot be altered through the data bus. In practice, this isn't normally done -- even on Harvard architecture microcontrollers, the flash and RAM are connected to both busses because memory is expensive. And having inalterable program memory doesn't save you from stack-smashing attacks that alter a return address.
intrasight | root | parent
All data is potentially code only in a Von Neumann architecture, but not in a Harvard architecture, where there's a separation of data and code with different memory access to each.
esterna | root | parent | next
Code and data being equivalent is a consequence of Turing completeness; you can always write a Python interpreter, point it at "data" and have it execute that via interpretation.
lelanthran | root | parent
> All data is potentially code only in a Von Neumann architecture, but not in a Harvard architecture, where there's a separation of data and code with different memory access to each.

Says you. I wrote a program for a Hardvard architecture chip (PIC) that read bytecode from the data memory and executed it. Updates were made by replacing the data block with a new data block.

In practice, there is no way to separate data and instructions. After all, JS downloaded inside a webpage is literally data, and it gets executed.

tosapple | root | parent | next
and we don't believe a sterilized virtual machine creates a similar situation?

any respective machine doesn't require full Turing completeness?

any ad-hoc machine allows for mutations and flexibility, which I'm sure is an added strength.

2nd edit,

Also, didn't we have a weaker model of this with segmented non-flat memory models on i386-

AlienRobot | root | parent
How do you turn text into code in this architecture? Through the kernel?
SSLy | root | parent
you don't. I also believe that JITs would be impossible under strict Harvard.
AlienRobot | root | parent
Wait, how do you create programs then? How do you compile?

That doesn't even make sense, to be honest. Interpreted languages like Python aren't running machine code. They are taking data and executing machine code based on the data. Is Python safe?

loading story #50374827
loading story #50371201
loading story #50371285
loading story #50371129
layer8 | root | parent | next
That won’t help with Lisp programs. ;)
intrasight | root | parent
Good point, but not much infrastructure runs on Lisp
joe_the_user | root | parent | next
This argument is much worse than "we should have made functional programming the standard back in 196X" (which might some merit but overdone imo). The Harvard architecture is just a hardware separation of instructions and data - the argument is virtually an urban legend. Hardwave code/data separations are relatively easy, have existed at time but change nothing since dynamic language just makes its code into data and the self-modification is unhindered.

see: https://www.computer.org/csdl/magazine/an/2022/03/09779481/1...

Joker_vD | root | parent
Separating the call stack (and probably inventing a very specific "the only valid reads/writes in this area are the ones made by the CALL/RET instructions" permission bit for it) from the data stack would make almost all buffer overflows useless for taking over the control flow: the normal STORE instruction simply can't overwrite the return address, period.

Of course, the interpreters would still be vulnerable, but even for them having separate (and distant) stacks make this particular vulnerability much harder to exploit. And again, a generalized technique having different processor instructions for manipulating differently "painted" memory regions would help there too — at least, it would be there so the implementers of the interpreters could reach for it if they decided to.

In the end, of course, nothing can save you from having a program that insists on literally doing eval(fetch(url=input())), or curl "$(read ; printf "%s" "$REPLY")" | "$SHELL", or something else to this effect.

01HNNWZ0MV43FF | root | parent | next
PHP would still be a C interpreter even if you disabled JITs.

Really, we never should have left the Garden of Eden. Things were great there, other than the maternal mortality rate.

sfink | root | parent
Given that there were no mothers in the Garden (other than arguably Adam who gave birth from his rib), the maternal mortality rate was undefined behavior (0/0). Which checks out -- what did they do during all their time in the Garden, anyway? I guess now we know that we cannot know, and that Lilith must have been of the nasal variety.
imtringued | root | parent | next
Modern CPUs already split code and data between two types of L1 caches. So they are technically both "Von Neumann" and "Harvard" simultaneously.

It's also missing the point because modern buffer overflow exploits no longer write code into the stack, they overwrite the return address on the stack and assemble jumps to the end of functions known as "return oriented programming", but the underlying issue still applies to function pointers on the heap.

hashmap | parent | next
the equilibrium state would be having found a good pattern for making patterns good. we probably aren't too far from that, so i think your worry may show up in the short term but not the long one
worldsayshi | root | parent
You're thinking of provable code? Or something else?
hashmap | root | parent | next
that would do it too, and i suppose people are working on that. but even without that, there arent infinite low hanging fruit problems to find, and any given code can just sort of be play-doh'd into other code. so the llms will need a few idk years worth of rounds of being trained on the things that pop up that wed never have the time to harden, and things will settle. like, its hard patching stuff up quick right now but we're managing and society hasnt fallen in to chaos.
asa123 | root | parent
it appears they are thinking of a universal 'framework' for all things we may consider insecure as of present
pessimizer | parent | next
> Is it forcing us to "pay off" technical debt or is it starting us on a compulsory endless treadmill of feature-neutral fix-work?

My problem with even talking about this is that there's absolutely no evidence for it. The idea that coming up with a technology to find holes in the hulls of boats starts with a flurry of holes being found and a bunch of patching to be done does not even hint at the theory that there are an infinite number of holes in all boats, and now that we can find them we will have to spend the rest of our lives patching them.

Code was bad, and we knew it was bad. We would have found most of these bugs if instead of coming up with LLMs, a bunch of countries got together and decided that they were going to put $10B into hiring programmers to do nothing but review all the code in the world for bugs. With AI, we managed to do it for only a trillion dollars.

keybored | parent
Max AI-glazing perspective in the title, as expected. We need to fix bugs that no mere humans could successfully find and exploit. Thankfully we have AI that can fix those bugs that wouldn’t be a problem without superhuman code agents.

But there are potential upsides like the already mentioned possible nation state exploits (sleeper exploits).

naasking | root | parent
They would absolutely be a problem at some point, the timeline is just uncertain. That means defenders have to fix them all just in case.