Hacker News new | past | comments | ask | show | jobs | submit
stalfie | on: Kimi K3.1 Pro and K3.1 Pro Highspeed
Well, in the publication they claim to have identified specific cybersecurity post training tasks that nerfed capabilities when they removed them, which seems like a reasonable methodology.

Seeing as no one besides blackhats want frontier cybercrime capability to be open source, this seems like a very sensible thing to do.

So they have published a reasonable means, and they have a motive. I'd put my money on "probably true".

InsideOutSanta | parent | next
> Seeing as no one besides blackhats want frontier cybercrime capability to be open source

Sure. The problem is that there is no practical difference between "cybercrime capability" and "cybercrime defense capability."

All of my PRs go through an adversarial security review by GLM-5.3, because it's the only smart-enough model with a good understanding of cybersecurity issues that won't regularly refuse the job. Removing that ability from frontier models helps nobody except cyber criminals.

stalfie | root | parent
I'm not sure I follow. Let's say that GLM remains the frontier of open source cyber capabilities because Kimi is nerfed. Your situation, and the situation of your attackers, remain exactly the same. You run GLM to audit your code, they do the same looking for exploits.

Let us say that un-nerfed Kimi was released, and it turns out to be at the absolute frontier of cyber capabilities, Mythos 5.5 level, a step ahead of anything else. What now? Well now your situation is exactly the same as before, just that auditing and hacking got more expensive. You run a Kimi security audit, they do the same to try to pwn you. However, now the cabal of Linux kernel defenders with secret access to Mythos 5.5 all of a sudden have no advantage, where before they used to have one. And even worse now they are in a literal race to be the first ones to find every bug that Kimi is capable of finding, and patching it yesterday.

So for you, this makes no difference in practice, your capabilities are not elevated, and neither are those of the attackers. The only result of the other scenario is an overall defender disadvantage.

loading story #50219199
loading story #50217294
ApolloFortyNine | parent | next
The problem is, as anyone trying to address security concerns in claude can attest to, the guardrails trigger even when trying to address real bugs. Good cyber defense requires an understanding of the offense.

How you avoid the "fix my code" attack, the code being someone else's code, I do not know. But nerfing the model doesn't seem like the best path forward either.

zozbot234 | root | parent | next
This is not about guardrails, it's just not post-training the model on tasks that specifically relate to cyber-offense. It's a very lightweight intervention, and it makes the model less of a target for bad actors.
tough | root | parent
claude code "auto-mode" uses a smaller sonnet-like model for classifying, which means it's basically dumb, and the harness biases it into denials
tkz1312 | parent | next
I am a defender, and I really really want to have access to high quality models that are good at finding vulns and hacking. Leaving them in the hands only of government agencies and orgs with enough resources to do extensive training / fine tuning seems like the absolute worst possible case for the future security of global software systems.
stalfie | root | parent
Well, and I am genuinely curious here, wouldn't defender and attacker capabilities being accessible equally by both parties be objectively worse than having a limited amount of privileged defenders? Of course, you could have every piece of code go through an adversarial audit before it's pushed, but in an equal world you would have to always devote as much or more resources to this as your attackers. And historically speaking people just don't do this in practice.

In the scenario where only "verified" defenders get access to the best cyber capabilities, unverified plebian defenders and blackhats are on the same level exactly as described above, there's just also a few defenders that get to run things on godmode.

So to me, it seems that the only result of frontier cyber capabilities is removing a defender advantage. Although that advantage is not for everyone, it can only exists as long as capabilities are unequally distributed. Am I missing something?

loading story #50215501
loading story #50215676
ben_plotke | parent | next
> Seeing as no one besides blackhats want frontier cybercrime capability to be open source

I'm not interested in black hat anything, but I am very invested in making sure the products I work on are not compromised. Given that state agencies have "frontier cybercrime capability", I would like to have the same.

stalfie | root | parent
I wrote a few lengthier comments on some other similar replies. Summarized my argument is that it seems to me to be strictly better that select defenders have an advantage, rather than everyone being equalized. If you're not one of the privileged few that might be frustrating, but open source models upgrade the attack and defense by the same amount, so open source cyber capabilities being at the frontier only results in a defender disadvantage.

Replying here also as I'm curious about the counterargument.

LtdJorge | parent | next
It would also be very sensible to offer a BlackHat Pro version to national officials, for them.
CBLT | parent | next
Please don't change "cybersecurity" to "cybercrime" from one sentence to the next as a rhetorical flourish. It comes off as disingenuous and makes it harder to get your real point across.
stalfie | root | parent | next
Noted. I also did it intentionally to sound a bit spicy, which seemed like a good idea at the time. So I am guilty, and I regret my actions.
zozbot234 | root | parent
"Cybersecurity" as a general term often includes offensive "cyber" which is what is being talked about here.
bbg2401 | root | parent
But the person being addressed specifically used the term cybercrime.