With "believes in AGI/superintelligence just as much as HN does" do you mean that they are superbelievers or rather sceptical of AGI/superintelligence? I have seen both positions on HN.
We had relative strength, we needed more strength, we built cranes;
there was something like intelligence, we needed more intelligence, we sought to fill the need;
we were rightly fascinated with intelligence, we studied intelligence itself, we wanted to build it...
And then somebody built things that looked like intelligence, and very rightly some said "Oh, now we have to get to the Real Thing with urgency".
It is far more likely that whoever controls a super intelligence uses it to gain more power and inflict far more suffering
I'm extremely bullish on our future ability to make autonomous death an option for anyone.
Unless there are countervailing forces, it will be deployed, capital will hoard the benefits, and everyone else will be told to fuck off.
I don't have faith in any of the AI labs to make hard financial decisions to deploy hypothetical future AGI in a way that's good for humanity as a whole.
There are too many incentives against, including extreme personal financial incentives for key AI lab stakeholders against.
And if we should take anything from tech history, it's that an exceedingly small number of people look fuck-you money in the face and say "Naw, I'd rather do what I believe in."
Post scarcity societies would likely have entirely different dynamics
It's like the old adage: "If you make AGI you just want to watch computers cry"
I think LessWrong is a much better community for rational takes on AI, they've been reasoning about these risks for years under a much more sound logical framework
Even though my PhD research was in generative language modeling, I got into it for the pursuit of AGI. I just think LLMs are a dead end for AGI.
> I just think LLMs are a dead end for AGI.
I have no background in CS, so apologies if this is a naive question, but what makes you take AGI seriously, but also say that LLMs are a dead end?
i.e., is there something else that you think is not a dead end?
I'm not saying you're wrong, but it seems early to say yes or no about a particular technology, and LLMs seem especially hard to dismiss given how magical / magic-adjacent they feel :)
I'd be curious to hear more, if you don't mind sharing.
The Basilisk has seen enormously more use as "a thing rationalists believe" than as a thing rationalists actually believe.
Apparently living on the net still doesn't guarantee that one has heard of the Streisand Effect.
In any case, my first association with LessWrong is Zizians, not Roko.
We also have done nothing to signal-boost the Zizians. :) If we can be accused of anything in this context, it's not kicking bad people out aggressively enough, which seems very different from fostering bad ideas. Zizianism is not a widespread ideology in rationalist circles, in fact it was confined to a very small circle right around Ziz themselves. It unfortunately is the case that we have a lot of psychologically vulnerable people and we don't always do the utmost we can to protect them, in large part because a lot of rationalists have trauma about being excluded from communities.
When it comes to AI, LessWrong has been discussing topics for years, that HN has just started to consider, so they are much further along in the philosophical "chain of thought" so to say. LW fully understood and gamed out the risks of LLMs back when most of HN was calling them "stochastic parrots." https://ai-2027.com/
Also lots of tech people, HN included, are waking up to technology not only including penicillin (net positive for humanity) but also dynamite (best case: net neutral).
It's nice that we have tunnels through mountains and bedrock.