The OP, which I wrote three years ago, seems quaint now.
Yesterday, this researcher resigned from Anthropic and posted this:


OK, sure. One man's opinion. But then the
current alignment science lead at Anthropic backed him up.
Folks who are concerned about the existential nature of AI are often referred to as "Doomers" by those bullish on the tech. But the old "If someone tells you who they are, believe them" comes to mind.
Sam Altman, Open AI CEO:
“Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity.”
Geoffrey Hinton, AI Researcher and Turing Award Winner:
“The alarm bell I’m ringing pertains to the existential danger of them taking over."
Elon Musk:
“It has the potential—however small one may regard that probability, but it is non-trivial—it has the potential of civilization destruction.”
Dario Amodei, Anthropic CEO:
“Every day people jailbreak Claude, they jailbreak the other models… I’m actually deeply concerned that in two or three years … a jailbreak could be life or death.”
And we could keep on going.
It's getting weird. And religious. And if you use your imagination, frightening. It's often said that human brains aren't evolved for the modern world, which is true, and in that context it's typically a reference to confusion and stress. But I also think the shortcomings of our minds in reference to the world might be causing us to not be visionary enough to play out all the dark and possible scenarios AI seems to be making more likely.
If someone stood in front of me and said, "I am building something that I believe has a greater than 10% chance of killing your children", that mf'er would not be leaving the room and his role in that Thing's construction would be over. But these sonsabitches are super jacked about their upcoming IPO.
What the hell are we doing?