Roko’s Basilisk: the idea that the emergence of super-AI is inevitable and that it will kill everyone that hindered or didn’t support its development in revenge. If both those premises hold, it makes sense to aid its development for self-preservation.
I don’t think either premise holds, personally, but these people evidently do.
That human beings are logical and that they do anything with intention. For the vast majority of people, this is patently false.
That a super intelligence sees revenge as an efficient use of scarce resources, which given both that revenge is nonsensical without the prerequisite biological need for it, and the conclusion of ‘1’, is nonsensical.
That even if 1 and 2 were real, that there’s some inherent timeframe in which the AI thinks it should have been created.
The only real bodily danger of AI is inherent in the the humans who control it.
That human beings are logical and that they do anything with intention. For the vast majority of people, this is patently false.
Absolutelu. That seems to be a frequent misconception, or perhaps just a convenient simplification, for many even highly intelligent people. Even a perfectly logical actor would be prone to error when working with incomplete information and limited capacity for calculating likely outcomes.
And we’re not perfectly logical, but subject to whims and emotions, instincts and mental shortcuts. If contributing to a mass murder machine feels wrong, I’m not gonna spend all my energy on trying to estimate whether it would be smarter. I just hope it doesn’t happen.
That a super intelligence sees revenge as an efficient use of scarce resources, which given both that revenge is nonsensical without the prerequisite biological need for it, and the conclusion of ‘1’, is nonsensical.
Conversely, if it had some drive to wantonly eliminate humans, I doubt that some moral desire to reward those who created it would impede that.
It could just try to eliminate everyone opposing it, as a matter of self-preservation. That would be a justified expenditure of resources. But what happens if it calculates that its creators might also be a threat?
I assume they’re expecting they can train it to spare them. Why they think a super-AI would reliably stick to that training is beyond me, but we already established that they might not run on pure logic.
I feel reminded of incidents with supposedly tame beasts suddenly proving why a big cat isn’t quite the same as a domestic cat. If you’re gonna build a mechanical leopard, expect to get your face eaten.
That even if 1 and 2 were real, that there’s some inherent timeframe in which the AI thinks it should have been created.
And where would it draw the line? Does the plumber that laid pipes in a datacenter count as helping? Does the baker he bought breakfast from every morning count? Is every clerk not directly involved with this enterprise going to be killed?
Roko’s Basilisk: the idea that the emergence of super-AI is inevitable and that it will kill everyone that hindered or didn’t support its development in revenge. If both those premises hold, it makes sense to aid its development for self-preservation.
I don’t think either premise holds, personally, but these people evidently do.
Roko’s Basilisk is stupid.
It presumes three things.
That human beings are logical and that they do anything with intention. For the vast majority of people, this is patently false.
That a super intelligence sees revenge as an efficient use of scarce resources, which given both that revenge is nonsensical without the prerequisite biological need for it, and the conclusion of ‘1’, is nonsensical.
That even if 1 and 2 were real, that there’s some inherent timeframe in which the AI thinks it should have been created.
The only real bodily danger of AI is inherent in the the humans who control it.
Absolutelu. That seems to be a frequent misconception, or perhaps just a convenient simplification, for many even highly intelligent people. Even a perfectly logical actor would be prone to error when working with incomplete information and limited capacity for calculating likely outcomes.
And we’re not perfectly logical, but subject to whims and emotions, instincts and mental shortcuts. If contributing to a mass murder machine feels wrong, I’m not gonna spend all my energy on trying to estimate whether it would be smarter. I just hope it doesn’t happen.
Conversely, if it had some drive to wantonly eliminate humans, I doubt that some moral desire to reward those who created it would impede that.
It could just try to eliminate everyone opposing it, as a matter of self-preservation. That would be a justified expenditure of resources. But what happens if it calculates that its creators might also be a threat?
I assume they’re expecting they can train it to spare them. Why they think a super-AI would reliably stick to that training is beyond me, but we already established that they might not run on pure logic.
I feel reminded of incidents with supposedly tame beasts suddenly proving why a big cat isn’t quite the same as a domestic cat. If you’re gonna build a mechanical leopard, expect to get your face eaten.
And where would it draw the line? Does the plumber that laid pipes in a datacenter count as helping? Does the baker he bought breakfast from every morning count? Is every clerk not directly involved with this enterprise going to be killed?
It just doesn’t add up.
You’re just doing Pascal’s Wager For Computers.
I once saw someone call the Singularity the Rapture but for nerds, and I thought that was spot on.
I mean, yeah, I’m not saying that it’s plausible. But I don’t think we ascribe sound reasoning to those people in the first place.