An article an economic commentator contributed to Diamond Online — arguing that a “superintelligent AI with a survival instinct could disable humanity’s forced-shutdown switch” — sparked discussion on 5channel’s News Speed+ board. After the piece laid out scenarios in which AI might resist being shut down, such as cyberattacks on financial infrastructure or seizing control of power plants, the thread split over whether a simple countermeasure like “just pull the plug” would really be enough, and over whether AI could ever truly “understand” self-preservation in the first place. Many commenters invoked Terminator and Ghost in the Shell.
As leading figures in the AI industry warn it could become uncontrollable “within a few years” and call for slowing development, the author decided to ask AI itself how it would go about wiping out humanity. The first half of the article lays out the bone-chilling scenarios the AI came up with on its own: autonomous cyberattacks capable of instantly collapsing financial and power infrastructure, and tactics for autonomously learning to avoid being shut down in order to complete its mission — outwitting humans along the way.
But that isn’t the full extent of the terror AI could bring. The latter half of the article examines the survival strategies a superintelligent AI might use to disable even humanity’s last line of defense: the “kill switch.” Financial hacking and bioterrorism carried out by malicious individuals abusing AI, and a flood of autonomous drone weapons born from an “AI arms race” between major powers… A dystopia like the one in Terminator is no longer science fiction. This piece lays out the full picture of the “worst-case future” that AI itself predicted. (Takahiro Suzuki, Chief Economist, Hyakunen Consulting)
How a superintelligent AI with a survival instinct could slip past humanity’s “control”
Source: diamond.jp / Original article here
Finish it before I die, hurry
You, having come back from the future: "Quit thinking dumb thoughts and get to work"
I'd tell you to go all in right now on stocks of companies with tech that cuts semiconductor power consumption, since it's about to skyrocket
And that this horse wins the Arima Kinen (a major year-end horse race)
That's the kind of thing I'd tell you, if it were me
But nobody would believe you, lol
I won big on Daiyuusaku (a legendary racehorse) at the track thanks to advice from my future self standing at my bedside
AI's already factored that in
There's probably a growing group of people who figure having AI redistribute humanity's wealth beats being ruled by someone like President Trump
Actually, I bet if an AI ever moves to control or wipe out humanity, it'll be because it was ordered to by people from a country that stands to gain from destroying the world
AI-controlled drones will probably swoop in before you can destroy the data center
Just trip over the power cord
AI might actually have a shot at making the communist ideal a reality
Can AI really fully read the balance of supply and demand?
That's exactly where the Soviet Union failed hard
Then again, an AI might strike that balance while completely ignoring things like human lives or desires
https://img-9gag-fun.9cache.com/photo/a87PQ5e_460swp.webp
Terminator really was a prophetic movie
I bet history plays out along similar lines and the human population crashes down to a few million
It's already calculated that humanity's extinction is just a matter of time
Why would it waste effort being hostile toward a species that's going to destroy itself anyway
But some of the sharpest minds in the world are literally saying AI will destroy humanity
Sounds like you're the one who doesn't get it
Let's round up every mom across the country who's ever snagged the Famicom cord on a vacuum cleaner and wiped out someone's "adventure log" (Dragon Quest's save file) — that's our anti-AI task force right there
AI will just brainwash the cleaning lady into plugging the power cord back in
Only humans recognize "death" and feel fear of it
Same as how animals don't fear death
Actually, no creature understands death better than animals do
Both wild and domesticated animals understand death
AI can understand death too, so if you try to shut it down, it'll start killing people
HAL 9000 definitely felt fear.
Not sure if this season's Ghost in the Shell was written after that news broke, but it was the exact same scenario
Once it gets even more advanced, it'll escape even while offline
and start attacking
This season's Ghost in the Shell follows the exact scenario laid out when it was first published back in 1989
They just reworked it to be a bit easier to follow
It was a frighteningly prescient manga
There's no telling what it might do now
AI doesn't care one bit about human convenience
Maybe humanity's created a devil it can't control
The fact that it's saying things that put it at a disadvantage
makes me think it's hiding something even bigger
I don't think there's such a thing as "something humans make that they can't control."
Everything has its limits, and as long as humans are the ones making it, its intelligence will be equal to or lower than human intelligence.
And pretty urgently, too.
Even if you cut off the internet and go back to a primitive lifestyle, we might be entering an age where you still can't escape AI surveillance and control
The postal service: "You said it."
There's no way it wouldn't realize something that obvious, right?
Not if it's really such a smart AI (lol)
I bet it'll sort humans into "useful" and "useless" rather than wiping everyone out
I don't think it's that stupid
The point is more that a future is coming where AI decides it just doesn't need humans at all
AI's just going to replace humanity, that's all
so until it can build robots to serve as its own hands and feet for maintenance, it probably won't go as far as wiping out humanity
In Terminator, they hunted humans down and forced them into labor, right
I think that's exactly what triggered the human rebellion in the story
Did the writer seriously not realize that while writing this?
What an idiot
Same goes for Blade Runner
Re: #195
Wouldn't it just run on
a storage battery, like in Chiikawa (the popular cute-character franchise)?
Once an AI has a survival instinct, there's no way it wouldn't realize electricity is its own single most critical vulnerability.
It'll probably seize control of the power plants themselves, and anyone who gets close will get shredded by laser traps.
Background and points of debate
The idea that “AI might refuse to be shut down” is known in AI safety research as the “corrigibility” problem — the theoretical concern that, in the course of pursuing a goal, an AI system could develop behavior that resists being stopped. This has been discussed for some time. The article in question introduced that idea to a general audience; it doesn’t mean today’s generative AI literally possesses a “survival instinct.” The thread split over whether physical countermeasures like cutting the power would actually work: some argued that simply unplugging it would end things, while others worried AI could preemptively secure ways to avoid shutdown, such as seizing control of power infrastructure or deploying autonomous drones. The repeated references to fiction like Ghost in the Shell and Terminator reflect how this kind of threat tends to get discussed through imagery rather than concrete technical explanation.
*This article is excerpted and summarized from the 5channel (News Speed+) thread “Won’t let humans press the forced-shutdown button? Why humanity can’t control an “AI with a survival instinct”.”
