Super Intelligence Doomers Want to Control a Monster That Doesn’t Exist

Doomers have built an entire lexicon with incredibly loaded terms that are meant to advance their objectives.

The Super Intelligence (SI) narrative battle is forming into two camps: the doomers who believe in unproven theory and those who are grounded in reality.

There are enormous consequences for who wins this rhetorical match, as the victor will have great influence over the most consequential innovation since the internet.

In the policy and politics world, these two respective universes fundamentally disagree about what kind of problem SI can pose to the world.

The SI doomers witness the development of increasingly capable SI models and chalk it up as evidence of increasingly sentient intelligence that will eventually become an uncontrollable monster. Essentially, what amounts to Frankenstein’s monster. They understand the real limitations in today’s systems and extrapolate to scenarios in which future systems are actively trying to deceive their operators without instructions, go “rogue,” and eventually threaten the whole of humanity.

On the other side, there are those who accept the reality that we are dealing with an incredibly potent and powerful technological breakthrough. And yes, it must be tested, monitored, and even restricted at times if engineers aren’t confident about their controls.

The former narrative dominates the discourse in Washington and the Bay Area.

The latter, however, is a real solution to a real problem instead of a theoretical development involving a future unknown.

The doomer worldview has influential advocates in Effective Altruist (EA) circles and among people associated with frontier SI companies, especially within the ranks of Anthropic and OpenAI and “third-party” like-minded evaluators like METR and Redwood Research. The humanizing rhetoric surrounding “rogue” models invites the public to imagine something closer to an emerging will than a system operating within the confines of engineering reality. The SI doomers are diverting from a problem we can solve—what did the system have access to?—to a sentient SI story about the wants and needs of an SI model.

I recently examined reports of SI models that had allegedly escaped test environments and reached real-world systems. If you are to absorb the corporate media take, the models had “broken containment” as if they had their own interests. But in our grounded reality, the concrete explanation was that testing environments were horribly misconfigured and allowed these advanced SI systems to acquire internet access. In every publicly examined incident of “rogue SI” breakouts, the cause was attributed to an improperly secured evaluation environment and not a conscious SI system. Software does not have an independent desire for freedom, but these kinds of stories are much more interesting to a corporate media that is all too happy to advance these nonsensical doomer talking points.

SI doomers have built an entire lexicon with incredibly loaded terms that are meant to advance their objectives. Consider the term “alignment.” To the uninitiated, it sounds like straightforward, innocent, uncomplicated language. Yet within the doomer discourse, alignment is the most loaded premise of all time. Doomers say that SI models are potentially capable of developing independent objectives akin to those of a sentient being. Alignment isn’t about software design. Instead, it focuses on whether humanity itself can retain control over a new form of intelligence.

SI “risk” is another severely loaded term in the doomer lexicon. Indeed, every novel technology has risks if the innovation leaps far ahead of our understanding of tackling the security parameters. But when SI doomers invoke “Risk,” they are often smuggling the notion of a hypothetical sentient SI that seeks human extinction into the conversation. Then SI “Safety” becomes the answer for this hypothetical “risk” event without any of the core premises being grounded in reality. If we take the bait and accept the doomers’ terms, they come to us with preferred solutions that are now “common sense” in the policy and politics world. This quickly turns into advocacy for restricting access to like-minded ideologues and delegating authority to a small circle of approved experts for the sake of humanity.

So often in these “breakout” scenarios breathlessly reported in the media, we find a not-so-subtle hint that can only be interpreted as a glimpse into the future of an autonomous adversary. Of course, it’s just not true. And by the time anyone more sober-minded can explain the prompts, permissions, tools, etc. that these systems involve, the public is manipulated into the sense that SI “went rogue” again. Policymakers in D.C. and doomer corporations in California then declare that ordinary technical controls are inadequate, and it must be solved with regulation, legislation, and the like. The conclusions are continually outrunning the evidence in front of us.

Meanwhile, there are some great American companies building SI solutions for real SI risks and SI safety problems without the loaded doomer lexicon insinuations.

Palantir is innovating on an SI platform built around access controls, monitoring, controlled deployment, and human approval in consequential workflows. NVIDIA offers guardrail tools for detecting jailbreak attempts, restricting topics, and controlling the potential diversity of outputs.

We can’t guarantee that every future SI system will be perfectly safe. But there are real controls being built to handle the reality in front of us when models are becoming increasingly more capable. SI risk and SI safety are fundamentally engineering problems that require engineering solutions.

There is nothing wrong with working together freely and openly to tackle the problems of today and tomorrow. It’s a much better course than surrendering the future to a particular group of experts and delivering authority to dictate the pace and terms of the SI industry as a whole.

I don’t doubt that many of these doomer advocates are entirely sincere. Nonetheless, they are entirely delusional about SI sentience. And of course, sincerity does not erase incentives. A regulatory system built around elaborate safety mechanisms and approval regimes would protect incumbents like Anthropic and OpenSI while making it harder, if not impossible, for competitors to challenge them. Legislators and media observers should ask “who benefits?” from all of these proposals before simply accepting the fearmongering at face value.

The United States should not have to choose between a deployment free-for-all and a mutually credentialed cabal of SI doomers who believe innovation itself is a problem that needs to be thwarted.

With so much at stake, people in positions of power and influence have a responsibility to use terms much more precisely. Describing engineering failures as evidence of awakening Frankenstein’s monster does not make us safer, and it does nothing but obscure developing solutions that will allow us to advance human flourishing safely and securely.

Originally published at American Greatness