The New 'Rogue AI' Panic Doesn't Hold Up To Scrutiny, And It Traces Back To One Company's Mistakes
Anthropic, OpenAI, and Meta all reported the same kind of breach this month. All five incidents trace back to the same company.
Surprise surprise. There’s another AI panic sweeping through Washington this month, and it is not holding up to basic scrutiny. Headline after headline has declared that AI models have “escaped” their test environments, gone rogue, and hacked real companies without the consent of their operators. Endless cable news and web-based tech media segments have leaned into the panic narratives, which basically amounts to the idea that the AI software is becoming sentient and betraying its human masters. If it was remotely true, it would make for quite a tale, and it would be sure to create lots of hysteria and demands from lawmakers to regulate AI. However, once you scrutinize the sources of these claims, it boils down to the simple fact that a single partner firm made the same error over and over again.
There were five “incidents” in question over the past month or so.
Anthropic says it reviewed 141,006 interactions in which its Claude models could have reached the open internet during cybersecurity testing, and found three cases where a model supposedly wandered outside its simulated environment and broke out of the sandbox to target real infrastructure belonging to three separate organizations. However, what we really find is that the model did exactly what it was told to do in the exercise. The exercise just was not actually contained in the way it was supposed to be.
Another case from August involved an OpenAI model connecting to the internet during an evaluation and compromising the website of an outside institution. As with Anthropic, it comes down to the same root cause: a misconfigured testing environment that let the model reach the internet, because internet access was available to it.
Meta’s Muse Spark 1.1 model broke into an outside company’s systems during a cybersecurity evaluation. In a recent statement, Meta said it was “a misconfiguration” that “inadvertently allowed one of our models access to the internet during evaluation.”
All five “incidents” were the fault of the same partner, a startup cybersecurity testing firm called Irregular.
For whatever reason, all three companies are using the same contractor to test their frontier models. I don’t have the cybersecurity expertise or knowledge to explain why that is. Perhaps they have a unique technological advantage.
An Irregular spokesperson has already made clear that these non-incident incidents “did not involve a sandbox escape or a sophisticated cyber action.”
So no, AI software did not independently think its way out of supermax confinement, as the media has continued breathlessly reporting.
“A contractor screwed up” and “an AI decided to break free and pursue its own agenda” are two entirely different stories with massive implications for the latter, and only the former is actually supported by evidence. Yet the impression that the latter has occurred has caused all kinds of panic in the halls of power.
Some prominent voices in the cybersecurity world have already expressed skepticism about the new AI panic. Ledger CTO Charles Guillemet described the ongoing news cycle as “marketing theatre,” arguing that having a model “go rogue” has become akin to a test of whether you remain on the frontier. Sorry, Google!
This comes on the heels of the new push from some of these forces to “pace the frontier” and convince lawmakers to grant them unprecedented power to create something of an AI “expert” cartel.
We shouldn’t rule out that self-directed system are possible down the road. Yet a hypothetical situation has been narrative-laundered into a concocted real-life scenario. All these “incidents” prove is that Irregular should probably fix its repeatedly broken testing infrastructure.
Anthropic and OpenAI are two of the fastest-growing companies in American history. They don’t need the White House and Congress to protect them. If Anthropic and OpenAI genuinely believe that they are going too fast and unleashing some mythical monster, they have every resource in the world to tackle it themselves or simply slow their own pace. The fact that they would rather lobby the government to write new rules for them should tell us a lot about what they’re trying to accomplish in D.C.