A Senate staffer friend mentioned a rumor circulating among the Union Pub crowd: Anthropic’s prediction of cyber cataclysm from a free-range Mythos model was actually just some final boss level marketing. Soon after he pointed me to a BBC podcast discussing it - the rumor had broken containment. No mention of the rumor’s omicron variant where an Anthropic competitor (*cough* you know the one) was in fact the source of the rumor.
All I could do is shrug and think, same as it ever was.
Companies market, tech bros gonna tech bro, and from the origin of myths themselves, thru Stanley “the Rattlesnake King” Clark (the originator of “Snake Oil”), to Lyle Lanley’s genuine, bona fide monorails, to today, outlandish or maximalist claims are a time-worn razzle-dazzle way to break through the din and move units. Especially in a product cycle with a high implied discount rate.
I don’t know whether the claims about Mythos are true, any or all of them. Multiple things can be true at once–and a person, say a frontier AI model developer, can honestly believe their latest model is a remarkable threat. That claim is neither refutable nor provable ex ante, so there’s some filling in the blanks from those best positioned to know. And maybe it doesn’t hurt that whether epistemically valid or marketing schtick, it all makes darn good copy.
And it’s all beside the point. I’ll return to this in a moment.
Still it was pretty funny the way Anthropic was kind of saying to the Pentagon, “You can’t designate us a national security risk, because I’m designating us a national security risk!”
How to blow up a nuclear program
Couple years ago I took part in what we in DC called a “tabletop exercise” (what normies would call “a discussion”) on what the path of AI advancement tells about the likelihood of a weaponized nuclear Iran. It was convened by foreign policy, military strategy, and technology experts. Again this was a couple years ago, which might as well be decades in Trump Standard Time, or DOGE-years, or whatever. Since then…a few things have occurred.
However the exercise was useful in clarifying still pertinent questions for modeling this issue. Two in particular are worth mentioning. Knowing these two parameters go a long way to knowing whether AI would enable Iran’s nuclear capability faster than we could prevent it.
Does the “cost” of cybersecurity go up or down because of AI over the relevant time horizon?
Here cost is not that of frontier development, advances toward AGI and so on. We presume Iran is free-riding on innovation elsewhere. This cost is akin to ease of deployment. Yes access to compute and electricity, but also the ability for unsophisticated actors to effectively do increasingly complex tasks. In other words, for every “unit” of cybersecurity capability (good or bad), does AI increase or decrease their marginal cost? Consensus: down (high confidence)
Does AI advance cyber offensive capabilities faster or slower than defensive capabilities?1
This one’s trickier. Not only are you trying to get a handle on the path of innovation, but further trying to compartmentalize the nature of specific aspects of it. And those aspects themselves lack clear definitions. For reasons below, and foregoing a bunch of intermediate hand-waving, I think there’s a soft answer. Consensus: offense > defense (low confidence)
There’s a lot of other reasoning that explains how we got to these questions, arrived at decisions, gamed things out, and all laddered up to the macro question. There’s a paper somewhere I think that gets into it.
The only thing that can stop a bad guy with an AI, is a good guy with an AI
Faced with breathless exclamations of AI-midwifed doom, cries of “won’t someone please think of the children,” and general techno armageddon-ry, the e/acc crowd frequently retreat to the axiom, “Technology is a tool. It’s neither good nor bad.”2 I’ve said it in media interviews myself.
Except it’s not always true.3 Some technologies, well not bad per se, only have a purpose that runs one direction in the first order effect. A missile for instance has only a destructive purpose. There’s no benign ICBM program (unless it just doesn’t work). Of course we use that missile for a larger purpose, like deterring worse destruction by an evildoer. The point is not everything is so fully neutrum ex nihilo.4 (Woody Guthrie even accused his guitar of homicide)
By the way, the point of the palantir in that one story is, while not intrinsically evil, its magical power of insight could be misleading and manipulative. Jussayin’.
But we don’t have to be so obtuse. Seemingly every frontier model release precedes stories of developers befuddled by some emergent capabilities and “behaviors”: new modes of deception, shutdown resistance, and so on. Sure these things are not altogether “bad” - deception has productive uses - but it’s akin to the contractor discovering their missiles actually fly further and are much more explosive than they planned.5
And it’s all because, getting back to the Mythos conundrum, the Iron Triangle of Painful Tradeoffs (second only to the Conjoined Triangles of Success in triangle-based business models). Whether we label one of the corners “safety,” “reliability,” “security,” or “predictability,” shortchanging that bucket results from plain old incentives.6 Indeed, “move fast and break things” is just a reflection of ignoring that corner of the triangle.
The stochastic nature of LLMs, those surprise discoveries, means potential vulnerabilities can remain unseen without investment (including time) into discovery. Notwithstanding the user’s intent, an otherwise neutral aspect of a model is effectively a vulnerability (ex: not testing the missile under certain conditions). A company doesn’t have to willfully ignore a bug, because it’s less profitable to discover if there even is one.
Safety in this case isn’t like putting a bigger bumper on a car–it’s not even knowing where the bumper goes.
You see it all the time.
When’s the last time you read through a data breach notification? Even noticed one? In the ante-ChatGPT world, the nation’s cybersecurity was already woefully inadequate to the degree that major and constant breaches are treated like the cosmic background radiation of modern commerce. Yes, yes, the optimal number of almost anything is not zero. So insurance structures emerge, tort law adapts, people die in car accidents, and the world keeps turning.
Besides, if it was really such a problem, says my imaginary male interlocutor made of straw, it would be a huge market opportunity for more secure software–the numbers may seem big but the market tells us they’re smaller than the unseen benefits. Plundering the info of 192 million UnitedHealth customers isn’t that big of a deal is it? – the IT department probably had other mission-critical priorities. Plus you can improve security through numerous services and products. Except now even those are hacked (see, Solarwinds, Okta, Crowdstrike).
Let’s not be so precious. You can spin a plausible story to explain almost all of it away that goes “...something something revealed preference” or “market discovery process yadda yadda.” But don’t tell me hacks disrupting over 40% of the East Coast’s fuel supply (Colonial Pipeline), leaving 240,000 people without power (Ukraine), disabling potentially every Windows computer (WannaCry, incidentally stopped by a 24-year-old who happened to check in during vacation, himself previously a malware-ist), or its reprise the next year that disabled TSMC (the world’s most important chip fabricator), are all part of a system working. Perhaps I tend to overindex the impact of compromising 22 million feds (OPM), having been one.7
By the way, what happened with Salt Typhoon - likely compromising every domestic smartphone user, extraction of over 1 million call records, jeopardizing law enforcement wiretaps, army assets, Congressional staff and top White House staff? It got resolved and is definitely not ongoing right? Cool)
I’ll concede the disruption at over 3,000 colleges “preventing” students from taking finals may be a net wash.
It’s not surprising. It’s the incentives.
Not mustache-twirling execu-villains plotting to release dangerous software on the unsuspecting people of Gotham. Not even an unscrupulous non-mastachioed executive. Just a product manager telling a normal executive, “given our time and manpower, there are no known vulnerabilities in this build.”
Because finding bugs is hard! And thus expensive! And super boring (probably, I failed Intro to BASIC).
Until Mythos, we are told.
Mythos and its successors don’t have to live up to the marketing hype. But very well could. It doesn’t matter either way for making our world safer.
Addendum: On that whole incentives thing, it’s also asymmetric with respect to the exploiters and the exploited. You see, if I’m the proverbial bad guy with the [weapon], the marginal expected return on my efforts to find exploits is almost always higher than the good guy to find and defend them. And if this AI is really as good as they say, the barrier to entry for bad guys just got much lower.
On a contemporaneous episode of “Hard Fork” (The New York Times) Palo Alto Networks CEO Nikesh Arora confidently assessed offensive capability improved faster than defensive because of asymmetric incentives in a post-Mythos environment. That is, it is a continuation of the condition from prior to its release. Note: he runs a massive cybersecurity services firm, so assess however you will.
Disclosure: I’ve been identified by some people in policy circles as part of the AI-doomer cult. Likely having to do with once working at a place with “Responsible” in the name, not owning a Patagonia vest, advocating for an obscure federal non-regulatory office to help with clarifying definitions, and exhibiting some squeamishness around enabling guys to create deepfake porn of their neighbor. That said, I believe the most likely steady-state is neither “doom” nor “utopia” and somewhere closer to the middle. Benefits outweighing costs, with some costs nonetheless addressable.
If insisting dogmatically otherwise, consider whether it applies equally to something like viral gain-of-function research.
Another way to put it: we don’t measure the innovation of our missile-building industry by how many more missiles are being fired. It is a thing we acknowledge we actually want less of, because it means (through much attenuation) there’s less warfare and more peace. “Our missiles are so accurate and lethal, no one would dare attack us, and we never have to use them.”
But while we’re here, it turns out American originated AI-powered technology drove improvements in air-to-air missile range and accuracy for China’s People’s Liberation Army. https://www.fdd.org/analysis/2025/10/29/new-tech-transfer-between-uae-and-china-should-throw-sand-in-the-gears-of-u-s-ai-exports-to-the-gulf/
Stipulated: no product is ever 100% reliable (or equivalent dimension). Even the East Coast Television and Microwave Programming department only goes to a six-sigma tolerance level (very few people are capable of Reaganing). The triangle model tells us this is always the case, improvement in one sacrifices one of the others. What I argue here is that in software broadly, the “safety” corner is systematically biased against.
Thanks for the free year of credit monitoring!