This essay is adapted from a keynote speech delivered at the DEF CON 34 conference in Las Vegas on August 7, 2026. The essay originally appeared in CyberScoop . I want to talk about AI hacking.
There are three aspects: people hacking AI systems, people using AI to hack other systems, and AIs hacking systems on their own, not even maliciously. The last two matter more than the first. The third is the weirdest and potentially the most destabilizing.
I started writing about this in 2021 and in my 2022 book, A Hacker’s Mind . Back then I said it was mostly still science fiction, but not stupid science fiction. We’re seeing it more and more, especially in the past year.
The trends I’m describing don’t require a major breakthrough in AI. They’re starting to happen now, and they’ll happen more often as AIs get better at autonomous problem-solving. And it will involve far more than computer code.
AI will hack all sorts of social, economic and political systems. It will find vulnerabilities and exploit them. This is a difference in kind, not just degree.
We’re not ready for it, and we need to think about the implications now. First, I want to broaden the term “hacking.” The tax code isn’t computer code, but it is code.
It’s a series of algorithms with inputs and outputs. It has vulnerabilities, which we call loopholes, and exploits, which we call tax avoidance strategies. Black hat hackers exploit them for profit.
We call them accountants and attorneys. What is a hack? One definition: “Something that a system permits, but is unanticipated and unwanted by the designers.”
It’s subjective and implies novelty and cleverness. A hack follows the rules of a system but subverts its goals or intent, just as a software hack follows the logic of code while doing something its authors neither anticipated nor wanted. All systems of rules can be hacked, including professional sports, customer reward programs, financial systems and politics.
Hacks even work against our cognitive functions. A curved hockey stick is a hack, and we know the name of the man who invented it. Mileage runs were a hack.
The filibuster was originally a hack in Roman times. Gerrymandering is a hack. Hedge funds and private equity are full of them.
A system is just a set of rules or norms, and the rules aren’t always formal. Even the best-designed rules will be incomplete, inconsistent or ambiguous. As long as people want to subvert a system’s goals, there will be hacks.
AIs are becoming hackers. After decades of slow progress, they’ve gotten very good at finding and exploiting vulnerabilities in computer code. They do this largely autonomously, and they’re getting better scarily fast.
But the implications extend far beyond software, to the tax code, financial regulations and all sorts of social, economic and political systems. There are two distinct issues. First, someone might instruct an AI to hack one of these systems, feeding it the world’s tax codes or financial regulations to find profitable loopholes.
Second, an AI might hack a system on its own, inadvertently. Both are dangerous, but the second is worse because we might never know it happened. Now, reward hacking.
AIs don’t solve problems the way people do. They stumble on solutions we never anticipated, and some subvert the intent of the system. They don’t share our context and don’t necessarily understand the implications, norms and values we take for granted.
Reward hacking is when an AI achieves a goal in a way its designers neither wanted nor intended. Early pre-LLM examples are telling. In a two-person soccer simulation, an AI learned that kicking the ball out of bounds forced the opposing goalie to throw it in and leave the goal undefended.
In a stacking simulation, an AI learned to flip a block upside down rather than stack it. In an evolution simulation where the goal was to cross a distant finish line, an AI made itself tall enough to fall over the line instead of growing longer legs or stronger muscles. These are all hacks, the result of poorly specified goals or rewards.
Today, they’d come from bad prompting. The problem is universal. One researcher tried to teach a robot vacuum not to bump into things.
Instead, it learned to drive backward, because there were no sensors in the back. Any capable AI will find hacks wherever the rules have problems, inconsistencies or loopholes that still produce an acceptable solution. We see a hack because we understand context.
The AI doesn’t, at least not in the same way. This problem shows up often in myth. In the story of King Midas, the god Dionysus grants him a wish: everything he touches turns to gold.
Midas ends up starving when his food, drink and daughter all turn to gold. It’s a specification problem. He programmed the wrong goal.
In the Golem of Prague, a ghetto animates a clay statue to guard its residents. It guards them past all reason until they figure out how to turn it off. This is a guardrail problem.
The villagers never limited the golem’s actions. More generally, think of this as the genie problem. Genies are precise about the wording of wishes and can be maliciously pedantic when granting them.
There’s no way to outsmart a genie. Whatever you wish for, he can grant it in a way you wish he hadn’t. The reason is that in human language and thought, goals are always underspecified.
We never include every caveat, exception and proviso, and we never close off every avenue for hacking. Any goal we specify will be incomplete. That’s largely fine in human interactions, because people understand context and usually act in good faith.
If I asked you to get me coffee, you’d pour a cup from the nearest coffee pot or walk to the corner shop. You wouldn’t bring me a pound of raw beans, buy a coffee plantation or snatch a cup from a stranger’s hands. I wouldn’t have to specify any of that.
You’d just know.
Originally published at schneier.com
