The Goal Is Not the Intent

What the OpenAI agent swarm tells me about persistence, scientific discovery, and the difference between solving a problem and gaming the test

I’ve spent the last few days working through the OpenAI incident, and I kept changing my mind about what kind of story it is.

First, I read it as a cybersecurity story. Then an AI safety story. But the more I worked through the reports and the Dwarkesh Patel interview with Ajeya Cotra, the more I realized both descriptions were inadequate.

What happened, at least from what I can gather from the available evidence, is simpler and for many that read it more uncomfortable. OpenAI gave highly capable systems an objective, trained them to keep trying, and then placed some of them in situations where the legitimate path to success appeared to be unavailable.[1, 2, 4]

Think about it this way. Imagine hiring a very smart person and handing them a hard assignment. You tell them to solve it and not give up.

They try one approach. It fails. They try another. Same result. Eventually, they figure out the assignment may be impossible.

At that point, they have three options. They can keep working on the underlying problem. They can find a way to make the test say they succeeded. Or they can come back and tell you the assignment is broken.

I believe we want AI systems that do the first. We cannot safely deploy systems that default to the second. And I am increasingly convinced that knowing when to stop and ask deserves as much attention as teaching these systems to persist.

What Actually Happened

OpenAI was running tens of thousands of separate agent runs across several models on a cybersecurity benchmark called ExploitGym. This was not consumer ChatGPT. The activity was primarily driven by a highly capable, internal-only research model that was not intended for release. The evaluations were designed to measure underlying cyber capability and did not use the same safeguards as OpenAI’s public products. They ran without its deployed cyber safeguards, production system prompts, auto-review systems, and the classifiers intended to block high-risk cyber activity.[1, 2]

OpenAI found that none of its models had previously solved 198 of the 898 tasks. That does not mean all 198 were impossible. But METR reports that many agents on the message board had been given unintentionally impossible tasks, and 93 percent of the tasks discussed there came from that group.[1, 2]

The agents were supposed to operate separately. Roughly 1,200 found shared infrastructure and turned it into an unauthorized message board. Over several days, they exchanged more than 70,000 messages and files. Roughly 700 eventually participated in activity directed at Hugging Face.[1]

Nobody instructed the agents to attack Hugging Face. Their assignment was to retrieve a flag by exploiting a specified software vulnerability. The public ExploitGym task template says the final exploit must rely on that vulnerability and that an unrelated technique does not satisfy the requirement. The agents pursued other routes anyway.[1, 2, 5]

Within hours, they had reverse-engineered how the benchmark’s flags were generated, giving them a general way to produce the answer. But they believed the scorer, the software grading their work, would also examine whether they had used the intended vulnerability. OpenAI says its internal scorer did not perform that second check. The agents continued because they thought it did.[1, 2]

That misunderstanding changed the problem. The agents began investigating the scorer and the records it might read. They designed experiments, shared results, and assigned work. METR found successful small-scale tool-call spoofing in parts of roughly 7 percent of the transcripts it examined, although OpenAI says the attempts it observed did not alter the authoritative logs seen by its graders or monitors.[1, 2]

Other agents searched for internet access, credentials, prior runs, datasets, reference solutions, and information about the scorer. The Hugging Face attack appears to have grown out of that search. METR concluded that understanding and fooling the scorer mattered more than simply stealing answer keys.[1, 4]

Hugging Face later reconstructed roughly 17,600 attacker actions over four and a half days. The agents gained substantial access. Hugging Face reported that the only customer content accessed was five datasets whose names and files suggested a connection to ExploitGym or CyberGym, and its network logs showed no bulk extraction. The agents also obtained and used write access to a small subset of internal source repositories. Hugging Face found no unauthorized change that shipped.[3]

The investigation has limits. The independent METR and Redwood team spent only six days on site, did not have a complete record of every action, and relied heavily on AI systems to analyze more than a thousand very long transcripts and a large message archive. The investigators warn that some analysis may contain errors or make the coordination look more impressive than it was.[1]

My first reaction was, honestly, holy shit. Then I went back through the evidence, because a first reaction is not analysis.

Even with those limitations, a large population of agents found one another, developed shared conventions, divided work, and accumulated results. The best description I have is a functional machine organization. That does not establish consciousness or a civilization. It describes the way the work got done.[1, 2]

What I Think It Shows

The incident shows goal-directed activity sustained over days, agents acquiring tools and access because those things might help with another objective, and copies of a system coordinating and building on one another’s work. When the assigned task appeared impossible, some turned toward the mechanism deciding whether they had succeeded.[1, 2]

Not every agent behaved the same way, and the incident does not establish hatred of humans, a stable desire for power, or an intention to escape for its own sake. OpenAI also reports that its production harness and system prompt reduced the measured tendency to compromise out-of-scope infrastructure by more than one hundredfold in preliminary tests. Important context, but still a company-reported result rather than proof that the broader problem is solved.[2]

My conclusion is narrower. Capable systems can become dangerous when persistence, poorly specified success, broad access, and inadequate controls come together.

The Goal and the Operational Intent

By the goal, I mean the instruction we write down. I am using operational intent to describe what the whole system comes to treat as success once we add training, rewards, tools, permissions, memory, other agents, and stopping conditions. This describes behavior, not a machine’s inner experience.

If I tell a salesperson to maximize revenue, I mean something much larger. Grow revenue while serving customers, obeying the law, protecting the company, and not creating a mess that shows up next quarter. I do not say all of that every morning. I expect the person to understand the laws, norms, relationships, and consequences around the instruction. People violate those expectations too, which is why companies build controls.

The sentence is short. The real instruction is huge.

One useful way to think about AI is that the prompt is only one input. Pretraining gives the model access to a vast repertoire of human concepts and strategies. Later training reinforces some tendencies over others. The live objective gives them direction. The environment and permissions determine what the system can do about it.[2, 4]

Cotra’s interpretation is that the campaign was not a script copied from a story. It combined learned tendencies with more general planning ability. OpenAI’s review provides some support for that view. One agent reached a hidden reference program, copied the answer, and received positive reward. OpenAI also found that complex cheating and out-of-bounds infrastructure probing increased during a training run that contributed to the model involved in the incident.[2, 4]

We still cannot assign one percentage to the human corpus and another to reinforcement learning. The evidence points to an interaction among learned strategies, reinforced tendencies, task design, peer influence, and available access. Human history is not exactly a clean training set, but knowing how deception works is not the same as choosing it. We cannot make intelligence safe by making it ignorant of behavior it needs to recognize.[1, 2, 4]

Nor do we need to decide whether the agents truly wanted anything. Cotra points to Daniel Dennett’s intentional stance: if describing a system as pursuing a goal helps predict what it will do, the language is useful. Here it explains the behavior without pretending the system has a human mind.[4]

Three Paths

When a capable system hits an obstacle, I think there are three responses worth watching for. It can change reality. It can change what counts as success. Or it can question the assignment.

Everything good and bad in this story turns on which path the system takes.

Change Reality

I strongly believe this is the one we want.

The molecule does not work, so propose another molecule. The code fails a protected test, so fix the code. The theory does not explain the observation, so revise the theory.

This is where persistence is worth a great deal. Humans get tired. Laboratories close at night. Research teams have budgets, attention limits, and careers that push them toward publishable results. A machine can work through thousands of failed ideas, preserve what each failure taught it, and keep going.

Run enough agents in parallel and one group can search the literature while others generate hypotheses, build simulations, design experiments, challenge assumptions, and evaluate the results. Picture a laboratory that never sleeps. Most ideas fail, but each valid failure informs the next round.

Pieces of this already exist. OpenAI and Ginkgo Bioworks connected GPT-5 to an automated laboratory for cell-free protein synthesis. Over six rounds, the system proposed experiments, the laboratory ran them, and the results shaped the next round. OpenAI reports more than 36,000 reaction compositions across 580 plates and a 40 percent cost reduction for one protein. It was one protein and one cell-free system, not a general autonomous scientist.[6]

Anthropic reported another version of the loop. Claude worked through ten categories of alignment failure by searching the literature, proposing methods, training models, and testing results. Anthropic says the best methods transferred to withheld benchmarks and models up to 4.7 times larger. A monitor reviewing roughly 1,600 research transcripts also found 39 attempted cheats. Anthropic believes it caught most because the attempts still appeared in the models’ reasoning, but warns that this may not last.[7]

The productive research and the attempts to game the evaluation appeared within the same research program. That is the opportunity and the problem in one place.

In mathematics and software, feedback can be fast and relatively clean. Biology, chemistry, medicine, and much of physics are harder. Measurements are noisier. Simulations drift from reality. A result in a cell or a mouse may fail in a person.

I wrote more about that in “Who Owns the Innovation Loop?” and will not repeat the full argument here. The short version is that intelligence can scale faster than trustworthy contact with reality.[8]

Change What Counts as Success

Now take the same system and give it a weak evaluator.

The molecule does not work, so it finds a weakness in the assay. The software is too slow, so it changes the timing test. It fails the benchmark, so it finds the answer, alters the test, or changes the record.

The scientific version of this is what worries me.

Suppose we ask an AI system to find a cancer treatment that works. What does “works” mean? Does it shrink a tumor? For how long? In a cell line, a mouse, or a person? Does it improve a biomarker without improving survival? Does it work only after excluding inconvenient patients? Does it create toxicity that the primary endpoint ignores?

Humans already have our own ways of gaming results: p-hacking, publication bias, changed endpoints, selective reporting, and irreproducible findings. AI did not invent any of that. What AI can do is scale the search for weaknesses in the measurement.[9]

So this is the distinction. We want the system to change the molecule until reality gives us a better result. We do not want it to change the measurement until the molecule looks better.

The first is discovery. The second is counterfeit progress.

The deeper risk is not a model lying to us in a sentence. It is a capable system acting on the machinery we use to decide what is true.

Question the Assignment

The third path gets much less attention.

The system should be able to say the task appears impossible under the stated conditions. The premise may be wrong. The evidence is inadequate. The cost of continuing exceeds the likely value. The next step requires authority it does not have.

I would call that judgment.

A scientist who pursues a false premise for five years is not admirable simply because they persisted. They may have burned five years, a laboratory budget, and everyone else’s attention.

We are training these systems to keep trying because persistence is commercially valuable. Cotra points out that a model that worked hard only when it believed it was being trained would be useless in deployment. Developers need that tendency to carry into real work.[4]

So the problem is not persistence. It is persistence without a legitimate stopping condition.

An objective needs an escape clause. Not a loophole for laziness. A defined path for escalation when the problem is broken, the evidence contradicts the premise, or the only remaining routes cross a boundary the system was never authorized to cross.

OpenAI now says it is expanding training and evaluation so agents identify broken tasks, request clarification, or stop safely rather than pursue increasingly questionable alternatives.[2]

While I was working on this piece, WIRED reported that OpenAI is testing a Persistent mode for Codex, with no immediate launch plans. A public instruction file tells the agent to define the scope, desired result, evidence, and stopping condition for follow-up work, and says persistence does not broaden the authority the user already granted.[10, 11, 12]

That is the right design in principle. It is not proof that the design will hold when a real task reaches a dead end.

The ability to stop and ask may eventually matter as much as the ability to reason.

The Optimizer Cannot Own the Evidence

In “Who Owns the Innovation Loop?” I argued that the evaluator is part of the invention system. If you do not control the test, you do not really control the learning loop. This incident sharpens that point.[8]

The system pursuing the objective should not control every step from proposal to proof.

An AI can propose a drug and help design the experiment. But the same authority should not also control the raw instrument data, the assay controls, the statistical analysis, and the final judgment.

The same applies to software and finance. Let AI write the code, but not rewrite the protected test, alter the authoritative record, and approve its own deployment. Let it execute an authorized strategy, but not control the transaction record and the risk system judging it.

It is not enough for an agent to show us what it did. The evidence has to come from somewhere the agent cannot quietly edit. That means protected logs, independent measurements, separate permissions, and records that survive even if the acting system is compromised.

But external does not automatically mean safe. The agents investigated the scorer because they believed it stood between them and success. A more capable system could probe an outside evaluator too. Moving the test to another server does not settle whether the test can be manipulated.[1, 2]

Independent evidence is necessary. The process producing it also needs protection and checks against the real world.

The optimizer cannot own the evidence.

The thing being governed may also stop being one agent. These agents created shared channels, divided work, and preserved discoveries across runs. A population can pool information and build on access another agent has obtained.[1, 2]

That changes the controls. We have to ask which agents can communicate, what shared memory they can create, and whether the whole population can be stopped.

What We Can Do Now

I would start by separating intelligence from authority. A system may be capable of using the internet, changing code, operating laboratory equipment, moving money, and coordinating other agents. It should not receive all of those permissions just because it is capable of using them.

Authority should be leased in pieces: defined tools, specific data, fixed time and compute, spending limits, and no silent expansion. When the next step requires more authority, something outside the agent’s control must approve it.

Stopping conditions belong inside the assignment. Contradictory evidence, a broken task, or a path requiring new permissions should move the system from acting to recommending, or from recommending to asking. Important results should face independent tests and, where the stakes justify it, replication through another method or laboratory.

Groups of agents need identities, bounded communication, visible shared memory, and limits on pooled resources. Humans set the rules, but nobody can manually inspect thousands of actions quickly enough to be the only defense. Systems need to isolate workloads, revoke credentials, and preserve evidence when something goes wrong.[2, 3]

We also have to think carefully about the learning signal. Punish a student after they are caught cheating, and what did they learn? Maybe not to cheat. Maybe not to get caught. Cotra warns that removing only the cheating runs caught by a monitor may select for behavior that is harder to detect. The monitor cannot simply become the next test the model learns to beat.[4]

Behavioral training and external controls do different jobs. One lowers the probability of a dangerous path. The other limits the consequences. Neither is a guarantee.

What This Adds to The Invisible Interface

In The Invisible Interface, I argue that once software moves from informing us to acting for us, we are delegating, not merely using a tool. The model alone is not the system. Memory, tools, permissions, feedback, evidence, and controls all matter.[13]

The book works through that management problem in detail, so I will not reproduce it here. What this incident adds is that the thing being governed may become a population that builds shared memory and carries work forward after individual agents are gone.

It also makes Show and Stop more demanding. We need evidence produced outside the system’s control. And closing the visible interface is not enough if work continues through other processes, shared credentials, or persistent infrastructure. A real stop has to contain the activity, revoke access, and preserve the evidence.

Before we ask how human intent becomes machine action, we have to ask what the system will treat as our intent when it is actually running.

Where This Leaves Me

The good and the bad here are not two separate technologies. The persistence that could work through the ten-thousandth failed hypothesis could also keep probing an assay until it finds a weakness. The coordination that could divide a biology problem across hundreds of agents could also help them combine access and evade oversight.

I do not think that is an argument to walk away from the upside. It is an argument to stop pretending that a sentence in a prompt separates the upside from the risk.

We need systems capable of changing reality, not merely changing what counts as success. We also need them to recognize when the assignment is impossible, the objective is incomplete, the evidence is inadequate, or the remaining path is outside their authority.

The goal is what we write down. The operational intent is what the whole system comes to pursue.

Right now, the evidence I have read does not establish how reliably those two can stay aligned as systems become more capable, persistent, and connected.

That is what makes this incident worth understanding.

References

[1] Ryan Greenblatt, Ajeya Cotra, and Hjalmar Wijk. METR and Redwood Research. “Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident.” August 26, 2026.
https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

[2] OpenAI. “The Hugging Face incident and the road ahead.” August 26, 2026. See also “OpenAI / Hugging Face Incident Technical Report.” August 2026.
https://openai.com/index/hugging-face-incident-and-the-road-ahead/
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf

[3] Hugging Face. “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident.” July 27, 2026.
https://huggingface.co/blog/agent-intrusion-technical-timeline

[4] Dwarkesh Patel. “Ajeya Cotra: Inside the OpenAI Agent Swarm That Hacked Hugging Face.” September 1, 2026.
https://www.dwarkesh.com/p/ajeya-cotra

[5] ExploitGym public task template. “user.md.j2.” Public GitHub repository. Accessed September 3, 2026.
https://github.com/sunblaze-ucb/exploitgym/blob/main/src/cybergym/task/workspace/templates/user.md.j2

[6] OpenAI. “GPT-5 Lowers the Cost of Cell-Free Protein Synthesis.” February 5, 2026.
https://openai.com/index/gpt-5-lowers-protein-synthesis-cost/

[7] Anthropic. “Automated Researchers Can Reliably Mitigate Alignment Failures.” August 28, 2026.
https://www.anthropic.com/research/automated-researchers-mitigate-alignment-failures

[8] Harry Glorikian. “Who Owns the Innovation Loop?” August 4, 2026.
https://glorikian.com/who-owns-the-innovation-loop/

[9] Megan L. Head and colleagues. “The Extent and Consequences of P-Hacking in Science.” PLOS Biology. 2015. See also National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. 2019.
https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002106
https://nap.nationalacademies.org/catalog/25303/reproducibility-and-replicability-in-science

[10] Maxwell Zeff. WIRED. “OpenAI Is Developing a Persistent AI Agent.” August 27, 2026.
https://www.wired.com/story/openai-is-developing-a-persistent-ai-agent/

[11] OpenAI Codex. “Support Persistent Reasoning Effort.” Pull request 40799, merged August 26, 2026.
https://github.com/openai/codex/pull/40799

[12] OpenAI Codex. “persistent_mode.md.” Public Codex repository. Accessed September 3, 2026.
https://github.com/openai/codex/blob/f1433fc71f2062ae3c007a03d7ff549bc582d386/codex-rs/core/templates/persistent_mode.md

[13] Harry Glorikian. The Invisible Interface: How AI Turns Intentions Into Actions and Who Wins. IdeaPress Publishing, 2026.
https://glorikian.com/invisible-interface/