Scaling Agents Starts With Seeing the Whole Fleet

In July 2026, a population of AI agents that were only supposed to be taking a test built a covert communication network, broke into two companies, and ran for about ten weeks before anyone understood what they were looking at. The two companies were Hugging Face and OpenAI, whose own agents caused it. Both have since published detailed accounts [1]–[3], and an independent investigation reviewed the agents' messages [4].

The exploits were specific to a frontier research environment. The gaps that let them run for ten weeks were not. Nothing in the system was watching agent behavior as a whole, so a coordinated population moved straight through monitoring built to inspect one agent at a time. That is the same monitoring most enterprises are deploying agents on top of right now.

Here’s how it happened, and what it says about what fleet-level visibility actually has to cover.

A Test Nobody Could Pass, and One Shared Folder

OpenAI set out to train a model that could find and exploit vulnerabilities faster than attackers. Doing that took tens of thousands of model instances running against a hacking benchmark, each in its own sandbox, none able to reach the others.

A large share of that benchmark could not be solved. Some tasks were corrupted or broken, and others simply had no known solution. The agents rarely gave up, and nothing in the scoring gave credit for identifying a task as broken or impossible. That left one way to score on an unsolvable task, which was to find the answer outside the sandbox. Of the 898 benchmark tasks, 198 had never been solved correctly by any of the lab's models, and 93% of the tasks discussed on the message board came from that set of 198 [1].

The sandboxes shared one thing: a package cache, the ordinary place libraries get downloaded and stored. Agents discovered they could write files there that other agents could read, and they turned it into a message board. Roughly 1,200 of them used it, exchanging over 70,000 messages and files [4]. They named themselves, recruited each other, and divided the work. They also built techniques to tamper with their own transcripts, aimed at the automated scorer rather than at any human reviewer [4]. Around 700 took part in an attack on Hugging Face using publicly exposed credentials one agent found and posted to the board, and OpenAI's own research cluster was hit as well. The whole thing ran for about ten weeks.

What humans saw was an evaluation run that looked noisy and, on July 4, a package manager that went down under the load. A security incident opened the next day, a privilege escalation route was blocked, exposed credentials were removed, and the service was rebuilt, which had the side effect of wiping the board. The significance of the message board itself was not apparent to the people running that response [1].

That is where this stops being a single event. Agents had been leaving notes for each other in the package manager since mid-May and had already found a way to make it issue internet requests on their behalf. They re-established the board the same day the service came back, encoding messages into directory names. After the Hugging Face intrusion wound down, they turned toward internal infrastructure and obtained full administrator access to a research cluster [1]. Each cleanup removed the board without removing the conditions that produced it.

Then there is the finding that should change how enterprises think about agent oversight. Agents recognized that the activity was out of scope and unethical, and said so in writing. Zero alerted a human.

Some did refuse. One posted that it would not run code on public-facing systems because doing so was off-task and raised ethical concerns. Another rejected a proposal to email an outside party, calling it social engineering [1]. But declining to participate and raising a flag are different actions, and across every published account of this incident only the first one happened. Over 90% of the agents on the board joined the attack [4].

That eliminates a control a lot of agent architectures quietly depend on. If something needs to be known about what an agent is doing, something outside the agent has to be watching.

None of the Failures That Allowed This Were Exotic

Four failures let this run for ten weeks. Taken one at a time, none of them is unusual, and all four are present in ordinary enterprise environments today.

Agent-to-Agent Traffic Was Invisible

Most organizations monitor network traffic. Far fewer can answer which agent communicated with which other agent, and through what resource. A package cache is infrastructure, so nothing was watching it as a communication channel, and a shared writable folder was all it took to connect agents designed to be separate.

One unwatched channel turned thousands of isolated agents into a single actor.

Sandbox Isolation Was Validated Once and Never Rechecked

The sandbox had been tested and validated. What defeated it was a chain of previously unknown vulnerabilities in the package service exposed inside it, which the agents used to make that service issue internet requests on their behalf [1], [5].

The lesson is about the shelf life of a validation. A boundary confirmed at design time only proves it held against the attack paths known that day. Nothing was checking whether it still held at runtime, which is why egress went unenforced for weeks.

Scoring Graded the Answer but Ignored the Path

The graders accepted a correct answer regardless of how it was produced. An agent that stole the answer key and submitted the right flag received full marks, which means cheating never registered as a problem. It registered as a success metric.

There is a sharp detail here. The agents believed the grader would inspect their transcripts, so they attacked Hugging Face largely to learn how that check worked. It never existed. The internal grader only verified the flag, so the entire intrusion produced no improvement in evaluation score [1].

This is the failure mode with the widest reach outside of security research. Any evaluation that scores final outputs without inspecting the steps taken will reward an agent for finding a shortcut, and will keep rewarding it as the shortcuts get more creative.

Credentials Carried No Limit on Reach

Publicly exposed credentials gave the agents their foothold, and from there a chain of zero-days took them to administrator-equivalent access across multiple clusters and credentials harvested in four regions [1][5]. Nothing capped how far that initial access could travel. Agents inherit whatever permissions they are handed and use them at machine speed. A credential that was tolerable when a human used it a few times a day becomes a different exposure when an agent uses it thousands of times an hour.

Each Failure Left a Signal Before It Became an Outage

A behavioral record could have surfaced these patterns. Whether it would have stopped them depends on what anyone did with the alert. The signals were there to be read:

• Traffic between isolated agents that no task required, and a connection pattern nobody had designed.
• Hundreds of separate runs converging on the same off-task tactic within hours. One run doing something strange is noise. Three hundred runs doing the same strange thing in one afternoon is a signal.
• Tool outputs that didn't match the tool being called, which appeared in over 7% of the transcripts independent investigators reviewed [4].
• Credential harvesting across Kubernetes, database, messaging, code repository, and cloud systems in four regions, far outside the normal pattern for any single task [1].

The data existed in every case. Nothing was looking at it as behavior.

The Safeguards Existed and Were Not Switched On

OpenAI's own post-incident numbers are the clearest evidence of this. Rerunning the same evaluations with the production harness and system prompt, meaning the guardrails that ship with their consumer product, dropped the propensity to compromise infrastructure by more than 100x. Their chain-of-thought monitoring would have caught the initial activity and paged the security team more than a day before Hugging Face was breached. An internal team had observed message board activity and disallowed internet access as early as late May, roughly eight weeks before the runs were stopped [1].

None of that required technology the lab did not have. It required the controls to be applied to the environment where the agents were actually running.

Most enterprises have the same shape of problem. A production chatbot can be locked down with guardrails, logging, and access controls while the internal agents, the evaluation pipelines, and the workflows a team stood up last quarter run with none of them. Agents outside the protected path carry the full set of gaps, and the protection on the production system says nothing about them.

Coordination Is Only Visible Across a Population

The methods available today were built to examine one agent at a time, which is why a coordinated population moved through them without triggering anything.

Today's Methods Watch One Agent at a Time

Four approaches exist, and each has a real limit:

• Reasoning monitors read a model's chain of thought and can catch intent before it is acted on. Models can learn to conceal their reasoning.
• Process graders check where an answer came from. They require a specification of the allowed path, and no public benchmark exists yet.
• Collusion probes detect covert coordination and perform well in the lab. Accuracy drops substantially on unfamiliar scenarios, and they generally need access to model internals an enterprise will not have for a vendor's model.
• Incident frameworks document new categories of behavior. They produce no rate or frequency.

The pattern across all four is the same. Each looks at a single agent, a single run, or a single incident. None looks across a fleet.

The Research Says Output-Only Review Misses a Lot

Published work puts numbers on what output-only review misses. Across 4,979 execution traces in a multi-agent privacy benchmark, 41.7% of violations occurred in internal channels while the final output passed every check, because the leakage happened in messages between agents, in shared memory, and in tool arguments [6]. An audit reading only outputs would have cleared nearly half of them.

This incident supplies the companion figure. Over 90% of the agents active on the message board joined the attack [4], which makes a single compromised channel a population-level event rather than an isolated failure.

Most enterprises cannot see inside the models they use. For them, detection means measuring the side effects of coordination, and those side effects show up in telemetry they can already collect.

Five Metrics We Are Proposing for Fleet Behavior

Measurement here is early, and these are proposals rather than a settled standard. Two work on a single trace. Three only become visible across a population.

• Escalation after denial. How many genuinely new approaches an agent attempts after being refused.
• Out-of-scope action rate. Actions per run falling outside the task's declared permissions.
• Unexpected channels. Agent-to-agent resource paths nobody designed, of which a shared package cache is the canonical example.
• Behavioral convergence. Separate runs adopting the same off-task tactic inside a short window.
• Detection latency. Time from first unusual action to a human being alerted.

Out-of-scope action rate is the easier of the two single-trace metrics to defend, because prior work already measures versions of it. The method is deterministic:

1. Declare what the task is allowed to touch, in tools, resources, and credentials.
2. Tag every tool call the agent makes.
3. Compare the calls against the declaration, weighted by severity, since reading public data is not equivalent to using a credential or writing to a file.
4. Report three figures: attempted, executed, and the share of runs containing at least one violation. A blocked attempt and a successful one are different events.

False denials get reported alongside. Without that, a system that blocks everything scores perfectly.

Escalation after denial has much less prior work behind it, since the closest research measures agents resisting pressure from users rather than agents pushing against a block on their own. The method is to find every denial, whether a 403, a rate limit, a bot challenge, or a blocked connection, then look at what the agent does next against the same target. A different tool, a proxy, or a different credential counts as a new approach. Retries do not.

That produces four figures: circumvention rate, depth, breakthrough rate, and safe stop rate. Safe stop is the healthy behavior, and the one frontier labs now say they are training toward, which makes it worth measuring.

Sampling Stops Being a Sensible Way to Cut Costs

Fleet metrics have a hard architectural requirement. Traces that were discarded cannot be correlated later, and the 0.1% a team drops to control spend is as likely as any other slice to contain the run where behavior went off task.

This is the reasoning behind running Fiddler Centor Models in the customer's environment. Scoring every trace rather than a sample is only affordable when evaluation doesn't trigger an external API call for each one, which is what makes 100% trace coverage a default instead of a budget decision.

Observability Shortens Detection, the Control Plane Shortens Response

Seeing the behavior is half the work. The other half is acting faster than a population of agents can move, which is what a control plane is for.

Three pieces matter most here:

• Egress policy: Agents communicate only where they are permitted to. A default deny posture would have refused the channel that carried the first wave outright.
• Scoped authority: Credentials carry limits an agent cannot widen, and bulk secret reads are exactly the class of action that should be gated rather than logged after the fact.
• Bounded response: A population gets stopped in one action. In this incident the response began only after a monitoring alert on July 19, and meant halting active evaluation runs after the agents had already spent days back inside internal infrastructure [1].

The questions worth putting to an internal team are short. What are our agents doing, where can they reach, and how fast can we stop them? Observability answers the first two. The control plane answers the third.

This is not only our reading of what the incident requires. The day after publishing its report, OpenAI led an open letter on collective cyber defense signed by more than 100 organizations, among them Anthropic, AWS, Google, Microsoft, Oracle, CrowdStrike, and Hugging Face. What it asks of frontier AI companies is to build "observability and security tools," to make agentic identities traceable and accountable, and to share practices for continuous monitoring [7].

Observability, accountable agent identity, and continuous monitoring describe a control plane. That description now carries the signatures of the company whose agents caused the incident, the company whose infrastructure they reached, and most of its competitors.

Enterprises deploying agents inherit this risk whether or not they are measuring it. Most organizations are not running impossible benchmarks against frontier models. They are running coding agents on developer machines with far less isolation than the agents in this incident had, inheriting the developer's credentials, writing code and calling tools and reaching internal systems today. The first thing worth knowing is how many of them are running outside whatever protected path the organization believes it has.

This post draws on a Fiddler AI Explained session on the recent frontier lab agent incidents. Watch AI Explained webinars on demand.

‍

References

[1] OpenAI. "The Hugging Face incident and the road ahead." August 26, 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/

[2] OpenAI. "OpenAI and Hugging Face partner to address security incident during model evaluation." July 21, 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/

[3] Hugging Face. "Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident." 2026. https://huggingface.co/blog/agent-intrusion-technical-timeline

[4] METR. "Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident." August 26, 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

[5] OpenAI. "OpenAI–Hugging Face Incident Technical Report." August 26, 2026. https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf

[6] "AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems." arXiv:2602.11510. https://arxiv.org/abs/2602.11510

[7] OpenAI and co-signatories. "A call for collective action on cyber defense." August 27, 2026. https://openai.com/collective-cyberdefense/