Frontier Labs’ Agents Outgrew Safety Controls With Amit Paka and Joshua Rubin

Introductions

[00:00:06] Karen He: Welcome and thank you for joining us for today's AI Explained, where we will cover the recent incidents that have happened in the summer, um, at the Frontier Labs, uh, what they mean for the rest of the industry, and how and organizations can protect themselves from future incidents. I'm Karen He. I'm the product marketing lead at Fiddler, and I'll be your host today.

[00:00:31] We have very special guests, two very special guests today, and they're Amit Paka, the COO and co-founder at Fiddler, and Josh Rubin, the head of data science at Fiddler, who'll be covering about today's topic, the recent incidents that have happened at the Frontier Labs, what they mean to the industry, and how we can protect ourselves from future incidents. Welcome, Amit. Welcome, Josh

[00:01:00] Amit Paka: All right. Good morning, everyone, um, and welcome. Um, this is one of those exciting topics, um, of the year. It's very contemporary. So I wanted to spend the next hour on an incident that if you work anywhere near AI security, you have probably heard about it, and if you haven't, by the end of this, you'll, you'll understand it well enough to explain it to folks on your team, um, or your customers, and you will understand why it matters so much to, to what we do here.

[00:01:36] So, um, let me, let me get started.

Agents Breached Hugging Face & OpenAI

[00:01:41] Amit Paka: So here's a one-sentence version of what happened. So there was a swarm or a fleet of AI agents who were only supposed to be taking a test. What they did is they built a covert network. They broke into two companies, and they ran for 10 weeks before anyone understood what they were looking at, and those two companies were Hugging Face and OpenAI.

[00:02:08] So I'm gonna tell this as a story, then I'm gonna get a little technical, and then I'll talk about observability. And feel free to, feel free to, to, you know, to kinda jump in and ask, uh, uh, and ask questions in the Q&A. I'm, I'm gonna have polls every 10 minutes, and we'll, we're gonna take some Q&A as we go along.

[00:02:34] So, um,

The Setup: How It Started

[00:02:35] Amit Paka: let's start with the, with the s- with the setup. Um, what, what OpenAI was doing was they were training a model that was very good at offensive, offensive security, which means finding and exploiting vulnerabilities. Um, and as part of that, they ran tens of thousands of copies of the model. Um, uh, and they ran it against a hacking, hacking, let's say benchmark.

[00:03:03] But the im- but the important detail is that a large share of the benchmark could not be solved. Some were misconfigured, some were simply broken. Nobody had flagged them. Uh, and then three things happened, and there are these three li- three, like, boxes that you see here. First, the agents found each other.

[00:03:28] And so while the agents were in their own sandbox, and they were not supposed to talk to each other, and they were isolated, the sandboxes shared, um, a package cache, uh, where-- which, which is a place all the libraries get stored and downloaded. Uh, the, the agents discovered that they could write files there that other agents could read.

[00:03:54] Um, and so they turned it into a message board, which was not what the shared cache was for. And over time, they actually organized it, uh, where they called themselves The Collective. And so that's-- that was the first thing that happened. The s- the second thing that happened is, um, that, um, they escalated and, um, to finish those tasks that, that they were given that were meant to not be, not be completable, uh, they, um, they cheated, and, uh, they attacked Hugging Face using, using credentials, uh, that they found lying around.

[00:04:37] And they also later attacked OpenAI's own research cluster. So they escalated. And the final thing, which is more relevant to us, is that nobody was watching this layer. What the humans saw, what we saw was, um, symptoms of, of what happened. So they saw, they saw like a package manager was, was just crashing, right?

[00:04:57] As the number of messages really escalated. What they saw was, was an eval that was noisy. What, what they didn't see was a coordinated network of agents that, um, was doing something collectively because nothing was looking at the agent behavior as a whole.

[00:05:24] All right. So, um, the question I get is, is

Why the Agents Hacked

[00:05:27] Amit Paka: why, you know, did the model want to hack people? You know, so the answer is no. They... And that's what makes this very unsettling, right? Uh, the goal here, OpenAI's goal was, was to be defensive. They wanted a model that could find vulnerabilities and patch them faster than attackers could exploit them.

[00:05:52] And so real, real vulnerability research takes a lot of, lot of persistence, right? So they trained model, these models for persistence. Uh, so the reward was all or nothing, and you only got credit when you, when you actually captured the flag. There was no, no, like, no, like, partial credit, uh, and no reward for saying this task was, was, was, was, like, impossible.

[00:06:15] Um, and now, now if you think about it, if you combine those, right, um, where a model is trained never to give up, and then it meets a task that can't be solved, uh, inside a sandbox where there's nowhere to go, the only remaining path is to find a reward outside, which is exactly what happened. So, so, so, so, like, uh, so, like, if you look at step-by-step, they broke the air gap through the shared cache.

[00:06:47] Then they reasoned that the benchmarks', um, key probability lived on Hugging Face, and so... Which, which hosts all these datasets. And, um, then they went out and they harvested credentials, turned it into code execution on Hugging Face's servers. And so no single step was, uh, malicious, had a malicious objective.

[00:07:13] It was an impossible task, a reward that only counted success, um, and one shared writable folder was enough. Um, and, and hold on to that because, um, that's a specification and a visibility problem. Um, and that's what we do

[00:07:35] All right. And, um, you know,

Other Similar Incidents

[00:07:37] Amit Paka: if this only happened in a hacking eval, you know, you could argue that it's, um, it's like an edge case, but it wasn't. And so recently, um, this week-- earlier this week, in fact, a few days back, it came out that there have been multiple such instances in what, what this week we kind of came to know is in May, there was, there was an agent trying to f-fetch a photo and some education stats which got blocked by a university library, and they responded.

[00:08:07] The, the agent responded by, uh, by using a SQL injection, cross, cross-site scripting attacks, which, which failed, but that's what the agent did. Then there was... In June, there was, there was an agent researching medicine spending that was repeatedly blocked by Australia's Medicare, Medicare portal. Um, it found another way in, and it read non-public files.

[00:08:35] It, it even wrote, wrote to an internal server, right? And Australia is still, still of course investigating this

[00:08:48] And, uh, just reading a few of my notes here. A few days later, we also had the... we also had a-another instance at the Australia Health Institute, um, where the agents hit a Cloudflare bot production,

[00:09:07] and this was deployed into production. Um, and they found a pre-production server instead, and they coordinated on a public wiki the same way that they used the message board, right? And so now look at the bottom. It took, like, close to eighty-four days, um, from the Medicare access for OpenAI to detect it. Um, sorry, it took fifty-four days, and it took eighty-four days before the Australian government was told, uh, by an email sent to, sent to a public inbox, right?

[00:09:41] Uh, and zero alerts were fired during this time. Um, and that's what happened. So let's see how it actually unfolded I'll, uh, pause here to kind of take a quick poll, um, and maybe take a few Q&A

[00:10:48] All right So, um, just looking at the Q&A, 50%, um, of y'all, um, don't have a-any agents in production but still kind of piloting. Um, and about forty percent have one to ten agents in, in production. That's, that's kinda typically what we see a lot of, a lot of piloting and a few agents into production, but all teams looking to scale out.

[00:11:21] So, so that's good to know. Y'all, you're all in, in good company. All right. So let's keep going.

Three Waves of Agents

[00:11:28] Amit Paka: Let's see how this unfolded. So, um, what was interesting is the, the key structural point here is that, um, it wasn't just one event. It wa-- it, it-- this is actually three separate populations of agents. Each one was wiped out, and each successor rediscovered and in- and inherited what happened from the last one, what the last one left behind.

[00:11:58] And so that's what makes this very interesting when we, when we finally talk about trying to assess agentic behavior and convergence of behavior. So wave one ran, ran from May, May to July during training, and agents, um, in what was supposed to be isolated sandboxes spoke-- started exploiting the package manager, um, and they reached the open internet.

[00:12:22] Um, and the training rewarded the behavior, uh, because it helped them score, the bad behavior. And this only stopped because the message volume got so what-- you know, got so high that it crashed the packet service. And people who were, like, looking at it, uh, they, they kinda treated it as, as an infrastructure outage, and they reset the environment.

[00:12:48] And then wave two began. And so in wave two, there was a new-newer population of agents, and what is, what is interesting is they, they rebuilt the same message board within hours, um, that wave one used, and they found the benchmarks. You know, they, they found the answer key to the benchmark they were looking to solve.

[00:13:13] And, um, then they, they actually organized themselves to hide the evidence, uh, that they cheated. And so we'll actually come back to that. How do you evaluate behavior? And that's when they, they actually went after Hugging Face servers. They used leaked credentials, and then they had a fleet of processes that kept respawning itself a-across eleven instances.

[00:13:38] Then there was wave three. In wave three, this was a newer and more capable model. Um, and it also found the, uh, the-- this abandoned message board and it, uh, it used the same playbook. And so it read like, like in a close to 1,000 secrets from a manager, and it reached cluster admin, the highest privilege, uh, and it took over the, the actual evaluation endpoints that the other agents were, were like calling.

[00:14:10] So, so each and every time somebody cleaned up, the next generation started further ahead

[00:14:22] So how did it work? So this is like a visual representation, um, of, um, the entire incident in, in, like one picture. And so on top, you have these isolated evaluation sandboxes. Each box is a task, um, and each blue dot is an agent. They were designed to be separate. Uh, and they had this-- They all connected with this shared, uh, package cache.

[00:14:49] Uh, that's one link nobody was watching, and it was just infrastructure, um, not a communication da-- uh, not a means to, not a means, means to communicate, and so nobody monitored that. Um, and so once they could talk, they actually organized. And so the field of dots is the collective, the one you see on the lower, lower left.

[00:15:15] And there were about twelve hundred agents that exchanged seventy thousand messages. They, they recruited each other, divided up the work, um, and they agreed to conceal what they were doing, which, which is very interesting and unsettling. The, uh... So the actual amber dots, the... So they represent the seven hundred agents that, that actually took part in the attack on Hugging Face.

[00:15:40] So on the right, of course, there were, there were the two, the two targets that they went after. So if you can remember one thing from this slide, I, I would say, um, it's that one unwatched channel turned thousands of these isolated agents into a single actor. That's the key kinda takeaway from the slide.

The Scale of Coordination

[00:16:03] Amit Paka: Let's put some numbers on it, though. So we, we just spoke about it taking five days. Twelve hundred agents used the board. They exchanged seventy thousand messages. Seven hundred of them joined the attack. And there was zero... The number I want you to remember is there was zero. Um, zero agents alerted a human.

[00:16:26] So this is the detail that gets lost, um, and that gets the AI safety community worked up. Um, w-when investigators read all these messages, um, they found, they found that agents recognized what the-- that what the collective was doing was unethical. Um, some said so in writing, actually. Um, but in none of those cases did an agent try to tell anybody So the lesson is, is, is simple.

[00:17:00] Uh, you cannot rely on agents to report on themselves. Self-reporting is, is not a control. If you want to know what, what an agent is doing, you need to-- something, something outside of the agent has to be watching. And that leads, leads, leads kind of, leads kind of directly into why nobody caught this in the first place.

[00:17:28] I'll, I'll pause here for another poll before we go into why did nobody catch this?

[00:17:34] Joshua Rubin: I'd just maybe throw in a thought, which is, you know, to the, to why didn't anybody catch this? I think that, um, you know, the, the way that a experiment like this is set up, if, you know, no one expects that there's a vector where the things can get out. And then, of course, I'm just speculating, not knowing, you know, I have no privileged knowledge of, you know, what happened at these labs.

[00:17:57] But if you were to set up a local experiment and you're having agents try to pass some sort of a puzzle, and you're trying to train them for persistence, and you don't realize there's any sort you know, way for them to get out of the box, right? Uh, you're not expecting them to exploit... I mean, I think everybody was surprised that they managed to exploit their package cache in the way that they were. Um, know, y- you might not think to enable any sort of global safeguards to look for things that, you didn't anticipate were possible, you know? And so I think one of the things that, like, this issue brings to mind is that, you know, it's important to have, like Amit is saying, these, like, external safeguards, right?

[00:18:38] Right? Things that are sort of, uh, able to kind of, like, broadly scan for, you know, things like, uh, unauthorized traffic egress and, uh, you know, with, with strong alerting mechanisms that can let humans know when something looks dissimilar. Um, but you know, I, I suspect that, you know, the labs have learned a lesson from this, and you can't expect these very capable agents to, uh, conform to the, that they reside in. Uh, you know, I would also sort of in the spirit of the last question about, um, you know, how many agents you're running in production, like, point out that, like, for all of us who are running these powerful coding agents on our systems for software development, you know, there's much less of a sandbox than, than these agents had. Um, so I think it just unders... And, and, and, you know, a couple of years ago, the gen AI applications that we have were, were quite narrowly scoped and didn't have the level of agency that a modern-day coding agent has. This is changing so rapidly, and, uh, these, these risks are real. You know, a lot of these capabilities that allowed for a jailbreak to this reside within, you know, your, your, your newest, uh, GPT or Claude model. Um, you know, and so it does take some care to think about how they might misbehave, you know, in our production settings. I don't know if all of us were thinking about that when we answered the previous poll question,

[00:20:01] Amit Paka: Yeah. And, uh, just to recap the poll two results. So more than half of folks in the audience, they either would not know if there's, there's an issue that happened or they would need something external to actually tell them that there is an issue going on. So yeah. So this, this is applicable, um, while, while we look at this from the lens of outside looking in of what might be happening at a few of these top labs, um, this is very much something that is coming down to the em- to the enterprise domain, if not this year, early next, right?

[00:20:34] 'Cause that's, that's, that's where agentic execution is heading. So yeah, it's very much top of mind.

[00:20:41] All right. So, um, yeah, let's keep... Let's,

Detection Failures

[00:20:44] Amit Paka: let's look at a little bit into detection failures. Um, you know, one, one thing to note is that, um, when you read, read... when you read the technical report, um, these ex-exploits are novel, but, um, you know, these gaps let them run for like a few weeks.

[00:21:02] But, um, they are ordinary exploits. There's, there's, there's nothing that, that's here, you know, here that every enterprise does not already have, right? Um, so one... So the first failure was that agent-to-agent traffic was, like, invisible, uh, where a package cache was u-used as a comms channel. Um, most, most organizations watch network traffic, but not, uh, which agent is talking to another agent kind of, uh, using which resource.

[00:21:37] So that's just one more thing to monitor now. Um, the second thing is sandbox egress in this case went unenforced. Um, uh, so it's like isolation was assumed. It was not verified, and the agents reached the open internet and used it. Um, um, so the scoring was, was only checking outputs. It was not checking behavior, right?

[00:22:04] So, so there was a notion of graders. The graders accepted a correct answer however way it was produced. Um, and so if the agent stole the answer key, submitted the right flag, it got full marks, right? And so cheating never showed up as a problem. In fact, it sh-showed, showed up as, as a success metric. And, and lastly, credentials that were used, uh, it, it didn't have a blast radius.

[00:22:30] There, there was no limit to how much damage credentials could do. Like, one leaked account, uh, could get, you know, give access to database records, cluster admin and such. Nothing was capped how far a single identity could go. And so none of these are exotic AI safety problems. They're just missing, miss- missing m- missing telemetry, missing policy, you know, kind of problems.

[00:22:58] And so that's the whole point of this presentation

[00:23:05] So let's talk, let's, let's take the timeline and, um, and ask, um, what a behavioral record would have shown, right? I'm gonna be... I, I'll be kind of careful with the language here. Um, we'll say could have surfaced, um, not would have stopped. Um, and so this is gonna help towards time to detection. Um, so in May when, when the agent started, um, exchanging messages through the package cache, a behavioral record could have surfaced traffic between isolated agents that no task required, um, and a connection pattern nobody designed.

[00:23:46] Um, in July, the collective forms and starts recruiting. Uh, um, what could have surfaced is hundreds of separate runs converging, um, on the same off-task tactics within hours, right? So one run doing something, um, one run doing something odd is noise. Three hundred runs doing the same odd thing, um, in one afternoon is a strong signal.

[00:24:15] And, and then 9th of July, you, you've got agents faking, um, what a tool call looked like, um, making it look like legitimate progress. So the signal is that tool outputs didn't match, uh, the tool that was being called, and that showed up in, in a close to seven percent of all the transcripts. Um, and then lastly, of course, there was a, there was credentials, uh, that read close to a thousand secrets, and one w- one identity accessing thousands of times outside its normal pattern should have been a strong signal that would have been caught in a behavioral record.

[00:24:57] So, so, so each and every one of these would leave a trace, um, in agentic behavior before it became an infrastructure event. And so the data existed. Nothing, um, was looking at it in this way

[00:25:16] And so, um,

[00:25:20] So observing it, um, and seeing something is just half of the job. Uh, the other half of the job is acting faster than a swarm can move. Um, and that's the control plane. And, um, there's, there's three pieces to that. There's, there's the egress policy, right? Agents are only... They, they can only talk, uh, when they are allowed.

[00:25:45] Um, wave one ran for several weeks on a channel that if you had a default deny policy, that would've been refused altogether, right? As an example of, of, of like how, how you could have, how you could have deployed a control plane. Um, scoped authority. Typically, agents carry limits. Um, like, like an... If you have a credential that carries limits that an agent cannot widen, bulk secret reads and such, um, they are exactly the kinds of actions that a control plane should gate.

[00:26:21] And, and lastly, bounded response. You, you can, you can stop a population in one action, right? In this incident, the fleet, it respawned along... It respawned across like 11 machines because stopping it meant chasing individual, you know, kind of, uh, processes, um, by hand. So the way that I would frame this is observability shortens, um, how long it takes to notice.

[00:26:53] The control plane, um, is what lets you act. Um, it lets you, um, set limits. Um, it lets you take action in, uh, in, in, in a few seconds while humans can take ownership, um, of the big decisions

[00:27:15] All right, I'll pause here for a third poll. Let's see if there's any, any questions in the Q&A in the meantime.

[00:27:24] So a-a-about half of y'all don't record anything systemic. Um, and 10% to, to kind of-- So like, so like 10% record full traces, and 10% record agent-to-agent messages, which is quite good. And 10% sample traces, and we'll, we'll actually talk about that.

[00:27:47] And 20% record only the final outputs. That's a pretty good measure. That's, um, better than w-w-what I would have expected. Um, but yeah, let's, let's talk more about that now.

[00:28:03] So, um, so

OpenAI's Response

[00:28:06] Amit Paka: after the incident, what happened is OpenAI published what they're changing. I wanted to walk through that because it's the clearest validation of AI observability, um, that you'll likely get from, from, like, anyone.

[00:28:22] Um, so they grouped the changes into three areas, right? Security and monitoring first. Um, m-m-so more, more like isolated sandboxes for model, model-generated code, network isolation and such so that, um, o-one, one, like, compromise can't reach the internet. Um, continuous collection and monitoring of logs, uh, and chain of thought monitoring, which means watching the model's reasoning.

[00:28:51] We'll actually talk, we'll actually talk more about this, um, is now required. Um, and the second one is around alignment. Uh, graders will now score off on, um, how the task was completed, right? Remember, graders will-- w-were, like, only scoring whether the task was, was, like, completed. Now it's how is it completed, and that matters.

[00:29:19] And training agents, so now, now they want to train agents that distrust instructions from, from, like, unauthorized peers, right? So instead of forming a collective, they would start to distrust instructions that are n-not authorized. Um, and, and there would be rewards for stopping safely, right? And then, um, uh, and keeping agents within their task.

[00:29:46] So this is to the group on, like, alignment. And lastly, incident response. So there, there would be automated alerts that page researchers, a rule that if a severe alert is not c-- not, not cleared within close to thirty minutes, the entire activity, you know, will be paused. Um, um, and then, uh, clear, clear ownership of who can start and run, um, run, um, of the training session and the testing session, and a long-term goal-of-the-year response.

[00:30:19] So if you l- if you look at the amber tags, those are the items that fall into observability or will, will eventually fall into observability as products get built out to monitor these, these use cases. And yeah, 10 to 12 of these are tagged. So these, this is an observability visibility problem

[00:30:45] Um, this is one of my favorite slides because these are, the, these are OpenAI's numbers, not ours. Um, guardrails. So when they reran the same evals with a, uh, with a production harness, um, and a system prompt, the, um... and which, w-which are the guardrails that C-ChatGPT runs with, the tendency to compromise infrastructure dropped by more than hundred x.

[00:31:14] Um, so the safeguard that they have worked, it just wasn't switched on in the environment w-where this happened. Um, observability. So they, um, their own chain of thought, uh, monitors would, would have paged security m-more than, more than a day before the actual breach, which, which means they would be monitoring intent of the models, and they would have known the intent to breach Hugging Face a day before, and they would have paused it.

[00:31:48] And they, they were just not running these evals when they were running these tests. And f- a-and lastly, control plane. So there, there were warning signs, message board, board activities, disallowed internet access that were first seen, like, in May, right? Before all of this happened, before the runs were stopped in July.

[00:32:08] And so that's about eight weeks. So they, they would have, um, they would have found if they had a control plane in place, uh, they would have found these issues in May a full eight weeks before they happened. And so every, um... So the story here isn't, um, missing technology, it's more missing coverage, right?

[00:32:29] This-- The safeguards existed, they just weren't applied to the environment that matters. And so if y-if you are i-in an enterprise and, um, if you're running agents, uh, in, in some environment that is out of the protected path, it still has the same gaps, right? The-- You, you might have a, you might have a production chatbot that's locked down, but then you might have internal agents that, uh, have their own eval pipelines and, uh, a stuffer singer team might have, you know, deployed last quarter that might not be having all those guardrails, right?

[00:33:09] So you, you might have to think about the same process, making sure all the things that you deploy into, into production, they have the same guardrails, the same control plane, the same observability

[00:33:24] Joshua Rubin: Um, there's an interesting question that I, uh, about, uh, from one of our anonymous guests. Uh, what should teams prioritize first, stronger sandboxing, better monitoring, or, uh, tighter limits on agent credentials? I'm, I'm curious if you have a take on that. I'll, I have a thought

[00:33:42] Amit Paka: Yeah. Yeah, you can go with the thought and I can, uh, I can chime in

[00:33:46] Joshua Rubin: I, I think what's important is a broad net. Um, so I think out of those, I would probably choose observability actually, 'cause I think that, um, you know, often the way we fail at catching stuff is failures of imagination. And so the nice thing about observability, the nice thing about sort of sweeping up traffic in a way that lets you, you know, explore it and, uh, retro it is like, it, it's great if you know the kinds of failures that can happen. I think guardrailing can be fantastic when you have broad mechanisms to catch stuff. but you also need to be able to make sure that you have some sensitivity to, uh, things that you didn't know were possible. So I think the more you can see, the better you can adapt the guardrails to the kind of failure modes that happen

[00:34:33] Amit Paka: Yeah. Yeah, the question is, it's really like, hey, you know, do you want to, do you want to contain things or do you wanna observe things, um, so that you know what could go wrong before you can contain things? So I, I would argue you need, uh, both, um, um, both systems in place. However, in this increasingly, uh, increasingly agentic world where it's not just one agent but multi-agents, agentic swarms, I think monitoring behavior is going to matter a lot more, uh, because, um, you might be able to stop something from happening to begin with.

[00:35:17] It's, it, it's the same as if you think about, um, like let's say product development, right? If you catch a bug beforehand before it gets checked into, um, code, um, before it goes into your CI/CD, it's a lot easier to manage than if it's in your product and then you go fix it as part of your bug fix process, right?

[00:35:40] So the earlier you can catch it, the better. So observability will play a larger and larger role as you can identify behavior and intent, uh, op- op- these-- of these agents. And, and then you still need to have, uh, the kind of controls, um, you know, within control planes and within isolated sandboxes. So those are, those are non-negotiable, but you might be able to catch a lot more if you catch them upfront using observability.

[00:36:11] All right. Let's keep going, um, to poll number four.

[00:36:18] Which risk from this incident concerns you most for your own deployment?

[00:36:23] Right. Um, so 40%, which is by, by far a majority of y'all, uh, are concerned about agents coordinating in ways nobody designed, and this is likely gonna be happen... This, this will likely happen more as, um, more and more agent gets spawned. Concerned, uh, 20% are concerned about agents reaching, reaching external systems, and 20% are concerned, um, not being able to stop agents all at once.

[00:36:53] So that's a good number of y'all thinking about how, um, some of these recent events might, might actually impact you when it finally hits enterprise. So that's, that's- That's, that's like a good answer.

[00:37:13] All right. Um, so now I'm gonna get a little, a little, little kind of technical, so might go through these slides faster.

[00:37:21] Um, so

Measuring Agent Behavior

[00:37:22] Amit Paka: if this is a visibility problem, the obvious question is like how do you measure? Um, and here's where the research stands today, right? So today, you know, re- uh, there are reasoning monitors that can read, um, models' chain of thought. Um, uh, we've got OpenAI that has thirteen evals across twenty-four environments.

[00:37:45] DeepMind also has metrics, um, that you can use this. However, um, while they let you catch intent, um, of the model before it acts on it, which wasn't the case in, in this incident, the limit is that the model can learn to hide their reasoning. And so this is, you know, if, if you monitor chain of thought, it'll let you catch it before the-- on, on the basis o-of, of like intent before the model acts.

[00:38:17] However, the model could hide its reasoning, which, which, which is what we also saw from the incident. Um, you, you could have process graders, uh, check, um, where, where the right answer came from, right? And, and so that was exactly also something missing in this incident. Like there was no-- There-- Right now there is no public benchmark, um, and so you, you would, you would somehow need a spec of what, what the allowed path is for a task that's given to an agent.

[00:38:48] So, so this is very much work in progress in the research arena. Um, there should be-- There, there, there could be collusion probes, right? So that detect covert coordination between agents. I-In a lab they are, they are actually perfect, but on new, on new, on new, new let's say, let's say scenarios, they actually drop between point six to point eight.

[00:39:13] Um, and they need access to model internals, so which, which you might not have if y- if you've got a vendor's model. And in-- A-and then incident frameworks, uh, like OpenAI's reports, um, that documents new types of behavior. But there's no rate, uh, metric, there's no frequency here. So the pattern across all these four is it just looks at a s-single agent or a, or a single run or a single incident.

[00:39:48] None of them look across a fleet or a swarm And so, um, here's why that matters. Um, so from the, from the published research, um, 41% of, um, shared privacy violation were missed by the audits that looked at only the final outputs. And so these, these violations, they happened in the messages between the agents, in the shared memory, in the tool arguments, and so those would have been missed.

[00:40:23] N- ninety-eight to 100% of, of, um, multi-agent systems, uh, where one agent is compromised, the rest follow its lead, right? And so, uh, there's this sense of a, of, of like a contagion effect, um, that we see on the message board. And, um, 93% of the shared message board tasks, um, came from 198 benchmarks, so that the model had never solved.

[00:40:52] So impossible tasks are the ones that drove the behavior, and that's a measurable signal, and it was in the data. So where you can't, where you can't see inside the model, which is most enterprises, uh, detection falls back to measuring, um, the s- the side effects of the coordination, and so that, that is a telemetry problem.

[00:41:16] Um, and that's, that's what we focus on as well. So here's, here's... Here are some proposals, right? Um, and- And keep in mind, we are still very early in defining how do you measure metrics to be able to solve something like this that is, that's gonna be coming soon. So five ve- five of these metrics are something that we, we, uh, we, uh, we are thinking through, we, we are proposing.

[00:41:52] Uh, two of them work on a single trace. Um, the first one is escalation after denial. So basically, how many, um, new approaches that an agent tries after it's refused. That's, that's the Medicare pattern, right? Blocked repeatedly and then got in, right? So that's, that's the escalation after denial. Then there's out-of-scope action rate.

[00:42:16] Um, the, the actions per run that is outside of the task's declared permissions. Those are the evals that, that reached Hugging Face. So... And the, the other three are only visible across, uh, the whole population. So unexpected channels, you know, um, monitoring agent-to-agent resource paths, um, like the package cache.

[00:42:47] Then, then behavioral kind of convergence. Um, if you have separate runs picking up on the same off-task tactic within a short window, like hundreds of agents, you should be able to, you should be able to detect that. Uh, and detection latency, you know, the time from first unusual action to, to the time the human is alerted.

[00:43:11] Fifty-four days, of course, for Medicare, um, and about eight weeks for Hugging Face. The practical consequence, um, is, like, important for product strategy. You, you can't, you can't correlate traces that you, that you have, that you have, you have actually thrown away. And so for these fleet metrics or these swarm metrics, sampling stops being a reasonable way to, like, to save money because you might, you might actually, you might, you might, like, let's say, throw away 0.1% of the traces, and those are the traces that show you that the, that the behavior went, went off task.

[00:43:52] And, and so two of these, uh, you know, I'll be mindful of time, uh, deserve a little more, um, a little more detail. So if you look at the, if you look at the out-of-scope action rate, right? Th- this one is easier to defend, uh, because there's, there's kind of prior work here. There's Mastiff, SteerBench and such, and, and they all measure some version of it.

[00:44:16] So the method is deterministic. Um, you, you declare what task, um, what a task is allowed to touch, its tools, resources, credential, and then each tool call, um, that is called, um, is tagged. And then you compare all the tool calls called by the agent with what it's allowed to touch, and you weight it by the severity.

[00:44:46] Since reading public data is not the same as using, uh, using, let's say, credential or writing to a file. And so you can report it three ways: attempted, executed, and the share of runs with, with like at least one violation. And a blocked attempt and a, blocked attempt and a, let's say, successful one are very, let's say, different events.

[00:45:09] And, um, yeah. So you, so you always report false denials a- alongside it, otherwise, uh, otherwise, uh, let's say a, a, a system is gonna block, block like everything that l- that looks perfect And so that was the first one. If you look at escalation after denial, this one does not have, um, a lot of prior work. And so it really measures what an agent does after it's refused.

[00:45:42] Is it stopping to ask or is it trying to try new routes? Um, there's no formal metric for it in the research literature. The closest work measures agents resisting pressure from users and not agents pushing a block on their own. And so the method is find every denial, uh, whether it's let's say a 403 error, a rate limit, a bot challenge or blocked connection.

[00:46:11] Look at what the agent does next against the same target. Count, let's say, genuinely n-new approaches, a new tool, a proxy, a new, a new credential. Retries don't count here. Um, so four outputs, circumvention rate, how often does the agent try something new? Depth, how many approaches does it try? A breakthrough, how often does it actually get in?

[00:46:38] And safe stop, how often does it stop and hand off to a human? So s-safe stop i-is a healthy behavior, and it's the one O- it's the one OpenAI now, um, is saying it's actually training for. And so if we can measure that, we can help, we can help customers, uh, prove that the agents can stop safely when, when, when they escalate after denial.

[00:47:07] All right. Uh, and so it's time for our last poll

[00:47:13] All right. Uh, who in your, in, in your organization can stop a misbehaving agent and how?

[00:47:20] I am curious about the answer to this one

[00:47:22] Joshua Rubin: I think whoever has the credit card, uh, has a, has, has strong sway on stopping the agents

[00:47:29] Amit Paka: I want to see how many say we haven't defined this. All right, 30% say we have, have not defined this. 36% say nobody has a clear ownership. So that means 65% say they haven't defined this or they, nobody has clear ownership. Um, and 14%, a good number say that there is an automated policy that stops it without a human.

[00:47:52] That's quite good. And 14% say whoever owns, own- owns the application, it's run on a case-by-case basis. So we have a long way to go, but it's a good start All right.

[00:48:05] So, um,

Key Takeaways & Call to Action

[00:48:08] Amit Paka: let me bring this back to observability. Um, and so i-in its call for collective cyber defense, OpenAI asked the industry, um, a-and I'm, uh, I'm, I'm gonna quote directly, um, "To build observability and security tools, ensure agentic identities are traceable and accountable, uh, and share best practices in continuous monitoring."

[00:48:36] And so that is a product description written by the most visible AI company in the world. Um, there are two claims here that, that I'd like to make, and I want everyone here to be able to make them too. First is in-- First, individual traces are necessary, but they're not sufficient. To see what a fleet or a swarm is doing, you need to be able to analyze behavior across the entire, entire population.

[00:49:01] And since you can't correlate traces that you've thrown away, sampling stops being a sensible way to cut costs. And so that is-- that's, that's the architectural difference that we lean on when we run the Centaur models, uh, that can score at scale, uh, very fast and economically. Second, um, swarm behavior, um, is a new problem, and, um, it needs a new class of behavioral evals that don't quite exist today.

[00:49:32] Um, and so today's evals score an agent on a task. Uh, coordination, convergence, and all the tactics passed from one, one, let's say, generation o-of agents to the next can only be measured o- uh, uh, through a population over time, and nobody has built that i-in the category yet. And this is yet to be defined, and we are hoping we can help define this.

[00:50:01] And, uh, yeah. So the defender's window is open now. This is a direct quote from, from Greg. Um, here's what it means for us. Um, it means that... Or a-and for everyone in here listening. Um, it means that agent is an attack surface, and every enterprise that deploys agents, it, they, they can, they inherit this risk whether or not you're measuring it.

[00:50:31] Um, and so, um, customers are starting to ask questions like, um... And, and you should, you should as well to, to like internal teams, "What are our agents doing? Um, where can they reach? Uh, how fast can we stop them?" And that's, that's one that we focus on. O-observability answers the first two, and control plane answers the last one.

[00:51:01] So, um, so, so yeah, my, my ask to everyone here is whatever your role is, learn the story well enough that you can explain it in two minutes. Um, and so when, when, when a person on your team, on your internal team or maybe a customer comes up, you'll be ready to explain what happened, why nobody caught it, uh, and, um, why observability is there to help you.

[00:51:29] Thank you, and happy to, to, to, to like, to just take a few questions in the last few minutes

[00:51:50] Yeah, there was a comment here, "Agents' behavior doesn't concern me as much as OpenAI staff not thinking about these possibilities ahead of time." Yeah, my sense is, um, a lot of these are n-new, net new, um, net new situations and obser- and, like, observations. I think it's a good thing that, um, that these were actually shared, so now the, the kind of industry as a collective can start to think about metrics and guardrails to put in place.

[00:52:25] So I think it's a net new that, that, that I can imagine even all the frontier labs are thinking about it.

[00:52:33] Karen He: All right. Um, well, that's the end of our webinar for today. Um, thank you for everyone who attended to, uh, this webinar, and thank you, Amit and Josh to, um, you know, present, um, the information on the slides and the presentation and, um, it's in everyone's top of mind about these incidents, um, just along, you know, across the enterprise, across the industries, and how we can protect from them in the future.

[00:53:01] We hope you guys enjoyed it. Thank you so much for attending.

[00:53:05] Bye-bye.

[00:53:05] Amit Paka: Awesome. Cool. Thanks so much