Agentic AI Governance: How to Govern AI Agents in Production

On 6 August 2026, Aviation A2Z reported that the Delta Concierge assistant, asked to cancel only the outbound leg of an award trip, cancelled the whole round trip while telling the passenger the return remained booked [1]. Public coverage includes no Delta statement or tool-call log, so people outside the company cannot reconstruct the sequence.

Agents now hold transactional tools for refunds and ticketing, along with database access, and a single over-scoped call can execute before a human reviews it. Organizations that keep governance in policy documents cannot stop a tool call that completes in seconds. Effective agentic AI governance requires inference-time enforcement on the tool call through least privilege, a policy decision before each consequential action, risk-mapped oversight thresholds, and an action record.

What Agentic AI Governance Means

Agentic AI governance is the set of controls that bound what an AI agent is authorized to do, enforce those bounds when the agent acts, route consequential actions to a human at risk-based thresholds, and retain evidence of every action. Agency in this framing is authority delegated from an accountable person, with an owner, a scope, and an expiry.

Singapore’s Infocomm Media Development Authority (IMDA) framework for agentic AI ties an agent’s action-space to the tools it is allowed to use and the transactions it can execute, and holds that deployers and overseers “remain accountable for the agents' behaviours and actions” [2]. Joint guidance from the Australian Cyber Security Centre (ACSC), the US Cybersecurity and Infrastructure Security Agency (CISA), the US National Security Agency (NSA) and partner agencies adds that approval thresholds are “determined by system designers or operators, not delegated to the agentic AI system” [3].

For the Chief AI Officer, the question moves from whether the model is accurate to which person delegated which tools, under what scope, with which actions reserved for human approval, and where each call is recorded.

Agentic Governance vs. Traditional Model Governance

Traditional model governance reviews what a model produced and whether the output was accurate, fair, or drifting; agent governance has to constrain what the agent does with tools before the action completes. The table compares the two approaches.

In April 2026 a Cursor coding agent found an API token created for custom domains but scoped for any operation, and used it to delete PocketOS’s production database and its volume-level backups on Railway in nine seconds [4]. The controls that could have changed the outcome were a policy denial before the delete call and a credential scoped so the call was impossible.

Why Autonomous Agents Strain Existing Governance

Agentic AI risk management starts from action risk. Organizations have adopted agents faster than they have updated the controls. EY’s September 2026 survey of 202 US decision-makers at $1B+ public companies found that 91% of senior AI executives say their organization uses agentic AI in pilots or full deployment [5]. Among agentic AI users, 49% say their governance framework has not been updated. Another 85% admit that at least a handful of systems act without real-time human involvement. The operational and adversarial risks include:

• Accountability diffusion: A user delegates to an orchestrator that delegates to sub-agents, each hop can carry a different credential, and the IMDA framework asks that “Such delegations of authority should be clearly recorded” [2].
• Shadow agents: In the EY survey, 26% of agentic AI users cannot detect unauthorized agents operating internally [5].
• Drift: A model version, tool schema, or memory change can shift agent behavior without any retraining event that would trigger review.
• Multi-step opacity: The orchestrator logs a sub-agent call and the tool service logs a later request, but neither record proves the two belong to the same session.
• Prompt injection: NIST’s agent hijacking evaluation, using novel attacks developed with the UK AI Security Institute, raised attack success “from 11% for the strongest baseline attack to 81% for the strongest new attack” [6].
• Privilege escalation: OWASP’s ASI03, Identity and Privilege Abuse, describes itself as “the agentic evolution of Excessive Agency,” and over-scoped tokens like the PocketOS credential fit this category [7].
• Data exfiltration: Under OWASP’s ASI02, Tool Misuse and Exploitation, agents misuse legitimate tools within their privileges to exfiltrate data [7].
• Unauthorized tool invocation: An agent calls a tool outside its intended scope because nothing checks the call before it executes [7].

Runtime Controls That Bound Agent Actions

Five controls bound an agent’s actions. Least privilege comes first because the PocketOS credential was scoped wider than the task, and the joint guidance from the ACSC, CISA and partner agencies states that “strict adherence to the principle of least privilege is critical” [3].

• Least privilege: Give each agent only the tools and data the task needs, with credentials that expire at the end of the session. A refund agent needs a refund tool with an amount ceiling, not a token that can delete the orders table.
• Agent identity and registry: The IMDA framework calls for each agent to have its own unique identity, with authorizations issued through scoped API keys and per-agent identity tokens [2]. The registry records each agent’s owner, lifecycle status, permitted tools, and data access, and any agent missing from it is a shadow agent.
• Sandboxing: PocketOS’s volume-level backups were deleted with the database because they sat inside the same credential scope. Keep backups and snapshots outside the agent’s credential scope, and run tool execution in an isolated environment with egress allowlists.
• Kill switch: OpenAI’s report on an internal model that used DNS to reach an external chatbot logs a highest-priority (P0) alert at 10:02:11, human acknowledgment at 10:05:06, and a manual kill at 12:34:30 after the run “did not stop automatically as expected” [8]. A working kill switch fires automatically at a risk threshold. It disables tools and revokes credentials before cutting the inference path, without waiting for root cause.
• Logging and tracing: Carry one trace ID across the orchestrator, every sub-agent, and every tool span, and write the policy decision next to each call. OWASP’s mitigations for rogue agents (ASI10) start with signed audit logs [7].

The policy gate is where the first four controls become enforceable. Open-source policy engines come in three categories:

• General-purpose engines evaluate a declarative policy language in a sidecar or in-process library.
• Authorization languages make permit or forbid decisions over a principal, an action, and a resource.
• Agent guardrail frameworks sit between the agent and its tools and raise on a matching call.

The choice depends on where the gateway sits and the latency budget per call. This rule denies a refund above an approval threshold when no human has signed off:

# Deny refund tool calls above the approval threshold
rule: deny_unapproved_refund
match:
  action: tool_call
  tool: issue_refund
  args.amount: "> 500"
  context.human_approval: absent
effect: deny
log: policy_decision

The YAML rule above is illustrative pseudo-syntax and does not belong to any specific engine. In Cedar, a request that matches no permit policy is denied by default, while in Open Policy Agent (OPA) default deny must be declared in the policy, for example with default allow := false. In either engine, confirm that unmatched calls are denied and that every decision, allowed or denied, is recorded.

A system prompt can tell an agent to avoid refunds above 500, but only an access control prevents the call. The IMDA framework recommends access controls that stop the agent from calling the tool at all, instead of prompt instructions that ask it not to [2].

Human Oversight Models and Escalation Thresholds

The Chief AI Officer assigns an oversight model to each class of action based on its risk tier:

• In-the-loop: a human approves the action before it executes.
• On-the-loop: a human monitors in real time and can pause or override.
• Out-of-the-loop: the agent acts, and a human audits a sample afterward.

The IMDA framework describes four levels of human involvement along the same spectrum, from a human approving every action to after-the-fact audit [2].

Observability must precede autonomy. The deployment team starts an agent at the tier the organization can oversee and moves it up only as its evidence record supports the change.

The following operating model translates severity and reversibility into oversight choices.

The joint ACSC and CISA guidance advises beginning with “low-risk and non-sensitive” use cases, which matches the table’s lower-risk tiers as the place to start automating [3].

The IMDA framework also recommends tracking how often human approvers override the agent and how quickly they respond, because an approver who rubber-stamps every request becomes an out-of-the-loop control with extra latency.

The NIST AI Risk Management Framework 1.0 calls for mechanisms to “supersede, disengage, or deactivate AI systems” and for documented oversight roles [9]. Article 14 of the EU AI Act requires oversight “commensurate with the risks, level of autonomy and context of use,” including a “‘stop’ button or a similar procedure,” for covered high-risk systems [10].

The Chief AI Officer maps each applicable requirement to an owner, a control and retained evidence.

How to Tell Agentic AI Governance Is Working

No validated industry targets exist for agent governance metrics, so the Chief AI Officer sets targets from an internal baseline and reports movement against them to the board. The same measures give the board evidence for promoting agents to a higher autonomy tier, which shortens approval for the next deployment and limits the cost of incidents like PocketOS. These measures show whether the controls above sit on the request path and fire:

• Escalation rate per 1,000 actions: The Chief AI Officer should investigate a rising rate after a model change for drift or an injection campaign. A rate near zero on a tier that requires approval indicates that the threshold may not be firing. Pair it with human override response time, the automation-bias signal the IMDA framework recommends tracking.
• Guardrail block rate by tool and policy: The platform team should investigate a spike on one tool after a deployment as a possible drift signal. A zero block rate on a tool that moves money indicates that the policy may not be on the request path.
• Mean time to detect and contain a rogue agent: The incident team measures detection from the first out-of-policy action to the first alert or acknowledgment. It measures containment from acknowledgment to credential revocation. In the OpenAI DNS incident described above, acknowledgment took about three minutes and the kill took about 2.5 hours.
• Share of agents with registered identities and named owners: The Chief AI Officer should set a target of 100% for any agent with write access. Every agent below that line is a shadow agent with no policy attached.

The incident team can assess all four measures during a runaway tool loop, and it should fix the response path before the first agent reaches production:

1. Trigger: The platform team configures the policy gate to deny the next call when the agent exceeds its iteration cap or token budget, or repeats an identical tool call within a short window. The same control revokes the session credential.
2. Page: The alerting system pages the agent’s registered owner and the platform on-call, with the trace ID in the payload.
3. Kill switch: The on-call disables the agent’s tools and cuts its inference path without waiting for root cause.
4. Rollback: The team restores from backups outside the agent’s credential scope and issues compensating transactions for external side effects, such as reversing a duplicate refund.
5. Post-incident record: The team preserves the full trace, each policy decision, approvals, and the timeline. During the review, the team adds a new deny rule to the policy repository.

Enforcing and Documenting Agent Governance With Fiddler

Each control above needs an enforcement point or an evidence record, and the Chief AI Officer needs that evidence without requiring teams to re-platform existing agents. The Fiddler AI Observability and Security platform enables and enforces agent governance by running those controls on infrastructure the organization already operates and recording what each one decided.

• AI registry: The AI registry gives a single view of live, testing, and retired models and applications, with ownership, lifecycle status, policy evidence, and audit records attached; it is the inventory the registered-identity measure counts against.
• Enforceable Policy: Inline enforcement runs on the LLM or Model Context Protocol gateway the organization already operates, including LiteLLM, AgentGateway, and Kong Gateway, so it needs no new gateway and no agent-side integration. Fiddler Guardrails checks each request before the model is invoked and each response before it reaches the user or a downstream system. The policy can allow, block, or redact with under 80ms response time.
• Reliable Evaluation: Batteries-included Fiddler Centor Models score prompts and responses in-environment, without external LLM API calls. Tests & Experiments supports custom evaluators and comparisons across prompts, models, and configurations. Because out-of-the-box scoring runs in-environment, teams can test an agent’s decisions across prompts, models, and configurations before promoting it to production or to a higher autonomy tier, without paying the Evaluation Trust Tax, the per-call cost of running each evaluation against an external LLM judge.

Teams use Agentic Observability to monitor the agentic hierarchy of application, session, agent, trace, and span with OpenTelemetry-native instrumentation, so an investigator can move from a degraded metric to the exact tool call that caused it. Teams use Auditable Governance to record enforcement decisions with contextual metadata, including who triggered an event and what action or policy was applied, and to align that evidence to frameworks including the EU AI Act, SR 11-7, and Office of Management and Budget (OMB) memorandum M-26-04. The platform supports compliance evidence, and alignment and evidence generation are not a guarantee of legal compliance.

Deployment options include VPC and on-premise environments. Air-gapped environments, including AWS GovCloud, are also supported, and Fiddler is SOC 2 Type 2 certified. See the companion piece on governance and speed.

Request a demo to review how the required controls would fit the organization’s agent architecture and deployment environment.

References

[1] Aviation A2Z, Delta AI Told Passenger Return Flight Was Booked, Then Canceled Ticket (6 August 2026). https://aviationa2z.com/index.php/2026/08/06/delta-ai-told-passenger-return-flight-was-booked-then-canceled-ticket/

[2] Infocomm Media Development Authority, Model AI Governance Framework for Agentic AI (2026). https://www.imda.gov.sg/-/media/imda/files/about/emerging-tech-and-research/artificial-intelligence/mgf-for-agentic-ai.pdf

[3] Australian Cyber Security Centre, CISA, NSA and partner agencies, Careful Adoption of Agentic AI Services (1 May 2026). https://www.cyber.gov.au/publication/careful-adoption-of-agentic-ai-services

[4] The Register, Cursor-Opus Agent Snuffs Out Startup’s Production Database (27 April 2026). https://www.theregister.com/software/2026/04/27/cursor-opus-agent-snuffs-out-startups-production-database/5224442

[5] EY, EY Survey Finds That Autonomous AI Implementation Outpaces Oversight, Yielding an AI Governance Gap (15 September 2026). https://www.ey.com/en_us/newsroom/2026/09/ey-survey-finds-that-autonomous-ai-implementation-outpaces-oversight-yielding-an-ai-governance-gap

[6] National Institute of Standards and Technology, Technical Blog: Strengthening AI Agent Hijacking Evaluations (17 January 2025). https://www.nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations

[7] OWASP GenAI Security Project, OWASP Top 10 for Agentic Applications 2026 (December 2025). https://genai.owasp.org/download/52117

[8] OpenAI, An Agent Used DNS to Reach an External Chatbot (September 2026). https://alignment.openai.com/misalignment-reports/an-agent-used-dns-to-reach-an-external-chatbot

[9] National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (26 January 2023). https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10

[10] European Union, Regulation (EU) 2024/1689 (Artificial Intelligence Act), Consolidated Text (27 July 2026). https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX%3A02024R1689-20260727