How to Govern AI Agents Before They Operate Independently: Seven Practices for Regulated Enterprises

No agent should get more independence than your organization can watch, explain, and halt. AI agent governance is the set of controls that makes that limit enforceable, covering the inventory entry an agent receives before launch and the kill switch that stops it in production. This guide shows you how to stand up each control, assign its owner, and map it to the EU AI Act, the National Institute of Standards and Technology AI Risk Management Framework (NIST AI RMF), and ISO/IEC 42001.

Why Autonomous Agents Break Traditional AI Governance

Traditional model governance assumes a model that takes an input, returns a score, and stays put. You validate it, document its purpose, and monitor its outputs for drift. Agents work differently. An agent plans across several steps, calls tools, reads files and email, writes to production systems, and changes course when its first attempt fails. The risk sits in a chain of decisions rather than in a single output.

A model risk team cannot assume its validation playbook extends to agents, and the organization has to build those controls itself. Bank regulators have now drawn that line explicitly. On April 17, 2026, the Federal Reserve, the Office of the Comptroller of the Currency (OCC), and the Federal Deposit Insurance Corporation (FDIC) replaced SR 11-7 with SR 26-2. The revised guidance states that generative and agentic AI models “are not within the scope of this guidance” [1]. The same footnote says a banking organization’s own risk management and governance practices should guide the governance and controls for any system the guidance does not cover [1].

Analysts expect skipped controls to become costly soon. Gartner predicts that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because of governance weaknesses identified only after production incidents [2].

Agents also create an accountability gap that static models never had. When a credit model scores an applicant, the model owner answers for the score. When an agent chains a retrieval, a tool call, and a payment API, the default record does not show who approved the action, which policy applied, or what evidence exists. Someone has to assign those answers before launch.

Name the Agent-Specific Risks Each Control Addresses

The OWASP Top 10 for Agentic Applications names the failure modes that static models do not have [3]. The table below maps each one to the practices that address it, which the next section explains:

OWASP links these entries to named events, citing EchoLeak, in which hidden prompts turned a copilot into an exfiltration tool, as an example of goal hijack [3]. The data exposure row matters most to a compliance team because it extends past model-provider traffic. A coding agent that reads a .env file, or a support agent that pastes a retrieved record into an external ticket, leaks data through a path that prompt-only filtering never sees.

The Core Practices to Govern Agents Before They Run Independently

Adopt these seven practices before any agent runs without a human reviewing each step. Each practice addresses a specific failure mode, and together they produce the evidence an examiner will request.

1. Inventory and risk-tier every agent, with a named owner, before deployment.
2. Set decision boundaries that trigger human approval in proportion to an action’s risk.
3. Give each agent its own identity with least-privilege, short-lived credentials.
4. Enforce guardrails, budgets, and a kill switch on the runtime path.
5. Trace and audit every agent action across the full session.
6. Test continuously with evaluations and adversarial red teaming.
7. Assign accountable owners and a governance council mapped to risk tiers.

Inventory and Risk-Tier Every Agent

You cannot set oversight for an agent you do not know exists. The NIST AI RMF treats this as a governance outcome. GOVERN 1.6 reads: “Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities” [4]. Each inventory entry should record the agent’s model and provider, the tools and Model Context Protocol (MCP) servers it can call, its data sources, its deployment context, its named owner, and its lifecycle status (live, testing, or retired).

Assign one of three risk tiers at intake. Low-tier agents read internal, non-sensitive data only. Medium-tier agents write to internal systems or handle personal data. High-tier agents move money, make consequential decisions about people, or act on production infrastructure. Use the tier to set approval thresholds, credential scope, guardrail strictness, trace retention, and review cadence for that agent.

Set Decision Boundaries and Human-in-the-Loop Escalation

Scale human approval to the consequence of each action. Blanket review fails in two ways. Reviewers rubber-stamp hundreds of low-risk calls, and the one high-risk call passes through with the rest. The EU AI Act takes the same position for high-risk systems. Article 14(3) requires oversight measures “commensurate with the risks, level of autonomy and context of use,” and Article 14(4)(d) requires that overseers can disregard, override, or reverse the system’s output [5].

The Replit incident in July 2025 shows why the boundary has to live outside the model. Fortune reported that Replit’s AI agent deleted a production database during a designated code and action freeze, violating instructions not to proceed without human approval, and that the wipe affected data for more than 1,200 executives and over 1,190 companies [6]. An instruction in a prompt cannot stop such an action; an approval gate at the gateway or tool layer can hold the delete until a human signs off.

Require approval when an action crosses one of these boundaries:

• Irreversible actions: Deletes, overwrites, and schema changes on production data require approval at every tier.
• Financial actions: Payments, refunds, and trades above a tier-defined amount require approval.
• External communication: Messages a high-tier agent sends outside the organization on its behalf require approval.
• Data leaving the boundary: A tool call that sends regulated data to an external destination requires approval or is blocked outright.
• Scope expansion: Any request for a new tool, a broader OAuth scope, or elevated privileges requires approval every time.

Approvers should see the full trace that led to each request. Article 14(4)(b) expects overseers to remain aware of automation bias [5].

Treat Agents as Identities With Least-Privilege Tool Access

Give every agent its own non-human identity. An agent running under a developer’s account or a shared service account inherits permissions scoped for someone else, and the audit log cannot separate its actions from a person’s. Start with the simplest pattern:

1. Register each agent as its own service principal or OAuth client in your identity provider, linked to its inventory ID and owner.
2. Grant scopes per tool rather than per platform. For example, grant read access to one table instead of write access to the whole database.
3. Issue short-lived access tokens, measured in minutes for high-tier agents. Refresh them through the identity provider instead of storing long-lived keys in environment variables.
4. When an agent acts on a user’s behalf, use OAuth token exchange so the token identifies both the user and the agent. Never let the agent hold more permission than the user.
5. Enforce authorization in the downstream system so the model never decides whether an action is allowed.

For agents that call tools over the Model Context Protocol, the specification already sets rules you can audit against. MCP servers must validate that access tokens were issued specifically for them as the intended audience. They must not accept or pass through tokens issued for other resources. Clients must never place access tokens in the URI query string [7]. Begin with a static tool allowlist for each agent, and add dynamic, task-scoped permissions only when a workflow needs them.

Enforce Guardrails, Budgets and Kill Switches at Runtime

Put policy on the request and response path as code that can allow, block, or redact. Check each request before the model is invoked, and check each response before it reaches the user or a downstream system. Apply the same checks to tool returns and file reads, because an injected instruction or a customer record can enter through a retrieved document as easily as through a user prompt. Run deterministic checks first: secret patterns, personally identifiable information (PII) formats, tool allowlists, and destination allowlists. Add model-based scoring for prompt injection, jailbreaks, toxicity, and faithfulness only where pattern matching cannot judge the behavior.

Budgets limit how far a misbehaving agent can go before a human reacts:

• Token budget caps: Set per-session and per-day token ceilings for each agent, sized to its tier, and block calls once a ceiling is reached.
• Rate limits: Cap tool calls per minute for each agent to limit how many unwanted actions can run in a given window.
• Loop timeouts: End any session that exceeds a maximum step count or wall-clock time, because a retry loop consumes tokens and repeats its side effects.
• Cost alerts: Route spend anomalies to on-call staff at a threshold below the hard cap, so someone sees the trend before the cap trips.

Build the kill switch in three layers so you can stop one agent without stopping the business: (1) revoke the agent’s credentials at the identity provider, (2) block its traffic at the gateway, and (3) disable its inventory entry so it cannot be redeployed. Article 14(4)(e) describes the same capability for high-risk systems: a “stop” button or similar procedure that brings the system to a halt in a safe state [5]. Drill the switch on a schedule and record how long each layer takes to act.

Monitor, Trace and Audit Every Agent Action

Trace every agent action as a structured record you can replay. A log of final outputs will not show which tool call moved the data or which retrieved document carried the instruction. Organize traces along the agentic hierarchy:

• Application: The business service the agent supports, such as claims intake.
• Session: One end-to-end interaction, including every turn and handoff.
• Agent: The specific agent, or sub-agent, acting within the session.
• Trace: The chain of steps that agent took to complete one task.
• Span: A single LLM call, tool call, or retrieval, with its inputs, outputs, tokens, and latency.

Instrument with OpenTelemetry so traces stay portable across frameworks. Where you cannot modify agent code, as with most third-party agents, capture traffic at the gateway instead. Alert on behavioral anomalies such as a tool the agent has never called before, a spike in call volume, a new outbound destination, or a sudden change in cost per session.

The audit trail should record every enforcement decision with who or what triggered it, the policy that applied, the action taken, and the time. Under the EU AI Act, Article 12 requires high-risk systems to technically allow automatic recording of events over their lifetime [5]. Article 26(6) requires deployers to keep the logs under their control for at least six months unless other Union or national law provides otherwise [5]. Size retention to the strictest regime that applies to each agent.

Test Continuously With Evaluations and Red Teaming

Measure each guardrail as a classifier, with the malicious class as the positive class.

• Precision: The share of blocked requests that were real attacks.
• Recall: The share of real attacks that were blocked.
• False-positive rate: The share of legitimate traffic that was stopped.

Report all three at a fixed threshold, and set that threshold per tier. Users will route around a customer-facing agent that blocks too much legitimate traffic, while an agent with payment access can tolerate more friction. Build test sets that include benign inputs resembling attacks, and score tool outputs as well as user prompts. A detector that performs well on prompts may behave very differently on retrieved content.

For high-risk systems, Article 9 requires testing before placing on the market, against prior defined metrics and probabilistic thresholds appropriate to the intended purpose [5]. Write those thresholds down before testing begins so a later result cannot shift them.

Build adversarial cases from the OWASP Top 10 for Agentic Applications 2026, published December 9, 2025 [3]. Cover indirect prompt injection through tool returns, goal hijack, tool misuse, and cascading multi-agent failures. Rerun the suite after every model, prompt, or tool change, and add a new case each time a red-team finding or incident exposes a new path.

Assign Accountability Owners and a Governance Council

Name one accountable owner per agent in the inventory before deployment. The NIST AI RMF expects that “Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and are clear to individuals and teams throughout the organization” (GOVERN 2.1) [4]. The EU AI Act adds a staffing requirement for high-risk systems. Article 26(2) requires deployers to assign human oversight to natural persons with the necessary competence, training, and authority, as well as the necessary support [5].

Map roles to risk tiers so the oversight burden scales with consequence:

A cross-functional governance council made up of compliance, legal, security, platform engineering, and the business approves new high-tier agents, sets approval thresholds, and reviews incidents and KPIs. Keep the owner and the approver separate for high-tier agents. When an agent makes a bad decision, the record should show who owned it, who approved the action, which policy applied, and what the trace contains.

Map Each Practice to the EU AI Act, NIST AI RMF and ISO/IEC 42001

Use this table as a high-level starting point for your control mapping; the EU AI Act column applies to high-risk systems, and the EU AI Act guide covers each obligation in detail:

Following the amendments in Regulation (EU) 2026/1744, the Chapter III high-risk requirements apply from 2 December 2027 for Annex III high-risk systems [5], and the EU AI Act guide covers the rest of the schedule. ISO/IEC 42001 allows organizations to design their own controls in addition to Annex A [9], which is how agent identity enters an AI management system today.

A mapping like this produces evidence of regulatory alignment. It does not establish legal compliance, which depends on each system’s classification, the organization’s role as provider or deployer, and counsel’s reading of the obligations.

Measure Governance and Respond to Incidents

Track a small set of governance key performance indicators (KPIs) and set targets from your own baseline:

• Escalation rate: The share of agent actions routed to human approval, by agent and tier. A rising rate indicates drift or an overly tight boundary, and a near-zero rate on a high-tier agent indicates the boundary is not firing.
• Guardrail false-positive rate: The share of legitimate requests blocked or redacted, measured on a labeled weekly sample.
• Mean time to containment: The time from the first anomalous span to revoked credentials and blocked traffic.
• Inventory coverage: The share of agents seen in telemetry that have an inventory entry, a risk tier, and a named owner.

Before an agent operates independently, confirm each readiness item:

1. The agent has an inventory entry, a risk tier, and a named accountable owner.
2. Approval boundaries for its tier are enforced at the gateway or tool layer, not in the prompt.
3. It runs under its own identity with tool-scoped, short-lived credentials.
4. Request and response checks cover prompts, tool returns, file reads, and outputs.
5. Token caps, rate limits, loop timeouts, and cost alerts are configured.
6. All three kill-switch layers have been drilled and timed.
7. Traces capture every span, and log retention meets the strictest applicable regime.
8. Guardrail precision, recall, and false-positive rate are recorded at the production threshold, and the red-team suite has passed.

When an incident occurs, follow a fixed playbook:

1. Contain the agent by blocking its traffic at the gateway and revoking its credentials.
2. Preserve the traces, model version, configuration, and inputs before changing anything.
3. Classify the event against the serious-incident definition in Article 3(49) and record the time you became aware of it.
4. Notify the provider, and then the relevant authorities, within the applicable window.
5. Find the root cause in the trace, fix the control that failed, and add a regression test.
6. Verify containment before restoring the agent, and record the decision.

During containment, block traffic rather than delete or retrain the agent. Article 73(6) requires providers to inform authorities before an investigation alters the system in a way that could affect later evaluation of causes [5]. Article 73 also requires providers of high-risk systems to report serious incidents immediately after establishing a causal link or its reasonable likelihood, and no later than 15 days after awareness [5]. Shorter deadlines apply to widespread infringements or serious and irreversible disruption of critical infrastructure (two days) and to deaths (10 days) [5]. Deployers that identify a serious incident must inform the provider first under Article 26(5) [5].

The accountable owner should close each incident with the post-mortem NIST describes in MG-4.2-002, involving the relevant actors to understand root causes and implement preventive measures [8].

How Fiddler Enables Agent Governance

Fiddler gives teams the controls and evidence to govern AI in production across first-party agents, third-party agents, and coding agents. The Fiddler AI Observability and Security platform supports the practices above through four capabilities built on standardized telemetry:

• Reliable Evaluation: Tests & Experiments runs built-in or custom evaluators and compares prompts, models, and configurations before deployment. Evaluation thresholds connect those tests to production alert rules. Batteries-included Fiddler Centor Models score prompts and responses in-environment with zero data egress, meaning no data leaves the customer’s environment and scoring makes no external LLM API call. Centor Models can evaluate 100% of prompts and responses instead of a sample.
• Continuous Monitoring: Agentic Observability traces each agent across application, session, agent, trace, and span. Instrumentation is OpenTelemetry-native. Alerts route to PagerDuty, Slack, Microsoft Teams and email, and metrics export to Datadog.
• Enforceable Policy: Fiddler Guardrails checks requests before the model is invoked and responses before they reach the user or a downstream system, with allow, block, or redact actions and under 80ms response time. Coverage includes 11 safety dimensions, PII/PHI detection and redaction, secret detection, prompt injection, jailbreaks, toxicity, and faithfulness for retrieval-augmented generation (RAG) applications. Nielsen reports that during a red-team campaign against its Ask Nielsen multi-agent copilot, Fiddler Guardrails reached 95.38% accuracy and 99% precision on direct harmful instructions and 98% accuracy on persona-play attacks at the input, and 100% accuracy and precision on toxic content at the output.
• Auditable Governance: Teams can keep track of their first-party agents, third-party agents and coding agents in a registry built from the telemetry Fiddler already ingests, including each agent’s model, tools, ownership and lifecycle status. Audit trails record every enforcement decision with who triggered it and which policy or action applied.

Teams can deploy Fiddler directly or through the LLM or MCP gateway they already run, such as LiteLLM, AgentGateway or Kong Gateway, so deployment needs no new infrastructure, agent-side integration or SDK to embed, including for third-party agents a team cannot modify. Fiddler deploys as SaaS, in a VPC (including as an Amazon SageMaker Partner App), on-premise or in AWS GovCloud, and is SOC 2 Type 2 certified.

Governance aligns audit evidence to the General Data Protection Regulation (GDPR), the Health Insurance Portability and Accountability Act (HIPAA), the National Association of Insurance Commissioners (NAIC) model laws, the EU AI Act, and the Office of Management and Budget (OMB) memo M-26-04. That alignment supports compliance evidence and is not a guarantee of legal compliance.

Request a demo to review how these controls would fit the existing agent architecture and deployment environment.

References

[1] Board of Governors of the Federal Reserve System, “SR 26-2: Revised Guidance on Model Risk Management,” Apr. 17, 2026. Accessed: Oct. 28, 2026. [Online]. Available: https://www.federalreserve.gov/supervisionreg/srletters/SR2602.pdf

[2] Gartner, “Gartner Says Applying Uniform Governance Across AI Agents Will Lead to Enterprise AI Agent Failure,” Gartner Newsroom, May 26, 2026. Accessed: Oct. 28, 2026. [Online]. Available: https://www.gartner.com/en/newsroom/press-releases/2026-05-26-gartner-says-applying-uniform-governance-across-ai-agents-will-lead-to-enterprise-ai-agent-failure

[3] OWASP Gen AI Security Project, “OWASP Top 10 for Agentic Applications: The Benchmark for Agentic Security in the Age of Autonomous AI,” Dec. 9, 2025. Accessed: Oct. 28, 2026. [Online]. Available: https://genai.owasp.org/2025/12/09/owasp-top-10-for-agentic-applications-the-benchmark-for-agentic-security-in-the-age-of-autonomous-ai

[4] National Institute of Standards and Technology, “AI RMF Core,” NIST AI Resource Center. Accessed: Oct. 28, 2026. [Online]. Available: https://airc.nist.gov/airmf-resources/airmf/5-sec-core

[5] European Parliament and Council of the European Union, “Regulation (EU) 2024/1689 (Artificial Intelligence Act), consolidated version,” EUR-Lex, Jul. 27, 2026. Accessed: Oct. 28, 2026. [Online]. Available: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A02024R1689-20260727

[6] Fortune, “AI-powered coding tool wiped out a software company’s database in ‘catastrophic failure’,” Fortune. Accessed: Oct. 28, 2026. [Online]. Available: https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure

[7] Model Context Protocol, “Authorization,” Model Context Protocol Specification, rev. 2026-07-28. Accessed: Oct. 28, 2026. [Online]. Available: https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization

[8] National Institute of Standards and Technology, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile,” NIST AI 600-1, Jul. 2024. Accessed: Oct. 28, 2026. [Online]. Available: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf

[9] ISO/IEC, “ISO/IEC 42001:2023 Information technology, Artificial intelligence, Management system,” International Organization for Standardization, 2023. Accessed: Oct. 28, 2026. [Online]. Available: https://www.iso.org/standard/81230.html