Fiddler AI Control Plane for Coding Agents
The Fiddler AI Control Plane for Coding Agents provides the only inline enforcement at the agent's request and response paths, detecting and redacting PII/PHI and secrets. It also delivers fleet-wide intelligence across every developer, token, and dollar, so you can track and plan AI spend. All through the gateway they already run for added flexibility. No new infrastructure, no agent-side integration, no SDK to embed.
This demo shows PII guardrails that detect and redact sensitive user-provided data in a Claude Code session before it reaches the LLM, supporting DLP initiatives, and secret detection that redacts access tokens, API keys, and auth keys while notifying users. All sessions stream into Fiddler’s observability dashboards, which track KPIs like cost and cache-related savings and help identify outliers, heavy token users, and sessions with detected PII or secrets. It also highlights evaluator rules (e.g., identifying GitHub actions), an Explorer view for detailed trace inspection, and an easy workflow for creating and applying custom evaluation rules across every session and span.
AI Control Plane Overview
[00:00:00] Introducing Fiddler AI Control Plane, where you can observe and control every line that your coding agents write across all of your engineers in your organization. Fiddler employs both an observability layer that allows you to have audit traceability of every single session that your coding agents wrote, as well as inline guardrails that can protect your application and your sessions inline in real time.
Inline PII Redaction
[00:00:25] Let me jump into the guardrails to first show you how that operates. Here I have a Claude Code session that's hooked up with Fiddler Guardrails. In particular, I have two guardrails I want to demonstrate here. First is PII guardrails. Fiddler is able to detect inline within a hundred milliseconds all of the PII that is being created or provided into this Claude Code session.
[00:00:49] In this case, I put a simple prompt that included a name, an email, a phone number, et cetera, and Fiddler is finding that information and immediately redacting it. Now, this is occurring at the Claude Code harness layer before it actually enters the LLM, so it helps prevent those DLP initiatives your organization may have.
Secret Detection Guardrails
[00:01:10] Additionally, Fiddler provides guardrails for inline secret detection as well. And in this case, I have a script with a dozen or so various types of secrets, including access tokens, API keys, and auth keys. And we can see here that Fiddler is providing redaction of all of these secrets in line, again, before it's entering the LLM of Claude itself.
[00:01:38] So this is all occurring and notifying the user that there may be potential secrets that have to be addressed before this code can be pushed into production. All of these sessions and information are also entering the Fiddler observability layer.
Org Wide Observability Dashboard
[00:02:00] We can see here a dashboard of all the different Claude Code sessions that my engineers are doing here across the Fiddler organization.
Cost And Outlier Monitoring
[00:02:02] This includes some metrics such as high-level KPIs. For example, cost is a very important metric for anyone using coding agents today, as it's a very expensive endeavor. So we are actively monitoring costs as well as cache costs that could be considered savings in the way that Claude Code operates.
[00:02:23] Additionally, we can get more creative with the way that we support and measure costs, such as, "Hey, show me what kind of sessions and outliers are really driving up my costs. What is the most expensive? What kind of users on my team are really using the most tokens here so that I can actively monitor those specific sessions as needed?"
[00:02:43] We can also look at any sessions where PII was detected, or perhaps those secrets that were detected and redacted. So Fiddler is giving you complete observability of every detail within those traces and rolling those up into an appropriate dashboard for your team. Beyond these telemetry metrics, Fiddler also provides more creative evaluations as well.
Evaluator Rules For GitHub
[00:03:07] So, for example, I have an evaluator rule that's identifying any time we have a GitHub action involved. So in this case, this allows my team of engineers and leaders to review code here within Fiddler, where there was AI-generated code and a commit push or branch into GitHub. All of this information can also be tracked within our Explorer view.
Explorer Trace Deep Dive
[00:03:31] So if you want to find specific sessions, you can do that very easily within our Explorer here. For example, this last session that I just gave you an example of streamed into the application within seconds, and we can see here all the details around the attributes such as the input from the user, the output from the LLM, and then a number of different attributes that enhances our data and provides more insights and analytics into exactly what's happening within every coding agent session.
Build Custom Evaluations
[00:04:01] And the last thing I'll leave you with is the ease of use in creating evaluator rules and applying these to your coding sessions. So for example, for our coding applications, we wanted to detect any time GitHub was enacted or whenever a redaction was occurred, or even doing reviews on quality or misalignment within the Claude code.
[00:04:19] In this case, we are able to prompt our models within Fiddler to determine a specific rule that's important for me. So you can get creative and create your own rules and apply these to your evaluation of every single session and span within your particular Claude Code use of your organization.