How to Govern Coding Agents as Adoption Scales
As coding agent adoption scales across the organization, business and security stakeholders share common concerns around code quality, review capacity, codebase comprehension, cost, return on investment, sensitive data exposure, and governance.
The Fiddler AI Control Plane turns telemetry from coding agent sessions into operational and behavioral insights that help teams address these concerns.
What you'll learn:
- Gather telemetry from coding agent sessions using Fiddler’s Claude Code plugin.
- Explore traces, sessions, and turns to understand agent behavior.
- Track model use, cost, token consumption, and cache hit ratios, and spot cost inefficiencies and deviations from baseline.
- Evaluate instruction adherence, test tampering, code quality, prompt injection susceptibility, sensitive information access, and agent loops.
- Investigate trends by looking at contributing traces and detailed evaluation results.
Related resources:
[00:00:00] As coding agent adoption expands across the organization, business as well as security stakeholders share some common concerns that hover around factors such as code quality, review capacity, code base comprehension, cost, as well as return on investment, access to and exposure of sensitive information, and overall governance.
[00:00:23] In this video, we will look at how the Fiddler AI Control Plane helps address some of these concerns. Now, it all starts by gathering telemetry from your coding agent sessions. There are many ways to do so. We will look at Fiddler's Claude Code plugin. A plugin such as this can be easily deployed to a fleet of agents, and once deployed, it records all the agent's activities, everything from model calls to the tools it executed in your environments.
[00:00:56] These activities are sent as traces to the Fiddler platform, where they're organized into sessions and turns that you can easily peruse to understand the behavior of your agents. In this example, you can see a user sending a request to an agent to develop a simple program for a temperature converter.
[00:01:19] You can look at the agent's response, as well as the files that it edited, all the way up to the tests that it performed to verify the outcome
[00:01:31] now, these traces can be quite volumetric, and when you have them in the Fiddler AI Control Plane, you can extract some insights out of them. We typically see organizations create two types of dashboards. The first one is an operational dashboard that helps them measure things like model use, cost, tokens that are created, cash hit ratios, et cetera.
[00:01:53] A dashboard such as this can help you in several ways. For example, we've had organizations that were able to save cost by simply updating their agents and using more efficient models. This can have a significant impact if you look at your rollout across thousands of coding agents. Similarly, you can identify model abuse where users may be using powerful frontier, more costlier models for simple tasks.
[00:02:21] Overall, it helps identify deviations from baseline use that could indicate to some kind of issue in your coding agent rollout.
[00:02:30] Now, the second type of dashboard tends to be behavioral, where you run evaluations against these traces and sessions based on criteria you have defined. Now, the criteria could be around factors such as instruction adherence, test tampering, code quality. If you have a rubric defined, it could be around susceptibility to prompt injection, access to sensitive information, or to also measure whether the agent went on a loop, especially if you gave it an impossible task whereby you can incur unnecessary costs.
[00:03:02] Now, you can zoom in on any of these data points to look at the traces contributing towards a particular trend and get a detailed output of the evaluations that were performed against those traces. This is how the Fiddler AI control plane helps address the concerns that organizations face as coding agent use expands.
[00:03:28] I hope this video was useful. Thanks for watching, and I'll see you in the next one. Bye for now.