Visualize and Understand Your Agentic Applications in a Unified Dashboard
In this demo, we show you how to track performance metrics, safety scores, and compliance indicators across your AI agents and ML models, with the ability to drill down from aggregate charts to individual trace logs.
What you'll see:
- Real-time monitoring of faithfulness scores and prompt safety.
- Custom evaluators for PII detection, answer relevancy, and topic distribution.
- Drill-down from metrics to trace-level logs for root cause analysis.
- Drift detection, accuracy tracking, and bias detection for governance use cases.
[00:00:00] Hey everyone. My name's Kevin, and I'm a Solutions Engineer here at Fiddler. Today I'm going to be walking through how you can leverage Fiddler dashboards for production level monitoring across your AI applications.
[00:00:12] In this example, we have a demo agentic chatbot that was built using LangGraph and is being instrumented with OpenTelemetry, and we can start to see some of the traffic activity around the different spans, including tool calls and LLM calls.
[00:00:27] And as I move down here, we can also see some custom configured charts around metrics that we care most about, including faithfulness scores and prompt safety across different dimensions. And if at any point I want to drill into a spike here, for example, in prompt safety, I can click into the chart and view the individual logs driving that aggregated score and go straight into the trace view from here for each individual log to see the entire agentic flow from start to finish.
[00:00:59] You can use a wide variety of evaluator rules within Fiddler to create these custom level charts for things like PII detection or answer relevancy. And you can even bring your own custom prompt to use for chart creation. Additionally, we've configured charts around topic distribution and answer conciseness across time.
[00:01:24] This is a demonstration of an agentic level use case, but you can also apply charts and dashboards to traditional machine learning, which Fiddler also provides monitoring for. And this includes charts for things like drift detection or accuracy, and even things like bias detection that can be used for compliance and governance related use cases.
[00:01:47] So this is a quick introduction into the dashboarding capability within Fiddler.