We’re excited to announce the Fiddler and Domino partnership! Together, we’re helping companies accelerate the production of AI solutions and streamline their end-to-end MLOps and LLMOps observability.
Complete observability for your Domino models with Fiddler
The Domino Platform and the Fiddler AI Observability Platform allow your team to streamline ML and LLMOps workflows. Fiddler creates a continuous feedback loop from pre-production validation to post-production monitoring to ensure your ML models and large language models (LLMs) applications are optimized, high-performing, and safe.
Data scientists and AI practitioners in MLOps can explore data and train models in Domino’s Platform. Using Fiddler’s integration with Domino’s platform they can use the Fiddler AI’s Observability platform to validate the models before launching them into production. Fiddler monitors production models for model drift, data drift, performance, data integrity, and traffic behind the scenes, and alerts ML teams as soon as high-priority models’ performance dips.
Fiddler goes beyond measuring model metrics. It arms ML teams with a 360° view of their models using rich diagnostics and explainable AI. Contextual model insights connect model performance metrics to model issues and anomalies, creating a feedback loop in the MLOps workflow between production and pre-production. Fiddler helps ML teams pinpoint areas of model improvement. They can then go back to earlier stages of the MLOps workflow in Domino, to explore and gather new data for model retraining.
Data and software engineers and AI practitioners in LLMOps can evaluate the robustness, safety, and correctness of LLM applications in pre-production using Fiddler Auditor — the open-source LLM robustness library. Fiddler Auditor is available on GitHub and on Domino AI Hub and Domino users can red-team and monitor LLMs without leaving their environment.
Once LLM applications are in production, users can monitor them for correctness, safety, and privacy metrics. LLMOps teams can also perform root cause analysis using a 3D UMAP to pinpoint problematic prompts and responses to understand how they can improve their applications.
Get started on MLOps with Fiddler in three steps
Let’s walk through how you can start monitoring Domino ML models in Fiddler. Install and initiate the Fiddler client to validate and monitor ML models built on Domino’s Platform in minutes by following the steps below or as described in our documentation:
1. Upload a baseline dataset
Retrieve your pre-processed training data from Domino’s TrainingSets. Then load it into a dataframe and pass it to Fiddler:
2. Add metadata about the model
Share model metadata: Use Domino Data Lab’s ML Flow implementation to query the model registry and get the model signature which describes the inputs and outputs as a dictionary:
Now you can share the model signature with Fiddler as part of the Fiddler ModelInfo object:
3. Publish events
You can query the data sources in your Domino environment to pull the model inferences and put the new inferences into a data frame to publish to fiddler:
That’s it! Now you can jump into your Fiddler environment to start observing the model data we just published. Fiddler will be able to alert you whenever there are issues with your model.
We’re here to help. Contact our AI experts to learn how enterprises are accelerating AI solutions with streamlined end-to-end MLOps and LLMOps using Domino and Fiddler together.