Fiddler AI Observability Platform

Fiddler is the pioneer in AI Observability, offering a full stack platform that enables Data Science, MLOps, Risk, Compliance, Analytics, and LOB teams to streamline end to end MLOps and LLMOps workflows, and create a continuous feedback loop for improved model outcomes. Actionable AI Observability provides teams with actionable model insights to improve predictive models, large language models (LLMs), and generative AI models in pre-production and in-production. 

Fiddler provides full stack AI Observability that aligns teams across the organization with real-time alerts and model monitoring, rich model diagnostics and built in explainable AI (XAI) that help you identify the underlying issues impacting model outcomes, and build a framework for responsible AI practices. AI Observability is reliant not only on metrics but also on how actionable model insights are for ML teams to retrain and fine-tune models when something eventually goes wrong.

Fortune 500 organizations use Fiddler to standardize MLOps and LLMOps practices, deliver high performance AI, reduce costs and increase ROI, and be responsible with governance. 

‍

How it Works 

Getting started with Fiddler is easy. Simply upload your baseline dataset and add your model with Fiddler. Publish production events and start analyzing your model’s performance. 

• Fiddler is model, framework, and data-agnostic. 
• Upload training and production data from any local or cloud data source of your choice, including Snowflake, AWS, Google, PostgreSQL, or Databricks 
• Publish data with real-time event streaming or schedule batches of events that fit your business needs 

Monitor performance, detect bias, and analyze models trained using Amazon SageMaker, TensorFlow, PyTorch, SAS, Python, H20.ai, and Scikit Learn

Fiddler gives you the flexibility to deploy your models wherever you need them. Run your models in a dedicated VPC cluster in Fiddler’s AWS cloud or your private cloud of choice, such as AWS, Azure, Google, and IBM. 

‍

A Full Stack AI Observability Platform

‍

Predictive Models

Monitoring

Fiddler helps you reduce costs and increase efficiencies by tracking how all your ML deployments are performing in pre-production and production. Mitigate model performance issues before they impact your business with actionable model monitoring.

• Performance Monitoring: Track your predictive model’s performance with out-of-the-box metrics, including accuracy, recall, precision, F1-score, regression, mean absolute error (MSE), and mean average precision (MAP) for binary classification, multi-class classification, regression, and ranking models 
• Unstructured Monitoring: Detect drift in natural language processing (NLP) and computer vision models to improve model performance 
• 3D UMAP Visualizer: Gain contextual insights into complex data drift by locating and identifying drift in high dimensional spaces
• Data Drift: Easily monitor data drift, and compare data  distributions between baseline and production datasets to assess how shifts in data impact model outcomes
• Prediction Drift and Impact: Use popular model drift metrics like Jensen-Shannon Divergence (JSD) and Population Stability Index (PSI) to uncover any data drift and help calculate which drifting features are impacting your model’s predictions
• Data Integrity: Uncover data integrity issues in your data pipeline causing models to underperform, such as missing data, range violations, and data type mismatches
• Feature Quality: Pinpoint key features attributing to model outcomes for further deep dive into the model and inputs/outputs
• Class Imbalance: Detect changes in low-frequency predictions due to class imbalance in each stage of your ML workflow
• Ground Truth Updates: Update ground truth labels in a delayed, asynchronous fashion
• Alerts: Configure and receive real-time model monitoring alerts to identify and troubleshoot high-priority issues caused by performance, data drift, data integrity, and traffic metrics 

Explainable AI (XAI)

ML practitioners and business teams are empowered to make better decisions and build trust into AI with Fiddler’s explainable AI (XAI). Fiddler’s XAI technology provides 360° view and context into your predictive model’s behavior and predictions, enabling you to comprehend the outputs of their models using Fiddler's practical and widely adopted XAI methods. 

• Shapley Values (SHAP) and Fiddler SHAP: Increase your model’s transparency and interpretability using SHAP values, including our award-winning Fiddler SHAP metric 
• Integrated Gradients: Comprehend how data features contribute to data skew and model predictions
• ‘What-If’ Analysis: Gain a better understanding of your model’s predictions by changing any value and studying the impact on scenario outcomes
• Global and Local Explanations: Understand how each feature you select contributes to the model’s predictions (global) and uncover the root cause of an individual issue (local)
• Surrogate Models: Improve the interpretability of your models before they go into production by using automatically generated surrogate models
• Custom Explanations: Create customized explanations specific to your use case via APIs

Analytics

Fiddler’s model analytics tool provides you with rich model diagnostics to perform root cause analysis and gain actionable insights to create a continuous feedback loop for MLOps.  

• Dashboards: Increase business alignment and confidence in decision-making by enabling teams across the organization to glean insights and connect ML metrics to business KPIs in a unified view
• Insights: Build custom reports with the insights you need to gain deep understanding of your models and their impact on business outcomes, from monitoring metrics, feature impact, correlation, and distribution to partial dependence plot (PDP) charts
• Root Cause Analysis: Drill down on problem areas to uncover the root cause of underperforming segments
• Slice and Explain: Drill down into specific segments to perform exploratory or targeted analysis, and find underperforming cohorts
• Model Validation: Evaluate your model’s performance and validate it before deploying it into production
• Fiddler Report Generator: Create and share custom reports for periodic risk and compliance reviews

Fairness

Build transparent, accountable, and ethical practices for your business with responsible AI. Fiddler enables effective model governance with continuous monitoring, while detecting and mitigating bias in datasets and predictive models.

• Algorithmic Bias Detection: Detect algorithmic bias using powerful visualizations and metrics 
• Intersectional Bias Detection: Discover potential model bias by examining intersectional fairness across multiple dimensions simultaneously (e.g. gender, race, etc.)
• Model Fairness: Obtain fairness information by comparing model outcomes and model performance for each subgroup of interest
• Dataset Fairness: Check for fairness in your dataset before training your model by catching feature dependencies and ensuring your labels are balanced across subgroups
• Fairness Metrics: Use out-of-the-box fairness metrics, such as disparate impact, demographic parity, equal opportunity, and group benefit, to help you increase transparency in your models 

‍

Generative AI Models and Applications

Fiddler Auditor for Robustness Validation

Fiddler Auditor, the open-source robustness library for red-teaming of LLMs, enables AI teams to evaluate LLMs in pre-production. 

• Robustness Testing: Evaluate the robustness of LLMs and NLP models
• Prompt Testing: Assess prompt robustness and minimize risks in prompt injection attacks
• Insights: Analyze prompt perturbations and identify weaknesses to improve performance

LLM-Based Embedding Monitoring 

Monitor drift of LLM-based embeddings in production and detect issues as soon as drift happens to minimize risks impacting users from adversarial model outcomes. 

• Actionable Alerts: Get early warnings on performance of embeddings
• Drift monitoring: Continuously detect dips in performance caused by data drift
• Dashboards and Charts: Pinpoint performance issues for deeper analysis on LLM outputs. Measure metrics, such as toxicity, costs, and safety 

‍

Enterprise-level Scale

Fiddler helps companies securely operationalize and scale model deployment. 

• ‍Scalability: Accelerate the deployment of advanced models that require large volumes of data ingestion. Gain deeper insights with scalable baseline dataset ingestion.‍
• Single Pane of Glass: Monitor, explain, analyze, and improve your models within a unified platform with a single pane of glass. Easily view all your models in production and track what’s most important to your business.‍
• Choose Your Deployment: Choose where you deploy your models to meet your company’s needs — it can be in the Fiddler cloud or you can bring your own cloud.‍
• Security: Fiddler provides SOC 2 Type 2 security and HIPAA compliance. Users across the business will receive level-specific permissions to access protected environments through role-based access control (RBAC), and SSO.