How Fiddler’s MPM Uses ONNX to Support More Diverse Model Frameworks

This post was written with the help of our Engineering Intern, Matan Broner.

One of the primary reasons Fiddler’s Model Performance Monitoring (MPM) Platform is loved and trusted by customers is because of its versatile integration that simply plugs in and starts producing insights. In this post, we discuss how Fiddler uses ONNX to support various model frameworks and versions, seamlessly. 

Fiddler’s MPM Platform provides robust machine learning model monitoring for companies of all sizes. In fact, the platform addresses the four major challenges of current machine learning operations:

1. Transparency: Many businesses struggle to maximize the benefit of ML models because they cannot explain or open up black box models.
2. Drift: ML models are unique because they continue to work regardless of the shift, or drift, in the data which can cost millions of dollars if not handled appropriately (as we saw during the recent pandemic).
3. Bias: Due to the lack of awareness, process, and tooling, some businesses unknowingly, or knowingly, use biased datasets that do not offer fair opportunities to all customers or end-users.
4. Compliance: More industries are seeing additional regulations and compliance requirements on ML models to ensure responsible use.

As we continue our customer growth, we encountered more diverse model frameworks and versions. And to complicate things further, this problem was rampant inside individual organizations. Due to the lack of a centralized discipline, different teams inside an organization were using different model frameworks and versions that were ideal for their individual use case. As a result, when it was time to scale and establish a centralized model monitoring solution, their options were limited. This is where Fiddler comes in. Fiddler’s mission is to empower every organization to build trust and reliability in AI. In order to do so, our platform provides simple integration for any organization to plug their data and models in and get immediate actionable insights.

Why ONNX?

Fiddler is a pluggable platform that works with any model framework and data source. However, AI/ML models have framework compatibility issues. For example, a sales team might use a ScikitLearn model to predict the likelihood of a deal to close, while a marketing team might use a Tensorflow model to predict customer segments. As you can see, things can get complicated fast for a platform to handle or support all packages. This is where ONNX comes in. Regardless of the framework - Tensorflow, PyTorch, or ScikitLearn, ONNX serves as a middle ground between Fiddler’s MPM platform and its customers by converting and creating a uniform model type for the Fiddler platform to ingest. As a result, Fiddler users can easily plug in their models to Fiddler’s platform and enjoy all the robust features. On the Fiddler side, it accelerates our backend to ingest the model and provide a seamless user experience in monitoring services and AI explainability functionality.

How does Fiddler work with ONNX?

We will go over an example we created using our Quickstart that builds a ScikitLearn model. Following the steps, we will load all the libraries, connect with the Fiddler client, and upload the baseline dataset. Before we register our model, we need a few extra steps, starting with creating a basic sklearn pipeline. We will also train a gradient boosting model.

from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.ensemble import GradientBoostingRegressor
import sklearn.pipeline
from sklearn.compose import ColumnTransformer

category_transformer = sklearn.pipeline.Pipeline(steps=[('onehot', OneHotEncoder(handle_unknown='ignore'))])

preprocessor = ColumnTransformer(transformers=[('cat', category_transformer, categorical_features),
                                              ('cont', StandardScaler(), continuous_features)])

model = GradientBoostingRegressor(learning_rate=0.1,
                                  n_estimators=100,
                                  max_depth=7)
model_pipeline = sklearn.pipeline.Pipeline(steps=[('preprocessor', preprocessor),
                                                  ('model', model)])

# THIS LINE IS INCORRECT IN NOTEBOOK?
# Why does the notebook train on columns that are not features?
model_pipeline.fit(df_train[feature_columns], df_train[target])

Once the model is trained, we will use the skl2onnx package to convert the data types first.

from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import (
    FloatTensorType, 
    StringTensorType,
    Int64TensorType
)

# ref: http://onnx.ai/sklearn-onnx/auto_examples/plot_complex_pipeline.html#example-complex-pipeline
def convert_dataframe_schema(df, drop=None):
    inputs = []
    for k, v in zip(df.columns, df.dtypes):
        if drop is not None and k in drop:
            continue
        if v == 'int64':
            t = Int64TensorType([None, 1])
        elif v == 'float64':
            t = FloatTensorType([None, 1])
        else:
            t = StringTensorType([None, 1])
        inputs.append((k, t))
    return inputs

inputs = convert_dataframe_schema(df_train[feature_columns])

And convert the model to ONNX.

# convert to onnx
model_onnx = convert_sklearn(model_pipeline, 'pipeline_auto_insurance', inputs)

The way Fiddler’s MPM platform is able to interface with an ingested model is via a package.py. At a high level, Fiddler gets a handle to the model object via get_model() method and calls predict() with a dataframe as input. In this case, as the model is ONNX serialized, the model is loaded into the onnxruntime, and when the user wants to explain a specific prediction using Fiddler, the data is input to the model handle from package.py. The incoming pandas dataframe is transformed into the format expected by the user as a part of the predict() and then the inference is run via the run() and the output is piped back to Fiddler for the rest of the workflows. With this simple modification, the prediction pipeline is created in the backend, and a user won’t see any difference in our UI. Everything will be very smooth and easily controllable.

%%writefile gb_regression_onnx/package.py

import pandas as pd
import numpy as np
from pathlib import Path
import onnxruntime as rt
import os

PACKAGE_PATH = Path(__file__).parent
TARGET = 'customer_lifetime_value'
PREDICTION = 'predicted_customer_lifetime_value'
CONTINUOUS = ['income', 'monthly_premium_auto', 'months_since_last_claim', 'months_since_policy_inception',
                        'number_of_open_complaints', 'number_of_policies', 'total_claim_amount']

class OnnxGBRegressor:
    """A Gradient Boosting Regressor predictor for auto_insurance data.
       This loads the predictor once and runs for each call to predict.
    """

    def __init__(self, model_path, output_column=None):
        """
        :param model_path: The directory where the model is saved.
        :param output_column: list of column name(s) for the output.
        """
        self.model_path = model_path
        self.output_column = output_column

        file_path = os.path.join(self.model_path, 'model.onnx')
        self.model = rt.InferenceSession(file_path)
        
    def format_input_df(self, input_df):
        model_input_cols = [col.name for col in self.model.get_inputs()]
        inputs = {c: np.array(input_df[c]) for c in input_df.columns if c in model_input_cols}
        for c in inputs:
            if c in CONTINUOUS: # get d_type from feature in DF and remove cont. vars list
                if c == 'total_claim_amount':
                    inputs[c] = inputs[c].astype(np.float32)
                else:
                    inputs[c] = inputs[c].astype(np.int64)
            inputs[c] = inputs[c].reshape((inputs[c].shape[0], 1))
        return inputs

    def predict(self, input_df):
        input_obj = self.format_input_df(input_df)
        return pd.DataFrame(
            self.model.run(None, input_obj)[0], columns=self.output_column) # 0: prediction, 1: probability?

def get_model():
    return OnnxGBRegressor(model_path=PACKAGE_PATH, output_column=[PREDICTION])

A quick sanity check with our validator would also come in handy.

from fiddler import PackageValidator

validator = PackageValidator(model_info, df_schema, model_dir)
passed, errors = validator.run_chain()

With everything checked, the final step is to upload the model and predictions to the project.

# Let's first delete the model if it already exists in the project
if model_id in client.list_models(project_id):
    client.delete_model(project_id, model_id)
    print('Model deleted')
    
client.upload_model_package(artifact_path=model_dir, project_id=project_id, model_id=model_id)
f"Project '{project_id} now contains model '{model_id}'"

client.trigger_model_predictions(project_id, model_id, dataset_id)

At this point, the backend work is done. Now the dataset, model, and predictions will all be visible on our platform, ready for users to analyze predictions and get explanations using our All-purpose Explainable AI technology. In fact, a user will see no difference between the ONNX and ScikiLearn models on our platform. It will be a seamless, simple, and easy experience.

Next Steps for Fiddler + ONNX

As demonstrated, Fiddler’s MPM is a flexible platform that can work with any model framework and version by utilizing ONNX. We plan to build automating the model conversions, with support for more model types, so it becomes a much easier integration for users.