how.fm Reduces Time to Detect Model Issues From Days to Minutes

The Fiddler AI Observability (formerly known as Model Performance Management) platform empowers how.fm’s lean data science team to rapidly launch natural language processing (NLP) models and streamline the entire ML lifecycle, from data and feature exploration to model monitoring to refining models for optimal model outcomes.

About how.fm

how.fm is a software as a service training platform allowing 3PLs to quickly and easily onboard, upskill, and support their workers.

Customers use how.fm in more than 30 languages to put their entire staff training process on autopilot while increasing safety and quality in their operations.

Building a robust MLOps framework with best-of-breed AI Observability 

NLP models built by how.fm’s data scientists power their employee training and performance platform, enabling customers to onboard and train workers as quickly and seamlessly as possible. 

Though central to the company’s offering, the lean data science team could not  afford to spend long cycles in R&D building perfect models that need to be retrained when encountering new inputs. Instead, they embraced agile model development to launch models rapidly. Once in production, the models would be monitored and improved using production inputs.

The lead data scientist at how.fm, Richard Sieg, knew he needed to build a standardized MLOps framework with robust monitoring and explainability capabilities to create a continuous feedback loop in the MLOps lifecycle.

“We looked into building our own monitoring tool on top of our existing ML platform,” said Sieg. “Our existing ML platform gave us the gears we needed to build a simple model monitoring capability but it only supported tabular models with a specific pattern, format and columns. Our models were not compatible with those specifications since our models deal with NLP and CV data. We sought support from their documentation, but its support was rudimentary and limited.”

“That’s when we decided to purchase best-of-breed monitoring and explainable AI tools, which are two critical pieces of our MLOps framework that we couldn’t overlook to launch and iterate models quickly and at scale,” said Sieg. 

After evaluating several monitoring tools in the market, Sieg chose the Fiddler MPM platform as the company’s vendor of choice. “We started monitoring models and obtaining valuable model insights within a day of onboarding onto Fiddler. Fiddler is flexible and compatible with our agile model development and supports our  company goals.” 

Embracing agile model development with a continuous feedback loop in the ML lifecycle

The data science team has been more productive since adding Fiddler to their MLOps tech stack. “With Fiddler running in the background, I am at ease knowing that it does all the model monitoring for me. I can spend more time testing new hypotheses and experiments that’ll inform how to build the type of models that drive our company forward.”

The NLP model behind how.fm’s training platform helps their customers create better employee onboarding and training content. It inspects training documents by accurately classifying whether content is actual instruction or auxiliary tips and recommendations. In classifying the content, customers improve the quality of employee training by including valuable and pertinent information. 

“Prior to using Fiddler, I’d spend from several hours to days digging into API logs and identifying the specific log that was causing a problem whenever a model underperformed” said Sieg.

“With Fiddler, I am alerted on mission-critical issues, perform root cause analysis to find the underlying cause of the issue, and resolve the issue before our customers are impacted.”

— Richard Sieg, Lead Data Scientist, how.fm

Gaining complete visibility into model outcomes with rich analytics and explainable AI

In addition to automating model monitoring using Fiddler, Sieg queries specific slices of data to drill-down and uncover anomalies in the data that may be causing model underperformance or model drift. “I use point explanations daily to understand why a model is making certain predictions based on the production inputs,” said Sieg. “I can make informed assumptions and theories with this level of information and have more guidance in how to improve the model if the production inputs change slightly.” 

Sieg found an anomaly using Fiddler in their classification model that provides language translation. By drilling down into slices of data and feature attributions, Sieg found that the contribution of the text length to prediction erred across different languages. Because German sentences tend to be longer than English sentences, translations from the classification model were skewed. By discovering this root cause, Sieg and his team were able to quickly retrain the model to improve its performance. 

“Fiddler is a powerful tool in our MLOps tech stack, enabling us to iterate our ML workflows quickly,” said Sieg, “As a data scientist, I don’t have to make compromises of having monitoring-only or explainability-only capabilities. Fiddler gives me both so I can confidently deliver ML solutions for mission-critical projects at scale.”