Low accuracy in machine learning can lead to serious consequences. Model bias and data leakage in machine learning are prime examples of what can occur when a machine learning (ML) model is not properly trained and model monitoring isn’t performed.
Take the COMPAS case, for example. The Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS AI system, was programmed to assess the risk for recidivism of different offenders. But there was a problem, the COMPAS model’s algorithm was inherently biased against people of color. In the end, the COMPAS system predicted double the number of false positives for recidivism for Black Americans compared to White Americans. This resulted in extremely inaccurate risk assessment scores which were presented to judges during criminal sentencing. Naturally, the falsely-inflated risk assessment scores generated by the COMPAS algorithm caused many offenders to receive unjust sentences.
This is just one example of how dangerous AI can be when a model does not perform correctly. If AI is to reach its full potential, greater care must be taken to monitor model performance. But how can you improve the performance of a ML model? Optimizing Machine Learning Operations (MLOps) through ML model performance management (MPM) allows developers to continuously monitor and improve model performance in machine learning. In short, the MPM framework takes MLOps and model monitoring tools to the next level.
In this article we’ll briefly outline how model performance is currently evaluated and explain how an MPM platform can successfully improve performance and reduce the risk of harmful AI errors.
Currently ML models are evaluated using a myriad of machine learning performance metrics. Here are just a few examples of the most popular machine learning metrics used to evaluate performance:
Regression metrics are involved with predicting a numeric value. For example, the 9- Mean Squared Error (9-MSE) metric is one of the most popular regression metrics used in ML modeling. The 9-MSE metric is used to calculate the average squared error between the model’s predicted values and actual values.
Classification metrics have discrete output and are involved with evaluating a model’s performance and describe if a classification is good or bad. A prime example of a classification metric is the F1-score. An F1-score is a metric that measures the predictive performance of a model. This is done by joining precision and recall metrics. A precision metric evaluates true positives and false positives while recall evaluates true positives and false negatives. The calculated F-1 score is often called the “harmonic mean” of precision and recall. This means that the F-1 score is designed to give developers a full picture, considering how many times a model presents false positives and false negatives. The more false positives and negatives that occur, the lower the F-1 score of a model.
Each of these metrics are critical to understanding how to increase accuracy of machine learning models and improve overall performance. However, because models are so opaque in nature, developers often operate with low visibility as these metrics are being used to evaluate model performance. This is where bias and other performance issues can begin to creep in. So, what can be done? Let’s see how an MPM platform can solve this problem.
A Model Performance Management (MPM) platform acts as a control system at the core of the MLOps lifecycle. Because of frequent data changes, the performance of a machine learning model can fluctuate over time. This means that no one will know exactly how a model will perform until it is deployed and tested against real life scenarios. That’s why constant ML monitoring is needed to ensure that a model is performing as expected on an ongoing basis. Here is a visual representation of what an MPM platform looks like in practice:
As an example, let’s explore how an MPM platform can be used to overcome the challenge of model bias.
Since models are trained using existing data, they have the potential to propagate existing bias or even introduce new bias. But with an MPM platform, model bias can be detected and eradicated before any harm is done. An MPM platform can explain where issues are arising and trigger alerts that are shared with all stakeholders, thus improving accuracy and increasing transparency.
An MPM platform allows ML teams to augment their traditional monitoring processes with explainable AI, providing actionable, real-time insights into ML performance. In short, machine learning is no longer a black box.
Interested in seeing how Fiddler can help you maintain a high-performance model? Request a demo and discover what Fiddler can do for you today.
