In machine learning, model performance evaluation uses model monitoring to assess how well a model is performing at the specific task it was designed for. There are a variety of ways to carry out model evaluation in model monitoring, using metrics like classification and regression.
Evaluating model performance is essential during model development and testing, but is also important once a model has been deployed. Continued evaluation can identify things like data drift and model bias, allowing models to be retrained for improved performance.
Model performance in general refers to how well a model accomplishes its intended task, but it is important to define exactly what element of a model is being considered, and what “doing well” means for that element.
For instance, in a model designed to look for credit card fraud, identifying as many fraudulent transactions as possible will likely be the goal. The number of false positives (where non-fraudulent activity was misidentified as fraud) will be less important than the number of false negatives (where fraudulent activity is not identified). In this case, the recall of the model is likely to be the most important performance indicator. The MLOps team would then define the recall results they consider acceptable in order to determine if this model is performing well or not.
A commonly asked question is about model accuracy vs model performance, but this is a false dichotomy. Model accuracy is one way to measure model performance. Accuracy relates to the percentage of model predictions that are accurate, which is one way to define performance in machine learning. But it will not always be the most important metric of performance, depending on what the model is designed to do.
Performance evaluation is the quantitative measure of how well a trained model performs on specific model evaluation metrics in machine learning. This information can then be used to determine if a model is ready to move onto the next stage of testing, be deployed more broadly, or is in need of more training or retraining.
Two of the most important categories of evaluation methods are classification and regression model performance metrics. Understanding how these metrics are calculated will enable you to choose what is most important within a given model, and provide quantitative measures of performance within that metric.
Classification metrics are generally used for discrete values a model might produce when it has finished classifying all the given data. In order to clearly display the raw data needed to calculate desired classification metrics, a confusion matrix for a model can be created.
This matrix makes clear not only how often the model predictions were correct, but also in which ways it was correct or incorrect. These variables are listed in formulas as TN (true negative), FP (false positive), etc.
These are some of the most commonly useful classification metrics that can be calculated from the data contained in a confusion matrix.
Regression metrics are techniques generally better suited to be applied to continuous output of a machine learning model, as opposed to classification metrics which tend to work better to analyze discrete final results.
Some of the most useful regression metrics include:
Machine learning models are incredibly useful and powerful tools, but they need to be trained, monitored, and evaluated regularly to produce the benefits your business wants. Choosing the most applicable predictive performance measures and tracking them appropriately takes time and expertise, but is a critical step in machine learning success.