Many companies are now utilizing data science and machine learning, but there’s still a lot of room for improvement in terms of ROI. A 2021 VentureBeat analysis suggests that 87% of AI models never make it to a production environment and an MIT Sloan Management Review article found that 70% of companies reported minimal impact from AI projects. Yet despite these difficulties, Gartner forecasts investment in artificial intelligence to reach an unprecedented $62.5 billion in 2022, an increase of 21.3% from 2021.
#1 Build a Drift Detector
Although powerful, modern machine learning models can be sensitive. Seemingly subtle changes in the data can destroy the performance of otherwise state-of-the-art models, which can be especially problematic when ML models are deployed in production. Drift detection is a discipline focused on detecting such changes, understanding the ways in which drift can occur, and the types of drift. Building a drift detector using tools like Alibi Detect, an open-source Python library, will enable you to detect drift in a principled manner. Alibi Detect is a powerful library containing algorithms for adversarial, outlier, and drift detection. Since data can take many forms, such as image, text, or tabular data, one often needs to use existing machine learning models to preprocess data into a form suitable for drift detectors. To gain further insights into the causes of drift, one can employ state-of-the-art detectors which are able to perform fine-grained attribution to instances and features. To assess whether model performance has been affected by drift, you can experiment with using supervised and uncertainty-based detectors. Additionally, in production environments, data often arrives sequentially, so drift can be detected using online detectors.
#2 Understand Machine Learning Safety
Machine learning systems are rapidly increasing in size, acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for machines should be a major priority for any researcher or practitioner. Some of the problems in machine learning safety are well known and addressable but many unsolved problems remain. Recent large-scale models in particular are an emerging safety challenge in machine learning. The three pillars of machine learning safety are namely, withstanding hazards (“Robustness”), identifying hazards (“Monitoring”), and steering ML systems (“Alignment”).
#3 Learn To Mitigate Bias in Machine Learning
It’s well established that bias is everywhere–in data, in algorithms, in humans. As data increases exponentially in nearly all dimensions (velocity, volume, veracity) and machine learning systems trained on these data sets are becoming omnipresent, dealing with bias becomes more complex and important. Practitioners may underestimate the complexity that bias introduces into machine learning workflows. Thus, the first steps are awareness, understanding data quality, and proficiency in measuring and monitoring machine learning algorithms. Next is an understanding of how to mitigate these problems.
#4 Measure Technical Debt
Technical debt was a concept introduced by Ward Cunningham in 1992 to understand the long-term costs (debt) incurred by moving quickly in software engineering. This can be readily applied to machine learning and can take on many facets such as feedback loops (direct or hidden), data dependencies, high-debt design patterns, poor abstraction, and more as is explained in this 2015 paper by Sculle et al. 7 years after the publication this paper, “MLOps” is now a hot topic. MLOps’ goal is to quickly and reliably deploy and maintain machine learning models in production. Its widespread adoption, however, has the potential to substantially increase technical debt in production deployments. Understanding technical debt in machine learning workflow goes beyond just modeling libraries. It requires additional methods for monitoring, lineage, and deployment.
#5 Build New Methods To Handle Missing Values
Missing values are ubiquitous in data analysis and many methods to tackle this problem are widely understood. Classical methods for dealing with missing values include single imputation, multiple imputations, and likelihood-based methods developed in an inferential framework. The aim of these methods is to estimate at best the parameters and their variance in the presence of missing data.
Recent results from efforts to use a supervised-learning setting to predict a target when missing values appear in both training and testing data look promising.
One striking result from these efforts, is that naive imputation strategies (such as mean imputation) might be the best method to use, as the supervised-learning model does the hard work. That such a simple approach can be relevant may have important consequences in practice. Additionally, missing-value modeling can be readily incorporated into tree models, such as gradient-boosted trees, giving a learner that has been shown to perform very well, including in difficult missing-not-at-random settings.
#6 Build Defense Methods for Adversarial Attacks
Classifiers and other machine learning models can be easily tricked into making embarrassingly false predictions. When this is done systematically and intentionally, it is called an adversarial attack. Specifically, this kind of attack is called an evasion attack. Defense methods for these kinds of attacks include spatial smoothing preprocessing and adversarial training. Methods to ascertain that the model can withstand such attacks include the robustness evaluation method and the certification method.
#7 Go Beyond Basic Model Evaluation
Machine learning has made rapid progress, but at the cost of getting more complex and opaque. Despite widespread deployment, the practice of evaluating models remains limited to computing aggregate metrics on held-out test sets. This practice can fall short of surfacing failure modes of the model that may otherwise show up during real-world usage. A better method would be to ask why did the model make this prediction? One approach to answering this question is to attribute predictions to input features–a problem that has received a lot of attention in the last few years.
Integrated Gradients (ICML 2017), is an attribution method that is applicable to a variety of Deep Neural Networks (object recognition, text categorization, machine translation, etc.). Evaluation workflows based on feature attributions have several applications and attributions can be used for monitoring models in production.
