When AI Algorithms Go Off The Rails

5 min

Here’s a non-fictional scenario: An algorithm that has been running for years warning about the unusual behavior in credit card usage, suddenly identifies anomalies in the behavior of people who aren’t outliers (meaning a significant increase in false alarms). Credit cards are blocked for no reason; frustrated and nervous customers call the customer service center to complain.

After several hours of abnormal stress on customer service centers, service teams realize that there may be an unusual event happening and pass the problem on to their development teams. After some additional time, they recognize that the algorithm is malfunctioning and disable it until it is “back on the rails”. Meanwhile, other customers are affected by the lack of an alert for unusual behavior on their card, and the credit card companies suffer further damage to their finances and prestige.


This is a scenario that will happen for sure.

Along with the rise in popularity of artificial intelligence and the increased usage of algorithms in our daily lives and in various organizations’ critical processes, there has been a rise in reports of glitches over the past two years, i.e.artificial intelligence algorithms that have made crucial mistakes.

Take Amazon’s face recognition algorithm which “missed” several famous congressmen and identified them as famous criminals. In another unfortunate case, Google’s object detection algorithm tagged a woman as a gorilla.

This may sound amusing, but mistakes in image recognition can lead to fatal results, especially when it comes to the medical field — such as when another Google algorithm, which detected diabetes through retinal image analysis, failed to accurately detect in a real-world environment. In another incident IBM’s Watson system created false insights for cancer patients, forcing doctors to halt the use of the machine immediately.

“This product is a piece of s—“

Physician at Jupiter Hospital

The adverse and devastating effects of incorrect algorithm results are apparent. Although artificial intelligence cannot replace people yet, it can undoubtedly make poor decisions for us.

This is a scenario that will continue to happen for sure. The responsibility for creating it, as well as preventing it, rests with the data governance teams, such as the chief data officer.

But are the growing dependencies of organizations on algorithms likely to come back like a boomerang? The answer is, it depends of course. It depends on their ability to allow a continuous maintenance process and ideally the improvement of the algorithms’ performance with time. Furthermore, it depends on real-time fault detection. When the galloping train gets off the track, they must effectively handle the damage.

Source: Photo by Johannes Plenio on Unsplash

How can organizations be prepared for this scenario, producing control, prevention, and escalation processes? In this article, I will propose several methods for handling just that.

But before we discuss the solutions, it is crucial to understand the origin of the above scenario. Predictions of algorithms and artificial intelligence are based on a data training process that reflects a particular point in time and portrays a specific reality. When these algorithms meet the changing reality and hence changing data, yet they remain static in the learning process, which is the exact point where this problem arises.

Therefore, what is first and foremostly required is the monitoring capability of machine learning algorithms runtime.

The monitoring ability of algorithms is a necessity for proper and successful AI usage. Monitoring assumes that if something goes wrong, it’ll happen over time. That is, if algorithms are monitored as a black box, while the input and the output are known, on the timeline, real-time anomalies and biases can be detected (i.e. disruption of the expected distribution of data as observed in the training phase). Such monitoring can be detected before harming a large number of consumers.

Photo by Stephen Dawson on Unsplash

But this is not enough. Understanding that algorithms should be kept “alive,” we need to produce a frequent version upgrade mechanism that addresses up-to-date data. Such a tool is required to enable feedback on end-users’ results of the algorithm. This feedback streamlines the real-time learning process. Online learning adapts the model to the changing reality and reduces the damage. The algorithm update process requires its re-deployment and integration into production systems. As with any software update in the agile era, sanity and backward testing is required to ensure that this deployment does not create new secondary damages.

Finally, it is highly important to reflect and explain the algorithm’s performance and outcomes to end-users. This experience generates the required trust and explanation in case of any incompatibility. The tendency to attribute explanations solely for the purpose of clarifying the result (the right for explanations) is insufficient. The extra value in an explanation is the ability to monitor unwanted phenomena and allow them to be corrected, similar to quality assurance software processes.

Explanation use is intended for end-consumers directly, or for customer service teams. Beyond explaining the results, it allows immediate feedback from which the algorithm can be improved in real-time.

All this leads to the conclusion that artificial intelligence algorithms can no longer be considered as an inherent component of enterprise applications. The reason for this is because this method cannot be separated from the process of monitoring and improving those “black box” algorithms. At the same time, we want their results to be reflected in the decision-making processes within the organizational applications.

Therefore, the right technical solution is to produce AI micro-services (SaaS) that can be consumed, monitored, maintained, and improved — regardless of the application they are integrated with.

Photo by Bill Oxford on Unsplash

Let’s go back to the credit card scenario. Dissatisfied customers call the service center. Their problem is checked against the algorithm’s explanatory results, and immediate correction is made. The result is fed as feedback into the algorithm’s real-time learning processes. At the same time, monitoring for data operations and governance teams warn of an emerging exceptional behavior. Until the algorithm is stabilized, they work to minimize the harmful impact on thousands of customers, even at the cost of manually handling inquiries that are inspected as a warning.

In summary, artificial intelligence algorithms learned from snapshots at a point in time are likely to be out of use over time. To mitigate the potential damage from static usage of artificial intelligence algorithms, a continuous maintenance process is required, monitoring exceptional real-time events, presenting explanations of each outcome, and ensuring continuous learning of the algorithms. We also recommend packing algorithms as services for enabling this dynamic approach.