More Than an Audit Trail: Blockchain Model Governance for Auditable AI

“Set it and forget it” may be the greatest infomercial tagline of all time, but it is not a successful strategy for managing artificial intelligence (AI) decisioning models. Yet that is exactly what most organizations do; monitoring AI models after they’ve gone into production is very often an afterthought.

With data science teams under pressure to deliver constant algorithmic innovation, transparency suffers. AI monitoring often consists of periodic checks to track shifts in data distributions and key variables. Measuring performance of the AI model over time is another popular method, but still a lagging indicator of model issues that may impact customers for weeks and months before being detected. 

Unfortunately, neither approach can be documented into AI monitoring requirements or provide a transparent audit trail of AI model performance. Ad hoc and retrospective monitoring of AI systems both completely overlook the culprit that can impact model decisioning the most, and most dangerously: the latent features that stealthily drive the model score, and thus decisions based on its machine learning (ML) model.

AI Governance and Today’s Blockchain Monitoring Imperative

With 48% of businesses using some form of machine learning – a research finding that I think is low – the potential for AI to do damage is as great as its potential to do good. I am a fervent believer that a lack of a strong AI model development standard, and governance of adherence to this standard, can harm individuals and society.

In the past I have written about implementing a transparent model development governance blockchain to enforce an organization’s Responsible AI standards, which creates an immutable audit trail to demonstrate adherence and ensure accountability to these standards. This application of blockchain technology has been awarded a U.S. patent. While blockchain is critical for ensuring responsible development of AI within a governance framework, responsible, transparent use of artificial intelligence goes beyond model creation; it extends to ensuring that AI models’ behavior is monitored in production. 

How FICO Operationalizes Artificial Intelligence Model Monitoring

At FICO we recognize that artificial intelligence model output is driven by latent features, and the combinations of these latent features may fire. As part of our commitment to transparency and Responsible AI practices, we have operationalized the monitoring of latent features in our AI governance framework to detect changes in their distributions, firing combinations, outlier activations and other critical developments.

Importantly, this is a strategy, not an afterthought; the specific latent features, monitoring logic and thresholds for alerting are maintained in the same AI model development governance blockchain and defined during the model development process – the key to auditability and accountability for businesses. These latent features, thresholds and severity of alerting conditions are prescribed in the same blockchain that establishes the artificial intelligence model’s conditions and permitted use.

Artificial Intelligence Model Monitoring in Action

A machine learning model works by ingesting data and computing a set of derived variables based on that input data. It then transforms the derived variables into a set of latent features that drive the score computation logic, generating a set of outputs called scores (Figure 1).

Figure 1: Basic functionality of a machine learning model.

Through blockchain, monitoring can be integrated into this process. When the machine learning model is deployed in the AI production environment, a monitoring component is configured by accessing the AI blockchain that contains the data scientist’s predefined monitoring logic thresholds and alerting conditions, as determined during model development. There is no guessing here; the model comes with specific monitoring configurations supported by blockchain technology. To help ensure transparency, the model is not released without them. 

In production, if alerting conditions are met, an alert object is produced by the monitoring system. The alert object contains the specific conditions in violation of the AI blockchain, and alert severity. It is transported to a reporting and management system for triage and action by human operators, who will remediate the decisioning component as appropriate. For example, an alert could trigger an operational impact review and subsequent remedies such as adjusting corresponding score-based strategies, cauterizing a latent feature, or falling back to a secondary model in accordance with Humble AI practices.

Digging Deeper to Monitor Latent Features

Latent features learn nonlinear relationships between observed data and the outcome that the AI model is designed to provide; “nonlinearity” can easily translate into biased decisioning and other unintended, detrimental consequences. This is why monitoring latent features is a cornerstone of Responsible AI governance framework and Auditable AI. Monitoring enhances model explainability, ethics and stability testing, and gives us a more transparent, explainable picture of what drives the model’s output scores.

At FICO, we use interpretable neural networks models with interpretable latent features that provide transparency as to what drives the AI model outcomes. As shown in Figure 2, each latent feature explicitly combines no more than two incoming connections, and each of the resulting relationships can be monitored.

Figure 2: Constraining the interactions of latent features is a key to model explainability.

All the information contained in the input data elements and derived variables is distilled into interpretable latent features. Therefore, monitoring the interpretable latent features is what truly determines the success or failure of overall governance and monitoring.

Monitoring Thresholds Are Pre-Determined

During AI model development, it is important to understand the expectations of how each of these latent features behaves independently and in combination with each other, and how these individual and collective behaviors can impact the AI model outcome score.

For example, looking again at Figure 1, we may expect interpretable latent features LF1 and LF2 to fire together 1% of the time within a rolling one-hour window, with an acceptable lower threshold of firing of 0.5% and upper threshold of 1.2%. Any tandem firing frequency beyond these thresholds should raise an alert for data scientists.

Monitoring Is an Essential Part of Responsible AI

Figure 3 shows an end-to-end AI model development and model monitoring operational ecosystem for Responsible AI, centered on the AI model governance blockchain. In this way, monitoring requirements can be determined as part of the AI model development process, thwarting model drift, biased decisioning and other negative outcomes before they start.

Figure 4: An end-to-end model development and model monitoring ecosystem leverages the model governance blockchain.

Removing the (Hu)man in the Middle

Incorporating and persisting monitoring requirements into the model governance blockchain removes the man-in-the-middle from the operational equation. It ensures that the AI model monitoring is not based on conditions determined by human operators, which can vary widely over time and even intra-day. Automating alert generation logic further ensures transparency, with meaningful actions that are taken when the model behaves in an unexpected way.

Furthermore, model monitoring provides the information necessary to achieve Auditable AI – a criterion for AI deployment that will become increasingly important as companies weave AI decisioning into the fabric of their business. Because, as we all know, when it comes to operationalizing AI, you can’t just “set it and forget it.”

How FICO Can Help You Develop and Use Auditable AI

• Read about FICO’s Blockchain Award - How to Use Blockchain to Build Responsible AI: An Award-Winning Approach
• Learn more at FICO® Responsible AI
• Download our AI Playbook: A Step-by-Step Guide for Achieving Responsible AI
• Review The State of Responsible AI in Financial Services