BIS AI Explainability in Financial Regulation | Guide

The Growing Role of AI Models in Financial Institutions

Financial institutions are rapidly adopting artificial intelligence models across critical business functions, from credit decisioning and fraud detection to regulatory capital calculations and customer service automation. The Bank for International Settlements (BIS) analysis reveals that AI explainability has become the number one issue raised by financial institutions when engaging with regulators on AI deployment.

This surge in AI adoption brings unprecedented capabilities but also introduces new categories of model risk. Unlike traditional statistical models with transparent linear relationships, modern AI systems—particularly deep learning models and large language models—operate as “black boxes” where the relationship between inputs and outputs is opaque even to their developers.

The challenge extends beyond individual model performance to systemic risk implications. When multiple institutions deploy similar AI models with limited explainability, model failures can propagate across the financial system, amplifying risks that regulators struggle to detect and address promptly.

Why AI Explainability Matters for Financial Regulators

Regulatory scrutiny of AI explainability stems from fundamental principles of financial supervision: transparency, accountability, and consumer protection. When a bank’s AI system denies a loan application, both the regulator and the affected consumer need to understand the reasoning behind that decision.

The Bank of England and Financial Conduct Authority’s 2024 survey found that 50% of UK financial institutions reported only partial understanding of AI technologies they use, particularly when relying on third-party models. This knowledge gap creates multiple regulatory concerns:

• Consumer Protection: Customers cannot effectively challenge algorithmic decisions they cannot understand
• Bias Detection: Discriminatory patterns remain hidden within unexplained model outputs
• Prudential Supervision: Regulators cannot assess model reliability without understanding decision logic
• Systemic Risk: Correlated model failures may go undetected without transparency into model behavior

Moreover, existing consumer protection frameworks explicitly require explainable decision-making processes. BCBS 239 principles mandate that financial institutions maintain comprehensive risk data aggregation capabilities, which implicitly requires understanding how models process and interpret risk information.

Defining Explainability and Interpretability: Key Distinctions

The BIS analysis makes crucial distinctions between explainability and interpretability that shape regulatory approaches. Explainability focuses on answering “why” a model produced a specific output—explaining the reasoning behind individual decisions in human-understandable terms.

Interpretability concerns understanding “how” the model’s internal workings function—comprehending the mathematical relationships and feature interactions within the model architecture itself. While related, these concepts require different technical approaches and regulatory treatments.

The distinction matters because full interpretability of complex AI models may be neither feasible nor necessary for regulatory compliance. Cognitive load theory suggests humans can only understand approximately seven rules or decision nodes, making complete comprehension of sophisticated AI systems virtually impossible.

Instead, regulators increasingly focus on fit-for-purpose explainability—ensuring that explanations match the needs of specific stakeholders and use cases rather than demanding universal transparency. This pragmatic approach acknowledges that different audiences require different levels of technical detail in AI model explanations.

Current Model Risk Management Frameworks and Standards

Existing model risk management (MRM) guidelines from global standard-setting bodies and national regulators largely predate the widespread adoption of advanced AI models. The BIS analysis reviews frameworks from major jurisdictions including Canada (OSFI), Japan (FSA), UAE (CBUAE), UK (PRA), and US (FRB-OCC), finding that explainability requirements are typically implicit rather than explicit.

Most current MRM frameworks embed explainability through four key areas:

• Governance Requirements: Boards and senior management must understand models used for business decisions
• Documentation Standards: Models must be documented with theory, assumptions, logic, and limitations
• Validation Processes: Independent validation must assess conceptual soundness and implementation accuracy
• Monitoring Obligations: Ongoing performance monitoring must include questioning unexpected outputs

However, these requirements were designed primarily for traditional statistical models with interpretable parameters and linear relationships. Applying them to AI models with millions or billions of parameters creates significant compliance challenges that existing frameworks struggle to address effectively.

The Black Box Challenge: Why Complex AI Models Resist Explanation

Modern AI models, particularly deep neural networks and transformer-based language models, operate through complex non-linear interactions that resist traditional explanation methods. OpenAI’s GPT-3, for example, contains 175 billion parameters whose interactions determine model outputs—a scale that makes manual interpretation impossible.

The black box problem stems from several fundamental characteristics of advanced AI systems:

“The mathematical complexity of deep learning models means that even their creators cannot fully explain why specific inputs produce particular outputs. Each layer transforms data through millions of weighted connections, creating emergent behaviors that cannot be reduced to simple rules.”

Over-parametrization compounds the explainability challenge. Modern AI models often contain far more parameters than training examples, enabling them to memorize specific patterns while generalizing to new scenarios. This double descent phenomenon means that model behavior can appear random or inconsistent when analyzed through traditional statistical lenses.

Ensemble methods and model distillation further obscure explainability. When financial institutions combine multiple AI models or compress large models into smaller ones, the resulting systems may perform well while becoming even less interpretable than their individual components.

Post Hoc Explainability Techniques: SHAP, LIME, and Counterfactuals

Despite the inherent opacity of complex AI models, researchers have developed post hoc explainability techniques that attempt to illuminate black box decision-making after outputs are generated. The three primary approaches identified in the BIS analysis are:

SHAP (SHapley Additive exPlanations)

SHAP attributes a model’s prediction to individual input features by calculating each feature’s contribution to the deviation from the expected output. Based on cooperative game theory, SHAP satisfies several mathematical properties including efficiency, symmetry, and additivity that make explanations theoretically sound.

LIME (Local Interpretable Model-agnostic Explanations)

LIME fits a simpler, interpretable model around a specific prediction to identify which input features most influenced that particular output. By perturbing the input data and observing how predictions change, LIME creates locally linear approximations of complex model behavior.

Counterfactual Explanations

Counterfactual explanations identify the smallest change to input data that would alter the model’s output—answering “what would need to change for this loan application to be approved?” These explanations provide actionable insights for both consumers and model developers.

While these techniques represent significant advances in AI explainability, their practical implementation in regulated environments requires careful consideration of their limitations and potential failure modes.

Limitations and Risks of Current Explainability Methods

The BIS analysis identifies several critical limitations of post hoc explainability techniques that financial institutions and regulators must understand when evaluating AI model explanations:

Inaccuracy and Misrepresentation: Post hoc explanations may not accurately represent the model’s actual decision-making process. LIME’s local approximations can miss global patterns, while SHAP values may not reflect the true causal relationships within the model.

Instability: Small changes to input data can produce dramatically different explanations for similar outputs. This instability undermines confidence in explanation reliability and creates challenges for consistent regulatory review.

Lack of Generalization: Explanations that work well for one population or data distribution may not apply to different groups, potentially missing important fairness or bias considerations.

Absence of Ground Truth: Unlike traditional model validation where correct answers are known, explainability techniques lack universally accepted metrics to verify explanation correctness. This creates a fundamental challenge for regulatory assessment.

Manipulation Vulnerability: Adversarial actors can potentially manipulate models to produce misleading but plausible-looking explanations, creating false confidence in model transparency while obscuring actual bias or errors.

Challenges of Third-Party and Proprietary AI Models

Financial institutions increasingly rely on AI models developed by third-party vendors, creating additional layers of opacity that complicate explainability requirements. Proprietary algorithms from fintech companies, cloud providers, and specialized AI vendors often come with limited documentation about internal workings or decision logic.

The challenge extends beyond technical limitations to commercial considerations. AI model providers view their algorithms as competitive advantages and intellectual property, creating tension between business interests and regulatory transparency requirements.

Key third-party explainability challenges include:

• Limited Access: Financial institutions cannot examine proprietary model code or training processes
• Vendor Lock-in: Switching costs discourage thorough explainability evaluation before adoption
• Update Opacity: Model updates may change decision logic without transparent documentation
• Liability Gaps: Unclear responsibility for explanation accuracy between vendors and users

Explainability of Large Language Models (LLMs)

Large language models present uniquely severe explainability challenges that may fundamentally limit their use in high-stakes financial applications. The BIS analysis highlights several characteristics that make LLMs particularly resistant to traditional explanation methods:

Scale and Complexity: Modern LLMs like GPT-4 contain hundreds of billions of parameters trained on datasets comparable to the entire internet. The sheer scale makes comprehensive analysis computationally infeasible and conceptually impossible for human understanding.

Probabilistic Outputs: LLMs generate text probabilistically, with random sampling determining each successive word from likely candidates. This means identical inputs can produce different outputs, and rare probability draws can generate “hallucinations” that appear coherent but are factually incorrect.

Training Data Opacity: Even for LLMs with publicly available model weights, the training data and processes often remain completely opaque. Financial institutions cannot understand what information influenced model development or identify potential biases in training datasets.

Emerging explainability techniques for LLMs include chain-of-thought prompting, which asks models to show their reasoning steps, and attribution methods like those developed by Anthropic. However, the BIS analysis cautions that these approaches may create illusions of transparency without reflecting the model’s actual internal decision process.

Proposed Adjustments to MRM Guidelines for the AI Era

One of the most significant insights from the BIS analysis is the recognition that explainability and model performance often exist in tension. Simpler, more explainable models may deliver inferior predictive accuracy compared to complex AI systems that resist interpretation. This trade-off creates difficult regulatory decisions—prohibiting complex AI models solely due to limited explainability could prevent financial institutions from using superior risk management tools.

The BIS recommends a risk-based approach considering use case criticality, performance advantages, compensating controls, and decision reversibility. Based on this analysis, the BIS proposes several specific adjustments to model risk management frameworks:

Extend MRM Guidelines Beyond Regulatory Capital: Current frameworks focus heavily on models used for capital calculations, but AI deployment spans all business areas. Regulators should explicitly address AI model governance across the entire institution.

Require Risk-Scaled Explainability Standards: Financial institutions should establish acceptable explainability thresholds based on use case criticality, model complexity, and stakeholder needs rather than applying universal standards.

Mandate Multiple Explainability Techniques: Given the limitations of individual explanation methods, regulators should require institutions to employ suites of complementary techniques rather than relying on single approaches.

Acknowledge Performance Trade-offs Explicitly: Updated guidelines should recognize that optimal risk management may require accepting complex models with limited explainability, subject to enhanced safeguards and compensating controls.

For regulatory capital applications specifically, the BIS suggests considering more stringent output floors than currently required under Basel III or restricting complex AI models to certain risk categories where explainability concerns are less acute.

Finally, the analysis emphasizes the need for regulatory capacity building. Supervisory staff require training in AI model evaluation to assess explainability submissions effectively and distinguish between genuine transparency and superficial explanations that may obscure rather than illuminate model behavior.