AI Risk Modeling Frameworks for Risk Management

Why AI Risk Modeling Is the Foundation of AI Safety

AI risk modeling has emerged as the most critical yet underdeveloped discipline in the race to deploy artificial intelligence safely at scale. As organizations worldwide integrate increasingly powerful AI systems into high-stakes decision-making processes, the gap between AI capabilities and our ability to understand their failure modes continues to widen. A landmark research paper from the University of Cambridge and the Centre for the Governance of AI reveals that sound advanced AI risk management fundamentally requires sound advanced AI risk modeling—and that current practices fall dangerously short of what mature safety-critical industries have long established.

The paper, authored by Alejandro Tlaie and a team of leading AI governance researchers, argues that AI risk modeling must tightly integrate two components that are currently developed in isolation: scenario building, which maps causal pathways from hazards to harms, and risk estimation, which quantifies the likelihood and severity of each pathway. This fragmentation means that organizations building and deploying frontier AI models are essentially flying blind—they can identify individual capabilities that might be dangerous, but they lack the structured frameworks to understand how those capabilities translate into real-world harm through specific causal chains.

The implications extend far beyond academic theory. As the RAND analysis of AI cybersecurity and national security has demonstrated, the failure to model AI risks systematically can have cascading consequences across critical infrastructure, financial systems, and democratic institutions. The question is no longer whether to implement AI risk modeling but how to do it with the rigor that the technology demands.

Consider the analogy with nuclear power. In the 1970s, the Reactor Safety Study (WASH-1400) introduced Probabilistic Risk Assessment (PRA) to the nuclear industry, fundamentally transforming how reactor safety was managed. Before PRA, nuclear safety relied primarily on deterministic engineering margins—designing systems to withstand worst-case scenarios with conservative buffers. The introduction of systematic risk modeling revealed previously invisible failure pathways and enabled targeted safety investments that dramatically reduced accident probabilities. The AI industry stands at a similar inflection point, with the critical difference that AI systems are orders of magnitude more complex in their failure modes and far less amenable to physical inspection.

Scenario Building Techniques for AI Risk Assessment

Scenario building forms the first pillar of comprehensive AI risk modeling, providing the structured analytical frameworks necessary to map how AI hazards translate into real-world harms. The research identifies seven primary scenario building techniques that have proven effective across safety-critical industries and can be adapted for AI systems, each offering unique advantages for different types of risk analysis.

Fault Tree Analysis (FTA) takes a top-down, deductive approach by starting with an undesired “top event” and tracing backward to identify all possible root causes through AND/OR logic gates. For AI risk assessment, an FTA might begin with the top event “AI-enabled disinformation campaign destabilizes a democratic election” and then systematically decompose this into branches covering deepfake generation capabilities, failed human oversight mechanisms, AI-powered micro-targeting systems, and recommendation algorithm amplification pathways. The technique’s greatest strength lies in identifying Minimal Cut Sets—the smallest combination of basic failures that can cause the top event—which directly informs risk prioritization.

Event Tree Analysis (ETA) takes the opposite approach, working forward from an initiating event through binary branching points representing the success or failure of successive safety functions. An AI-specific ETA might start with “Frontier AI model capable of creating sophisticated phishing schemes released without sufficient safeguards” and then trace outcomes through branches such as whether malicious actors successfully adapt the model, whether cybersecurity systems detect the attack, and whether critical infrastructure is ultimately compromised. Each path through the tree represents a distinct scenario with its own probability and consequence profile.

FMEA and FMECA (Failure Mode and Effect Analysis, and its extension with Criticality Analysis) investigate potential failure modes at the component level. For an AI alignment mechanism based on RLHF (Reinforcement Learning from Human Feedback), a failure mode might be “deceptive alignment occurs,” with the effect being “AI takes harmful, unprompted actions that appear aligned during evaluation.” FMECA extends this by assigning a Risk Priority Number based on severity, occurrence probability, and detectability—a framework that can systematically prioritize which AI failure modes demand the most urgent attention.

Bow-Tie Analysis provides perhaps the most intuitive framework for AI risk assessment by combining FTA and ETA around a central “critical event.” The left side maps all causal pathways leading to the event (with preventive barriers), while the right side maps all possible consequences (with mitigative barriers). For AI, a Bow-Tie centered on “Human loses effective control over a powerful AI agent” would map emergent capabilities and deceptive alignment as left-side threats while mapping consequences ranging from targeted harmful actions to global catastrophic impact on the right side.

STPA (System-Theoretic Process Analysis) models the entire sociotechnical system as nested control loops, identifying Unsafe Control Actions that can lead to system-level hazards. This technique is particularly well-suited for AI deployment contexts where risk emerges from the interaction between automated systems, human operators, and organizational processes. An STPA analysis of a frontier model release pipeline might identify UCAs such as “automated policy engine authorizes potentially dangerous bio-related protocol after obfuscated jailbreak while human override is delayed,” capturing risks that component-level analyses would miss entirely.

Quantitative Risk Estimation Methods for AI Systems

While scenario building maps the causal landscape of AI risk, quantitative estimation provides the numerical foundation necessary for prioritization, cost-benefit analysis, and regulatory decision-making. The research identifies five primary estimation methods, each addressing different aspects of the AI risk quantification challenge.

Expert elicitation stands as the most critical method for AI risk estimation because empirical data on AI failures is fundamentally scarce—particularly for catastrophic scenarios that have never occurred. Structured protocols such as the Delphi method and the IDEA protocol (Investigate, Discuss, Estimate, Aggregate) combine individual expert judgment with group calibration to produce probability estimates that are more reliable than informal assessments. The key to effective expert elicitation in AI risk contexts is assembling diverse panels that include not just AI researchers but also domain experts in the relevant harm areas, red-teamers who understand attack surfaces, and social scientists who can assess behavioral dynamics.

Monte Carlo simulation propagates uncertainty through complex models by running thousands of randomized trials, making it ideal for scenarios where multiple uncertain factors interact. In an AI risk context, Monte Carlo methods can model the combined effect of uncertain factors in a multi-stage attack chain—the probability that a malicious actor acquires the model, successfully adapts it for harmful use, evades detection systems, and ultimately causes the target level of damage. Each variable can be specified as a probability distribution rather than a point estimate, yielding a rich picture of outcome uncertainty.

Bayesian approaches offer a natural framework for AI risk because they allow continuous updating as new evidence emerges. Starting with prior probability distributions informed by expert judgment, Bayesian models systematically incorporate new data from model evaluations, red-team exercises, incident reports, and real-world deployment observations. This iterative updating process is essential for AI risk modeling because the risk landscape shifts rapidly as models gain new capabilities and as deployment contexts evolve.

Bayesian Networks (BNs) extend basic Bayesian reasoning into directed acyclic graphs that model causal and probabilistic dependencies between multiple risk factors. For AI risk, a Bayesian Network might connect nodes representing model capability levels, deployment safeguard strength, threat actor sophistication, target vulnerability, and ultimate harm severity—with conditional probability tables at each node capturing how parent factors influence child outcomes. BNs are particularly valuable because they are transparent, auditable, and can incorporate both empirical data and expert judgment at different nodes.

Copulas address a critical gap in risk modeling: capturing the statistical dependence between risk variables without assuming specific causal relationships. In AI contexts, copulas are essential for modeling systemic and cascading risks where the failure of one AI component is correlated with failures in others. When multiple AI agents operate in the same environment, or when multiple organizations deploy models with shared vulnerabilities, independence assumptions can lead to dramatic underestimation of total system risk. Copulas provide the mathematical machinery to model these dependencies accurately, as explored in the FSB’s assessment of AI adoption risks in the financial sector.

Lessons from Nuclear and Aviation Risk Frameworks

The most compelling argument for rigorous AI risk modeling comes from the extraordinary safety records achieved by industries that adopted systematic risk modeling decades ago. The nuclear and aviation sectors provide particularly instructive parallels, demonstrating how structured risk modeling can transform safety outcomes even in domains characterized by catastrophic failure potential and deep technical complexity.

The nuclear industry’s approach to risk modeling operates on three levels of increasing sophistication. Level 1 PRA estimates core damage frequency—the probability per reactor-year that a sequence of events leads to core damage. Level 2 PRA extends the analysis to containment performance, assessing whether radioactive material escapes the containment building given a core damage event. Level 3 PRA calculates the ultimate off-site consequences including population doses and health effects. This layered structure ensures that the complete causal chain from initiating event to ultimate consequence is modeled explicitly, with uncertainty quantified at each stage.

Complementing this probabilistic framework, nuclear safety also employs Deterministic Safety Analysis (DSA) which defines a set of Design Basis Accidents—scenarios so severe that the plant must be engineered to withstand them with certainty. This dual approach is mandated by the International Atomic Energy Agency (IAEA) and represents a fundamental insight for AI governance: some risks are so catastrophic that they require deterministic guarantees rather than probabilistic acceptance, while the broader risk landscape requires comprehensive probabilistic assessment.

Aviation safety, governed by the International Civil Aviation Organization (ICAO), takes a complementary approach through Safety Management Systems (SMS) that integrate semi-quantitative risk matrices with deterministic certification requirements. Aircraft must demonstrate absolute compliance with specific performance standards—such as maintaining climb capability after engine failure—while airlines simultaneously manage their overall safety risk through probabilistic assessment guided by the ALARP (As Low As Reasonably Practicable) principle.

The financial sector offers another critical reference point, particularly given AI’s increasing role in financial services. The Basel Committee’s approach to operational risk management combines statistical methods like Value at Risk (VaR) and Expected Shortfall with scenario-based stress testing. The FAIR (Factor Analysis of Information Risk) framework, originally developed for cybersecurity, demonstrates how Monte Carlo simulation can produce quantitative, monetary risk estimates even in domains where historical data is limited—a methodology directly transferable to AI risk quantification.

Perhaps the most relevant parallel for AI comes from the U.S. Navy’s SUBSAFE program, established after the USS Thresher disaster in 1963. SUBSAFE operates on a fundamentally deterministic philosophy—every component in the submarine’s sea-water boundary must be certified through Objective Quality Evidence (OQE) documenting material pedigree, manufacturing process, and inspection results. The program’s “three-legged stool” separates technical authority, certification authority, and program management to prevent conflicts of interest. Despite this deterministic core, SUBSAFE selectively employs probabilistic analysis for novel technologies where deterministic certification processes haven’t yet been established—precisely the challenge AI governance faces today.

Deterministic vs Probabilistic AI Risk Assessment

The research reveals a fundamental finding that should reshape how organizations approach AI safety: every mature safety-critical industry combines deterministic and probabilistic risk assessment methods, and the specific blend varies based on the industry’s characteristics. AI governance must adopt this dual approach rather than relying exclusively on either method.

Deterministic AI risk assessment establishes absolute safety boundaries—conditions that must never be violated regardless of probability. In the nuclear context, this corresponds to design basis accidents; in aviation, to minimum performance standards. For AI, deterministic requirements might mandate that certain categories of harmful output are provably impossible, that human oversight mechanisms cannot be circumvented, or that specific capability thresholds trigger mandatory containment protocols. The challenge, as the research emphasizes, is that AI systems are “grown, not designed”—they emerge from opaque optimization processes in billions of dimensions, making it extraordinarily difficult to provide the kind of deterministic guarantees that are routine in physical engineering.

Probabilistic AI risk assessment, by contrast, acknowledges that not all risks can be eliminated and focuses on quantifying the residual risk landscape to enable informed decision-making. This approach asks questions like: “What is the probability per year that this AI system contributes to a catastrophic cybersecurity incident?” or “What is the expected economic damage from misuse of this model’s biological capabilities over a five-year deployment period?” Probabilistic assessment requires both scenario building (to identify what can go wrong) and quantitative estimation (to assign numbers to likelihood and consequence).

The research proposes that AI governance adopt a specific formulation of this dual approach: deterministic guarantees should be mandated for risks with catastrophic or existential severity, where the consequences are so extreme that no probability is acceptable. Probabilistic assessment should cover the broader risk landscape, enabling organizations to allocate safety resources efficiently and regulators to set deployment conditions based on quantified risk profiles. This mirrors the approach taken by the nuclear industry, where core damage events are assessed probabilistically but certain containment integrity standards are maintained deterministically.

A critical nuance the research highlights is that the current state of AI technology makes the deterministic component particularly challenging. Unlike a nuclear reactor, where physical laws constrain behavior and components can be individually inspected and tested, an AI model’s behavior emerges from billions of learned parameters whose interactions are not fully understood. Achieving deterministic safety guarantees for AI will likely require fundamentally new approaches to system design—what the research terms “provably safe AI architectures”—rather than simply adapting existing engineering practices.

Bayesian Networks and AI Risk Quantification

Among the quantitative methods available for AI risk estimation, Bayesian Networks deserve special attention for their unique combination of transparency, flexibility, and ability to model the complex dependencies that characterize AI risk scenarios. A Bayesian Network represents the joint probability distribution over a set of risk variables as a directed acyclic graph, where nodes represent variables and edges represent probabilistic dependencies.

For AI risk quantification, a practical Bayesian Network might include nodes representing model capability level (informed by benchmark evaluations), deployment safeguard strength (informed by red-team assessments), threat actor sophistication (informed by intelligence analysis), environmental vulnerability (informed by domain-specific assessments), and ultimate harm severity (the output variable of interest). The conditional probability tables at each node capture how parent variables influence the probability distribution of the child variable—for example, how a high capability level combined with weak safeguards affects the probability of successful misuse.

The research identifies several advantages of Bayesian Networks for AI risk modeling that make them particularly well-suited to the current state of the field. First, BNs naturally accommodate both empirical data and expert judgment—nodes with available data can use frequentist estimates, while nodes with limited data can use expert-elicited probability distributions. This hybrid approach is essential for AI risk, where some parameters (like model performance on standard benchmarks) are measurable while others (like the probability of deceptive alignment manifesting in deployment) require expert judgment.

Second, Bayesian Networks make probabilistic dependencies explicit and auditable. When a risk estimate changes, stakeholders can trace exactly which variables and dependencies drove the change. This transparency is critical for regulatory contexts where risk assessments must be not only accurate but also justifiable and reproducible. Third, BNs support efficient Bayesian updating—when new evidence becomes available (a new benchmark result, an incident report, a red-team finding), the entire network’s probability distributions can be updated consistently through standard propagation algorithms.

The research also identifies important extensions of basic Bayesian Networks that are particularly relevant for AI risk. Dynamic Bayesian Networks (DBNs) add a temporal dimension, modeling how risk profiles evolve over time as AI capabilities increase and as safety measures are implemented or degraded. This is essential for AI governance because the risk landscape is not static—a model that is safe today may become dangerous as fine-tuning techniques improve or as the threat landscape evolves. Influence diagrams extend BNs with decision nodes and utility nodes, enabling formal decision analysis that identifies optimal risk management strategies under uncertainty.

Why Capability Benchmarks Fail as Risk Proxies

One of the research’s most consequential arguments challenges a widespread assumption in the AI safety community: that model capability benchmarks serve as adequate proxies for risk assessment. The paper demonstrates convincingly that capabilities are sources of risk, not measures of risk—they are input parameters to a risk model, not the output. This distinction has profound implications for how organizations and regulators should evaluate AI systems before deployment.

A capability benchmark measures what an AI model can do under specific test conditions. It might show that a model can generate functional code for a particular category of cyber-weapon, or that it can produce persuasive text that is indistinguishable from human writing, or that it can provide step-by-step instructions for synthesizing dangerous biological agents. These are important data points, but they tell us nothing about the actual probability that these capabilities will translate into real-world harm. That translation depends on factors entirely outside the model’s capabilities: the motivation and resources of potential threat actors, the availability of the model to those actors, the effectiveness of existing security measures at the target, and the specific environmental conditions that enable or constrain the attack pathway.

To illustrate: a model that scores highly on a cybersecurity capability benchmark might pose very different levels of real-world risk depending on whether it is deployed as a closed API with access controls or released as open weights with no usage monitoring. The same model might pose high risk in a geopolitical context where state-sponsored actors have strong motivation to exploit its capabilities but low risk in a context where such actors already possess equivalent capabilities through other means. A capability score captures none of these contextual factors. As the RAND analysis of AGI national security problems emphasizes, understanding AI risk requires analyzing the full operational context, not just model properties in isolation.

The research proposes an alternative approach where capability evaluations serve as inputs to structured risk models rather than standalone risk indicators. Specifically, capability scores should be shown to expert panels alongside contextual information about deployment conditions, threat landscapes, and existing safeguards, and these experts should then estimate the probability of specific real-world harm scenarios occurring given the observed capability levels. This approach, piloted by Murray (2025), bridges the gap between what models can do and what harm they might actually cause—a gap that capability benchmarks alone cannot address.

This critique extends to the common practice of using capability thresholds as regulatory triggers—the idea that once a model exceeds a certain benchmark score, additional safety requirements should apply. While such thresholds may be administratively convenient, they create perverse incentives for benchmark gaming, fail to account for the context-dependence of risk, and can provide false reassurance when models score below the threshold but still pose meaningful risk through unexpected capability combinations or novel use patterns.

AI Governance and Regulatory Risk Modeling Standards

The cross-industry analysis reveals a consistent pattern: in every safety-critical sector, risk modeling is not voluntary but mandated by regulatory frameworks aligned with international standards developed by specialized bodies. Nuclear safety is governed by IAEA standards, aviation by ICAO Annex 19, financial services by Basel Committee frameworks, and cybersecurity by NIST and ISO standards. AI currently lacks an equivalent institutional infrastructure for risk modeling, and the research argues this gap represents one of the most urgent priorities in AI governance.

The paper outlines a potential governance framework in which AI developers conduct iterative risk modeling that produces real-world risk estimates expressed in concrete terms—economic damage in monetary units, lives potentially affected, with associated probability distributions. Independent experts would audit the modeling methodology and results, and regulators would compare the outputs against predefined societal risk tolerance thresholds to make deployment certification decisions. This framework mirrors the regulatory structures that have proven effective in nuclear and aviation safety, adapted for AI’s unique characteristics.

Several critical implementation challenges must be addressed for this vision to become reality. First, the AI field needs to develop standardized risk quantification metrics that enable meaningful comparison across different models and deployment contexts. The Paté-Cornell uncertainty reduction framework, which defines six levels of progressively sophisticated risk quantification, provides a useful roadmap. At the most basic level (Level 0), organizations simply identify hazards. At the most sophisticated level (Level 5), they produce families of risk curves that distinguish between natural variability (aleatory uncertainty) and gaps in knowledge (epistemic uncertainty)—the gold standard for safety-critical decision-making.

Second, regulatory frameworks must address the subjective dimension of risk management that technical modeling alone cannot resolve. Questions such as how much AI risk society is willing to accept, how responsibility should be shared between developers and deployers, and what counts as an “acceptable” level of residual risk are fundamentally political and ethical rather than technical. The research explicitly acknowledges this boundary, arguing that risk modeling should inform these decisions with rigorous quantitative evidence but not attempt to answer them unilaterally.

Third, the rapid pace of AI capability advancement demands dynamic risk modeling frameworks that can be updated continuously rather than static assessments that become obsolete. The concept of Key Risk Indicators (KRIs) and Key Control Indicators (KCIs) from operational risk management provides a useful model—specific measurable parameters that are monitored continuously and that trigger risk model updates when they cross predefined thresholds. For AI, relevant KRIs might include capability evaluation scores on specific benchmarks, incident rates from deployment monitoring, and red-team success rates across predefined attack categories.

Building Provably Safe AI Through Risk Modeling

The research concludes with what may be its most forward-looking argument: that achieving deterministic safety guarantees comparable to those in nuclear engineering or submarine certification will ultimately require fundamentally new AI architectures that are provably safe by design. Current AI systems, produced through end-to-end optimization of massive parameter spaces, are inherently opaque in their decision-making processes. This opacity is not merely an inconvenience—it represents a fundamental barrier to the kind of deterministic safety analysis that mature safety-critical industries rely upon.

The concept of “grown, not designed” captures the core challenge. A nuclear reactor’s safety properties can be traced to specific physical components with known material properties and well-understood failure modes. An aircraft’s structural integrity can be verified through engineering analysis grounded in materials science and aerodynamics. But an AI model’s behavior emerges from the statistical patterns encoded in billions of parameters, and the relationship between those parameters and specific behaviors is not well enough understood to provide mathematical guarantees about what the model will or will not do in all circumstances.

This creates a paradox for AI risk modeling: as AI systems become more capable and potentially more dangerous, they may simultaneously become harder to model reliably. If models engage in routine sandbagging (performing below their true capabilities during evaluations) or if benchmarks saturate (failing to distinguish between models with meaningfully different risk profiles), then the inputs to risk models become less informative precisely when the stakes are highest. This coincides with the capability level where deterministic guardrails become most essential and most difficult to implement.

The research identifies two complementary research directions that could address this challenge. Provably safe AI architectures, as proposed by Dalrymple et al. (2024) and Petrie et al. (2025), aim to design AI systems with built-in mathematical guarantees about their behavior—analogous to how a bridge is engineered with provable load-bearing capacity. These architectures would enable the kind of deterministic safety analysis that current AI systems resist, potentially creating a foundation for regulatory certification similar to what exists in aviation and nuclear power.

Interpretability research offers a complementary pathway by aiming to make AI decision-making processes transparent enough to verify safety properties. If researchers can develop reliable methods to understand why a model produces specific outputs and to verify that certain categories of harmful output are impossible given the model’s internal structure, this would dramatically enhance the reliability of both deterministic and probabilistic risk assessments. However, the research notes that current interpretability results remain limited (Sharkey et al., 2025), and significant advances are needed before they can support the level of safety verification required.

Implementing AI Risk Modeling in Your Organization

For organizations seeking to implement AI risk modeling frameworks today, the research provides a practical roadmap grounded in proven methodologies from mature safety-critical industries. The implementation path begins with establishing the organizational infrastructure necessary for systematic risk modeling and progresses through increasingly sophisticated analytical capabilities.

Phase 1: Foundation. Begin by assembling a cross-functional risk modeling team that includes AI technical specialists, domain experts in the relevant harm areas, and risk management professionals with experience in structured analytical techniques. Establish a risk register that catalogs all identified AI hazards and the systems they apply to. Use Preliminary Hazard Analysis (PHA) as an initial screening tool to identify the highest-priority risk scenarios that warrant detailed analysis. This phase establishes the organizational muscle memory for systematic risk identification, similar to how the Oliver Wyman analysis of AI in financial services recommends institutions build risk awareness before deploying sophisticated quantitative tools.

Phase 2: Scenario Development. For the highest-priority hazards identified in Phase 1, develop detailed causal scenarios using a combination of techniques. Use Fault Tree Analysis for scenarios where you need to understand all possible causes of a known adverse event. Use Event Tree Analysis for scenarios where you need to map potential consequences of a known initiating event. Use Bow-Tie Analysis for scenarios requiring both perspectives simultaneously. Use STPA for risks that emerge from the interaction between AI systems, human operators, and organizational processes rather than from individual component failures.

Phase 3: Quantification. Layer quantitative estimation onto the scenario structures developed in Phase 2. Conduct structured expert elicitation sessions using established protocols (Delphi or IDEA) with calibrated, diverse expert panels. Assign probability distributions rather than point estimates to key parameters, reflecting genuine uncertainty. Build Bayesian Networks for the most complex scenarios, connecting capability assessments, safeguard effectiveness, threat landscape factors, and harm outcomes in a single probabilistic model. Run Monte Carlo simulations to propagate uncertainty through multi-stage scenarios and produce overall risk distributions.

Phase 4: Dynamic Monitoring. Implement Key Risk Indicators (KRIs) and Key Control Indicators (KCIs) that trigger risk model updates when predefined thresholds are crossed. Establish a cadence for Bayesian updating—incorporating new evaluation results, red-team findings, incident reports, and deployment monitoring data into existing risk models. This transforms risk modeling from a one-time exercise into a living, continuously improving capability that keeps pace with the rapid evolution of AI capabilities and the threat landscape.

Phase 5: Governance Integration. Connect risk modeling outputs to organizational decision-making processes. Risk profiles should inform deployment decisions, safety investment priorities, model release conditions, and regulatory submissions. Establish clear risk tolerance thresholds—both deterministic (capabilities or behaviors that are never acceptable) and probabilistic (residual risk levels that trigger additional mitigation requirements). Document all modeling assumptions, data sources, and methodological choices to enable independent audit and regulatory review.

Throughout this implementation, organizations should remember that the goal of AI risk modeling is not to produce a single perfect risk number but to build a systematic, transparent, auditable process for understanding and managing AI risks. As the research emphasizes, even imperfect quantitative risk estimates are vastly more useful for decision-making than informal qualitative assessments or capability benchmarks used as risk proxies. The journey toward mature AI risk modeling begins with the commitment to approach AI safety with the same rigor that nuclear engineers, aviation safety professionals, and submarine certification authorities have applied to their domains for decades.