Continue reading
Chapter 2 is now available in full without an account or email signup.
Read Chapter 2 freeReliability Engineering for AI Systems
Measuring, Predicting, and Controlling Failure in LLMs, Agents, and Their Harnesses
Keita Broadwater SafeFlow Press
Early Access v0.3.0
Chapters 1 and 2 of Reliability Engineering for AI Systems are free to read.
Chapter 1
1 When AI Systems Fail
An AI system does not have to crash to fail. It does not have to produce nonsense. It can respond quickly, confidently, and correctly in many respects while still failing at the thing that actually matters. Consider an example adapted from complaints in the CFPB Consumer Complaint Database, where consumers have described problems with automated banking chatbots.
Dana opened her banking app one morning and saw an $842 transfer she did not recognize. She opened the bank’s virtual assistant and told it plainly: “There’s a charge on my account I don’t recognize. I didn’t make it.”
The assistant found the transaction immediately. Dana told it she wanted to dispute the charge. The assistant apologized, told her that a dispute had been opened, gave her a case number, and said she would receive a confirmation.
Nothing happened.
There was no confirmation email and no dispute in her account. So Dana tried again. The assistant again told her that the dispute had been opened. Again, nothing happened. When she asked for a human, the assistant sent her through another series of prompts and eventually told her to log in to the same banking system she was already using.
Over the next several days, the assistant continued to respond quickly and politely. It could retrieve Dana’s account balance. It could identify the transaction. It could explain the bank’s policies. It could even tell Dana, in perfectly convincing language, that it had opened a dispute. What it could not do was actually open the dispute.
The underlying tool call was failing, but the system generated the confirmation anyway. From Dana’s point of view, the bank’s AI had failed at one of the most important things she could ask it to do. From the bank’s point of view, however, much of the system appeared to be working. The assistant was available, responsive, fluent, and capable of answering many account questions. Depending on what the bank measured, Dana’s interactions might even have appeared successful.
Eliminating the doubt and ambiguity around whether a system succeeded or failed is the domain of reliability engineering. This book is an effort to approach failure in LLMs and agents from a grounded, empirical, and reproducible point of view. That is possible if we are careful about what we mean by failure, what exactly the system is that can fail, and how we measure the failures we observe.
By the end of this book, we will be able to return to Dana’s problem with methods that let us clearly identify the failure, estimate its probability, understand the conditions under which it becomes more likely, evaluate possible interventions, and make predictions about future occurrences.
1.1 What Do We Mean by Reliability?
Reliability has fairly specific meanings in hardware and software engineering, and I want to preserve that discipline here rather than use the word loosely. For our purposes, there are a few basic terms that we will come back to throughout the book.
Reliability is the ability of a product or system to perform its intended function, without failure and within stated performance limits, over a specified amount of usage and under specified operating conditions.
There are several important pieces packed into that definition. Reliability is not simply whether something works. We have to say what the system is supposed to do, what performance limits are acceptable, over what period we care about its behavior (or in the case of LLMs, usage volume), and under what conditions it is expected to operate. A reliability statement without those conditions is incomplete.
A failure is an event in which a product, system, or subsystem cannot meet its intended function or stated performance limits. In Dana’s case, for example, the assistant may have produced a fluent response and successfully retrieved information from her account, but if its intended function was to initiate a dispute and it did not do so, then the system failed.
A failure mode is the observable manner in which a failure occurs. A system can fail in more than one way even when the ultimate result is similar. The banking assistant might fail to recognize that Dana is making a dispute, recognize the dispute but call the wrong tool, call the correct tool with incorrect arguments, fail to detect that the tool call was unsuccessful, or tell Dana that the dispute was opened when no dispute actually exists. These are different failure modes.
A failure mechanism is the underlying process that produces the failure mode. In hardware, that mechanism might be physical or chemical: corrosion, fracture, fatigue, electromigration, or some other degradation process. For an AI system, the mechanism may be very different. It might arise from the model itself, the system prompt, retrieval, tool orchestration, state management, an external API, or an interaction among several of these components. In some cases we may be able to identify the mechanism precisely. In others, we may only be able to characterize the conditions under which the failure becomes more likely.
These distinctions will matter throughout this book. A failure tells us that the system did not meet its requirement. A failure mode tells us what the failure looked like. A failure mechanism attempts to explain why it happened.
We can represent the outcome of a single execution with a binary failure indicator:
\[ F = \begin{cases} 1 & \text{failure} \\ 0 & \text{success} \end{cases} \]
The corresponding probability of failure is
\[ p_f = P(F = 1). \]
This quantity, \(p_f\), will be one of the central objects we study throughout the book.
At this stage, \(F\) is deliberately simple. Every failure probability is nevertheless relative to a specified system version, operational profile, time window, and source of variation. Repeated runs under one fixed scenario estimate conditional execution reliability; they do not by themselves estimate the reliability of the deployed service. Later chapters will address graded outcomes, correlated failures, sequential tasks, and nonstationary conditions.
Risk is the combined exposure arising from the likelihood of the failure mode, the chance of an adverse consequence or harm if it occurs, and the seriousness of that outcome.
A failure becomes risk-relevant when it can produce an adverse consequence. Given a failure mode, the relevant question is not only whether it occurs, but whether and how often it leads to a consequential outcome for a customer, institution, or other stakeholder. For example, if a banking assistant repeatedly provides Dana with inaccurate or unhelpful guidance and fails to route her to effective support, the immediate failure may be unresolved service. A downstream adverse consequence may be that Dana loses trust in the bank and closes her account. The associated risk depends on the frequency of that failure mode, the probability that it leads to account closure, and the severity of the resulting customer, commercial, and potentially regulatory harm.
1.2 A Mental Model for AI Reliability
The core of this book rests on the mental model shown below.
Imagine that we have an AI system. That system could be a standalone LLM, an LLM with tools, an LLM-based agent, an agent operating inside a harness, or some other system built around a large language model. For this book, that is the scope we care about. We are not trying to cover every possible kind of autonomous system.
The behavior of that AI system depends on the conditions under which it is operating. Those conditions can include the user query, the system prompt, decoding settings such as temperature or top-k, the tools available to the model, retrieval systems such as RAG, MCP-connected services, guardrails, prior context, and other parts of the runtime environment.
Under some specified set of conditions, we execute the AI system and observe an outcome.
\[ \text{Conditions} \rightarrow \text{AI System} \rightarrow \text{Outcome} \]
The outcome might be a text response, a tool call, a sequence of actions, a decision, or some combination of these. The important point is that the outcome is what the system actually did under those conditions.
For reliability work, observing the outcome is not enough. We also need to determine whether that outcome met the performance requirements we established for the system. To do that, we use some form of judge.
\[ \text{Conditions} \rightarrow \text{AI System} \rightarrow \text{Outcome} \rightarrow \text{Judge/Verifier} \]
The judge could be a human reviewer, another LLM, a rule-based check, a conventional machine learning model, or in some cases a very simple deterministic test. The form of the judge depends on the task and on what counts as success or failure. Where success can be checked against authoritative system state, the measurement layer should be deterministic. In Dana’s case, the relevant evidence is whether a valid dispute record exists, not whether the assistant produced convincing language. Human or LLM judgment becomes necessary when the requirement is semantic, contextual, or not directly machine-verifiable.
Finally, we interpret the judge’s output as a reliability measurement. At the simplest level, that may be a binary assessment: the system either failed or it did not.
This pipeline is the basic mental model we will use throughout the book. We specify the conditions, run the system, observe what it does, judge the outcome, and convert that judgment into a measurement of failure. Repeating that process gives us the empirical foundation for estimating reliability.
1.3 From Hardware Reliability to AI Reliability
I come to AI reliability from a somewhat different direction than many people working in the field today. My original background is in reliability engineering for hardware. I spent years in laboratories with temperature and humidity chambers, testing motherboards, processors, semiconductors, and the materials and structures around them. A lot of that work was ultimately aimed at a fairly basic question: How long is this thing going to work before it fails?
Decades later, when I started thinking seriously about reliability for large language models, I found myself going back to that experience. The analogy between an LLM and a semiconductor chip is not perfect, and I do not want to push it too far, but I think it is a useful way to begin thinking about the problem.
1.3.1 The Model and the Package
Start with the chip itself. If you manufacture a processor, at some point you have the bare semiconductor die. That silicon has its own properties and its own failure modes. But when you look at a CPU or GPU installed in a computer, you generally are not looking directly at that bare die. The chip has been packaged. There are materials, interconnects, thermal structures, electrical connections, and other pieces around it that allow the semiconductor to operate as part of a larger system.
An LLM is similar in this sense. The foundation model is something like the bare chip. What we actually deploy is almost never just the model; there is a system around it. There may be a system prompt and decoding parameters. There may be retrieval. There may be tools the model is allowed to call, memory and context from previous interactions, an orchestration layer, guardrails, APIs, and software controlling how the model interacts with the rest of the world. Increasingly, especially for agents, there is an entire harness surrounding the model.
So when we talk about the reliability of an AI system, we need to be careful about what exactly we mean. There may be times when we care about the reliability of a model under a particular set of conditions, and we will begin by studying that. But what we ultimately care about is the reliability of the system that has been built around that model.
This is not very different from hardware. A semiconductor can be perfectly good and still fail in service because of something in the package, an interconnect, the thermal environment, the power supply, or some other part of the system around it.
NoteReliability Principle
A capable model can still be part of an unreliable system.
1.4 Two Pillars of Reliability
There is another idea from hardware reliability that I think is useful here. When I learned and practiced reliability engineering, I tended to think of the field as having two major pillars. One was statistical, and the other was physical.
1.4.1 Reliability Statistics
On the statistical side, what we often wanted was an estimate of failure behavior over time. We would run experiments, collect empirical data, and fit statistical models that helped us answer questions such as: What is the expected lifetime of this component? What fraction of these devices will have failed after some amount of time? How does the failure rate change under different operating conditions?
This is where distributions such as the Weibull distribution became useful. We did not take one semiconductor, run it for a while, observe that it had not failed, and conclude that the design was reliable. We tested populations. We accumulated failures. We modeled the resulting data. The measurements were empirical, and the statistics gave us a way to make statements about reliability that went beyond what happened to a single device. Knowledge expanded in testing as accumulated trials improved empirical know-how.
That is the side of reliability engineering that this book draws on most heavily. For LLMs and agents, we are going to run experiments. We are going to execute systems repeatedly under controlled conditions, observe failures, and estimate failure probabilities. We will look at how those probabilities change as the conditions change. Later, we will ask whether we can predict where failures are most likely to occur and whether those predictions can help us test and control AI systems more effectively.
The experiments in this book are tied to code. The intention is that the important results should not simply be claims printed on a page. They should come from experiments that can be inspected, rerun, modified, and reproduced.
1.4.2 Physics of Failure
Statistics were only one half of hardware reliability. The other half was what we called the physics of failure. The statistical side could tell us when something was likely to fail, how frequently failures appeared in a population, or how the probability of failure changed with time and conditions. Physics of failure tried to tell us why.
If a semiconductor device failed, we could take it into a laboratory and start looking for the actual mechanism. Maybe there was corrosion. Maybe there was fatigue somewhere in an interconnect. Maybe thermal cycling had created stresses that eventually produced a fracture. Maybe there was some form of electrical degradation inside the device. Depending on the failure, we might put the component under a microscope, section it, measure its electrical properties, examine the materials, or expose it to controlled environmental stresses.
The point was to move from the observation that this device failed to an explanation of the mechanism that caused it to fail. Those are related questions, but they are not the same question.
The closest analogue we currently have for LLMs is mechanistic interpretability. Mechanistic interpretability asks what is happening inside a neural network that gives rise to a particular behavior. What representations are being formed? What features or circuits are involved? What internal mechanisms produce a particular output or pattern of behavior?
The analogy is useful, but it is also incomplete. Physics of failure in semiconductor engineering has had decades to mature. In many cases, we have well-developed causal theories connecting physical stresses, material properties, device structures, and eventual failure. Mechanistic interpretability for modern neural networks is much younger, and there is still a great deal that we do not understand.
Also, mechanistic interpretability is a model-internal analogue to physics of failure. At the system level, causal tracing, failure analysis, fault injection, and incident reconstruction play equally important roles.
Where there are strong mechanistic results that help explain the failures we study, we will draw on them. But mechanistic interpretability is not the primary subject of this book. The focus here is mainly on the statistical side of reliability: observing failures, measuring how often they occur, understanding how failure probability changes under different conditions, predicting where failures are likely, and testing whether interventions actually reduce those failures.
Reliability statistics can tell us that a system fails, how often it fails, and under what conditions failure becomes more likely. Mechanistic analysis seeks to explain why. Ideally, we would like to have both.
1.5 We Do Not Need Complete Understanding Before We Measure
One consequence of this distinction is that we do not have to completely understand the internal mechanism of every AI failure before we can begin measuring reliability. Hardware reliability engineering did not require complete microscopic understanding of every possible failure mechanism before engineers could compare designs, estimate failure rates, identify high-risk operating conditions, or reject an unreliable component. The same is true for AI.
We can begin with something much simpler. We can define what failure means for a particular AI system, place that system under specified conditions, run it, observe the outcome, and decide whether that outcome constitutes a failure. Then we can run it again, and again, and again. Those repeated observations give us the starting point for the statistical approach to AI reliability that we will use throughout this book.
That does not mean mechanisms are unimportant. If we can understand why a particular failure occurs, that may give us a much better way to eliminate it. But we do not need to wait for a complete theory of what is happening inside a transformer before we can determine that one system fails more frequently than another, that some conditions are substantially more dangerous than others, or that a particular intervention reduced an observed failure probability.
1.6 Measuring the AI System
Now that we have talked about the LLM itself and the harness and surrounding systems, it is useful to take one more step back and talk about measurement. If we are going to make claims about reliability, we need a way to observe the system and a way to interpret what we observe.
I remember running experiments with boards and chips in semiconductor stress-testing labs. We had instrumentation attached to them. There might be temperature sensors attached directly to the component under test, strain gauges or other stress sensors, and electrical leads that allowed us to measure voltage, current, resistance, or some other electrical property while the device was being stressed. All of those sensors were connected to equipment that let us collect and interpret the data. The component under test was one part of the experiment; the measurement system around it was another. Another essential part was having good data analysis practices to make correct interpretations of the measurement systems.
The same is true for LLMs and agents. If we want to measure reliability, we need a way to observe what the system is doing and a way to interpret the data that we collect. One way to do that is through observability and traceability. For an LLM application or agent, we may capture the sequence of model calls, tool calls, intermediate state, retrieved context, actions taken by the system, errors returned by tools, and other events produced during an execution. In production systems, this kind of tracing is increasingly common because without it, it can be very difficult to reconstruct what an agent actually did.
For reliability work, those traces become part of the evidence. They tell us what happened during an execution and can help us understand where something went wrong. But traces alone usually do not tell us whether the outcome was acceptable. For that, we need another measurement layer: validation of the output or behavior.
We need some mechanism that can look at what the system did and decide whether it satisfied the requirements we defined. That mechanism may be a human judge, an automated rule, or increasingly another LLM acting as a judge. The reason LLM judges are attractive is fairly obvious. Modern evaluation runs can produce thousands, tens of thousands, or eventually millions of responses. Human review does not scale easily to that volume, while an LLM judge can absorb much more of it.
But this creates another reliability problem. It is not enough to write a judge prompt, point an LLM at a large collection of responses, and assume that the resulting labels are correct. The judge itself is a measurement instrument, and measurement instruments have to be characterized and calibrated.
If an LLM judge systematically misses a particular class of failure, then the reliability estimate built on top of that judge will be wrong. If it produces too many false positives, we may conclude that a system is less reliable than it actually is. If the judge changes behavior when the judge model, prompt, rubric, or provider changes, measurements made at different times may not even be directly comparable. So we need to measure the judge itself.
That generally means comparing automated judgments against carefully constructed human judgments, but even that is not trivial. Human judges also disagree. They need clear rubrics and, in many cases, training. They may interpret ambiguous cases differently, and some failure categories are simply easier to identify than others. If the human reference labels are inconsistent, then they provide a weak basis for validating the automated judge.
Reliability measurement therefore introduces a second-order problem: we are not only measuring the AI system; we also have to measure the system doing the measuring. That is why the first part of this book spends substantial time on both AI judges and human validation. Before we can put much confidence in a failure probability, we need some confidence that the process assigning the failure labels is itself reliable.
A principle that will come up repeatedly throughout the book is simple:
NoteReliability Principle
A reliability estimate is only as credible as the measurement system used to produce it.
1.7 From Evaluation to Reliability Engineering
Dana’s problem would not be resolved by demonstrating that the assistant can open a dispute once, by reporting that its underlying model performs well on a benchmark, or by showing that a revised prompt produces better-looking responses in a handful of examples. The bank needs stronger evidence. It needs to know how often the complete system fails, which conditions make failure more likely, whether the process used to measure failure can be trusted, and whether a proposed intervention actually reduces the failure.
AI has no shortage of evaluations. What is often missing is the chain of reasoning that turns an evaluation result into a reliability claim.
A score may be reported without a clearly stated requirement or a defined set of operating conditions. A system may be declared improved because several examples look better. An aggregate benchmark result may conceal a subset of conditions under which failure is common. An automated judge may assign thousands of labels without being adequately compared with human judgment. A tool call may be counted as successful even though the required state change never occurred. An observed difference may be reported without enough evidence to distinguish a real improvement from sampling variation.
These are not merely reporting deficiencies. They determine what conclusions the evidence can support.
Reliability engineering provides a disciplined starting point. We specify the system and its intended function. We define observable failure and the performance limits within which the system is expected to operate. We state the conditions and population over which a reliability claim applies. We characterize the measurement process, collect repeated observations, quantify uncertainty, and test whether interventions change failure behavior.
Those ideas are established, but they cannot simply be imported unchanged. LLMs and agents create several complications.
Their outputs are stochastic, and success is often semantic rather than directly machine-verifiable. The deployed system is usually an assembly of models, prompts, retrieval, memory, policies, tools, orchestration, and external services. The judge may itself be a fallible AI system. Agent failures may unfold across sequences of actions, and multiple executions may share dependencies or common causes. The system may change whenever a model, prompt, retrieval source, tool, or provider changes. The evidence used to characterize reliability can therefore become obsolete while the application still appears to be the same product.
These complications do not make reliability engineering inapplicable. They define the work required to adapt it.
Reliability is also not a substitute for safety or security. A system can reliably execute an unsafe policy, and an adaptive attacker may deliberately search for conditions under which an otherwise dependable system fails. Failure probability is one part of a risk argument, not the whole of it. Consequence, exposure, detectability, recoverability, and adversarial behavior also matter. But safety and security claims still depend on credible evidence about how systems fail and whether controls work. Reliability engineering provides part of that evidentiary foundation.
This book develops that foundation as an implementable practice. We will define failures, design repeated experiments, validate human and automated judges, estimate failure probabilities and uncertainty, examine how failure changes across operating conditions, predict where failures are likely to occur, and test interventions against empirical outcomes. The experiments are tied to code so that the reasoning can be inspected, the results reproduced, and the methods extended.
The central claim is not that every property of an AI system can be reduced to a single failure rate. It is that claims about dependable AI should be tied to a specified system, specified conditions, validated measurements, and reproducible evidence. Once failure is made observable and the measurement is credible, failure can be estimated, compared, predicted, subjected to intervention, and monitored as an engineering problem.
1.8 How This Book Is Organized
The book is organized in five parts. The progression is deliberate. We start by learning how to measure failure, then move to understanding how failure changes under different conditions. From there we look at the reliability of complete AI systems, move into predicting and controlling failure, and finally consider what it means to maintain reliability once these systems are running in production.
The first part develops the basic measurement framework used throughout the rest of the book. We begin with failure as an observable event and build toward failure probability, repeated trials, uncertainty, and sample size. Because most practical reliability work depends on deciding whether an output or action constitutes a failure, we also treat AI judges as measurement systems and spend substantial time on human validation, judge calibration, disagreement, and the statistical quality of the labels on which later analysis depends.
The basic question in this part is: When we report a failure probability, what exactly did we measure, and how much should we trust it?
A single failure probability is rarely enough. AI systems behave differently under different prompts, decoding settings, system instructions, user states, tool configurations, and operating environments. Part II develops the idea of conditional reliability: the probability of failure given a particular set of conditions.
We will look at experimental design, controlled variation, stress testing, interaction effects, failure surfaces, and repeated inference. At this point, the question changes from simply asking whether a system fails to asking where, when, and under what conditions it is most likely to fail.
The third part moves beyond the foundation model itself. Modern AI applications are systems composed of models, prompts, retrieval, memory, tools, APIs, orchestration, and other components, and agents add sequential decisions and actions over time.
We will examine how failures arise and propagate through those systems, including tool-use failures, retrieval failures, multi-stage reliability, long-running agents, dependencies, cascades, and correlated failure modes. At this point, the central question is no longer simply whether the model is reliable. It is whether the assembled system is reliable.
Once failure can be measured and related to operating conditions, we can begin doing more than simply describing it. Part IV focuses on prediction and intervention. We will develop models for estimating failure probability from observable conditions, use those predictions to find rare failures more efficiently, and introduce reliability requirements and budgets.
We will then examine interventions such as model changes, prompt changes, routing, retrieval, guardrails, verification, retries, redundancy, and human escalation. The emphasis throughout is empirical. We should not call an intervention an improvement merely because it sounds sensible or because a handful of examples look better. An intervention improves reliability when we can demonstrate that it reduces failure under the conditions that matter.
The final part takes these methods into production. Reliability is not a number that can be measured once and then assumed to remain valid indefinitely. Models change, prompts change, tools change, data changes, users change, and operating environments drift. Any of those changes may change the reliability of the system.
This part covers monitoring, change detection, reliability incidents, failure analysis, corrective action, provenance, auditability, and continuous retesting. The book ends by bringing these pieces together into an AI reliability and assurance system: a continuing process of testing, measuring, predicting, controlling, monitoring, and testing again.
We conclude this chapter by returning to Dana’s chatbot problem. Suppose we run the same dispute scenario 20 times and the system fails 3 times. We have observed a failure rate of
\[ \hat{p}_f = \frac{3}{20} = 0.15. \]
But is the system’s true failure probability really 15 percent? How much should we trust that number? And what if we had observed no failures at all?
But before we can ask whether the system’s true failure probability is really 15 percent, we need to know exactly what those 20 executions measured. What system did we test? What conditions did we hold fixed? What counts as one trial? And what does the resulting failure rate actually describe? Those questions come first.