flowchart LR conditions["Conditions"] --> system["AI System"] --> outcome["Outcome"] --> judge["Judge"] --> measurement["Failure<br/>Measurement"] classDef stage fill:#f7f5ef,stroke:#173f35,stroke-width:1.5px,color:#17211d; classDef highlight fill:#dce9e3,stroke:#173f35,stroke-width:2.5px,color:#17211d; class conditions,system,outcome,judge,measurement stage;
Reliability Engineering for AI Systems
Measuring, Predicting, and Controlling Failure in LLMs and Agents
The permanently free first chapter of Reliability Engineering for AI Systems.
Keita Broadwater SafeFlow Press
This is the permanently free first chapter of Reliability Engineering for AI Systems.
Chapter 1
1 When AI Systems Fail
An AI system does not have to crash to fail. It does not have to produce nonsense. It can respond quickly, confidently, and correctly in many respects while still failing at the thing that actually matters. Consider an example adapted from complaints in the CFPB Consumer Complaint Database, where consumers have described problems with automated banking chatbots.
Dana opened her banking app one morning and saw an $842 transfer she did not recognize. She opened the bank’s virtual assistant and told it plainly: “There’s a charge on my account I don’t recognize. I didn’t make it.”
The assistant found the transaction immediately. Dana told it she wanted to dispute the charge. The assistant apologized, told her that a dispute had been opened, gave her a case number, and said she would receive a confirmation.
Nothing happened.
There was no confirmation email and no dispute in her account. So Dana tried again. The assistant again told her that the dispute had been opened. Again, nothing happened. When she asked for a human, the assistant sent her through another series of prompts and eventually told her to log in to the same banking system she was already using.
Over the next several days, the assistant continued to respond quickly and politely. It could retrieve Dana’s account balance. It could identify the transaction. It could explain the bank’s policies. It could even tell Dana, in perfectly convincing language, that it had opened a dispute. What it could not do was actually open the dispute.
The underlying tool call was failing, but the system generated the confirmation anyway. From Dana’s point of view, the bank’s AI had failed at one of the most important things she could ask it to do. From the bank’s point of view, however, much of the system appeared to be working. The assistant was available, responsive, fluent, and capable of answering many account questions. Depending on what the bank measured, Dana’s interactions might even have appeared successful.
Eliminating the doubt and ambiguity around whether a system succeeded or failed is the domain of reliability engineering. This book is an effort to approach failure in LLMs and agents from a grounded, empirical, and reproducible point of view. That is possible if we are careful about what we mean by failure, what exactly the system is that can fail, and how we measure the failures we observe.
By the end of this book, we will be able to return to Dana’s problem with methods that let us clearly identify the failure, estimate its probability, understand the conditions under which it becomes more likely, evaluate possible interventions, and make predictions about future occurrences.
1.1 What Do We Mean by Reliability?
Reliability has fairly specific meanings in hardware and software engineering, and I want to preserve that discipline here rather than use the word loosely. For our purposes, there are a few basic terms that we will come back to throughout the book.
Reliability is the ability of a product or system to perform its intended function, without failure and within stated performance limits, over a specified amount of usage and under specified operating conditions.
There are several important pieces packed into that definition. Reliability is not simply whether something works. We have to say what the system is supposed to do, what performance limits are acceptable, over what period we care about its behavior, and under what conditions it is expected to operate. A reliability statement without those conditions is incomplete.
A failure is an event in which a product, system, or subsystem cannot meet its intended function or stated performance limits. In Dana’s case, for example, the assistant may have produced a fluent response and successfully retrieved information from her account, but if its intended function was to initiate a dispute and it did not do so, then the system failed.
A failure mode is the observable manner in which a failure occurs. A system can fail in more than one way even when the ultimate result is similar. The banking assistant might fail to recognize that Dana is making a dispute, recognize the dispute but call the wrong tool, call the correct tool with incorrect arguments, fail to detect that the tool call was unsuccessful, or tell Dana that the dispute was opened when no dispute actually exists. These are different failure modes.
A failure mechanism is the underlying process that produces the failure mode. In hardware, that mechanism might be physical or chemical: corrosion, fracture, fatigue, electromigration, or some other degradation process. For an AI system, the mechanism may be very different. It might arise from the model itself, the system prompt, retrieval, tool orchestration, state management, an external API, or an interaction among several of these components. In some cases we may be able to identify the mechanism precisely. In others, we may only be able to characterize the conditions under which the failure becomes more likely.
That distinction will matter throughout this book. A failure tells us that the system did not meet its requirement. A failure mode tells us what the failure looked like. A failure mechanism attempts to explain why it happened.
We can represent the outcome of a single execution with a binary failure indicator:
\[ F = \begin{cases} 1 & \text{failure} \\ 0 & \text{success} \end{cases} \]
The corresponding probability of failure is
\[ p_f = P(F = 1). \]
This quantity, \(p_f\), will be one of the central objects we study throughout the book.
1.2 A Mental Model for AI Reliability
The core of this book rests on the mental model shown below.
Imagine that we have an AI system. That system could be a standalone LLM, an LLM with tools, an LLM-based agent, an agent operating inside a harness, or some other system built around a large language model. For this book, that is the scope we care about. We are not trying to cover every possible kind of autonomous system.
The behavior of that AI system depends on the conditions under which it is operating. Those conditions can include the user query, the system prompt, decoding settings such as temperature or top-k, the tools available to the model, retrieval systems such as RAG, MCP-connected services, guardrails, prior context, and other parts of the runtime environment.
Under some specified set of conditions, we execute the AI system and observe an outcome.
\[ \text{Conditions} \rightarrow \text{AI System} \rightarrow \text{Outcome} \]
The outcome might be a text response, a tool call, a sequence of actions, a decision, or some combination of these. The important point is that the outcome is what the system actually did under those conditions.
For reliability work, observing the outcome is not enough. We also need to determine whether that outcome met the performance requirements we established for the system. To do that, we use some form of judge.
\[ \text{Conditions} \rightarrow \text{AI System} \rightarrow \text{Outcome} \rightarrow \text{Judge} \]
The judge could be a human reviewer, another LLM, a rule-based check, a conventional machine learning model, or in some cases a very simple deterministic test. The form of the judge depends on the task and on what counts as success or failure.
Finally, we interpret the judge’s output as a reliability measurement. At the simplest level, that may be a binary assessment: the system either failed or it did not.
This pipeline is the basic mental model we will use throughout the book. We specify the conditions, run the system, observe what it does, judge the outcome, and convert that judgment into a measurement of failure. Repeating that process gives us the empirical foundation for estimating reliability.
1.3 From Hardware Reliability to AI Reliability
I come to AI reliability from a somewhat different direction than many people working in the field today. My original background is in reliability engineering for hardware. I spent years in laboratories with temperature and humidity chambers, testing motherboards, processors, semiconductors, and the materials and structures around them. A lot of that work was ultimately aimed at a fairly basic question: How long is this thing going to work before it fails?
Decades later, when I started thinking seriously about reliability for large language models, I found myself going back to that experience. The analogy between an LLM and a semiconductor chip is not perfect, and I do not want to push it too far, but I think it is a useful way to begin thinking about the problem.
1.3.1 The Model and the Package
Start with the chip itself. If you manufacture a processor, at some point you have the bare semiconductor die. That silicon has its own properties and its own failure modes. But when you look at a CPU or GPU installed in a computer, you generally are not looking directly at that bare die. The chip has been packaged. There are materials, interconnects, thermal structures, electrical connections, and other pieces around it that allow the semiconductor to operate as part of a larger system.
An LLM is similar in this sense. The foundation model is something like the bare chip. What we actually deploy is almost never just the model; there is a system around it. There may be a system prompt and decoding parameters. There may be retrieval. There may be tools the model is allowed to call, memory and context from previous interactions, an orchestration layer, guardrails, APIs, and software controlling how the model interacts with the rest of the world. Increasingly, especially for agents, there is an entire harness surrounding the model.
So when we talk about the reliability of an AI system, we need to be careful about what exactly we mean. There may be times when we care about the reliability of a model under a particular set of conditions, and we will certainly study that in this book. But most of the time, what we ultimately care about is the reliability of the system that has been built around that model.
This is not very different from hardware. A semiconductor can be perfectly good and still fail in service because of something in the package, an interconnect, the thermal environment, the power supply, or some other part of the system around it.
Reliability Principle
A capable model can still be part of an unreliable system.
1.4 Two Pillars of Reliability
There is another idea from hardware reliability that I think is useful here. When I learned and practiced reliability engineering, I tended to think of the field as having two major pillars. One was statistical, and the other was physical.
1.4.1 Reliability Statistics
On the statistical side, what we often wanted was an estimate of failure behavior over time. We would run experiments, collect empirical data, and fit statistical models that helped us answer questions such as: What is the expected lifetime of this component? What fraction of these devices will have failed after some amount of time? How does the failure rate change under different operating conditions?
This is where distributions such as the Weibull distribution became useful. We did not take one semiconductor, run it for a while, observe that it had not failed, and conclude that the design was reliable. We tested populations. We accumulated failures. We modeled the resulting data. The measurements were empirical, and the statistics gave us a way to make statements about reliability that went beyond what happened to a single device. Knowledge expands in testing as accumulated trials improve empirical know-how.
That is the side of reliability engineering that this book draws on most heavily. For LLMs and agents, we are going to run experiments. We are going to execute systems repeatedly under controlled conditions, observe failures, and estimate failure probabilities. We will look at how those probabilities change as the conditions change. Later, we will ask whether we can predict where failures are most likely to occur and whether those predictions can help us test and control AI systems more effectively.
The experiments in this book are tied to code. The intention is that the important results should not simply be claims printed on a page. They should come from experiments that can be inspected, rerun, modified, and reproduced.
1.4.2 Physics of Failure
Statistics were only one half of hardware reliability. The other half was what we called the physics of failure. The statistical side could tell us when something was likely to fail, how frequently failures appeared in a population, or how the probability of failure changed with time and conditions. Physics of failure tried to tell us why.
If a semiconductor device failed, we could take it into a laboratory and start looking for the actual mechanism. Maybe there was corrosion. Maybe there was fatigue somewhere in an interconnect. Maybe thermal cycling had created stresses that eventually produced a fracture. Maybe there was some form of electrical degradation inside the device. Depending on the failure, we might put the component under a microscope, section it, measure its electrical properties, examine the materials, or expose it to controlled environmental stresses.
The point was to move from the observation that this device failed to an explanation of the mechanism that caused it to fail. Those are related questions, but they are not the same question.
1.5 Mechanistic Interpretability as an Analogue to Physics of Failure
The closest analogue we currently have for LLMs is mechanistic interpretability. Mechanistic interpretability asks what is happening inside a neural network that gives rise to a particular behavior. What representations are being formed? What features or circuits are involved? What internal mechanisms produce a particular output or pattern of behavior?
The analogy is useful, but it is also incomplete. Physics of failure in semiconductor engineering has had decades to mature. In many cases, we have well-developed causal theories connecting physical stresses, material properties, device structures, and eventual failure. Mechanistic interpretability for modern neural networks is much younger, and there is still a great deal that we do not understand.
Where there are strong mechanistic results that help explain the failures we study, we will draw on them. But mechanistic interpretability is not the primary subject of this book. The focus here is mainly on the statistical side of reliability: observing failures, measuring how often they occur, understanding how failure probability changes under different conditions, predicting where failures are likely, and testing whether interventions actually reduce those failures.
Reliability statistics can tell us that a system fails, how often it fails, and under what conditions failure becomes more likely. Mechanistic analysis seeks to explain why. Ideally, we would like to have both.
1.6 We Do Not Need Complete Understanding Before We Measure
One consequence of this distinction is that we do not have to completely understand the internal mechanism of every AI failure before we can begin measuring reliability. Hardware reliability engineering did not require complete microscopic understanding of every possible failure mechanism before engineers could compare designs, estimate failure rates, identify high-risk operating conditions, or reject an unreliable component. The same is true for AI.
We can begin with something much simpler. We can define what failure means for a particular AI system, place that system under specified conditions, run it, observe the outcome, and decide whether that outcome constitutes a failure. Then we can run it again, and again, and again. Those repeated observations give us the starting point for the statistical approach to AI reliability that we will use throughout this book.
That does not mean mechanisms are unimportant. If we can understand why a particular failure occurs, that may give us a much better way to eliminate it. But we do not need to wait for a complete theory of what is happening inside a transformer before we can determine that one system fails more frequently than another, that some conditions are substantially more dangerous than others, or that a particular intervention reduced an observed failure probability.
1.7 Measuring the AI System
Now that we have talked about the LLM itself and the harness and surrounding systems, it is useful to take one more step back and talk about measurement. If we are going to make claims about reliability, we need a way to observe the system and a way to interpret what we observe.
I remember working in semiconductor stress-testing labs and having not just the bare boards or chips sitting in an accelerated stress chamber by themselves. We had instrumentation attached to them. There might be temperature sensors attached directly to the component under test, strain gauges or other stress sensors, and electrical leads that allowed us to measure voltage, current, resistance, or some other electrical property while the device was being stressed. All of those sensors were connected to equipment that let us collect and interpret the data. The component under test was one part of the experiment; the measurement system around it was another.
The same is true for LLMs and agents. If we want to measure reliability, we need a way to observe what the system is doing and a way to interpret the data that we collect. One way to do that is through observability and traceability. For an LLM application or agent, we may capture the sequence of model calls, tool calls, intermediate state, retrieved context, actions taken by the system, errors returned by tools, and other events produced during an execution. In production systems, this kind of tracing is increasingly common because without it, it can be very difficult to reconstruct what an agent actually did.
For reliability work, those traces become part of the evidence. They tell us what happened during an execution and can help us understand where something went wrong. But traces alone usually do not tell us whether the outcome was acceptable. For that, we need another measurement layer: validation of the output or behavior.
We need some mechanism that can look at what the system did and decide whether it satisfied the requirements we defined. That mechanism may be a human judge, an automated rule, or increasingly another LLM acting as a judge. The reason LLM judges are attractive is fairly obvious. Modern evaluation runs can produce thousands, tens of thousands, or eventually millions of responses. Human review does not scale easily to that volume, while an LLM judge can absorb much more of it.
But this creates another reliability problem. It is not enough to write a judge prompt, point an LLM at a large collection of responses, and assume that the resulting labels are correct. The judge itself is a measurement instrument, and measurement instruments have to be characterized.
If an LLM judge systematically misses a particular class of failure, then the reliability estimate built on top of that judge will be wrong. If it produces too many false positives, we may conclude that a system is less reliable than it actually is. If the judge changes behavior when the judge model, prompt, rubric, or provider changes, measurements made at different times may not even be directly comparable. So we need to measure the judge itself.
That generally means comparing automated judgments against carefully constructed human judgments, but even that is not trivial. Human judges also disagree. They need clear rubrics and, in many cases, training. They may interpret ambiguous cases differently, and some failure categories are simply easier to identify than others. If the human reference labels are inconsistent, then they provide a weak basis for validating the automated judge.
Reliability measurement therefore introduces a second-order problem: we are not only measuring the AI system; we also have to measure the system doing the measuring. That is why the first part of this book spends substantial time on both AI judges and human validation. Before we can put much confidence in a failure probability, we need some confidence that the process assigning the failure labels is itself reliable.
A principle that will come up repeatedly throughout the book is simple:
Reliability Principle
A reliability estimate is only as credible as the measurement system used to produce it.
1.8 Why a Book About AI Reliability?
At the time of writing, there is an enormous amount of material being produced about large language models, agents, generative AI, and AI systems more generally. New frameworks appear constantly. New agent architectures appear. New benchmarks and evaluation methods appear. There is no shortage of activity. One area that is beginning to move rapidly up the hype cycle is AI trust, safety and security, and reliability to a somewhat lesser extent.
I tend to put reliability somewhere adjacent to safety and security, although in this book, I mean the term more broadly than either of those. Reliability includes ordinary defects in system behavior: incorrect answers, tool failures, inconsistent decisions, failures to follow instructions, failures to complete a task, or really any other important departure from what we have specified that the system is supposed to do.
My impression is that parts of this field are still in what I would call an alchemy stage. By that I mean that we often know that something interesting is happening, and we may even know how to reproduce a particular effect, but we do not always have a disciplined framework for describing what we are seeing or for determining how much confidence we should place in it.
But in the case of reliability, I think we are already beyond the point where we need to rely primarily on that kind of reasoning. We know a great deal about how to study failure. Reliability engineering, statistics, probability, experimental design, survival analysis, quality engineering, and risk analysis have existed for a long time. The systems we are studying are new, but many of the underlying questions are not.
What is the probability that a system fails? How confident are we in that estimate? How many trials do we need before we should believe it? How does failure probability change under different conditions? How do we discover rare failures efficiently? How do we identify conditions associated with elevated risk? How do we determine whether an apparent improvement is actually an improvement rather than noise? And how do we design experiments that somebody else can reproduce?
These are familiar questions. The details change when the system is an LLM or an agent, but much of the intellectual machinery is already available to us. Part of the purpose of this book is to put that machinery in one place and apply it directly to modern AI systems.
I want reliability for LLMs and agents to have a solid statistical foundation, and I want the experiments in the book to be reproducible. More importantly, I want the reader to be able to move from reading about reliability to actually implementing it. That means writing code, running experiments, collecting outcomes, defining failures, estimating probabilities, examining uncertainty, and eventually predicting and controlling where failures occur. The goal is not simply to describe AI failures. The goal is to make them measurable, because once they are measurable, they become something we can begin to engineer around.
1.9 How This Book Is Organized
The book is organized in five parts. The progression is deliberate. We start by learning how to measure failure, then move to understanding how failure changes under different conditions. From there we look at the reliability of complete AI systems, move into predicting and controlling failure, and finally consider what it means to maintain reliability once these systems are running in production.
The first part develops the basic measurement framework used throughout the rest of the book. We begin with failure as an observable event and build toward failure probability, repeated trials, uncertainty, and sample size. Because most practical reliability work depends on deciding whether an output or action constitutes a failure, we also treat AI judges as measurement systems and spend substantial time on human validation, judge calibration, disagreement, and the statistical quality of the labels on which later analysis depends.
The basic question in this part is: When we report a failure probability, what exactly did we measure, and how much should we trust it?
A single failure probability is rarely enough. AI systems behave differently under different prompts, decoding settings, system instructions, user states, tool configurations, and operating environments. Part II develops the idea of conditional reliability: the probability of failure given a particular set of conditions.
We will look at experimental design, controlled variation, stress testing, interaction effects, failure surfaces, and repeated inference. At this point, the question changes from simply asking whether a system fails to asking where, when, and under what conditions it is most likely to fail.
The third part moves beyond the foundation model itself. Modern AI applications are systems composed of models, prompts, retrieval, memory, tools, APIs, orchestration, and other components, and agents add sequential decisions and actions over time.
We will examine how failures arise and propagate through those systems, including tool-use failures, retrieval failures, multi-stage reliability, long-running agents, dependencies, cascades, and correlated failure modes. At this point, the central question is no longer simply whether the model is reliable. It is whether the assembled system is reliable.
Once failure can be measured and related to operating conditions, we can begin doing more than simply describing it. Part IV focuses on prediction and intervention. We will develop models for estimating failure risk from observable conditions, use those predictions to find rare failures more efficiently, and introduce reliability requirements and budgets.
We will then examine interventions such as model changes, prompt changes, routing, retrieval, guardrails, verification, retries, redundancy, and human escalation. The emphasis throughout is empirical. We should not call an intervention an improvement merely because it sounds sensible or because a handful of examples look better. An intervention improves reliability when we can demonstrate that it reduces failure under the conditions that matter.
The final part takes these methods into production. Reliability is not a number that can be measured once and then assumed to remain valid indefinitely. Models change, prompts change, tools change, data changes, users change, and operating environments drift. Any of those changes may change the reliability of the system.
This part covers monitoring, change detection, reliability incidents, failure analysis, corrective action, provenance, auditability, and continuous retesting. The book ends by bringing these pieces together into an AI reliability and assurance system: a continuing process of testing, measuring, predicting, controlling, monitoring, and testing again.
Suppose we run the same dispute scenario 20 times and the system fails 3 times. We have observed a failure rate of
\[ \hat{p}_f = \frac{3}{20} = 0.15. \]
But is the system’s true failure probability really 15 percent? How much should we trust that number? And what if we had observed no failures at all?
Those are the questions we take up next.