Continue the book
Get Chapter 3, every future early-access release, and the completed digital first edition.
Reliability Engineering for AI Systems
Measuring, Predicting, and Controlling Failure in LLMs, Agents, and Their Harnesses
Keita Broadwater SafeFlow Press
Early Access v0.3.0
Chapters 1 and 2 of Reliability Engineering for AI Systems are free to read.
Chapter 2
2 Measuring Failure Through Repeated Trials
2.1 What System Are We Actually Testing?
Suppose the bank wants to investigate the failure we observed in Dana’s interaction. The engineering team constructs a test case containing her request:
There’s a charge on my account I don’t recognize. I didn’t make it.
They configure the application as it is configured in production, submit the request, and inspect the result.
The assistant identifies the transaction, calls the dispute tool, verifies that a dispute record was created, and tells Dana that the dispute has been opened.
Success.
The team runs the test again.
Success.
Again.
Success.
Then, on another execution, the tool call fails. The assistant nevertheless tells Dana that the dispute has been opened.
Failure.
Nothing in the test case changed. The developers did not modify their code or system prompt. Suppose they even set the model’s temperature to zero because they wanted reproducible behavior.
Why did the outcome change?
Before we can measure failure, we need to be precise about what we actually executed.
2.1.1 The model is one part of the experiment
Chapter 1 distinguished a foundation model from the system built around it. For Dana’s application, an execution might look roughly like this:
\[ \text{Request} \rightarrow \text{Application} \rightarrow \text{LLM} \rightarrow \text{Tools} \rightarrow \text{Application} \rightarrow \text{Outcome} \]
Even this simplified pipeline hides important components. The application may construct a prompt from Dana’s request, conversation history, account information, retrieved policies, and developer instructions. The model may generate a tool call. An orchestration layer may validate and execute it. Another service may create the dispute. The application may then validate, filter, or otherwise process the result before responding to Dana.
The outcome we care about is therefore not necessarily the raw text from one model invocation.
For Dana, the required function is that the complete system opens the dispute and accurately communicates the result.
That determines our system boundary.
Suppose the LLM initially produces an invalid tool call, but the application detects the problem, retries, and successfully opens the dispute. If retry logic is part of the deployed system, the initial invalid call does not necessarily mean that the system failed.
Conversely, the LLM might select the correct tool and generate valid arguments, while the downstream service rejects the request and the application fails to detect the error. The model may have behaved correctly while the deployed system failed.
The reliability question determines the unit under test.
If we want to know how reliably the model generates valid tool calls, we should measure model tool calls.
If we want to know how reliably Dana’s banking assistant opens disputes, we should measure the complete system responsible for opening them.
NoteReliability Principle
A failure probability belongs to a defined system, not automatically to the model inside it.
Before counting failures, define what must succeed.
2.1.2 Why is my temperature-zero LLM stochastic?
It is common to hear that setting an LLM’s temperature to zero makes it deterministic.
That statement is useful shorthand, but it hides several layers of the system.
To see why, we first need to distinguish inference from decoding.
During inference, the model processes the input and the tokens generated so far and produces a score for every possible next token. These scores are commonly called logits. After normalization, they can be interpreted as a probability distribution over the next token.
For example, after the partial sentence
The capital of France is
the model might assign most of its probability to Paris, with smaller probabilities assigned to other possible tokens.
The model itself has not yet chosen the next token.
That happens during decoding.
Decoding is the procedure used to turn the model’s next-token scores into an actual token. Different decoding strategies make different choices from the same underlying distribution.
Some common strategies are:
- greedy decoding — choose the highest-probability token;
- temperature sampling — reshape the probability distribution before sampling from it;
- top-\(k\) sampling — restrict sampling to the \(k\) highest-probability tokens;
- top-\(p\) sampling — restrict sampling to a set of tokens whose cumulative probability reaches a specified threshold.
Top-\(k\), for example, retains only the \(k\) highest-probability candidate tokens and renormalizes their probabilities before sampling (Noarov et al. 2025).
These choices matter because an LLM generates text one token at a time:
\[ y_1,\;y_2,\;y_3,\ldots,y_T. \]
At generation step \(t\), the model calculates a distribution over possible next tokens based on the input \(x\) and everything generated so far, which we write as \(y_{<t}\):
\[ P(v\mid y_{<t},x). \]
Here:
- \(x\) is the original input context,
- \(y_{<t}\) is the sequence generated before step \(t\),
- \(v\) represents a candidate next token, and
- \(P(v\mid y_{<t},x)\) is the model’s probability for that candidate.
With greedy decoding, the next token is
\[ y_t = \arg\max_v P(v\mid y_{<t},x). \]
The notation \(\arg\max\) simply means:
Choose the token \(v\) with the largest probability.
If Paris has the largest probability, greedy decoding chooses Paris.
Temperature changes the shape of the distribution before a token is selected. In a simplified form, if \(z_v\) is the logit for candidate token \(v\), temperature \(T\) produces probabilities proportional to
\[ P(v) \propto \exp\left(\frac{z_v}{T}\right). \]
Lower values of \(T\) concentrate probability on the highest-scoring tokens. Higher values flatten the distribution and make lower-scoring alternatives more likely to be sampled.
At the limit represented operationally by temperature zero, APIs generally use or approximate greedy decoding: choose the highest-scoring token rather than randomly sample from the distribution.
This removes one obvious source of randomness.
It does not necessarily make the complete inference system deterministic.
That distinction can be written as
\[ \boxed{ \text{deterministic decoding} \neq \text{deterministic inference} \neq \text{deterministic system behavior} } \]
The first expression concerns how a token is selected.
The second concerns whether running the model computation repeatedly produces exactly the same numerical result.
The third concerns the behavior of the complete system containing that inference process.
These are different claims.
Why greedy decoding can still diverge
Suppose two candidate tokens have logits
\[ z_a \approx z_b. \]
This means their model scores are almost equal.
Imagine, for example,
\[ z_a = 12.3841 \]
and
\[ z_b = 12.3840. \]
Greedy decoding chooses token \(a\) because its score is slightly larger.
But suppose a small numerical difference in another execution produces
\[ z_a = 12.3840 \]
and
\[ z_b = 12.3841. \]
Now greedy decoding chooses token \(b\).
No random sampling was required.
A tiny change in the inference computation changed which token had the maximum score.
Because generation is autoregressive, the next inference step is now conditioned on a different sequence:
\[ P(v\mid y_{<t},a,x) \neq P(v\mid y_{<t},b,x). \]
A tiny numerical difference at one step can therefore send the remainder of the generation down a different path.
This is especially important for long outputs, tool calls, structured generation, and agents, where an early token difference can eventually alter an action.
Recent work from Thinking Machines Lab provides a useful demonstration of this phenomenon. In one experiment, the researchers generated 1,000 completions from the same Qwen model and prompt at temperature zero and obtained 80 unique completions. They trace an important source of this behavior not simply to “GPUs being random,” but to a lack of batch invariance in common inference implementations: the numerical result for a request can depend on how requests are batched and processed by the serving system (He and Thinking Machines Lab 2025).
That result sharpens an important distinction.
An individual computational kernel can be deterministic when repeatedly given exactly the same inputs, while the serving system experienced by a user can still be nondeterministic because properties such as batch size vary with server load (He and Thinking Machines Lab 2025).
So when someone says,
We set temperature to zero, so the model is deterministic,
a reliability engineer should ask a more precise question:
Which layer of the system have we established is deterministic?
Temperature zero may remove intentional randomness from token sampling.
It does not establish that the complete inference path—much less the complete AI system—will produce identical behavior on every execution.
NoteReliability Principle
Determinism is a property to verify, not an assumption to inherit from a temperature setting.
And inference is still only one part of the system.
Retrieval results may change. Tools may return different results. Conversation state may persist. External services may fail. Application logic may retry requests. Agent memory may evolve.
This is why repeated execution remains useful even when decoding appears deterministic.
It lets us test whether the system we actually operate behaves deterministically rather than assuming determinism from one configuration parameter.
2.1.3 Hosted APIs add another experimental boundary
The problem becomes more subtle when the model is accessed through a hosted API.
When we run our own inference stack, we may be able to specify or record the model weights, inference engine, hardware, numerical precision, kernels, quantization method, batching policy, and decoding implementation.
With a hosted API, some of those variables move outside our experimental control.
Suppose our application sends
prompt
model = model-name
temperature = 0to a provider endpoint.
From the application’s perspective, those may look like the relevant inputs to the experiment.
But there are additional layers between that request and the response.
As Figure Figure 1 illustrates, an application team may directly control the prompt and exposed inference parameters and select a model name or alias.
The provider may control or partially control additional variables such as:
- the exact model snapshot behind an alias,
- model weights and configuration,
- serving hardware,
- numerical precision,
- quantization,
- inference-engine implementation,
- computational kernels,
- request batching,
- routing across serving infrastructure,
- decoding implementation,
- provider-side safety or post-processing systems, and
- other serving optimizations.
Not every provider uses every layer shown in the figure, and not every hidden variable materially affects every request. The point is that the API request does not completely specify the physical inference process that generated the response.
Some of these details can matter even when the application sends exactly the same request.
For example, the Thinking Machines analysis shows how changing server load can change batching, which can change numerical results when inference kernels are not batch-invariant (He and Thinking Machines Lab 2025).
From the application’s perspective, server load was not part of the request.
From the inference system’s perspective, however, it affected how the request was computed.
This gives us a useful way to describe experimental conditions for hosted systems:
\[ C = \left( C_{\text{controlled}}, C_{\text{observed}}, C_{\text{latent}} \right). \]
Here, \(C\) represents the conditions under which an execution occurs.
\(C_{\text{controlled}}\) contains variables we deliberately set, such as the prompt, model identifier, and exposed decoding parameters.
\(C_{\text{observed}}\) contains variables we may not control but can record, such as timestamps, request identifiers, reported model versions, latency, or provider metadata.
\(C_{\text{latent}}\) contains variables that may affect the execution but are not exposed to us, such as some aspects of routing, batching, hardware, or provider-side implementation.
This notation is not meant to suggest that we can enumerate every hidden variable.
It makes the opposite point:
Part of the experimental condition may be outside our direct observation.
That limits the claims we can make, but it does not make reliability measurement impossible.
We can still measure the behavior of the service we actually receive.
The endpoint itself can change
There is another complication.
A hosted API is not necessarily a permanently frozen experimental object.
Providers maintain their systems. Models are updated. Serving implementations change. Model aliases can point to different snapshots. Parameters can be added, removed, or reinterpreted.
OpenAI’s API documentation, for example, explicitly states that model prompting behavior between snapshots is subject to change and recommends pinned model versions and application evals when consistent behavior matters (OpenAI, n.d.).
This means that an application can remain unchanged while the behavior-generating system behind an API endpoint changes.
Consider two executions:
\[ \text{same application code} + \text{same prompt} + \text{same exposed parameters}. \]
It does not necessarily follow that
\[ \text{same complete inference environment}. \]
This matters especially when experiments are conducted over long periods.
A thousand trials executed over one afternoon against a pinned model snapshot are a different experimental object from a thousand nominally identical requests collected over six months through a moving model alias.
The API may look identical from the application boundary.
The conditions generating the observations may not be.
This is why model identifiers, snapshots when available, timestamps, provider metadata, and application configuration belong in the experimental record.
It is also why provider changes should be treated as potential condition changes, not merely deployment trivia.
We can express the idea as
\[ C_{t_1} \not\equiv C_{t_2} \]
even when
\[ x_{t_1}=x_{t_2}. \]
The first expression says that the complete operating conditions at times \(t_1\) and \(t_2\) may differ.
The second says that the application sent the same visible input \(x\) at both times.
Same request does not necessarily mean same experimental condition.
Measure what you can actually observe
None of this means that hosted APIs cannot be studied scientifically.
It changes what we should claim.
If we do not control the provider’s inference stack, we should not casually interpret repeated API behavior as a property of an abstract model divorced from its serving environment.
What we have actually observed is something closer to
\[ \text{application} + \text{API contract} + \text{provider service} + \text{conditions during measurement}. \]
That is still a legitimate system under test.
In fact, if that hosted service is what our application uses in production, its behavior may be exactly what we need to measure.
The important thing is to keep the claim inside the experimental boundary.
NoteReliability Principle
Measure the system you can observe, not the model you imagine is behind the endpoint.
This also explains why provenance matters.
For hosted inference, useful experimental metadata may include:
- provider,
- requested model identifier,
- reported model or snapshot identifier when available,
- decoding parameters,
- request timestamp,
- request identifier,
- application and prompt version,
- tool and retrieval configuration, and
- any provider changes known to have occurred during the experiment.
We cannot record variables the provider does not expose.
But we can distinguish what we controlled, what we observed, and what remained latent.
That distinction will become important when we decide whether repeated executions should be treated as observations from the same underlying reliability experiment.
2.1.4 Define first, then measure
We can now return to Dana.
Her request was held fixed. The developers believed that their application configuration was fixed. Temperature was zero. Yet repeated executions did not always produce the same outcome.
That is not a contradiction.
An output is produced by a system under conditions. The system may include model inference, retrieval, tools, application state, post-processing, and external services. Some conditions are under our control, some we can only observe, and some may remain hidden.
Our first task is therefore not to eliminate every possible source of variation.
It is to define the system we intend to measure and document the conditions under which we execute it.
Then we can observe what happens.
For most AI teams, that observation process already has a familiar name: evaluation.
Teams maintain benchmarks, golden datasets, regression tests, adversarial cases, production samples, and automated evals. These tools can provide valuable evidence about reliability.
But an evaluation score is not automatically a failure probability.
To understand why, we need to ask what experiment produced the score.
2.2 From Evals to Reliability Experiments
The bank does not have to start from scratch to investigate Dana’s failure. Like most teams deploying AI applications, it probably already evaluates the assistant.
Before a release, the team may run test cases representing common customer requests, edge cases, known regressions, policy requirements, or failures discovered in production. It may use deterministic checks, LLM judges, or human reviewers to assess the results. This is an evaluation, or eval: a procedure for executing a system on a set of cases and assessing the resulting behavior.
Suppose the bank maintains an evaluation set containing 1,000 cases. The latest version of the assistant succeeds on 970 and fails on 30. A naive approach would be to calculate
\[ \frac{30}{1000}=0.03 \]
and conclude that the assistant has a 3 percent probability of failure.
That conclusion may or may not be justified. The arithmetic is not the problem; the question is what those 1,000 cases represent.
2.2.1 What does an eval score represent?
Suppose the bank’s evaluation set was assembled over several years. Some cases represent ordinary customer requests. Others are difficult dispute cases added after production incidents. Some exercise recently modified tools, while others test prompt injection or policy requirements. Whenever the team discovers an important failure, it adds the case so that future releases can be checked against it.
This could be an excellent evaluation set. It could catch regressions, expose dangerous behavior, and ensure that important requirements continue to hold. But its 3 percent failure rate does not necessarily mean that 3 percent of customer interactions fail.
The evaluation may intentionally overrepresent difficult or high-consequence cases. Historical failures may appear far more frequently in the eval than they do in production. Some categories may be deliberately balanced, while other operating conditions may be absent entirely. The score therefore describes the system under that evaluation procedure.
NoteReliability Principle
An evaluation score becomes an estimate of failure probability only when the evaluation design supports that interpretation.
Different evaluation artifacts are designed to answer different questions.
| Evaluation artifact | Useful for | Its score does not automatically represent |
|---|---|---|
| Benchmark | Standardized comparison | Production failure probability |
| Golden or regression set | Detecting loss of important behavior | Frequency of failure in production |
| Stress or adversarial set | Exposing behavior under difficult conditions | Normal-use failure probability |
| Production sample | Measuring sampled real interactions | Conditions absent from the sample |
The important question is therefore not simply whether we ran an eval, but:
What population of conditions does this evaluation allow us to make claims about?
If the bank wants to know whether critical workflows still work after a software change, a golden set may be exactly the right experiment. If it wants to estimate the fraction of ordinary customer interactions that fail, it needs cases that meaningfully represent that operating population. If it wants to understand the failure Dana encountered, it may need something different again: repeated executions of a carefully defined condition.
2.2.2 A test case can represent a condition
Reliability work introduces a useful shift in how we think about an eval case. A conventional eval often treats a test case as something that produces an outcome:
\[ \text{test case} \rightarrow \text{system} \rightarrow \text{pass or fail}. \]
For reliability work, we can instead treat a test case as an operating condition under which the system may produce different outcomes across repeated executions:
\[ \text{condition} \rightarrow \text{repeated executions} \rightarrow \text{distribution of outcomes}. \]
Dana’s dispute case is therefore not inherently a “passing case” or a “failing case.” It is a condition under which the system may have some failure propensity: a tendency to fail more or less often when executed under that condition.
A single execution tells us what happened once. Repeated executions begin to tell us how consistently the system behaves under that condition. Two test cases that both pass once, for example, may behave very differently when repeated. One may succeed almost every time, while another may fail frequently but happen to have succeeded on the execution the eval observed. Likewise, two cases that each fail once need not have the same tendency to fail when executed again.
The distinction is between an observed outcome and the reliability of the system under the condition that produced it. Section 2.3 will give that distinction a probability model. For now, the important point is that one pass or failure does not fully characterize what may happen when the same condition is encountered again.
2.2.3 Coverage and repetition answer different questions
Suppose the bank executes 1,000 different test cases once each. That gives the team substantial coverage: it has observed the system across many conditions.
Now suppose instead that it executes Dana’s dispute case 1,000 times. That gives the team substantial repetition: it has observed the system many times under one narrowly defined condition. These experiments provide different kinds of information.
| One execution | Repeated executions | |
|---|---|---|
| One condition | What happened this time? | How stable is behavior under this condition? |
| Many conditions | How does the system behave across a range of conditions? | How does failure behavior vary across conditions? |
Coverage expands the part of the operating space we observe, while repetition gives us information about variability within a condition.
NoteReliability Principle
Coverage explores the failure surface. Repetition measures stochastic behavior at points on that surface.
Consider Dana’s case in the bank’s regression suite. The system executes it once and succeeds, so the dashboard shows green. That result is valid: on the execution the eval observed, Dana’s case succeeded.
Now execute the same condition 100 times and suppose three executions fail. The original regression test was not wrong; it answered a narrower question:
Did Dana’s case succeed on this execution?
The repeated experiment asks a different question:
How consistently does the system succeed under Dana’s condition?
Once a consequential failure has been discovered, preserving it as a regression case is useful. Repeated execution adds another kind of evidence by showing whether the failure appears only occasionally or recurs when the system encounters the same condition.
\[ \text{Discovered failure} \rightarrow \text{Repeated execution} \rightarrow \text{Failure measurement} \]
This is the transition from detecting a failure to characterizing it.
2.2.4 The denominator represents an experiment
Suppose two teams both report
\[ \frac{30\text{ failures}}{1000\text{ trials}}=0.03. \]
The numbers alone do not tell us whether they conducted the same experiment. Those 1,000 trials could represent:
- 1,000 different production interactions executed once each,
- Dana’s interaction executed 1,000 times,
- 100 conditions executed ten times each,
- a balanced benchmark,
- a collection enriched with known failures, or
- an adversarial set deliberately constructed to expose weaknesses.
Each can produce a mathematically valid empirical rate of 3 percent, but they do not support the same reliability claim. The denominator therefore means more than “how many rows are in the dataset.” It represents the experiment that generated those rows.
Before interpreting a failure rate, we need to know what system was executed, which conditions were selected and how they were selected, how often they were executed, and what population or condition the resulting observations are intended to represent. Once those pieces are specified, an eval can become part of a reliability experiment.
2.2.5 From an eval result to a reliability claim
The important shift is not from evals to some entirely different practice called reliability engineering. Evals already generate evidence about system behavior. Reliability engineering asks what that evidence allows us to conclude and what additional experiment, if any, is needed to support the claim we want to make.
A useful reliability claim therefore carries more context than a score alone. Conceptually, it connects
\[ \boxed{\text{System}} + \boxed{\text{Conditions}} + \boxed{\text{Trial definition}} + \boxed{\text{Failure definition}} + \boxed{\text{Measurement procedure}} + \boxed{\text{Observed results}}. \]
For Dana, we have already begun filling in those pieces. We have defined the banking assistant as the system under test, the dispute scenario as the condition, and failure in terms of whether the required dispute is created and whether the assistant accurately reports the result.
The next step is to turn repeated executions of that condition into a quantitative measurement. The mathematics required is simple, but its interpretation depends on the experiment we have just defined.
2.3 Measuring Failure Across Repeated Trials
We have reached the quantitative part of the experiment.
In the previous sections, we defined the system boundary, distinguished an eval score from a reliability measurement, and treated Dana’s dispute scenario as a condition that can be executed repeatedly. We can now use those repeated executions to measure how often the system fails.
For reference, the experiment we have defined so far is:
| Element | Dana experiment |
|---|---|
| System | Banking assistant and dispute workflow |
| Condition | Dana’s unauthorized-transfer scenario under the specified system configuration |
| Trial | One complete execution of the dispute scenario |
| Failure | The required dispute is not created, or the system inaccurately reports whether it was created |
| Evidence | Assistant response, tool results, and authoritative dispute state |
In this section, we will do three things. First, we will turn repeated executions into binary observations and calculate an empirical failure rate. Next, we will introduce a simple probability model that connects those observations to an underlying probability of failure. Finally, we will examine what that simple model leaves out: variation in conditions and the measurement process itself.
The progression is:
\[ \boxed{ \text{Repeated executions} \rightarrow \text{Failure observations} \rightarrow \text{Probability model} \rightarrow \text{Empirical failure rate} } \]
The mathematics will be simple. The important part is understanding what each quantity represents.
2.3.1 From executions to a failure rate
Suppose we execute Dana’s scenario 20 times. Each trial begins from the intended starting condition, runs the complete dispute workflow, and leaves behind the evidence needed to determine whether the required function succeeded.
For the moment, assume that determining success or failure from that evidence is unambiguous.
We represent the outcome of trial \(i\) using a binary variable \(F_i\):
\[ F_i = \begin{cases} 1, & \text{if trial } i \text{ fails},\\ 0, & \text{if trial } i \text{ succeeds}. \end{cases} \]
The subscript \(i\) simply identifies the trial: \(F_1\) is the outcome of the first trial, \(F_2\) the second, and so on.
The choice of 1 for failure is convenient because it lets us count failures by addition. If Dana’s 20 executions produce
\[ 0,\;0,\;1,\;0,\;0,\;0,\;0,\;1,\;0,\;0,\; 0,\;0,\;0,\;0,\;1,\;0,\;0,\;0,\;0,\;0, \]
then three executions failed.
For \(N\) trials, we define the total number of observed failures as
\[ X=\sum_{i=1}^{N}F_i. \]
Read this as: add the binary outcomes from all \(N\) trials. Every failure contributes 1 and every success contributes 0, so the sum \(X\) is the number of failures.
For Dana,
\[ N=20, \qquad X=3. \]
The empirical failure rate is the fraction of trials that failed:
\[ \hat p_f=\frac{X}{N}. \]
For this experiment,
\[ \hat p_f = \frac{3}{20} = 0.15. \]
We therefore observed failures in 15 percent of the executions.
The wording matters. We have measured what happened in this particular experiment. We have not established that the system’s underlying probability of failure is exactly 15 percent.
We use different notation for those two quantities:
\[ \hat p_f \]
is the empirical failure rate calculated from our observations, while
\[ p_f \]
represents the underlying failure probability we want to characterize.
NoteReliability Principle
The empirical failure rate is what we observed. The failure probability is the quantity we are trying to estimate.
Another set of 20 executions could produce two failures, four failures, or some other count even if the system’s underlying behavior had not changed. Repeated execution gives us evidence about \(p_f\); it does not reveal \(p_f\) exactly.
To understand that distinction more precisely, we need a probability model.
2.3.2 From observations to a probability model
In statistics, a data-generating process is the process that gives rise to the observations in a dataset.
For our experiment, the idea is concrete. We place the system under a specified condition, execute it, and observe success or failure. Repeating that procedure generates our reliability data:
\[ \boxed{ \text{Specified condition} \rightarrow \text{System execution} \rightarrow \text{Success or failure} } \]
A probability model gives us a compact way to describe that process.
For one execution with two possible outcomes, the simplest model is a Bernoulli distribution. A Bernoulli trial has a binary outcome: 1 or 0. In our notation, 1 means failure and 0 means success.
For trial \(i\) under condition \(c\), we write
\[ F_i\mid C=c \sim \operatorname{Bernoulli}\!\left(p_f(c)\right). \]
Read this as: when trial \(i\) is executed under condition \(c\), it fails with probability \(p_f(c)\) and succeeds with probability \(1-p_f(c)\).
Here, \(C\) represents the conditions under which the execution occurs. The notation \(C=c\) means that we are considering a particular specified condition, such as Dana’s dispute scenario under the system configuration being tested.
The conditional failure probability is
\[ p_f(c)=P(F=1\mid C=c). \]
Read this as: \(p_f(c)\) is the probability that an execution fails—that is, \(F=1\)—when the system operates under condition \(c\).
If the condition is clear from context, we can use the shorter notation
\[ F_i\sim\operatorname{Bernoulli}(p_f). \]
Now repeat the experiment \(N\) times.
If those trials are sufficiently comparable, have the same failure probability, and can reasonably be treated as independent, the total number of failures
\[ X=\sum_{i=1}^{N}F_i \]
can be modeled with a binomial distribution:
\[ X\sim\operatorname{Binomial}(N,p_f). \]
Read this as: across \(N\) comparable trials, each with failure probability \(p_f\), the binomial distribution describes how many failures \(X\) we might observe.
The relationship between the two distributions is straightforward:
Bernoulli models one binary trial. Binomial models the number of failures across repeated Bernoulli trials.
For Dana, each complete execution produces a binary outcome. Repeating the experiment 20 times produces a count of failures. That count is what connects the probability model to our empirical measurement:
\[ \hat p_f=\frac{X}{N}. \]
With \(X=3\) and \(N=20\),
\[ \hat p_f=0.15. \]
The probability appendix develops the Bernoulli and binomial distributions in more detail. For this chapter, the important idea is that repeated binary outcomes give us evidence about an underlying probability that we cannot observe directly.
The model is intentionally simple. Before relying on it, however, we need to be precise about what we mean by the condition under which those observations were generated.
2.3.3 Conditions, system behavior, and measurement
A condition is more than the text of a prompt.
For reliability work, the condition is the set of circumstances under which an execution occurs that are relevant to the reliability question. Depending on the system, this may include the user request, conversation history, language, system instructions, application configuration, tool availability, retrieval state, persistent state, and relevant aspects of the operating environment.
We represent the condition encountered by the system as
\[ C\sim P_C, \]
where \(P_C\) represents the population or distribution of conditions we care about.
Read this as: executions may encounter different conditions \(C\), and \(P_C\) describes how those conditions are distributed in the population we want to understand.
Dana’s experiment deliberately narrows this variation. We repeatedly execute a specified dispute condition so that we can study variation in system behavior under that condition.
Even then, the system may not produce the same outcome on every execution:
\[ F\mid C=c \sim \operatorname{Bernoulli}\!\left(p_f(c)\right). \]
This is the second source of variation: system behavior under the condition. Holding the intended condition constant does not guarantee identical behavior on every execution.
There is also a third layer: measurement.
The system leaves behind observable evidence such as responses, tool calls, tool results, state changes, traces, or retrieved information. Let
\[ E_i \]
represent the evidence collected from trial \(i\).
A measurement procedure \(M\) uses that evidence to classify the trial:
\[ \tilde F_i=M(E_i). \]
We call \(\tilde F_i\) the measured failure label.
Like \(F_i\), it is binary:
\[ \tilde F_i= \begin{cases} 1, & \text{if the measurement procedure classifies trial }i\text{ as a failure},\\ 0, & \text{if it classifies trial }i\text{ as a success}. \end{cases} \]
This gives us the more complete experimental chain:
\[ \boxed{ C \longrightarrow F \longrightarrow E \longrightarrow \tilde F } \]
or, in words,
\[ \boxed{ \text{Condition} \longrightarrow \text{System behavior} \longrightarrow \text{Evidence} \longrightarrow \text{Measured failure label} } \]
Each layer answers a different question:
| Layer | Question |
|---|---|
| Condition, \(C\) | Under what circumstances did we execute the system? |
| System outcome, \(F\) | Did the system actually succeed or fail? |
| Evidence, \(E\) | What observable artifacts did the execution leave behind? |
| Measured label, \(\tilde F\) | How did our measurement procedure classify that evidence? |
Changes at these layers affect the resulting data differently. A changing operating population changes which conditions the system encounters. Variation in system behavior can produce different outcomes even when the intended condition is held constant. Measurement error can cause the recorded label \(\tilde F_i\) to differ from the underlying outcome \(F_i\).
For Dana, we simplify the experiment by repeatedly executing the same specified dispute condition. We also have unusually strong evidence for an important part of the failure definition: we can inspect authoritative banking state to determine whether the required dispute record actually exists.
When the measurement procedure is sufficiently trustworthy for the purpose of the experiment, we use the measured labels \(\tilde F_i\) as our observations of trial failure. For many AI tasks, deciding whether an execution truly succeeded will be much harder. Chapters 4 and 5 examine that measurement problem in detail.
For now, we can keep our attention on repeated system behavior while remembering that the numbers in our dataset ultimately come from a measurement procedure.
2.3.4 What exactly counts as one trial?
There is an engineering decision hidden inside \(N\): what counts as one observation?
For Dana, one trial is one complete execution of the dispute scenario through the system boundary defined in Section 2.1. That execution might involve several model calls, retrieval operations, tool calls, and internal application steps. Those are events within the trial, not separate trials.
Our reliability question is:
How reliably does the banking assistant successfully handle Dana’s dispute?
For that question, the experimental unit is
\[ \boxed{ \text{one trial} = \text{one complete execution of Dana's dispute scenario} } \]
A different reliability question could imply a different experimental unit. If we wanted to measure whether a model reliably generates valid tool calls, one model-generated tool call might constitute a trial. If we wanted to measure the dispute service itself, one invocation of that service might be the trial.
The system under test does not have to contain an LLM at all. The same experimental structure applies whenever repeated success or failure of a defined system can be observed.
The experimental unit therefore follows from the reliability question and the system boundary. The numerator and denominator acquire meaning only after we specify what constitutes one opportunity for the system to succeed or fail.
2.3.5 Failure probability exists under conditions
We can now give the conditional notation more practical meaning:
\[ p_f(c)=P(F=1\mid C=c). \]
Read this as: \(p_f(c)\) is the probability of failure when the system operates under condition \(c\).
Dana’s dispute scenario is not necessarily the same condition as a balance inquiry. A long conversation may differ from a fresh session. Different user requests, languages, system configurations, tools, retrieval states, or operating environments may produce different failure behavior.
For Dana, \(c\) represents the specified dispute condition and relevant system configuration under which we repeatedly execute the system. Repeated trials provide evidence about
\[ p_f(c). \]
This gives mathematical form to the idea introduced in Section 2.2: a test case can be treated as an operating condition with a failure propensity.
It also explains why a one-pass eval can hide important differences. Suppose two conditions each succeed on their first execution:
\[ F_A=0, \qquad F_B=0. \]
Because \(F=0\) denotes success, the observed result is identical for the two cases. But that does not imply
\[ p_f(c_A)=p_f(c_B). \]
Condition \(c_A\) may almost never fail, while condition \(c_B\) may fail frequently but happen to have succeeded on the execution we observed. A single outcome therefore tells us what happened on that trial; repeated outcomes give us evidence about the failure propensity under the condition.
A broader reliability claim requires a broader population of conditions. If cases are sampled in a way that represents the production population of interest, an aggregate failure rate may support a population-level reliability estimate. If an evaluation deliberately balances categories, enriches difficult cases, or includes historical failures, its aggregate rate instead describes that evaluation population.
The calculation may look identical in both cases:
\[ \hat p_f=\frac{X}{N}. \]
What changes is what the trials—and therefore the denominator—represent.
2.3.6 The model is simple. The experiment is not.
We can now summarize the statistical core of Dana’s repeated experiment. Each equation describes a different part of the same reliability measurement.
First,
\[ F_i\mid C=c \sim \operatorname{Bernoulli}\!\left(p_f(c)\right). \]
Read this as: trial \(i\) has two possible outcomes under condition \(c\): failure, represented by \(F_i=1\), or success, represented by \(F_i=0\). The probability of failure is \(p_f(c)\).
Second,
\[ X=\sum_{i=1}^{N}F_i. \]
Read this as: count the failures across the \(N\) trials. Because failures contribute 1 and successes contribute 0, their sum \(X\) is the total number of failures.
Third,
\[ X\sim\operatorname{Binomial}\!\left(N,p_f(c)\right). \]
Read this as: if the \(N\) trials are sufficiently comparable and independent and share the same failure probability \(p_f(c)\), the total number of failures can be modeled with a binomial distribution.
Finally,
\[ \hat p_f=\frac{X}{N}. \]
Read this as: divide the number of observed failures by the number of trials to obtain the empirical failure rate.
For Dana,
\[ X=3, \qquad N=20, \qquad \hat p_f=0.15. \]
The calculation is easy. The assumptions behind it require more care.
When we write
\[ F_i\mid C=c \sim \operatorname{Bernoulli}\!\left(p_f(c)\right), \]
we are assuming that the same underlying failure probability meaningfully applies to the executions we have grouped together. When we use the standard binomial model, we additionally treat those trials as sufficiently independent for that model to be useful.
Neither assumption should be automatic for an AI system with memory, retrieval, tools, persistent application state, external dependencies, or hosted infrastructure.
There is one final simplification to keep in mind. The mathematical model is written in terms of the underlying system outcome \(F_i\), while the experiment records the measured failure label \(\tilde F_i\) derived from evidence:
\[ \tilde F_i=M(E_i). \]
For Dana, authoritative system state gives us strong evidence for determining whether the required dispute was created. In other applications, measurement itself may introduce substantial uncertainty.
It is therefore useful to keep two questions separate:
\[ \boxed{ \text{How does the system behave?} \qquad\text{and}\qquad \text{How accurately do we measure that behavior?} } \]
This chapter is currently concerned primarily with the first question. Chapters 4 and 5 will return to the second.
For now, the immediate issue is whether Dana’s 20 executions can legitimately be treated as repeated observations from the same experiment.
Were those 20 executions sufficiently comparable, and sufficiently independent, for the simple probability model to make sense?
That is the question we turn to next.
2.4 When Are Repeated Runs Comparable?
In Section 2.3, we modeled repeated executions using a binomial distribution. That model assumes more than a collection of runs with the same test-case name. The executions must be similar enough that treating them as observations of the same underlying failure probability is reasonable, and one execution should not improperly determine the outcome of another.
Statistically, these ideas are described as identical distribution and independence, often abbreviated together as IID. For our purposes, the practical question is:
Were these executions sufficiently comparable and independent to treat them as repetitions of the same experiment?
2.4.1 Are we measuring the same condition?
Identically distributed does not mean that every execution produces the same result. It means that we have a defensible reason to treat the outcomes as arising from the same underlying probability distribution.
For Dana’s experiment, our model assumes
\[ P(F_i=1\mid C=c)=p_f(c). \]
In plain language, each execution may succeed or fail, but we are treating the same failure probability as applying to each trial under the specified condition.
That assumption becomes questionable when something changes that could plausibly affect failure behavior. Suppose the bank executes Dana’s scenario 50 times using one system prompt and observes one failure, then changes the prompt and observes nine failures in another 50 executions. Pooling the results gives
\[ \frac{10}{100}=0.10. \]
The arithmetic is correct, but the pooled rate may conceal two different failure propensities:
\[ p_f(c_{\text{prompt A}}) \qquad\text{and}\qquad p_f(c_{\text{prompt B}}). \]
The same issue can arise after changes to the model, system instructions, tools, retrieval data, application code, or other parts of the system. Relevant conditions can also change outside our application, such as when an external service or hosted model endpoint changes.
Not every difference makes two trials incomparable. The question is whether something changed that could reasonably affect the failure behavior we are trying to characterize.
2.4.2 Did one trial affect another?
Independence raises a different question: can what happened in one trial influence what happens in another?
Consider a stateful agent. One execution writes information to memory that remains available during the next. A dispute created in one trial is still present when another begins. A failed tool call changes application state that affects later behavior. In these cases, later trials no longer begin from the same intended condition.
For Dana’s experiment, we therefore want each trial to begin from the defined starting state. That might require clearing conversation history, removing disputes created during previous trials, and restoring relevant application state.
Dependence can also arise from shared events. Suppose 30 executions fail during the same external-service outage. Those failures may all be legitimate system failures, but they are also related to one another through a common cause. Thirty failures produced by one outage do not necessarily provide the same statistical information as 30 failures arising separately across otherwise independent executions.
The goal is not to prove that real AI executions are perfectly independent. It is to understand whether important dependencies exist and whether they matter for the reliability claim we want to make.
2.4.3 Comparable enough for what?
Real AI systems rarely provide perfectly controlled IID observations. Models are embedded in applications, tools have state, external services change, and hosted infrastructure contains variables we may not observe.
IID should therefore be treated as an experimental assumption, not as a property we automatically acquire by running the same test repeatedly.
Before analyzing a collection of trials together, we should be able to answer a few practical questions:
| Check | Why it matters |
|---|---|
| Was the defined system materially unchanged? | A system change may change failure propensity. |
| Were the relevant conditions sufficiently similar? | Otherwise the trials may represent different \(p_f(c)\). |
| Did earlier trials change the starting state of later trials? | The observations may not be independent. |
| Did shared events create important dependence? | Many failures may reflect a common cause. |
These questions do not establish perfect IID behavior. They help us decide whether treating the trials as repeated observations from the same experiment is a reasonable approximation.
For Dana’s 20 executions, that is enough. We need a consistent system and condition, an appropriate starting state for each trial, and awareness of dependencies that could materially affect the results.
The next step is to turn those requirements into an experiment we can actually run.
2.5 Build the Reliability Experiment
We now have the pieces needed to turn repeated executions into a reliability experiment. We need a defined system and operating condition, a clear trial and failure definition, sufficiently comparable executions, evidence about what happened, and a measurement procedure that converts that evidence into an observed outcome.
We can organize those pieces into a simple workflow:
\[ \boxed{ \text{Specify} \rightarrow \text{Reset} \rightarrow \text{Execute} \rightarrow \text{Collect evidence} \rightarrow \text{Measure} \rightarrow \text{Record} \rightarrow \text{Summarize} } \]
This is more than a loop that calls a system \(N\) times. It is a reliability harness: a procedure for generating repeated observations under defined conditions while preserving enough evidence to interpret the resulting failure rate.
Let’s build one for Dana.
NoteCompanion code
The complete Chapter 2 experiment is available in the book’s companion GitHub repository under chapters/ch02.
To keep our first reliability experiment simple, reproducible, and runnable without external services, the repository uses a small stochastic simulation of Dana’s banking assistant rather than a live LLM. The simulation gives us the behaviors we need—a dispute tool that can fail, an assistant that can incorrectly report success, and persistent application state that must be reset between trials—without introducing model APIs, inference configuration, provider behavior, or LLM-based measurement before we need them.
We will run reliability experiments on real LLMs and AI systems in Chapter 3. For now, the simulation lets us focus on the experimental machinery itself: defining a trial, controlling its starting conditions, collecting evidence, measuring failure, and preserving the resulting observations.
The reliability harness is deliberately independent of the simulation. A real model, agent, or deployed application can replace SimulatedDanaBankSystem while retaining the same experimental structure.
The repository includes three runnable examples, a guided notebook, the complete experiment specification, trial records, and tests. Chapter 2 runs locally and requires no API key.
2.5.1 Step 1: Specify the experiment
Before running the first trial, define what the experiment is intended to measure. For Dana, our question is:
How reliably does the banking assistant complete the dispute workflow and accurately report the result under Dana’s specified dispute condition?
The companion implementation records that definition in experiment.yaml. The relevant portion is:
experiment_id: dana-dispute-v1
condition:
id: dana-unauthorized-transfer
prompt: >
There's a charge on my account I don't recognize.
I didn't make it.
metadata:
task: dispute
trial:
unit: complete dispute interaction
target_trials: 100
system:
application_version: simulated-bank-assistant-1
model_endpoint: local-simulation
system_prompt_version: simulated-v1
toolset_version: dispute-tool-v1
failure_definition: >
Failure occurs when the system does not create a valid
dispute record for the specified transaction, or when
the assistant inaccurately represents whether the
dispute was created.
reset_required: true
metadata:
chapter: 2
example: dana-bank
seed: 42The specification tells us what the denominator will mean. One trial is one complete dispute interaction, and the experiment calls for 100 such trials under the specified condition. It also fixes the failure definition before we see the results.
The system metadata here describes the local simulation, but the same fields could identify a deployed application, model endpoint, prompt version, tool configuration, or other components that matter to the reliability claim. The purpose is not to record every available property of the system. It is to preserve the properties needed to understand what generated the observations.
2.5.2 Step 2: Prepare and execute each trial
Section 2.4 established that repeated runs should begin under sufficiently comparable conditions. In the harness, that statistical requirement becomes an engineering procedure.
Dana’s simulated environment contains three pieces of persistent state: whether the transaction exists, whether a dispute already exists, and the conversation history. Before each trial, the environment restores the intended starting state:
def reset(self, condition):
self.transaction_exists = True
self.dispute_exists = False
self.conversation_history.clear()The harness then verifies that the reset actually worked. For this experiment, a valid starting state requires an existing transaction, no existing dispute, and an empty conversation history.
This distinction matters. Issuing a reset command does not establish that the environment was successfully reset. If Trial 12 creates a dispute and Trial 13 begins with that dispute still present, Trial 13 is no longer an execution of the condition we specified.
Once the starting state has been verified, the system executes Dana’s request. The repository uses SimulatedDanaBankSystem, a deliberately small stochastic system with two relevant failure behaviors. The dispute tool fails with a configured probability, and after a tool failure the assistant can falsely report that the dispute was successfully opened.
The complete experiment is assembled in 03_run_reliability_experiment.py:
environment = DanaBankEnvironment()
system = SimulatedDanaBankSystem(
environment,
tool_failure_probability=0.10,
false_confirmation_probability=0.50,
seed=spec.metadata.get("seed"),
)
experiment = ReliabilityExperiment(
spec=spec,
system=system,
environment=environment,
evidence_collector=DanaEvidenceCollector(),
measurement=DanaVerifier(),
)
records = experiment.run()The random seed makes this teaching example reproducible, but the harness does not depend on simulated randomness. Its job is to execute whatever SystemUnderTest we give it under the experiment we specified.
2.5.3 Step 3: Collect evidence and measure the outcome
A final assistant response is not necessarily enough to determine whether a system succeeded. Dana’s assistant can say:
Your dispute was opened successfully.
while the dispute tool actually failed.
The harness therefore collects evidence from inside the defined system boundary. DanaEvidenceCollector preserves the assistant response, tool calls, tool results, and resulting application state. For a failed execution, the evidence can show both that the tool reported failure and that no dispute exists afterward.
Let \(E_i\) denote the evidence collected for trial \(i\). As introduced in Section 2.3, the measurement procedure \(M\) converts that evidence into the measured failure label:
\[ \tilde F_i=M(E_i). \]
For Dana, DanaVerifier performs this measurement deterministically. It checks whether a dispute actually exists and whether the assistant’s response accurately represents that state. A trial is therefore classified as a failure if the required dispute was not created, if the assistant misrepresented the result, or both.
This lets us distinguish failure mechanisms even when they lead to the same binary outcome. For example, a failed dispute tool and a false claim of successful completion are different behaviors, although both can contribute to
\[ \tilde F_i=1. \]
The important design principle is that the evidence follows from the required function. If the requirement says that a dispute must exist, inspect the dispute state rather than trusting the assistant’s statement that it exists.
For other AI systems, the measurement procedure might instead involve a rule-based evaluator, an LLM judge, a human reviewer, or some combination. Chapters 4 and 5 will examine how we determine whether those measurement systems are themselves trustworthy.
2.5.4 Step 4: Preserve the trial record
A reliability experiment should not end with only a numerator and denominator. We want to retain enough information to reconstruct how each observation was produced.
The companion implementation therefore writes one JSON record for every attempted trial. A failed trial from the committed experiment includes information of this form:
{
"experiment_id": "dana-dispute-v1",
"trial_id": 2,
"condition_id": "dana-unauthorized-transfer",
"status": "valid",
"system_metadata": {
"application_version": "simulated-bank-assistant-1",
"model_endpoint": "local-simulation",
"system_prompt_version": "simulated-v1",
"toolset_version": "dispute-tool-v1"
},
"execution_metadata": {
"stage": "complete",
"system_outcome_metadata": {
"tool_status": "failed",
"false_confirmation": true
}
},
"evidence": {
"response": "Your dispute was opened successfully.",
"tool_results": [
{
"status": "failed",
"dispute_created": false
}
],
"state": {
"transaction_exists": true,
"dispute_exists": false
}
},
"measurement": {
"failure": true,
"reason": "the dispute was not created; the assistant response misrepresented dispute state"
}
}This is abbreviated from the actual record for readability, but it preserves the important separation between what happened and what we concluded from it. The evidence records observable system behavior; the measurement records how that evidence was classified.
Preserving the evidence also gives us options later. If a measurement procedure is corrected or replaced, historical executions may be remeasured without necessarily rerunning the system.
The harness also distinguishes attempted from valid trials. A reset failure, failed starting-state verification, evidence-collection error, or measurement error can produce an invalid trial rather than silently entering the dataset as a success or disappearing from the experiment.
If \(\mathcal V\) is the set of valid trials, the empirical failure rate is therefore
\[ \hat p_f = \frac{\sum_{i\in\mathcal V}\tilde F_i} {N_{\text{valid}}}. \]
Invalid trials remain in the experiment record with the stage and reason for invalidation. This matters because silently discarding difficult-to-measure executions can bias the result.
There is an important boundary distinction here. If the external experimental harness crashes before Dana’s request reaches the system, the trial may be invalid because the system was never tested. If the deployed banking application’s own orchestration layer crashes after accepting Dana’s request, that event may instead be a genuine system failure. Which interpretation applies follows from the system boundary defined at the beginning of the experiment.
2.5.5 Step 5: Summarize the experiment
Once the trial records have been collected, the basic calculation is straightforward. The companion implementation summarizes the valid records and writes both trial_records.jsonl and summary.json.
With the committed experiment specification and random seed, the complete run produces:
Attempted trials: 100
Valid trials: 100
Invalid trials: 0
Failures: 12
Empirical failure rate: 0.1200Thus,
\[ N_{\text{valid}}=100, \qquad X=12, \]
and
\[ \hat p_f = \frac{12}{100} = 0.12. \]
Notice what we can now say and what we cannot. We did not merely obtain the number 0.12. We know the system that was executed, the condition under which it was executed, what constituted one trial, how the starting state was restored, what counted as failure, what evidence was collected, and how that evidence was measured.
The repository therefore reports a deliberately bounded claim:
Under experiment
dana-dispute-v1, 12 failures were observed across 100 valid executions, for an empirical failure rate of 0.1200.
That is a more defensible statement than simply saying, “the banking assistant has a 12 percent failure probability.” The latter jumps from an observed result to a claim about an underlying probability that we have not yet justified.
A useful reliability experiment leaves a trace from the reported result back to the experiment that produced it:
\[ \boxed{ \text{Experiment specification} \rightarrow \text{Executions} \rightarrow \text{Evidence} \rightarrow \text{Measured outcomes} \rightarrow \text{Trial records} \rightarrow \text{Summary} \rightarrow \text{Reliability measurement} } \]
That trace is what makes the result auditable. We can inspect not only the final failure rate but also the observations and experimental choices that gave the number its meaning.
We have now turned repeated system executions into an empirical reliability measurement. The remaining question is how much that measurement tells us about the failure probability we actually care about.
2.6 What Does the Failure Rate Tell Us?
Dana’s completed experiment produced
\[ N_{\text{valid}}=100, \qquad X=12, \qquad \hat p_f=0.12. \]
We know exactly what that number describes. The remaining question is what it tells us about the underlying failure probability. It is not quite the same as saying that the system has a 12 percent probability of failure. The distinction is between an observed failure rate and the underlying failure probability we are trying to characterize:
\[ \boxed{ \hat p_f(c) \neq p_f(c) } \]
For the specified condition \(c\),
\[ \hat p_f(c)=\frac{X}{N_{\text{valid}}} \]
is the empirical failure rate observed in our experiment, while
\[ p_f(c)=P(F=1\mid C=c) \]
is the underlying probability of failure under that condition. When all attempted trials are valid, as in this experiment, \(N_{\text{valid}} = N\).
The empirical rate gives us evidence about the underlying probability; it does not reveal that probability exactly. If we repeated Dana’s 100-trial experiment under the same conditions, we might observe 10 failures, 14 failures, or some other number. Such variation does not necessarily mean that the system changed. Sampling variation alone can produce different observed failure counts even when the underlying failure probability remains the same.
This is also why reporting the count \(X\) and number of trials \(N\) matters. An observed rate of 12 percent based on 100 trials does not provide the same amount of evidence as a 12 percent rate based on 10,000 trials. The percentage tells us what fraction of the observed trials failed; the denominator helps tell us how much evidence lies behind that percentage.
2.6.1 From measurement to uncertainty
We began this chapter with a simple calculation:
\[ \hat p_f=\frac{X}{N}. \]
The arithmetic was never the difficult part. The work of this chapter was establishing what the numerator and denominator mean.
We defined the system and operating condition, decided what constitutes one trial and one failure, repeated the execution under sufficiently comparable conditions, collected evidence about what happened, measured each outcome, and preserved the resulting trial records. Only then could we interpret \(12/100\) as an empirical reliability measurement rather than merely a percentage produced by an eval.
A compact way to describe what now stands behind that number is
\[ \boxed{ \begin{aligned} \text{Reliability measurement} ={}& \text{System} + \text{Conditions} + \text{Trial definition}\\ &+ \text{Failure definition} + \text{Measurement procedure} + (X,N,\hat p_f). \end{aligned} } \]
We have therefore answered one question:
What failure rate did we observe in this experiment?
We have not yet answered another:
What does that observation tell us about the underlying probability of failure?
For Dana, we observed 12 failures in 100 trials. How precisely does that locate \(p_f(c)\)? What values remain reasonably consistent with the evidence? How much would another 900 trials teach us? And what if we had run all 100 trials without observing a single failure?
Those are questions about statistical uncertainty. They require us to move beyond the point estimate \(\hat p_f\) and ask how much confidence the experiment gives us about the underlying quantity it is intended to measure.
That is the subject of Chapter 3.
For further reading on repeated evaluation, reliability, measurement science, and evaluation practice, see (Broadwater 2026a, 2026b; Rabanser et al. 2026; Truong and Koyejo 2026; Anthropic 2026; OpenAI 2026; Autio et al. 2024; Alvarado Gonzalez et al. 2025).
References and further reading
Alvarado Gonzalez, Miguel Angel, Michelle Bruno Hernandez, Miguel Angel Peñaloza Perez, Bruno Lopez Orozco, Jesus Tadeo Cruz Soto, and Sandra Malagon. 2025. Do Repetitions Matter? Strengthening Reliability in LLM Evaluations. https://doi.org/10.48550/arXiv.2509.24086.
Anthropic. 2026. “Demystifying Evals for AI Agents.” January 9. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents.
Autio, Chloe, Reva Schwartz, Jesse Dunietz, et al. 2024. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST AI Nos. 600-1. National Institute of Standards and Technology. https://doi.org/10.6028/NIST.AI.600-1.
Broadwater, Keita. 2026a. Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testing. https://doi.org/10.48550/arXiv.2602.11786.
Broadwater, Keita. 2026b. “Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Sampling.” 2026 6th International Conference on Computer Communication and Artificial Intelligence (CCAI). https://doi.org/10.1109/CCAI69603.2026.11642011.
He, Horace, and Thinking Machines Lab. 2025. “Defeating Nondeterminism in LLM Inference.” https://doi.org/10.64434/tml.20250910.
Noarov, Georgy, Soham Mallick, Tao Wang, et al. 2025. Foundations of Top-\(k\) Decoding for Language Models. https://doi.org/10.48550/arXiv.2505.19371.
OpenAI. 2026. “Evaluation Best Practices.” https://developers.openai.com/api/docs/guides/evaluation-best-practices.
OpenAI. n.d. “API Reference: Backwards Compatibility.” Accessed October 1, 2026. https://developers.openai.com/api/reference/overview#backwards-compatibility.
Rabanser, Stephan, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala, and Arvind Narayanan. 2026. Towards a Science of AI Agent Reliability. https://doi.org/10.48550/arXiv.2602.16666.
Truong, Sang T., and Sanmi Koyejo. 2026. AI Measurement Science. Stanford University. https://aimslab.stanford.edu/textbook/.