Uncertainty in Failure Probability

Suppose an AI system fails 12 times in 100 comparable trials. The empirical failure rate is 0.12, but a new set of 100 trials will not necessarily produce exactly 12 failures. Sampling variation is unavoidable whenever a finite experiment is used to learn about an underlying probability. Reliability claims therefore need two parts: a point estimate and an honest account of its uncertainty.

From an observed rate to an uncertainty interval

Chapter 3 begins with 1,500 executions across three language models. The experiment makes the distinction between observed outcomes and latent probabilities concrete: models can have different observed failure counts, while the precision of every estimate still depends on the number of trials and the location of the estimated probability.

A confidence interval summarizes which underlying probabilities remain compatible with the observed data under a specified statistical procedure. It does not mean that the system’s failure probability moves around inside the interval. The interval changes from sample to sample; the unknown probability is the quantity the experiment is trying to constrain.

For binomial outcomes, the chapter uses the Wilson score interval rather than relying on the familiar normal approximation. The Wilson interval behaves better when the sample is modest or the observed rate is near zero or one—conditions that are common in reliability work. The public guide on Wilson confidence intervals for AI evaluation derives the calculation and walks through an example.

Zero observed failures is not zero risk

A test with no observed failures is encouraging, but it does not establish that the true failure probability is zero. A rare failure mode may simply not have appeared in the tested sample. The relevant question becomes: how large could the underlying failure probability still be while remaining consistent with the zero-failure result?

At approximately 95% confidence, the rule of three gives a quick upper bound after N independent, comparable trials with no observed failures:

After 300 zero-failure trials, for example, the approximate upper bound is 0.01. The result supports a claim that the failure probability is likely below about 1% under the tested conditions; it does not support a universal claim that the system is failure-free. The rule-of-three guide explains the exact bound, the approximation, and the assumptions behind both.

Designing the sample size around the decision

Sample size should be chosen before the experiment by working backward from the engineering decision. If a team needs a narrow interval around an expected failure rate, it should plan for the desired precision. If the goal is to have a high probability of observing at least one failure at a specified rate, the calculation is different. A test designed to detect a 1% failure mode is not automatically adequate for a 0.1% failure mode.

This is why “we ran many tests” is not a reliability argument. The useful quantity is the relationship among the sample size, the failure rate of concern, the desired confidence or detection probability, and the operating conditions represented by the trials. The broader LLM reliability testing guide places these calculations inside a complete experimental workflow.