AI reliability field guide
Wilson Confidence Intervals for AI Evaluation
A failure rate computed from a finite AI evaluation is a point estimate. The Wilson interval shows how much statistical uncertainty remains around it.
If an LLM fails 12 of 100 valid trials, the empirical failure rate is 12%. Another comparable sample will usually produce a different count. A confidence interval describes the range of underlying binomial probabilities that remain plausible under the interval procedure, given the observed data.
Why not use the simple normal interval?
The textbook Wald interval adds and subtracts a normal-theory margin from the observed proportion. It is easy to calculate, but it can have poor coverage with small samples or probabilities near zero or one. It can even produce impossible limits below 0 or above 1. AI reliability evaluations frequently operate in exactly these regimes, especially when a system has few observed failures.
The Wilson score interval rearranges the score test for a binomial proportion. It remains within the valid probability range and generally has more dependable coverage. For an observed proportion p̂, sample size N, and normal critical value z, its center and half-width can be written as:
Worked AI-evaluation example
For 12 failures in 100 valid trials, p̂ = 0.12. Using z = 1.96 for a nominal 95% interval gives a Wilson interval of approximately 0.069 to 0.198. The point estimate is still 12%, but the experiment is also compatible with an underlying failure probability around 7% to 20% under this procedure.
That width matters. A team deciding whether the system meets a 10% reliability requirement cannot treat the observed 12% as exact. Nor could an observed 8% automatically establish that the true probability is below 10%; the interval and the decision rule need to be considered together.
How to interpret the interval correctly
A 95% frequentist confidence interval does not mean there is a 95% probability that this fixed interval contains the true probability. The repeated-sampling interpretation is that, under the model assumptions, 95% of intervals constructed by the same procedure over repeated samples would contain the true probability.
The interval accounts for binomial sampling uncertainty, not every source of uncertainty in an AI evaluation. It does not correct an invalid judge, nonrepresentative test cases, dependent trials, silent model changes, or a mismatch between laboratory and production conditions. Those are experimental-design and measurement-system problems.
Reporting Wilson intervals in practice
Report the failure count and valid denominator, the empirical rate, the confidence level, the interval method, and the resulting limits. A concise statement might read: “The system failed 12 of 100 valid trials (12%; 95% Wilson CI: 6.9%–19.8%) under the specified configuration and test population.”
When comparing systems, avoid declaring a meaningful improvement simply because two point estimates differ. Examine the experimental design and uncertainty around the difference. Paired inputs, stratification, and direct tests of the comparison can be more informative than visually comparing two separate intervals.