Experimental design
LLM reliability testing: a practical framework
Define failure, control operating conditions, repeat comparable trials, and turn observed outcomes into defensible reliability evidence.
A technical book written in public
Reliability Engineering for AI Systems
Measuring, Predicting, and Controlling Failure in LLMs, Agents, and Their Harnesses
AI systems can succeed repeatedly and still contain hidden failure modes. Their behavior depends on prompts, context, model configuration, tools, retrieved information, and other operating conditions.
Reliability Engineering for AI Systems develops a practical framework for measuring failure probability, discovering where risk concentrates, evaluating whether interventions work, and operating more reliable LLM and agent systems.
Starting with repeated trials and empirical failure estimation, the book builds toward conditional and latent failure models and applies them to tools, RAG systems, autonomous agents, guardrails, failure prediction, reliability budgets, monitoring, and production assurance.
This is a living technical book. Chapters 1 and 2 are available free in full. The Founding Reader edition now includes Chapter 3, every subsequent release, and the completed digital first edition.
Early-Access Edition
Reliability Engineering
for AI Systems
for AI Systems
Measuring, Predicting, and Controlling Failure in LLMs, Agents, and Their Harnesses
Free to read
Read the First Two Chapters Free
Learn how reliability engineering applies to large language models and AI agents, from defining failure to building reproducible experiments with repeated inference.
Chapter 1 introduces the foundations of AI reliability. Chapter 2 develops a practical experimental framework for measuring failure through repeated trials.
Chapter 3 is now available through Founding Reader access, together with every future early-access release.
Public technical guides
Practical methods for evaluating AI reliability
These focused guides explain the experimental and statistical tools behind the book. Each is free to read and designed to stand on its own.
Statistical uncertainty
Wilson confidence intervals for AI evaluation
Learn why a point estimate is not enough and how the Wilson interval expresses uncertainty in an observed AI failure rate.
Zero observed failures
The rule of three for zero failures
Translate a zero-failure test result into an approximate upper bound—and understand what the result does not prove.
About the book
Engineering reliability for probabilistic AI systems
Conventional software can often be tested by asking whether a fixed input produces the expected result. LLMs and agents complicate that model: the same system can produce different outcomes across repeated executions, and its failure probability can change with prompts, context, tools, retrieval, model configuration, and deployment conditions.
Reliability Engineering for AI Systems treats reliability as an empirical engineering problem: estimate the probability of failure, understand how that probability changes under operating conditions, and determine whether interventions actually reduce risk.
The book develops that reasoning progressively:
failure count→P(F)→P(F | X)→latent failure propensity
and then applies it to tool use, retrieval-augmented generation, agents, failure propagation, guardrails, prediction, reliability budgets, monitoring, and production assurance.
Practical outcomes
What you’ll learn
Readers will learn how to:
Estimate AI failure probability using repeated trials instead of relying only on single executions or aggregate benchmark scores.
Quantify uncertainty and distinguish an observed failure rate from the underlying probability it is trying to estimate.
Validate human and LLM-based judges before relying on them as measurement instruments.
Measure how prompts, models, configurations, tools, retrieval, and operating conditions change failure risk.
Analyze reliability across RAG pipelines, tool-using systems, and long-running agent trajectories.
Test interventions, predict high-risk conditions, allocate reliability budgets, and monitor reliability after deployment.
Build the methods in Python using working LLM examples, starting in Chapter 2.
Writing schedule
Release schedule
After Chapter 3, new chapters are planned approximately every three weeks.
This is a living technical book. Release dates may shift when experiments, validation, technical review, or editing require additional time.
- Free
Chapter 1
When AI Systems Fail
Free
- Free
Chapter 2
Measuring Failure Through Repeated Trials
Free
- Founding Readers
Chapter 3
Uncertainty in Failure Probability
Founding Readers
- Next
Chapter 4
AI Judges as Measurement Systems
Target: November 7, 2026
- Planned
Chapter 5
Human Validation and Judge Calibration
Approximately three weeks after Chapter 4
The manuscript
Table of contents
Twenty-one chapters move from measuring failure to production reliability and assurance.
Part I — Measuring Failure5 chapters
The reliability mental model, system boundaries, failure definitions, repeated execution, and the first reliability harness.
Bernoulli trials, binomial experiments, empirical failure rates, and estimating failure probability.
p_f = P(F = 1)- Uncertainty in Failure ProbabilityFounding Readers
Confidence intervals, zero-failure experiments, sample size, rare failures, and the difference between observed failure rate and underlying failure probability.
- AI Judges as Measurement SystemsNext
Rubrics, structured judgments, judge stochasticity, false positives, false negatives, and how judge error affects reliability estimates.
- Human Validation and Judge CalibrationPlanned
Blinded human review, confusion matrices, precision, recall, sensitivity, specificity, Cohen’s κ, disagreement analysis, bootstrap intervals, and deciding whether an automated judge is fit for use.
Part II — Conditional Reliability4 chapters
- Failure Probability Depends on ConditionsPlanned
Conditional failure probability, prompt and context effects, model and decoding settings, system prompts, harness configuration, user state, and environment.
p_f(x) = P(F = 1 | X = x) - Designing Reliability ExperimentsPlanned
Factors, controls, replication, randomization, blocking, factorial designs, ablations, frozen artifacts, and reproducibility.
- Mapping the Failure SurfacePlanned
Stress testing, interaction effects, stratification, high-risk regions, adversarial conditions, and response surfaces.
- Latent Failure Under Repeated InferencePlanned
Single-run versus repeated-run evaluation, hidden failures, k-of-N outcomes, repeated-use risk, and cumulative failure probability.
P(at least one failure in N) = 1 − (1 − p)ᴺ
Part III — Reliability of AI Systems5 chapters
- From Model Reliability to System ReliabilityPlanned
Foundation models, harnesses, prompts, orchestration, memory, guardrails, APIs, and the distinction between model-level and system-level failure.
- Tool-Using AgentsPlanned
Wrong-tool selection, incorrect arguments, tool execution failures, misinterpretation of results, false claims of successful action, and unsafe actions.
- Retrieval and Multi-Stage ReliabilityPlanned
RAG pipelines, retrieval failures, context quality, stage-wise reliability, and propagation of upstream errors.
- Reliability of Long-Running AgentsPlanned
Sequential decisions, accumulating risk, per-step reliability, task completion probability, recovery, and dependence across steps.
- Failure Propagation and CascadesPlanned
Correlated failures, common-cause failures, dependencies, cascading actions, redundancy, recovery, and system reliability models.
Part IV — Predicting and Controlling Failure4 chapters
- Predicting Failure ProbabilityPlanned
Risk models for AI behavior, conditional failure prediction, embeddings, metadata, graph features, telemetry, and supervised models for failure prediction.
P(F = 1 | X) - Finding Rare Failures EfficientlyPlanned
Risk-based sampling, ranking, top-k targeting, lift, failure-discovery recall, adaptive sampling, testing budgets, and stopping criteria.
- Reliability Requirements and BudgetsPlanned
Acceptable failure probabilities, severity classes, probability-consequence tradeoffs, component-level budgets, safety margins, escalation, and acceptance criteria.
- Improving ReliabilityPlanned
Prompt changes, model changes, decoding, routing, retrieval, verification, guardrails, redundancy, retries, human review, and experimental verification of whether an intervention actually lowers failure probability.
Part V — Production Reliability and Assurance3 chapters
- Reliability Changes Over TimePlanned
Drift, changing models, prompts, tools, user populations, retrieval corpora, distribution shift, rolling estimates, change detection, and statistical process control.
- Reliability IncidentsPlanned
Detection, reproduction, diagnosis, traces, logs, root-cause analysis, failure taxonomies, corrective action, and mechanistic evidence where available.
- Building an AI Reliability and Assurance SystemPlanned
Continuous testing, measurement, prediction, control, monitoring, judge validation, experiment provenance, audit trails, reliability reports, and retesting.
Pedagogical approach
Built around engineering problems, not abstract theory
The mathematics appears when the engineering problem requires it. Starting in Chapter 2, each chapter uses Python and working LLM systems to move from a reliability question through the mental model, probability model, experiment, code, results, and engineering decision.
- Engineering problem
- Mental model
- Probability model
- Experiment
- Code
- Results
- Engineering decision
Audience
Who should read this book
AI/ML engineers
For engineers building LLM, RAG, and agent systems who need to understand reliability beyond average benchmark performance.
AI evaluation and safety practitioners
For practitioners designing evaluations, stress tests, human-validation studies, red-team exercises, and assurance programs.
Reliability, platform, and systems engineers
For engineers bringing reliability, observability, fault analysis, and operational-risk thinking to probabilistic AI systems.
Technical leaders and model-risk practitioners
For people making deployment and governance decisions who need defensible evidence about failure probability, uncertainty, controls, and residual risk.
Publishing model
How Founding Reader access works
- 1
Read Chapters 1 and 2 free
The first two chapters are available now without an account or email signup.
- 2
Purchase early access
Buy the current manuscript with a one-time payment through Stripe.
- 3
Keep every future chapter
Founding Readers receive later chapters without paying future price increases as the manuscript grows.
- 4
Receive the completed digital book
The purchase includes all subsequently released chapters and the completed first edition.
Reader feedback
Help shape the book.
Found an error, have a question, or want to suggest a topic? Reader feedback is welcome while the manuscript develops.
Purchasing and access
Frequently asked questions
Are Chapters 1 and 2 really free?
Yes. Both chapters are available in full without an account, payment, or email signup.
What do I receive?
The current early-access manuscript, every subsequently released chapter, and the completed digital book. Early purchasers will not pay later price increases.
Is this a subscription?
No. Founding Reader access is a one-time $29 USD payment, not a recurring subscription.
Will the price change?
Founding Reader — Digital is currently $29. The price may increase as the manuscript grows, while your purchase price remains locked in.
Is a print copy included?
The current checkout is for the digital edition. Any future print offer will state its price, shipping, and fulfillment terms separately.
Is this the finished book?
No. This is a living technical book. Chapters, examples, organization, and wording may change as the manuscript develops.
Are release dates guaranteed?
No. Release dates are approximate and may shift when experiments, validation, technical review, or editing require additional time.