AI/ML engineers
For engineers building LLM, RAG, and agent systems who need to understand reliability beyond average benchmark performance.
A technical book written in public
Reliability Engineering for AI Systems
Measuring, Predicting, and Controlling Failure in LLMs and Agents
AI systems can succeed repeatedly and still contain hidden failure modes. Their behavior depends on prompts, context, model configuration, tools, retrieved information, and other operating conditions.
Reliability Engineering for AI Systems develops a practical framework for measuring failure probability, discovering where risk concentrates, evaluating whether interventions work, and operating more reliable LLM and agent systems.
Starting with repeated trials and empirical failure estimation, the book builds toward conditional and latent failure models and applies them to tools, RAG systems, autonomous agents, guardrails, failure prediction, reliability budgets, monitoring, and production assurance.
This is a living technical book. Chapter 1 is available free now. Founding Reader access will open with Chapter 2, targeted for October 1, 2026. After that, new chapters are planned approximately every three weeks as the manuscript develops.
Early-Access Edition
Reliability Engineering
for AI Systems
for AI Systems
Measuring, Predicting, and Controlling Failure in LLMs and Agents
About the book
Engineering reliability for probabilistic AI systems
Conventional software can often be tested by asking whether a fixed input produces the expected result. LLMs and agents complicate that model: the same system can produce different outcomes across repeated executions, and its failure probability can change with prompts, context, tools, retrieval, model configuration, and deployment conditions.
Reliability Engineering for AI Systems treats reliability as an empirical engineering problem: estimate the probability of failure, understand how that probability changes under operating conditions, and determine whether interventions actually reduce risk.
The book develops that reasoning progressively:
failure count→P(F)→P(F | X)→latent failure propensity
and then applies it to tool use, retrieval-augmented generation, agents, failure propagation, guardrails, prediction, reliability budgets, monitoring, and production assurance.
Practical outcomes
What you’ll learn
Readers will learn how to:
Estimate AI failure probability using repeated trials instead of relying only on single executions or aggregate benchmark scores.
Quantify uncertainty and distinguish an observed failure rate from the underlying probability it is trying to estimate.
Validate human and LLM-based judges before relying on them as measurement instruments.
Measure how prompts, models, configurations, tools, retrieval, and operating conditions change failure risk.
Analyze reliability across RAG pipelines, tool-using systems, and long-running agent trajectories.
Test interventions, predict high-risk conditions, allocate reliability budgets, and monitor reliability after deployment.
Writing schedule
Release schedule
After Chapter 2, new chapters are planned approximately every three weeks.
This is a living technical book. Release dates may shift when experiments, validation, technical review, or editing require additional time.
- Available
Chapter 1
When AI Systems Fail
Free
- Next
Chapter 2
Failure Probability from Repeated Trials
Target: October 1, 2026
- Planned
Chapter 3
Uncertainty in Failure Probability
Approximately three weeks after Chapter 2
- Planned
Chapter 4
AI Judges as Measurement Systems
Approximately three weeks after Chapter 3
- Planned
Chapter 5
Human Validation and Judge Calibration
Approximately three weeks after Chapter 4
The manuscript
Table of contents
Twenty-one chapters move from measuring failure to production reliability and assurance.
Part I — Measuring Failure5 chapters
- When AI Systems FailAvailable
The reliability mental model, system boundaries, failure definitions, repeated execution, and the first reliability harness.
- Failure Probability from Repeated TrialsNext
Bernoulli trials, binomial experiments, empirical failure rates, and estimating failure probability.
p_f = P(F = 1) - Uncertainty in Failure ProbabilityPlanned
Confidence intervals, zero-failure experiments, sample size, rare failures, and the difference between observed failure rate and underlying failure probability.
- AI Judges as Measurement SystemsPlanned
Rubrics, structured judgments, judge stochasticity, false positives, false negatives, and how judge error affects reliability estimates.
- Human Validation and Judge CalibrationPlanned
Blinded human review, confusion matrices, precision, recall, sensitivity, specificity, Cohen’s κ, disagreement analysis, bootstrap intervals, and deciding whether an automated judge is fit for use.
Part II — Conditional Reliability4 chapters
- Failure Probability Depends on ConditionsPlanned
Conditional failure probability, prompt and context effects, model and decoding settings, system prompts, harness configuration, user state, and environment.
p_f(x) = P(F = 1 | X = x) - Designing Reliability ExperimentsPlanned
Factors, controls, replication, randomization, blocking, factorial designs, ablations, frozen artifacts, and reproducibility.
- Mapping the Failure SurfacePlanned
Stress testing, interaction effects, stratification, high-risk regions, adversarial conditions, and response surfaces.
- Latent Failure Under Repeated InferencePlanned
Single-run versus repeated-run evaluation, hidden failures, k-of-N outcomes, repeated-use risk, and cumulative failure probability.
P(at least one failure in N) = 1 − (1 − p)ᴺ
Part III — Reliability of AI Systems5 chapters
- From Model Reliability to System ReliabilityPlanned
Foundation models, harnesses, prompts, orchestration, memory, guardrails, APIs, and the distinction between model-level and system-level failure.
- Tool-Using AgentsPlanned
Wrong-tool selection, incorrect arguments, tool execution failures, misinterpretation of results, false claims of successful action, and unsafe actions.
- Retrieval and Multi-Stage ReliabilityPlanned
RAG pipelines, retrieval failures, context quality, stage-wise reliability, and propagation of upstream errors.
- Reliability of Long-Running AgentsPlanned
Sequential decisions, accumulating risk, per-step reliability, task completion probability, recovery, and dependence across steps.
- Failure Propagation and CascadesPlanned
Correlated failures, common-cause failures, dependencies, cascading actions, redundancy, recovery, and system reliability models.
Part IV — Predicting and Controlling Failure4 chapters
- Predicting Failure ProbabilityPlanned
Risk models for AI behavior, conditional failure prediction, embeddings, metadata, graph features, telemetry, and supervised models for failure prediction.
P(F = 1 | X) - Finding Rare Failures EfficientlyPlanned
Risk-based sampling, ranking, top-k targeting, lift, failure-discovery recall, adaptive sampling, testing budgets, and stopping criteria.
- Reliability Requirements and BudgetsPlanned
Acceptable failure probabilities, severity classes, probability-consequence tradeoffs, component-level budgets, safety margins, escalation, and acceptance criteria.
- Improving ReliabilityPlanned
Prompt changes, model changes, decoding, routing, retrieval, verification, guardrails, redundancy, retries, human review, and experimental verification of whether an intervention actually lowers failure probability.
Part V — Production Reliability and Assurance3 chapters
- Reliability Changes Over TimePlanned
Drift, changing models, prompts, tools, user populations, retrieval corpora, distribution shift, rolling estimates, change detection, and statistical process control.
- Reliability IncidentsPlanned
Detection, reproduction, diagnosis, traces, logs, root-cause analysis, failure taxonomies, corrective action, and mechanistic evidence where available.
- Building an AI Reliability and Assurance SystemPlanned
Continuous testing, measurement, prediction, control, monitoring, judge validation, experiment provenance, audit trails, reliability reports, and retesting.
Pedagogical approach
Built around engineering problems, not abstract theory
The mathematics appears when the engineering problem requires it. Each chapter begins with a failure or reliability question, introduces the minimum statistical machinery needed to reason about it, implements the analysis, and ends with an engineering decision.
- Engineering problem
- Mental model
- Probability model
- Experiment
- Code
- Results
- Engineering decision
Audience
Who should read this book
AI evaluation and safety practitioners
For practitioners designing evaluations, stress tests, human-validation studies, red-team exercises, and assurance programs.
Reliability, platform, and systems engineers
For engineers bringing reliability, observability, fault analysis, and operational-risk thinking to probabilistic AI systems.
Technical leaders and model-risk practitioners
For people making deployment and governance decisions who need defensible evidence about failure probability, uncertainty, controls, and residual risk.
Publishing model
How Founding Reader access works
- 1
Read Chapter 1 free
The opening chapter is available now and will remain free.
- 2
Purchase when Chapter 2 is released
Choose Digital or Digital + Print with a single purchase.
- 3
Keep every future chapter
Founding Readers receive later chapters without paying future price increases as the manuscript grows.
- 4
Receive the completed digital book
The purchase includes all subsequently released chapters and the completed first edition.
Purchasing and access
Frequently asked questions
Is Chapter 1 really free?
Yes. Chapter 1 is available free now and will remain free.
When does Founding Reader access open?
It opens when Chapter 2 is published, currently targeted for October 1, 2026. The date is a target and may shift if the chapter needs additional experimental, technical, or editorial work.
What do I receive?
The current paid release, every subsequently released chapter, and the completed digital book. Early purchasers will not pay later price increases.
Will the price change?
Founding Reader — Digital will open at $29, and Founding Reader + Print will open at $59. Prices may increase as the manuscript grows, while your purchase price remains locked in.
What is included in the print tier?
Founding Reader + Print includes all digital releases and one physical copy of the completed book. Shipping and fulfillment details will be confirmed before the print edition is released.
Is this the finished book?
No. This is a living technical book. Chapters, examples, organization, and wording may change as the manuscript develops.