Chaos Engineering on a Database

In one line

You'd think test cases keep a database correct; in fact the deadliest bugs in a distributed database hide on paths where several faults happen to overlap, and they are quiet: no crash, just a wrong answer. This chapter is the PingCAP team's field report on TiDB: inject faults systematically at four layers (application, CPU/memory, network, filesystem), judge correctness against invariants such as "the sum of all balances is conserved," and hand the whole loop to an automated platform that runs day and night. That platform's know-how later became the open-source Chaos Mesh.

Glossary

Where it sits

Chapter 19, in Part V ("Evolution"), written by Liu Tang and Hao Weng of PingCAP. It follows Ch18 on the human side of systems and precedes Ch20 on security chaos engineering. It is one of the book's few complete practice reports from a database vendor, and maps onto any team building or running distributed storage (NewSQL, distributed key-value stores, message queues).

What problem this chapter solves

TiDB is a MySQL-compatible distributed database; production clusters often run tens to hundreds of nodes and carry heavy transactional load. For such a system, "never lose data, never miscount" is non-negotiable. A web service down for five minutes can apologize; a database that drops one transfer may force a business to halt and reconcile.

Three things make it hard. First, failure is normal: slow disks, crashed processes, broken links and jumping clocks happen daily at scale. Second, failures overlap: each single failure's handling path usually has a unit test, but combinations like "a network partition in the middle of a leader change" are hard to write down in advance. Third, database bugs are quiet: no alarm, just a wrong result that may spread before anyone notices. The chapter asks how to turn "correct under failure" from a promise into a fact verified every day.

The core ideas, in depth

1. Know your target: TiDB's three layers

To shake a system you first need its skeleton. TiDB splits into three parts:

• TiDB server: the stateless SQL layer. It parses SQL and plans queries but stores nothing, so you can add or remove instances freely.
• TiKV: a distributed transactional key-value store. Data is cut by key range into Regions; each Region has three replicas by default, forming one Raft group.
• PD (Placement Driver): the cluster's "brain." It tracks where every Region lives, schedules and balances load, and hands out globally increasing transaction timestamps (TSO, the timestamp oracle).

This picture defines the experiment checklist: kill one TiKV replica, and can Raft elect a new leader without losing committed writes? If the PD leader dies, do timestamps stay monotonic? If the link between the SQL layer and storage breaks, do transactions roll back or hang?

2. Injecting faults at four layers

The chapter groups injection by the layer it hits, from most precise to most realistic:

① Application: failpoints. "Fault switches" planted in code. Normally they do nothing; switched on in a test, they make a given line return an error, sleep, or panic. PingCAP wrote pingcap/failpoint for Go and fail-rs for Rust, modeled on FreeBSD's failpoints. Their value is precision: a razor-thin window like "a Raft snapshot fails halfway through sending" can be reproduced reliably.

② CPU and memory. Stress tools hog CPU and eat memory to simulate a noisy neighbor saturating the machine. Under resource starvation a database shows chain reactions such as heartbeat timeouts and frequent leader changes, which you never see on an idle box.

③ Network. Linux traffic control (tc) injects delay, loss and reordering; firewall rules (iptables) cut links between specific nodes to create partitions. This is the layer most worth shaking, because everything subtle about consensus lives in "what if a message is lost or late?"

④ Filesystem. Intercept I/O calls to slow them down or return errors (disk full, corrupt reads), checking whether the storage engine ever treats a half-finished write as success, or keeps running in a damaged state after an error.

Two more run throughout: killing processes / restarting nodes, and clock skew, setting a node's time forward or back. TiDB's transaction timestamps come from PD, so local clocks don't directly order transactions, but logic built on timeouts and leases can still be disturbed by a jumping clock, so it gets shaken too.

3. How Raft should behave under a partition

Take the canonical experiment: a three-replica Region, with leader node A cut off from B and C. The steady-state hypothesis: the minority side (A) can no longer commit writes; the majority side (B, C) elects a new leader after the election timeout and keeps serving; no committed data is lost; when the network heals, the old leader steps down and catches up. Any violation is a bug: A still believing it is leader and serving stale data (breaking linearizable reads), or both sides accepting writes (split brain).

4. How to tell something went wrong: metrics plus invariants

Injecting faults is half the job; the other half is detection. The chapter uses two kinds of signal:

• System metrics: watch QPS, latency, leader-change counts and error rates through monitoring such as Prometheus. A QPS dip right after injection is expected; a dip that never recovers, or latency that stays high, is an availability problem.
• Correctness invariants: the layer unique to databases, and the more important one. The classic is the bank test: create accounts, have many clients transfer random amounts between them concurrently, and keep reading all balances and summing them. Transfers are transactions, so the total must stay constant; if any read sees a different sum, there is a lost update, a read of an intermediate state, or another isolation/consistency bug. Similar checks include "an append-only log must never lose or reorder entries."

The beauty of invariants: you don't need to know what the bug looks like, only what must never happen. This is the same spirit as Jepsen, which records the operation history and checks it against the consistency model the system claims.

5. Automation: from manual drills to Schrodinger

Running one partition experiment by hand is easy. Running thousands, every day, against every new build, in every combination, is not. PingCAP built an internal platform called Schrodinger (after Schrödinger's cat: until you open the box, you don't know whether the system is alive or dead), running on Kubernetes. Roughly:

The fault-injection capabilities built into Schrodinger were later extracted and open-sourced as Chaos Mesh: you declare, as Kubernetes custom resources (CRDs), "which pods, what fault, for how long," and any system on Kubernetes can use it. That is the chapter's landing point: for a database, chaos engineering is not a one-off drill but a continuous verification pipeline running alongside CI.

Key trade-offs and choices

The core tension: the more precise the injection, the easier it is to reproduce and pinpoint, but the further it is from real failures; the more realistic, the more it uncovers the unexpected, but the harder it is to reproduce.

Fault-injection methods by layer

Routes to verifying "still correct under failure"

Rule of thumb: a distributed database needs at least "invariant checking plus continuous injection" together. Injection without auditing tests availability, not correctness.

Why it matters / in real systems

In an interview or design review, when asked "how do you prove this store doesn't lose data under failure," saying "we use Raft" is not enough: the algorithm is correct, the implementation may not be. A credible answer lists the invariants to verify (no committed write lost, conserved totals, monotonic reads), names which faults are injected at which layers, and says whether these experiments run automatically before every release. This chapter is a complete template you can copy.

Pitfalls and debates

• "We use Raft / Paxos, so we're consistent." The algorithm's proof covers the protocol, not your code. Snapshots, membership changes and leader leases are the corners where implementations break, and exactly where chaos experiments should aim.
• "The service stayed up, so the experiment passed." A database must be checked for correctness. Chaos without invariant checks finds crashes, not quiet errors.
• "Random injection is enough." Pure randomness is inefficient in a huge combination space; use knowledge of the architecture to target windows like leader changes, Region splits and snapshot transfers (echoing Ch12's experiment selection problem).
• "Found a bug, can't reproduce it." The biggest pain of realistic injection. Preserving evidence (logs, operation history, injection timeline) and narrowing the bug into a reproducible failpoint case matter just as much.
• A test cluster is not production. The chapter's practice mainly runs on dedicated test clusters, unlike Netflix-style experiments in production. For a database, gambling with real user data costs too much: a reasonable trade-off, but one to acknowledge.
• Era limits and what came after. The book dates from 2020, when Chaos Mesh had only just been open-sourced; it has since grown to cover HTTP, JVM, cloud-provider and more faults. The industry also increasingly pairs chaos testing with deterministic simulation and formal methods (such as TLA+), each covering a different blind spot.

Key takeaways