HOP Meets Chaos Engineering

One line

You think an incident review ends when you find "the person who made the mistake" — in fact that is where it should start. This chapter brings the five principles of Human and Organizational Performance (HOP) from industrial safety into software — error is normal, blame fixes nothing, context drives behavior, learning and improving is vital, intentional response matters — and argues that chaos engineering is itself a way of putting those principles into practice: it causes failure deliberately and under control, studies how the system (people included) copes under stress, and changes the setting instead of the person.

Glossary

Where it sits

Chapter 18, Part V "Evolution," written by HOP practitioner Bob Edwards, whose background is industrial safety rather than software. It follows Ch17, which pushes chaos engineering into cyber-physical systems, and precedes Ch19 on chaos experiments for databases. It is the only chapter in the book devoted to the people in the system, and maps to incident review, on-call culture and safety culture in practice.

What problem this chapter solves

Picture 3 a.m.: an on-call engineer follows the runbook to remove some servers, types one extra character, and takes out 200 machines instead of 2. The service is down for hours. A traditional review concludes quickly — "human error; more training, more approvals." Case closed. But the command that allows removing 200 machines at once, the rota that leaves one person alone at night, and the runbook that no longer matches real practice are all still there.

Industrial safety has watched the same ending play out again and again in chemical plants, utilities and factories: attribute the accident to an individual, close with discipline, and the next person in the same setting makes the same mistake. Worse, discipline stops the front line from reporting near misses and small problems, so the safety numbers management sees keep getting prettier — until a major accident shatters the illusion.

Software teams face the same problem, only faster: hundreds of changes a day, systems too complex for anyone to hold in their head. The chapter asks: how should we think about the people in a system so that incidents actually become less severe — and how does that change the way we design and review chaos experiments?

The core ideas, in depth

1. What HOP is: from fixing people to fixing systems

Human and Organizational Performance (HOP) is a philosophy that grew out of industrial safety, strongly influenced by safety thinkers such as James Reason, Sidney Dekker and Todd Conklin. It is not a toolkit but a shift in how you look at accidents:

• Old view: the system is safe by design; accidents come from people who break rules or are careless. The fix is more rules, more training, harsher punishment.
• New view: people are trying to do good work in a system full of time pressure, conflicting goals and incomplete information; errors are symptoms of deeper problems surfacing. The fix is to change the context and build the capacity to fail safely.

2. The five principles, one by one

① Error is normal. Even well-trained people make mistakes; it is a property of being human, not a matter of attitude. So the goal is not "zero errors" but a system that still fails safely when an error happens: limit the impact of any single action, catch errors early, make them quick to undo. Chaos engineering starts from exactly the same premise — failure will come, so verify in advance that the system can absorb it rather than hoping.

② Blame fixes nothing. Blame feels like resolution, but it only puts a different person at the edge of the same hole. The bigger cost is information: once admitting a mistake invites punishment, people hide near misses and the organization loses its cheapest early-warning signal. That is why the book insists on blameless reviews in several places.

③ Context drives behavior. People's choices are shaped by time pressure, tool design, incentives and process. To change behavior, change the context: rather than asking people to "be careful with parameters," make the tool refuse to remove more than 5% of capacity in one go. A useful pair of terms here: work-as-imagined, the way manuals and procedures describe the job; and work-as-done, the way the front line actually gets it done under real constraints. The gap between the two is where risk hides.

④ Learning and improving is vital. Organizations need to learn from the front line actively and continuously, not only after something breaks. One tool is the learning team: bring in the people who actually do the work and ask, without judgment, "how does this normally get done, where is it awkward, where does it rely on luck?"

⑤ Intentional response matters. The first sentence a manager says on hearing bad news decides whether anyone tells the truth next time. Responses should be deliberately designed: thank the reporter first, ask about context first, then talk about improvement.

3. HOP meets chaos engineering: how experiment design changes

The chapter's central argument is that chaos engineering and HOP point at the same thing — accept that failure is inevitable, and actively learn how the system fails before it does harm. For years HOP has studied systems "indirectly" on the industrial floor through interviews and learning teams; chaos engineering offers a way to observe directly, and under control, how a system (people included) responds to failure. Mapping the five principles onto experiments:

• Put people inside the system boundary. A fault-injection hypothesis is not only "the service fails over automatically" but also "the on-call engineer understands the alert and takes the right action within 15 minutes." A game day tests people and tooling together.
• Experiments look for gaps, not culprits. If someone strays from the runbook during a drill, the right question is "why didn't the runbook work at that moment?" — which often exposes the gap between work-as-imagined and work-as-done.
• Make the context realistic. Drills run only in daytime with senior engineers in the room tell you nothing about one person alone on call at 3 a.m.; but raise realism step by step with the blast radius, not with a surprise attack on day one.
• Review blamelessly. Weaknesses an experiment exposes must count as gains, not faults — or next time people will pick "safe" scenarios and go through the motions.
• Leadership's response is part of the experiment. How the organization reacts after a big gap is found decides whether chaos engineering can last.

Key trade-offs and choices

The central trade-off from a HOP point of view: once an incident or experiment exposes a problem, do you spend your improvement effort on people or on context?

Four typical responses after something goes wrong

How the five principles rewrite a chaos experiment

Rule of thumb: if an improvement only works when people never make mistakes, it is not an improvement. Prefer measures that still work when someone errs, and use chaos experiments to prove that they really do.

Why it matters / real systems

Blameless postmortems are now standard in SRE teams, but many teams copied the template without the HOP logic behind it: their action items still say "more training" or "operators must be careful." The chapter's test: every action item should change the context, not ask people to be more careful. When an interview or architecture review asks "this outage was human error — how do you prevent it?", the strong answer points the same way: limit the blast radius of any single action, add undo and rate limits, make errors visible, and verify all of it with drills.

Pitfalls and debates

• "Blameless = nobody is accountable." Blameless targets punishment, not accountability. People still explain what they did and help with the fix; what is removed is being punished for telling the truth. Malice or reckless disregard is another matter — that distinction is exactly what a just culture draws.
• "Human error was the root cause." Writing it down as the root cause means stopping the investigation exactly where it should go deeper. Complex systems usually have several contributing factors.
• "Drills are an exam for on-call engineers." Once drill results feed into performance reviews, people choose safe scenarios and hide real practice, and the experiments lose their point.
• "One more manual approval makes it safe." Approvers usually have less context than the person doing the work; more approvals become a formality. Put defenses in the tooling wherever possible.
• The move from factory to software has limits. HOP's experience comes from physical-harm settings with a rhythm of weeks and months; software changes happen in minutes, so interview-heavy methods like learning teams need to be made lighter. The chapter is also mostly principles and ideas with little quantitative evidence, and how to measure HOP's effect remains an open discussion in safety circles.

Key takeaways