The Case for Security Chaos Engineering

In one sentence

You think security comes from adding another layer of defense, but most defenses have never been tested under real conditions, and whether they work is only revealed on the day of a real incident. In this chapter Aaron Rinehart carries chaos engineering over to security: proactively and safely inject "security failures" (most typically, human misconfiguration), write down a hypothesis first, then verify that preventive, detective, and response controls actually behave as expected, so you find failing controls before an attacker does.

Glossary

Where it sits

The second-to-last chapter of Part V, "Evolution," after Ch19 "Chaos Engineering on a Database" and before the Ch21 conclusion. Earlier chapters ask "does the system still work when it breaks?"; this one applies the same empirical method to security, where the adversary is a person with intent. In practice it maps to cloud security operations, compliance, and incident response.

What problem this chapter solves

Security teams traditionally work by stacking defenses: buy another product, add another rule. But the more controls you stack, the more complex the system gets, and the less anyone can say whether the whole set will fire in a specific situation. Worse, there is no feedback: once a detection rule is written, its first real test is a real attack, and failing that test costs you an actual breach.

The argument has three steps. ① In complex systems people will make mistakes (a wrong setting, a port left open), and no amount of training eliminates that. ② Attackers mostly don't take exotic routes; they pick this low-hanging fruit. ③ So defenders need a continuous feedback loop that actively and repeatedly confirms "when these common mistakes happen, we stop them, see them, and handle them." Without it, your security posture is a belief, not a measured fact.

Core ideas, in depth

1. Definition: from "find vulnerabilities" to "find failing controls"

The chapter's definition is roughly: security chaos engineering is the identification of security control failures through proactive experimentation, to build confidence in the system's ability to defend against malicious conditions. It asks a different question from a pen test:

• A pen test asks: "Can an attacker get in?" The answer is a list of vulnerabilities.
• Security chaos engineering asks: "When a mistake we know will happen does happen, do the controls we believe in actually work?" The answer is evidence about the defenses themselves.

It reuses the experiment skeleton from earlier chapters: define steady state → hypothesize → inject a condition → observe → fix and retest. Only the steady state changes, from "checkout success rate" to "what should be blocked is blocked, the alert fires within the agreed time, and the on-call person gets enough context."

2. Human factors: misconfiguration is the norm, not the exception

The author puts security back inside the "complex systems" frame. Modern systems are stitched together from many microservices, cloud resources, and automation scripts; nobody holds the whole picture, and engineers make locally reasonable changes under time pressure. Misconfiguration is a natural by-product of the system running, not one person's negligence (the same line as Ch18's "human error is a symptom, not a root cause").

Two practices follow. First, remove the low-hanging fruit: attackers favor cheap entrances such as open ports, over-broad permissions, and public buckets, so confirm that these common mistakes get blocked or noticed the moment they appear, forcing attackers onto costlier paths. Second, close the feedback loop: security controls rot just like code, and the only way to keep knowing they are alive is to trigger them actively and repeatedly.

3. Compared with current methods: where red and purple teams fall short

The chapter takes apart the limits of the two mainstream exercises:

• Red team: the outcome is "did the attackers win," with the focus on attack techniques. It is point-in-time (a few weeks of work while the system keeps changing), and the output is often a report that embarrasses the defenders and rarely turns into systematic improvement of the controls.
• Purple team: putting attack and defense in the same room is far more collaborative, but it is still expensive, labor-intensive, and infrequent, with limited scenario coverage.
• Security chaos engineering: it targets the defenders' controls, is hypothesis-driven, and can run automatically and continuously. There is no winning or losing and no blame; the output is reproducible evidence of which control fails under which condition.

They are not mutually exclusive: red teams excel at finding attack paths you never imagined; security chaos engineering excels at confirming that the paths you believe are covered really are.

4. Security Game Days: run it through people first

Before automating, the author suggests starting with a security Game Day. The team picks a time, injects a security condition within a controlled scope (say, an unusual permission change), and watches the whole chain: does the alert fire, whose hands does it land in, can the on-call person read from the logs what happened, where, and how bad, and does the runbook work? Half the value is in technical controls and half in people and process: many incident responses fail not because there was no alert, but because the alert reached someone who couldn't act on it.

5. ChaoSlingr: turning the port experiment into a tool

The chapter's worked example is ChaoSlingr, a security chaos framework for AWS that the author's team built and open-sourced at UnitedHealth Group. It is made of a few serverless functions: Generatr picks a target among resources explicitly tagged as opted in to experiments; Slingr performs the injection, and the signature experiment, PortSlingr, opens or closes a port on a security group that should not change; Trackr records each step of the experiment and sends notifications for later comparison.

The first results the author recounts are the "strange story" from the Simple view. The team assumed the firewall would detect and block the change every time; in fact it caught only about 60%. A cloud-native configuration management tool, not bought for security at all, caught it every time. And when the alert reached the SOC, the logs lacked the context to tell which account or system the event came from. None of the three findings is a "vulnerability"; all are failures of controls and process, exactly the kind that rarely shows up in a pen test report and hurts most in a real incident.

Two design details are worth keeping. First, explicit opt-in: experiments only touch tagged resources, which limits the blast radius and lets teams enroll themselves. Second, every experiment leaves a trail: Trackr makes each injection auditable, so a drill isn't mistaken for a real intrusion and the review afterward has data.

Key trade-offs and choices

The central choice: are you asking "can an attacker get in?" or "do the controls I trust actually work?"

Method	Question it answers	Frequency	Cost / limits
Pen test	Which exploitable weaknesses exist within the agreed scope	Typically one to a few times a year	Point-in-time; doesn't test detection or response
Red team	Can a realistic adversary reach a goal undetected	Rare, project-based	Expensive; win/lose framing breeds defensiveness, findings hard to act on
Purple team	Which attack techniques are visible, attackers and defenders together	Rare	Labor-intensive, limited coverage
Security Game Day	Can people and process take over an incident	Monthly / quarterly	Needs cross-team coordination; hard to scale
Security chaos experiment	When known common mistakes happen, do controls behave as hypothesized	Automated, continuous	Only tests conditions you thought of; production injection needs a tight scope

In practice you also choose which layer to test first:

Layer	Typical injection	What to watch	Common trap
Prevent	Open a port, loosen a permission policy	Is it blocked or auto-reverted	Assuming "we have a firewall" means "it always blocks"
Detect	Disable a log, make an unusual permission change	Does an alert fire, and how fast	The alert exists but lacks the context to locate the source
Respond	Game Day style simulated incident	Can on-call locate and contain it within the agreed time	Stale runbooks, missing access, no clear owner

Rule of thumb: start with low-risk configuration experiments (ports, log switches) in non-production or explicitly opted-in resources, prove the stop / see / handle layers line up, then widen the scope; only move into production what has an automatic rollback behind it.

Why it matters / in real systems

The cloud turned infrastructure into code: one API call opens a port or makes a bucket public. Change frequency went from quarterly to hourly, so the snapshot from the annual audit is stale the next day. That makes continuous control verification a necessity rather than a nicety, and it's why the chapter's tools target cloud configuration. When an interview or architecture review asks "how do you know your detection rules work?", the answer shouldn't be "we configured them" but "we regularly inject known events and have measured hit rate and time to locate."

Pitfalls and debates

• "Security chaos means attacking production." No. You inject controlled, reversible "common mistakes," you target the defenders' controls, and you start with opted-in resources.
• "An alert fired, so detection worked." ChaoSlingr's lesson is exactly the opposite: the alert fired, but without context the SOC still couldn't act. Detection's acceptance criteria must include locatable and actionable.
• "It replaces pen tests and red teams." It doesn't. Chaos experiments only verify conditions you thought of; finding unknown attack paths still takes a creative adversary. They complement each other.
• "Find a gap, find someone to blame." Once results are used for blame, teams resist experiments and hide problems, and the feedback loop breaks. It must be blameless.
• "An injection will never be mistaken for a real incident." Without trails and notifications, the SOC may respond for real, wasting effort or causing side effects. That's why components like Trackr exist.
• Limits of its era, and what came after. Written in 2020, the examples center on AWS network configuration, and ChaoSlingr itself is now archived. The author and Kelly Shortridge later expanded the idea into a full book, Security Chaos Engineering (O'Reilly, 2023), shifting from "inject misconfigurations" to a broader view of system resilience.

Key takeaways