Open Minds, Open Science, and Open Chaos

In one sentence

You think chaos engineering's output is "a few bugs fixed"; in fact its most valuable output is knowledge — and knowledge only compounds when it flows and others can reproduce and challenge it. This chapter argues for importing science's norms of openness into chaos engineering: a collaborative mindset, open-source tools, reproducible experiment designs, and shareable results.

Glossary

Where it sits

Chapter 14 of the book, in Part IV "Business Factors", written by Tammy Bütow, then a principal SRE at Gremlin. Ch13 argued how to show chaos engineering is worth the money; this chapter asks how that value spills beyond a single team. Next comes Ch15's maturity model, whose "adoption" axis is exactly the spreading discussed here.

The problem this chapter solves

The previous dozen chapters cover how to run one experiment well. But a mid-sized company may have dozens of teams and hundreds of services, each experimenting. If every team's findings stay in its own weekly report, three kinds of waste appear:

• Repeated potholes: missing timeouts, retry storms, single-zone dependencies keep reappearing across teams, each time rediscovered at the cost of another experiment — or a real incident.
• Untrustworthy conclusions: "we tested a network cut, no problem" — which path? for how long? watching what? A conclusion nobody can reproduce is one nobody dares rely on and nobody can refute.
• Reinvented tooling: each company builds its own injection framework, nobody audits what it actually does to production, and nobody contributes improvements.

Left unsolved, chaos engineering stays a craft of a few heroic teams, and organizational resilience does not grow with the number of experiments.

The core ideas, in depth

1. A collaborative mindset: a finding is not someone's failure

The chapter opens with mindset. When a team discovers "our service can't handle a dependency timeout", the first instinct is often to hide it — for fear of looking incompetent, or because it feels like private business. The author contrasts this closed mindset with a collaborative one, which treats every disproved hypothesis as a shared organizational asset and invites other teams to look, ask and poke holes.

This continues the book's recurring "blameless" theme: if exposing weaknesses gets punished, people stop exposing them, and chaos experiments become theatre. Openness requires safety — make sure saying "we were wrong" carries no penalty, and sharing follows.

2. Open science: why borrow science's rules

Chaos engineering has called itself an experimental discipline from the start: define a steady state, form a hypothesis, inject real-world events, try to disprove the hypothesis. The author pushes that self-image one step further — if it is the scientific method, it should accept science's norms of openness: public methods, stated conditions, results others can reproduce and review.

The open science movement argues for publishing methods, data and results for a simple reason: a conclusion only stands once someone has independently redone it; a closed conclusion can be wrong with nobody able to notice. Chaos experiments face exactly the same problem — one run is one sample, and a different time, traffic shape or dependency version can flip the conclusion.

3. Open source: open tools are the foundation of trust

The second layer is tooling. Chaos tools carry a special trust problem: you are letting them break things in production. With a closed-source injector you can't see which network rules it changes, which processes it kills, or whether it leaves debris when it fails; open source makes that behavior auditable. Community contributions also mean each kind of injection (instances, network, containers, cloud APIs) need not be rewritten at every company.

The starting point is concrete: Netflix open-sourced Chaos Monkey in 2012, and injection tools kept appearing as open source after that. Open source also brings the chance to standardize the experiment format — once an experiment is a declarative file, it can be shared, versioned and code-reviewed.

4. Open chaos experiments: write designs others can rerun

The third layer, and the most actionable part of the chapter: the experiment itself must be written in a shareable form. "We tested a network cut" carries no information; a reproducible description must answer at least these questions:

The top half of the card is the design: hand it to someone and they can rerun it on their system. The most commonly omitted row is environment and versions — the same injection against a different dependency version or at peak hour can give the opposite answer. Without that row, when a rerun disagrees you can't tell whether the system changed or the experiment did.

5. Experiment findings, shareable results

The bottom half of the card is the result. The author stresses writing results so others can read and use them: what the hypothesis was, whether it was disproved, what was observed, the inferred cause, what was changed, and whether the fix was rechecked. Share the uneventful experiments too — "hypothesis held, all fine" is valuable, because it tells others that path has already been verified.

The audience can widen in stages: the team, then the whole company (a searchable experiment library, regular show-and-tells), then the outside world (engineering blogs, conference talks, open-source experiment templates). Each ring outward adds more people who can find your mistakes, and the conclusion gets harder.

Key trade-offs and choices

More openness is not automatically better; the trade-off is the gains from reuse and review versus the cost of writing and the risk of exposure.

How far to open up: gains and costs

Tooling: open source, commercial, or in-house

Rule of thumb: start with a company-wide card format and shared library — small investment, biggest reuse payoff; prefer tools with open formats so you can exchange experiments later; save public disclosure for conclusions that are rechecked and redactable.

Why it matters / in real systems

This looks like a "culture chapter", but in engineering terms it is concrete: experiments as code. Once an experiment is a declarative file it can live in version control, go through code review and be replayed in a pipeline — the precondition for Ch16's continuous verification. Asked in an interview "how would you roll out chaos engineering across an organization?", the strong answer isn't "run more experiments" but "standardize the experiment format, build a shared library, keep postmortems blameless, make conclusions reproducible."

Databases have a benchmark for open science: the Jepsen project (its open-source repository's tagline: "Breaking distributed systems so you don't have to") runs fault-injection tests against distributed databases and publishes both test code and analyses; many vendors have fixed bugs or rewritten consistency claims in their docs as a result — the outside pressure that "reproducible + publicly reviewed" can create. (Ch19 on chaos for databases returns to this style of work.)

Pitfalls and debates

• "Open means publish everything on the internet." The chapter's weight is actually inside the organization; public disclosure is the last ring and needs redaction.
• "Sharing the tools is enough." Open-source tooling solves one layer; what's actually scarce is reproducible designs and results. Everyone on the same tool, each writing unreproducible conclusions, is still a set of islands.
• "A failed rerun means the original was wrong." Not necessarily: the system, versions or traffic may have changed. That's exactly why environment and versions must be recorded — without them a failed rerun can't be interpreted.
• Tension between security and openness. Some results are vulnerability information; fix and assess risk before disclosing. The chapter says little about this.
• Without a blameless culture, openness becomes theatre. Sharing only the "successful" experiments while hiding disproved hypotheses misleads more than sharing nothing.
• Advocacy over data. The chapter offers no quantitative evidence linking openness to resilience gains, and the author works for a chaos engineering vendor; read with that in mind.

Takeaways