CHAPTER DEEP READ · SRE · CH 5

Eliminating Toil: the work that grows with you is the real enemy

Site Reliability Engineering · Ch 5 · Google (Vivek Rau) · 2016

What is this chapter about?

Behind every app on your phone there is a group of people whose job is to keep it working. This chapter isn't about technology — it's about how those people should spend their hours. And it lays down a rule that sounds surprisingly rigid: at least half of that time must be reserved for work that makes tomorrow's work smaller.

An analogy first

A noodle shop gets popular, the dirty dishes pile up, and the only answer is more dishwashers — twice the customers, twice the people at the sink. Buying a dishwasher is a different kind of work: do it once, save forever. That's the distinction this chapter is built on. The test isn't whether the work is tiring or annoying; it's whether it doubles when your customers double.

The strange part

Most people assume "the work we should eliminate" means "the work I hate." It doesn't. Meetings, expense reports and interviewing candidates are all annoying, and none of them count — they don't double when business doubles, and you shouldn't want a machine doing them anyway. Meanwhile some work feels rather pleasant: a few clicks, hard to get wrong, a small win before lunch. If it grows every time you get more customers, that is exactly the work this chapter is hunting.

The core idea: put a ceiling on it

Google's answer is blunt: this kind of work may not take more than half of anyone's time, and the other half must go into making it smaller. Why does it have to be a rule? Because without a line, it eats everything on its own. It is always urgent, someone is always waiting, and someone thanks you when it's done — whereas rebuilding the process takes two months during which nobody thanks you at all. Urgent beats important unless you lock the important work's hours in advance. Look up a year later and the whole team is doing what a machine could do: nobody is growing, the strongest people leave, and the ones who stay inherit more of it.

How you actually do it

The first step isn't automation — it's keeping the books: what share did this work take this month, and which two or three things account for most of it? Almost every team overestimates how automated it already is, and almost every team finds the bulk sitting in a couple of unglamorous places. Then attack only the biggest one or two — one Google team automated the swap-out procedure for network hardware over roughly a year, and it has saved them hundreds of hours a month ever since. The best outcome isn't even "let a machine do it"; it's making the task unnecessary: rather than building a robot to click approve, delete the approval step. The honest cost: automation has to be fed too — months to build, bugs to fix afterwards, and one day it will press the wrong button at 3 a.m. on your behalf. So if a task only costs you a dozen hours a year, leave it alone; the math doesn't work.

One line to remember

To judge whether a task deserves to die, don't ask "does this annoy me?" Ask "if we get ten times the customers, does this get ten times bigger?" If yes: cap it, measure it honestly, and attack the largest piece. If no, that may simply be the job.

Want the mechanisms, numbers and diagrams? → Switch to the deep read

In one line

You think toil means "grunt work, the stuff I don't want to do." In fact the deadliest clause of the definition has nothing to do with how it feels: toil scales linearly with the service. This chapter turns that into an operable discipline — a hard six-part definition, a clean separation from "engineering" and "overhead", a 50% ceiling, and a standing requirement to measure. Toil isn't the work that makes you unhappy; it's the work that forces headcount to grow with traffic. Leave it in place and you pay for scale in people.

Glossary

Toil: the chapter's core term — repetitive work tied directly to running a production service (six characteristics below). It is not a synonym for "grunt work".
Overhead: administrative work not tied to running a service — hiring, meetings, self-assessments, expense reports. Not toil.
On-call: taking the pager in rotation and responding first when the service breaks; usually a primary and a secondary shift.
Interrupt: a non-urgent but human-requiring request about the service — a ticket, an email, a "can you take a look?" in chat. One of the top sources of toil.
Release / push: getting new code or configuration into production. The more manual steps it has, the more it is toil.
Runbook: an operational procedure written down step by step so anyone can follow it — the halfway house between "by hand" and "automated".
O(n) growth: plain English, "double the scale, double the work." O(1) means the work stays flat no matter how big it gets. The chapter's whole test lives here.
SLO and error budget: the products of the two previous chapters — a reliability target and an allowance for how much may break. They answer how much reliability; this chapter answers where the hours go.

Where it sits

Part II, "Principles", right after Ch3 (Embracing Risk) and Ch4 (Service Level Objectives). Those chapters set how reliable the service should be; this one governs where human time goes, and it turns Ch1's slogan — ops work capped at 50% — into something you can actually audit. Downstream, Ch6 (Monitoring) designs the alerts (alerts and tickets being the largest toil sources), Ch7 (The Evolution of Automation) covers how to actually kill toil, and Ch9 (Simplicity) covers how to avoid creating it. Real-world match: any SRE, DevOps or platform team, plus every backend team that spends half its week writing code and the other half firefighting.

The problem it attacks

Disease 1: operational work eats the calendar by itself. It is always urgent (someone is waiting), always completable (you can close a ticket within the hour), and someone thanks you afterwards — whereas rebuilding the release process takes two months with zero feedback in between. Without an enforced quota, urgent always beats important. That's not a discipline problem, it's an incentive-structure problem, which is why the answer isn't exhortation but a hard line.

Disease 2: this work is O(n), and services grow. This is the real enemy of the chapter. A manual procedure that eats 5 hours a week isn't fatal; what's fatal is that it becomes 10 hours when traffic doubles and 20 when it doubles again. Nobody has to get worse at their job and no extra outage has to happen — business success alone is enough to grow it until it consumes everyone. Run the numbers (this arithmetic follows the book's framing but the figures are mine): an 8-person team has 320 person-hours a week, so the ceiling is 160 hours of toil. A 24×7 two-tier rotation absorbs ~28 hours of interrupts, three releases a week at 4 manual hours each is 12, and 30 quota/permission tickets is ~20 — about 60 hours, or 19%. Very healthy. But all three line items are O(n): double the scale and it's 120 hours (38%); double again and it's 240 hours (75%) — over the line, without a single person having done anything wrong.

Disease 3: nobody can say which work deserves to die. "The work I dislike" is simultaneously too broad (it swallows meetings and interviews) and too narrow (it misses the few-clicks-and-done tasks that are almost relaxing). Without a shared definition, "we have too much toil" is just a complaint — a manager can neither verify it nor act on it.

The core content

1. Six characteristics: the hard definition, and what toil is not

The book's definition is one long sentence: toil is work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows. The six characteristics aren't a menu where any one qualifies — they're a composite portrait, and each one blocks a different misjudgement:

Table 1 · The six characteristics, each with a case that qualifies and one that doesn't

Characteristic	What it means	Toil	Not toil
Manual	A human has to act — including running an automation script by hand	SSH in and run `deploy.sh` for every scale-up	The script fires on an event, unattended
Repetitive	Doing it for the first or second time doesn't count	The 14th quota change this month	First time debugging a new class of failure
Automatable	A machine could do it as well — or the need could be designed away	A 12-step swap procedure read off a runbook	An architecture review that needs judgement
Tactical	Interrupt-driven and reactive rather than strategy-driven	An "@you, can you look at this?" in chat	Proactively redoing the capacity model
No enduring value	The service is in the same state when you finish	Restarting a stuck job	A one-off cleanup so it stops getting stuck
`O(n)` growth	Grows linearly with scale, traffic or user count	Manually opening an account per new customer	Config done once regardless of customer count

What it is not matters just as much. The book opens by saying it outright: toil is not just "work I don't like to do." Three boundaries worth memorising: (1) dirty work isn't automatically toil — spending two days cleaning up a tangled configuration so it never breaks again is engineering with enduring value; (2) overhead isn't toil — hiring, meetings and self-assessments consume time but don't grow with the service, so they belong in a different bucket; (3) work that feels good can still be toil — the book specifically notes that small amounts of repetitive work can be calming: low risk, quick wins, a sense of accomplishment. The test is the growth curve, not the feeling.

2. Four buckets: dividing up a week

A definition of toil alone isn't usable — otherwise "what counts as the other 50%?" never gets settled. The book splits the time four ways: software engineering (writing automation, tools and frameworks, or adding scalability/reliability features), systems engineering (configuring production systems, changing configuration, documenting — characterised by a one-time effort producing lasting improvement: monitoring setup, load balancer configuration, OS tuning, plus architecture and productionisation consulting for dev teams), toil, and overhead. The first two together are "engineering", and that is what the 50% floor refers to.

The sorting test is "what is left behind when you finish": software engineering leaves code you can reuse, systems engineering leaves a lasting improvement bought with one push, and toil leaves nothing — the service returns to its previous state and waits for the next time. Overhead simply isn't tied to running the service at all. Neither of the last two counts toward the 50%, but don't inflate your toil number by counting overhead in it.

Fig 1 · Top: a healthy team, engineering (blue) outweighing toil (violet). Bottom: toil past the cap squeezes engineering to a sliver — note overhead is counted separately.

3. The `50%` cap: why it has to be a hard line

The rule itself is short: at least 50% of each SRE's time goes to engineering project work, so toil must stay under 50%. The important half is the second one — this is a ceiling, not a target. The internal survey quoted in the book puts Google SREs at roughly 33% toil, comfortably under the cap, with enormous variation between teams. Running the team as if 50% were the goal is the most common misreading. The same survey names the top sources: interrupts (non-urgent service-related messages and email), urgent on-call responses, and releases and pushes — which means most teams' toil isn't hiding anywhere exotic. It's in the ticket queue, the pager history and the release process, and fishing there always catches something.

So what happens when you're over? This isn't solvable by trying harder, and the book's answer is organisational: managers measure and intervene, pushing the overflow of operational load back to the product development team (Ch1's relief valve — work beyond the cap returns to the developers who wrote the service, and in the extreme the pager goes back with it), or adding people. It is the existence of that valve that turns 50% from a wish into a constraint.

4. `O(n)` is the enemy: why the line eventually breaks

The first three sections are taxonomy; this one is the physics. What makes toil dangerous is not how much of today it takes, but that its growth curve is bolted to your business success. The book sets a demanding bar: a well-designed, well-managed service should be able to grow by at least an order of magnitude with no more than one-time effort to add resources, day-to-day work staying flat. Put the other way round — if ten times the traffic means ten times the people, that isn't "business going well", that's a design defect.

Fig 2 · One cap, two fates: linearly growing toil hits the ceiling eventually, while toil that has been automated or designed away stays nearly flat.

5. Toil isn't always bad — the problem is the dose

This is the nuance most readers skip. The book says plainly that small amounts of toil don't make people unhappy; predictable repetitive tasks can even be calming — low risk, low stress, quick wins, a sense of accomplishment. It turns toxic through volume. The seven consequences the book lists are all organisational rather than technical: career stagnation (too little engineering output for growth), low morale, confusion (the team gets seen as a pure ops team), slowed progress, precedent (if SRE absorbs everything, developers keep sending more), attrition, and breach of faith (people who joined to do engineering find themselves doing chores).

Fig 3 · The positive feedback loop: toil consumes exactly the time needed to kill it, which is why the loop has to be broken from outside.

Key trade-offs

The trade-offs here aren't about picking technology. They're about how far up the ladder you push a given piece of repetitive work — and when you deliberately don't.

Table 2 · Five levels of response to a repetitive task: cost rises, residual manual work falls

Response	One-time cost	Cost / risk	When to pick it
1 · By hand	None	Grows linearly; steps live in someone's memory and go wrong	Things that happen two or three times a year
2 · Runbook	Hours	Runbooks rot; easily becomes the excuse never to automate	Procedure just stabilised, still changing; many new people
3 · Semi-automated script	Days	"Running an automation script by hand" is still toil; the script needs upkeep	Usually the best return — most teams should stop here
4 · Closed-loop automation	Weeks to months	Automation makes mistakes at machine speed and fleet scale; it is itself a new source of toil	High frequency, stable steps, failures safely reversible
5 · Design it away	Highest; often architectural	Touches product and process — rarely one team's call	The best answer: instead of a robot that clicks approve, delete the approval

Fig 4 · The five-level ladder and the manual work left at each step (illustrative magnitudes). Note that the human click at level 3 still counts as toil by definition.

So should you automate it? Treat it as an investment: annual saving = time per run × runs per year; cost = build hours + annual maintenance. A task eating 5 hours a week costs 260 hours a year; automating it at roughly 3 engineer-weeks (120 hours) plus 20 hours a year of upkeep pays back in about six months — do it. A task eating 1 hour a month (12 hours a year) against the same 120-hour build takes a decade to break even — keep doing it by hand. And there's a multiplier people forget: the service's remaining lifetime. Automating a system that gets decommissioned in nine months is wasted work no matter how good the spreadsheet looks. This is exactly the first advice Google's network repair team gave: measure first, attack the biggest block, and use the hours saved to compute the return.

Table 3 · Four organisational responses when toil goes past the cap, and what each costs

Response	How it works	Cost / precondition
Push the overflow back to developers	Ch1's relief valve: operational load beyond `50%` returns to the team that wrote the service; in the extreme, so does the pager	Requires organisational backing — the hardest thing in the book to copy; most SRE teams have no such leverage
Add people	Hire until per-person toil falls back under the line	Only moves the date: toil is `O(n)`, so headcount doesn't change the slope
Freeze onboarding	Stop taking on new services until existing toil is automated away	Sacrifices short-term support; needs explicit sponsorship
Narrow the commitment	Reduce scope or relax the SLO so some toil disappears at the source	Has to be negotiated with Ch4's SLOs — a product decision, not an engineering one

One last trade-off: measuring costs time too. Google's published guidance insists on keeping it lightweight — record three things (what type of work, difficulty easy/medium/hard, who did it) and estimate monthly or quarterly, without chasing precision. This matters: plenty of teams turn toil accounting into a new form of toil, spending half an hour a week on spreadsheets and eating the savings right back.

Why it matters / in real systems

The chapter's real contribution is giving operational work a unit of measure that management can read. Before it, "we're swamped" was an unverifiable complaint. After it, the sentence becomes "toil was 62% this quarter and 70% of that is quota tickets" — a number you can put in an OKR, use to ask for resources, and use to decline new work. The self-service portals, zero-ticket goals and unattended releases you see across platform engineering all trace back here. It also rewrote the right answer in interviews: asked "how do you handle operational automation", saying "we wrote a lot of scripts" fails. The passing answer is how you measure toil, what share it is, what the top three sources are — then which one you attacked, why that one, and how many hours came back.

EVIDENCE FROM INDUSTRY · not the book's claims, but what companies said publicly

Google's network engineering team published an end-to-end toil-elimination case: fault detection and component replacement across their global network used to be ticket-driven and performed manually by engineers. They abstracted "any ticket type we encountered repeatedly that follows a predetermined sequence of steps" behind a common interface and state machines, leaving datacenter technicians only the physical swap. The result is stated flatly — "network hardware repair automation saves us hundreds of hours every month." Their recommended order is worth copying: measure your toil, tackle the biggest sources first, don't try to solve everything at once, and measure the time savings to determine ROI.James O'Keeffe, Google Cloud, "Repairing network hardware at scale with SRE principles", 2018 ↗
Google Cloud later turned "how to measure toil" into public guidance, deliberately kept light: capture three pieces of metadata per ticket — what type of work it was (quota changes, pushes to production, ACL updates…), the degree of difficulty (easy, < 1 hour; medium, hours; hard, days), and who did it — and estimate monthly or quarterly. The post is explicit: "the emphasis is on lightweight in this step. Extreme precision has little value here."Eric Harvieux, Google Cloud, "Identifying and tracking toil using SRE principles", 2020 ↗
Lowe's is the evidence from outside Google: they adopted the definition wholesale — "we think of toil as work that is manual, repetitive, tactical, devoid of enduring value—but automatable" — and made it a goal that engineers are not the first point of contact for any alert, handing reproducible triage and resolution to machines. Their reported outcome: releases went from one every two weeks to 20+ daily, each taking under 30 minutes in a single click including automated quality checks.Vivek Balivada & Rahul Mohan Kola Kandy, Lowe's / Google Cloud, 2021 ↗
The concept has grown beyond SRE. Google Cloud's security team transplanted toil into the security operations center unchanged: after quoting the same definition, they argue you should "aim to minimize your ops time to 50%; try spending the remaining 50% on improving systems and detections with an 'automate-first' mindset."Iman Ghanizada & Anton Chuvakin, Google Cloud, "Achieving Autonomic Security Operations: Reducing toil", 2021 ↗

Pitfalls and disputes

Treating "work I dislike" as toil. The most common misuse. Meetings, hiring and self-assessments are overhead; counting them only inflates the number. Conversely, the few-clicks-and-done task that feels relaxing is often exactly toil.
Treating 50% as a target rather than a ceiling. It's the roof; the book's own survey lands near 33%. A team running at a steady 49% isn't healthy, it just hasn't hit the wall yet.
Treating automation as the only answer. The top of the ladder is designing it away: instead of a robot that approves quota requests, make quota self-service or delete the approval. An automated bad process is still a bad process.
Forgetting that automation has its own cost. Ch7 warns specifically that automation lets mistakes happen at machine speed and fleet scale. An unattended repair system that gets it wrong has a far larger blast radius than one flustered human.
"Push it back to the developers" depends entirely on organisational power. Google's SREs really can decline to onboard a service and hand back the pager; most companies' ops teams cannot, and copying the cap without the valve reduces it to a poster on the wall. A 50% cap without a relief valve isn't a constraint, it's a wish.
A counter-view worth knowing: the framework structurally favours people who write code — it files on-call, debugging and cross-team communication, all of which lean hard on experience and judgement, into the column marked "should be handed to machines". The book leaves room for nuance, but in practice "eliminating toil" is easily flattened into contempt for operations work.
Period limits: in 2016 managed cloud services, infrastructure-as-code and the Kubernetes ecosystem hadn't taken their current shape — and while they killed one generation of toil, they created another (who operates the automation?). The Site Reliability Workbook (2018) devotes its Chapter 6 to implementation detail and case studies. This chapter gives you the test and the discipline, not a tool list.

Takeaways

1 · Definition: toil is work tied to running a production service that is manual, repetitive, automatable, tactical, devoid of enduring value, and grows linearly with scale. The last clause is the test — not your feelings.

2 · What it isn't: not "work I dislike"; overhead (hiring, meetings, expenses) doesn't count; doing something the first or second time doesn't count; dirty work with enduring value is systems engineering.

3 · Four buckets: software engineering + systems engineering = "engineering", the other two are toil and overhead; the 50% floor refers to the first two. 50% is a ceiling, not a target — the book's survey sits near 33%.

4 · The top three sources are named: interrupts, urgent on-call responses, releases and pushes. Go fishing in the ticket queue, the pager history and the release process.

5 · O(n) is the enemy: a well-managed service should grow an order of magnitude on one-time resource work alone. Ten times the traffic needing ten times the people is a design defect, not a business win.

6 · Dose decides toxicity: small amounts of toil can even be calming; large amounts bring career stagnation, low morale, confusion, slowed progress, precedent, attrition and breach of faith — all organisational diseases.

7 · Five levels of response: by hand → runbook → semi-automated → closed loop → designed away. Running an automation script by hand is still toil, and the best outcome is usually that the task stops happening. Whether to automate is an investment question: annual saving vs build plus upkeep, multiplied by remaining service lifetime.

8 · Going over the cap needs mechanism, not willpower: managers measure and intervene, pushing overflow back to developers (Ch1's valve) or adding people. A cap without a valve is a wish. The chapter closes with an instruction — commit to eliminate a bit of toil each week with some good engineering.