BOOK DEEP READ · SRE · CHAPTER 2
Site Reliability Engineering · Ch 2 · Google (JC van Winkel) · 2016
You type a word into Google and get results in a fraction of a second. Behind that is not "one very powerful machine" but tens of thousands of ordinary machines, plus a stack of software that glues them into a single machine. Chapter 2 of the SRE book is a guided tour of that computer-the-size-of-a-datacenter. It teaches no practice at all; it establishes the physical premise for every practice in the dozen chapters that follow.
Think of a giant factory. Old-school operations treated machines like pets: each had a name, a personality, and someone who got out of bed at 3 a.m. when it got sick. That works for ten machines. At a hundred thousand it does not — statistically, some of them are broken every single day.
So Google hired a foreman. You no longer say "run this program on machine 37." You post a work order: "I need twenty copies of this, each with a two-core, four-gigabyte workstation, and don't put all twenty in the same row." The foreman does the scheduling. When a worker falls ill, it moves the work to someone else — and you never even hear about it.
The hard part isn't that machines break. It's that once there are enough of them, breaking stops being an incident and becomes the weather. Humans can't watch that, and shouldn't. The real job is to hide "which machine" entirely, so people only care whether the service is up and fast. This chapter's value is that it opens the hood on how that hiding is done.
The chapter opens by splitting one word in two. In everyday speech "server" means two different things: a box of metal, and a program running on it. The book forces them apart — the metal is a machine, the program is a server. One machine routinely runs several unrelated programs at once; conversely, one service's dozens of copies are scattered across dozens of machines. Once you hold those apart, the crucial sentence finally parses: the machine died, the service didn't.
The rest is three foundations. A scheduler that places programs onto machines and re-places them when things break. A stack of storage where you save one file and it is spread across thousands of disks, several of which can die without you noticing. And an absurdly fast internal network that makes "crossing between machines" cheap enough to do freely — which is exactly what lets the work be split across tens of thousands of small machines in the first place. Plus one unglamorous, load-bearing piece: a notice board everyone trusts, holding "who is the leader right now" and "where does this service live," so nobody disagrees about the basics.
Machine failure is demoted from incident to routine, nobody gets paged at night, and scaling becomes editing a number. The cost fits in one sentence: everyone stands on the same foundations, so if that notice board gets muddled, everybody gets muddled together.
Google's production environment is not "a lot of servers." It is one computer the size of a datacenter: scheduling, storage and networking are abstracted into a shared substrate, and people declare what they need instead of asking which machine it lands on.
Want the actual components, the layer diagrams and the trade-offs? → Switch to the deep read
You think SREs manage a pile of machines. They manage one computer the size of a datacenter. Chapter 2 teaches no practice; it states the physical premise. Google uses Borg (scheduling), Colossus (storage), Jupiter / B4 (networking), Chubby (coordination) and Stubby (RPC) to abstract tens of thousands of cheap machines into a single programmable machine. Understand that substrate and the monitoring, automation and overload chapters that follow stop being slogans.
This chapter belongs to Part I, Introduction, right after Chapter 1, and it is the book's only purely descriptive chapter — author JC van Winkel is handing you a site map. Almost everything later stands on it: monitoring (Ch 6, and Borgmon in Ch 10), automation (Ch 7, which operates Borg), overload and cascading failure (Ch 21–22, RPC and load balancing), critical state (Ch 23, Chubby). In today's terms, it is the prototype of the whole Kubernetes + object storage + service mesh arrangement.
It solves something unglamorous and fatal: if the vocabulary doesn't line up, nothing later parses. "Move that task to another machine" means a change request requiring approval to a traditional ops team, and a few thousand times a day to Borg. Without a shared mental model, readers misread Google's practices as mere bravado.
The deeper reason is that scale changes the nature of the problem. A cluster can hold tens of thousands of machines; at published industry disk failure rates (low single-digit percent annually), something is broken every day at that size. A service with 2000 tasks will see tasks restarted or preempted more or less daily. So Google's answer is not "stop machines from failing" but delete the concept of "machine" from the developer's field of view: you declare what you need and never ask where it lands. The cost of not doing this is blunt — people drown in machine count, and the SRE rule that operational work stays under 50% of the time does not survive a single day.
The chapter's opening definition: a machine is hardware (or a VM); a server is software implementing a service. This is not pedantry. One machine simultaneously runs a search backend, a log-cleanup task and someone else's MapReduce; meanwhile the "search frontend" service has hundreds of copies spread over hundreds of machines. The relationship is many-to-many, not one-to-one. Traditional ops scales badly precisely because it assumes one-to-one: one machine = one service = one hostname = one owner.
The hardware itself is layered: a rack holds dozens of machines, racks are arranged in rows, one or more rows form a cluster, a datacenter building holds several clusters, and adjacent buildings form a campus. That hierarchy is not decor — it is the boundary of failure domains and bandwidth. Two machines in one rack talk fast and cheap, but lose power together; across datacenters they don't die together, but latency jumps an order of magnitude. Every later argument about "where do the replicas go" uses this ruler.
Borg is a cluster operating system — the chapter likens it to Apache Mesos. What you submit is not "start a process on machine 37" but a declaration: this job needs 20 tasks, each with 2 cores, 4 GB of RAM and 8 GB of disk, and don't put them all in one rack. Borgmaster (the control plane, itself replicated with an elected leader) computes a placement; a Borglet on each machine fetches binaries and watches liveness. When a task dies or a machine vanishes, Borg rebuilds it elsewhere.
Three design points are worth keeping. First, it is declarative: you describe the desired state and the scheduler converges on it — the entire philosophy of today's Kubernetes lives in that sentence. Second, bin packing and overcommit: batch work and serving jobs share machines, with low-priority work filling the headroom serving jobs must reserve, so utilization lands far above the "one service, one fleet" approach. Third, preemption: when resources tighten, high-priority jobs evict low-priority ones. The cost sits right there — your batch job can be killed at any moment by design, so it has to be written as retryable and resumable.
One detail that is easy to skip and shouldn't be: tasks don't address each other by IP but by a BNS name (the Borg Naming Service, shaped like /bns/<cluster>/<user>/<job>/<task number>), resolved to the current IP and port. That layer of indirection is exactly why a task changing machines is invisible to its callers.
Google's storage is not one product but a pile. At the bottom sits D, a disk server running on nearly every machine, which only exposes that machine's disks and knows nothing about file systems. Above it, Colossus (successor to GFS) stitches thousands of D instances into a cluster-wide file system: chunking, replication or erasure coding, dead-disk replacement. Only above that come the familiar database layers: Bigtable (NoSQL, a sparse table addressed by row key, column and timestamp, scaling to petabytes), Spanner (SQL-like interface with global strong consistency) and Blobstore (large immutable objects such as images and video).
Why so many layers? Because each solves exactly one class of problem, so the layer above can assume the one below isn't lying to it. Colossus assumes disks die and adds redundancy; Bigtable assumes the file system is reliable and worries only about data model and sharding. This is the most common shape in distributed systems: not one enormous system, but a stack of small ones that trust each other.
The whole abstraction only holds if machines can talk cheaply. Google's datacenter network (the Jupiter generation) combines custom switching hardware with software-defined networking — SDN, where the control logic is centralized in software instead of each switch deciding for itself — to make a building behave roughly like "any two machines can talk at full speed." Across datacenters, the B4 software-defined WAN carries the traffic.
Traffic enters through GSLB, the global software load balancer, which splits load at three levels at once: geographic (DNS sends users to a nearby datacenter), service (across frontends in that location) and RPC (across backend instances). All three are needed: DNS alone is too coarse and cached beyond your control, while RPC-level balancing alone means the user already crossed half the planet before anything got balanced.
Programs talk to each other over Stubby (Google's internal RPC framework; gRPC is the public version), encoding data with protocol buffers — the book's numbers are 3 to 10 times smaller than XML and 20 to 100 times faster to serialize. That is not a garnish: when one user request fans out into dozens of internal calls, encoding overhead shows up directly in tail latency.
Chubby is a Paxos-based lock service and small strongly-consistent store. It does two jobs: holding the small amount of information everyone must see identically (such as where Borgmaster currently lives), and master election — deciding which replica leads, so two leaders never write at once. It stores almost no data and matters enormously: nearly every system uses it, and when it is unavailable a swathe of services goes with it.
Borgmon is monitoring: it scrapes metrics periodically from an endpoint each task exposes, stores them as time series, and computes alerts from rules. The key design is that alerts rest on aggregatable expressions ("this service's overall error rate exceeds x%") rather than per-machine thresholds — machines are constantly being swapped, and per-machine alerting drowns in noise on day one. Prometheus is the public re-implementation; Chapter 10 covers it in full.
The chapter threads every part together with a toy service: Shakespeare, an index over Shakespeare's complete works answering "which plays contain this word?" Offline, a MapReduce pipeline reads all the texts, builds the index and writes it into Bigtable. Online, a frontend job and a backend job each run several replicas across several clusters.
A request's path: DNS and GSLB land it on a frontend task in a nearby datacenter; the frontend sends a Stubby RPC to a backend task by BNS name; the backend queries Bigtable; the answer returns the same way. No hop in that chain knows a specific machine, and every hop can be replaced by Borg without interrupting service. That picture is the whole point of the chapter.
The most useful table in this chapter translates Google's internal parts into things you can actually use. Nearly every component has a public counterpart, and reading the mapping is reading where modern infrastructure came from.
Google internals and their public counterparts
| Inside Google | Public counterpart | What it actually solves |
|---|---|---|
| Borg | Kubernetes / Mesos / Nomad | hands "program to machine" placement to a scheduler; machine failure becomes automatic rebuild |
| Colossus (after GFS) | HDFS / Ceph / S3-style object stores | turns thousands of disks into one reliable namespace |
| Bigtable | HBase / Cassandra | petabyte-scale wide table sharded by row key, write throughput first |
| Spanner | CockroachDB / TiDB | cross-region strong consistency plus SQL, paid for in write latency |
| Chubby | ZooKeeper / etcd | strongly-consistent store for small data, plus master election |
| Borgmon | Prometheus | pull-based time-series metrics with rule-based alerting |
| Stubby + protobuf | gRPC + protobuf | efficient RPC and compact encoding, holding down fan-out tail latency |
| Blaze | Bazel | reproducible, cacheable distributed builds |
The storage layer is the second exercise the chapter leaves you: you cannot have consistency and latency at full strength at once.
Pick one of three: what to use when
| Choice | Fits | The honest cost |
|---|---|---|
| Bigtable-style wide table | huge key-value, time-series or index data accessed by row key, where single-row operations suffice | no cross-row transactions; cross-region replicas are typically eventually consistent, so the application absorbs stale reads |
| Spanner-style strong store | money, inventory, quota — data where a stale read is simply wrong | cross-region commits wait for a quorum; write latency runs from tens to hundreds of milliseconds, at higher cost |
| Blobstore-style object store | images, video, backups — large immutable data | overwrite rather than edit; a bad fit for small frequent updates |
A larger trade-off hides behind Borg: shared clusters versus dedicated machines. Sharing lifts utilization and drags isolation problems in with it — a neighbor's batch job eats memory bandwidth and your p99 wobbles. Google's answer is to accept that cost and manage it with priorities and resource isolation. If your business runs on a few dozen machines, that cost does not buy the same return, so don't copy it.
The chapter's value is turning an overused phrase into a concrete picture: what parts "treat the datacenter as a computer" actually requires. Scheduling, a cluster file system, strongly-consistent coordination, a uniform RPC layer, uniform monitoring — five pieces, none optional. When you assemble Kubernetes + S3 + etcd + gRPC + Prometheus in the cloud today, you are assembling that same diagram. Asked in an interview how you would build a platform for tens of thousands of machines, those five are the skeleton — and being able to say which layer owns which failure is worth far more than reciting component names.
1. In one line: this chapter is the book's physical premise — Google's production environment is not a pile of servers but one computer, the size of a datacenter, glued together by software.
2. The foundation is one definition: machine (hardware) is not server (software), and they are many-to-many. Traditional ops fails to scale precisely because it assumes one-to-one.
3. Machine, rack, row, cluster, datacenter, campus is a ruler for failure domains and latency, not a decorating scheme.
4. Borg is the cluster OS: declarative jobs and tasks, bin packing for utilization, priority preemption, automatic rebuild — and low-priority work that can be killed at any time.
5. Storage is a stack: D to Colossus to Bigtable / Spanner / Blobstore, each layer solving one class of problem so the layer above dares to trust it.
6. The network (Jupiter, B4) makes crossing machines cheap enough for the abstraction to hold, and GSLB balances at the geographic, service and RPC levels simultaneously.
7. Chubby (strong coordination and master election) and Borgmon (time series plus rule-based alerts) are tiny and universally depended upon — which makes them the ceiling on availability.
8. Life of a request: GSLB, frontend task, Stubby RPC, backend task, Bigtable — BNS names the whole way, never IPs.
9. The judgment to take away: this machinery was forced by scale, it is not a matter of taste. Copy it before the scale arrives and complexity is all you get.