IT PAPER DEEP-READ · PAPER 17
Ghemawat, Gobioff & Leung · Google · SOSP 2003
In 2003, three Google engineers published the storage system they'd built for themselves — GFS. The problem was concrete: Google's search, its crawled copy of the entire web, its logs — all of it was absurdly large, too big for any single machine, so it had to be spread across thousands of cheap machines. GFS is the software layer that glues those thousands of machines into "one giant hard drive." It later became the foundation for MapReduce and Bigtable, and directly inspired the open-source Hadoop.
When they built it, they accepted one thing: machines breaking is normal, not exceptional. Park thousands of cheap machines together and every day disks die, power supplies fail, cables come loose — it's not "will something break" but "how many are broken right now." Traditional storage systems assume hardware is basically reliable; at this scale that assumption simply collapses. Every design choice in GFS is reverse-engineered from "something is always breaking."
GFS's setup is strikingly simple — just two roles: one "manager" (the master) and a large crew of "warehouse workers" (chunkservers).
The clever part: the manager only keeps the ledger, never touches the goods. The actual file data is sliced into big blocks; each block is stored in three copies on three different machines. The manager holds just a "ledger" — which blocks make up which file, and where each of the three copies lives. To read a file, you first ask the manager "where is it," the manager hands back an address, and you go straight to that warehouse-worker machine to fetch the data — never routing through the manager.
A few plain but effective decisions. One: slice into big blocks. Files are cut into 64 MB blocks (not tiny few-KB cells), so the ledger stays small — the manager can keep the whole system's ledger in the memory of a single machine. Two: three copies of every block. One machine dies, the other two survive; when the manager notices a block is down a copy, it quietly re-copies one to restore the count, and you barely notice. Three: the manager never touches data. Every fetch is a direct, point-to-point connection between client and chunkserver, so the manager never jams up as "the place all traffic flows through." Four: optimized for appending. Most of Google's work is "keep adding new data to the end of a file" (a stream of logs, say), so GFS makes "many writers appending to the same file at once" both fast and safe.
GFS was the first to make people believe: a pile of machines that break constantly really can be assembled into storage that's huge, tough, and usable — not by making each machine more reliable, but by making the whole system tolerate machines breaking. This recipe — cheap machines + many copies + software backstop — became the template for big-data infrastructure for the next decade-plus.
Glue thousands of failure-prone cheap machines into one giant hard drive: slice data into big blocks, keep three copies of each, auto-restore when a machine dies; one "manager" keeps the ledger but never touches goods, so reads and writes go straight between client and warehouse worker — making the system huge, failure-proof, and jam-free. It's the first foundation stone of the big-data era.
Want the architecture diagram, the lease and record-append mechanics, and real-cluster numbers? → Switch to the deep read
GFS is a distributed file system Google custom-built for its own massive workloads. It treats "hardware failure is the norm" as the design premise, uses a single master for metadata plus many chunkservers for data, slices files into 64 MB chunks with three replicas each by default, and — via leases, atomic record append, and a relaxed consistency model — fully decouples control flow from data flow. With a crowd of failure-prone commodity machines it stably supports petabyte-scale storage dominated by "large sequential reads + appends." It is the substrate under MapReduce and Bigtable, and directly inspired the open-source HDFS.
The authors are Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, at Google; the paper appeared at SOSP 2003. It is the first of Google's "big-data trilogy" — the next year's MapReduce (large-scale parallel computation) and Bigtable (structured storage) two years later were both built directly on top of it. It inherits from traditional distributed file systems (AFS, xFS) but boldly drops the "general-purpose / POSIX-compatible" burden to serve only Google's own workloads; downstream it inspired the open-source HDFS (Hadoop's file system, essentially a public clone of GFS), and internally at Google it evolved into the more capable Colossus.
In the early 2000s, the data Google had to handle exceeded what any off-the-shelf storage could gracefully bear. Rather than force-fit a general file system, the team first honestly observed what their own workload actually looked like, arriving at four premises sharply different from traditional assumptions:
In short: don't build a general, perfect file system — build one that exactly matches this workload and assumes hardware breaks every day.
A GFS cluster has just two kinds of roles. The single master owns all metadata: the namespace (directory tree), access control, which chunks compose each file, and which machines currently hold each chunk's three replicas. The many chunkservers are where the actual data lives.
Files are cut into fixed-size chunks of 64 MB each; at creation the master assigns each a globally unique 64-bit chunk handle. Every chunk is stored as an ordinary Linux file on a chunkserver's local disk and, by default, replicated 3 times across different machines (and across racks where possible).
The most important design is "ledger-vs-goods separation": the master handles only the ledger and never touches data. To read a file, a client first asks the master "where are this chunk's replicas," giving a filename + chunk index; the master returns a list of chunkserver addresses (which the client caches for a while); the client then fetches data directly from a chunkserver, bypassing the master entirely. This way the master never becomes a bandwidth bottleneck from "all data flowing through me," so a single master can serve the whole cluster.
Why such large chunks (64 MB)? Three benefits: (1) big chunks → fewer chunks total → small metadata, so the master can keep all metadata in memory for fast lookups; (2) one location query lets a client read/write a large span, drastically cutting master interactions; (3) a client does many operations on one chunk, reusing its chunkserver connection. The honest cost: a small file occupies only one or two chunks, so it easily becomes a hot spot — many clients crowding the same chunkserver.
The master keeps three kinds of metadata entirely in memory: the namespace, the file→chunk mapping, and each chunk's replica locations. The first two are persisted — written to an operation log that is both the basis for crash recovery and the definition of the logical order of concurrent operations; the log is replicated to several remote machines, and on restart the master replays the log to recover its state (with periodic checkpoints to bound the replay).
Notably, chunk replica locations are NOT written to the log — because the chunkservers are the authority on that. At startup and thereafter, the master repeatedly asks each chunkserver "which chunks do you hold" via HeartBeat messages. This sidesteps a whole class of "master and chunkserver disagree" headaches: machines joining, leaving, renaming, and restarting is routine, so rather than laboriously keeping them consistent, GFS lets the chunkserver be the source of truth and has the master sync periodically.
A chunk has three replicas, and multiple clients may write at once — how do the three copies end up identical? GFS's answer is the lease: the master grants a short-lived "say-so" (a lease, ~60 s, renewable) over a chunk to one replica, called the primary. Thereafter the order of all mutations on that chunk is decided solely by the primary, and the other replicas apply them in that same order. As a result, the master need not take part in each write — it hands out one lease, delegates the heavy job of "ordering mutations" to the primary, and stays lean managing metadata.
When writing a chunk, GFS splits "command" and "haulage" onto two paths. Data does not go star-shaped (all pushed to the primary then fanned out); it flows down a chain, pipelined: the client pushes data only to the nearest replica, which forwards to the next in the chain as it receives, relay-style. This way each machine's outbound bandwidth is used entirely to send to "the next one", uncontended, maximizing network utilization. Only after the data is staged in every replica's memory does the client send the primary a tiny "write" control request; the primary assigns the order and tells the replicas to apply it in sequence. Data flow is routed by network proximity, control flow goes through the primary for ordering — the two are decoupled, which is the key engineering detail behind GFS's high throughput.
This is GFS's most distinctive operation and the one most tailored to Google's workload. A traditional "write at a given offset" needs either locks or mutual overwrites under concurrency. GFS offers atomic record append: the client merely says "add this data to the end of the file," and GFS itself picks an offset to write at, then tells the client the offset. So hundreds of clients can append to the same file simultaneously, no extra locking needed — ideal for "many producers pooling data into one result / queue file."
The cost is that it only guarantees "at-least-once": if an append fails on some replica, the client retries, possibly leaving duplicate records or padding gaps. GFS pushes this complexity to the application — apps use checksums, unique IDs to dedupe, and skip padding. This is exactly the "app + file-system co-design" idea: relax a guarantee a little, gain overall simplicity and high concurrency.
GFS does not chase strong consistency; it offers a "relaxed but sufficient" model. Metadata operations (e.g. file creation) are atomic, guarded by namespace locking at the master. For file-data regions, a state after a successful mutation may be: consistent — all clients see the same data regardless of replica; or defined — consistent AND clients see the mutation in its entirety. Concurrent successful writes may be "consistent but undefined" (all writes present, but interleaved and hard to disentangle); record append guarantees "defined," but may have duplicates or padding in between. The key trade-off: weaker consistency promises in exchange for staying fast and stable on a large, failure-prone cluster.
The paper argues from micro-benchmarks and two real production clusters. Each real cluster has hundreds of chunkservers, hundreds of TB of disk, and hundreds of thousands to over a million files, carrying Google's internal R&D and production loads. The data shows reads far outnumber writes, and appends far outnumber overwrites — closely matching the design assumptions; aggregate read throughput reaches the order of hundreds of MB/s, near the network limit. A recovery experiment is telling too: kill one chunkserver holding thousands of chunk replicas, and the master restores all replica counts within tens of minutes, with no service interruption. The most important conclusion is qualitative — from cheap machines that break every day, one really can assemble large-scale, high-throughput, failure-tolerant storage — and it was already running at scale inside Google, backing real business.
GFS is the first foundation stone of big-data infrastructure. The next year's MapReduce and the following year's Bigtable were built directly on it, forming Google's "big-data trilogy"; the open-source world cloned it almost directly into HDFS, which underpins the entire Hadoop ecosystem and, in turn, shaped nearly every big-data system for the next decade-plus. It truly established and popularized a whole set of ideas: move reliability up from "single-machine hardware" to "software + replication," scale out with lots of cheap machines, separate metadata from data, and relax general-purpose semantics for a specific workload. Much of what you hear today as "distributed storage," "replication fault tolerance," and "master-slave architecture" traces back to this paper.
1. In one line: a distributed file system designed for huge-scale workloads dominated by "large sequential reads + appends," taking "hardware failure is the norm" as its starting point.
2. Architecture: a single master for metadata + many chunkservers for data; files sliced into 64 MB chunks, 3 replicas each, spread across racks.
3. Ledger-vs-goods: the master only answers "where is the chunk," and data is hauled directly between client and chunkserver, so the master is never a data bottleneck; big chunks → small metadata → all in memory.
4. Master metadata lives in memory, persisted and recovered via an operation log + checkpoints; chunk locations are not persisted, reported by chunkserver heartbeats.
5. Leases delegate "ordering mutations" to the primary, keeping the master out of the write path; data flows pipelined down a chain, control flows through the primary for ordering — decoupled to maximize bandwidth.
6. Atomic record append lets many clients append concurrently without locks, but guarantees only "at-least-once" (possible duplicates / padding), pushing complexity to apps.
7. Consistency is relaxed but sufficient (consistent / defined), metadata ops are atomic; fault tolerance via re-replication, checksums, version numbers, fast recovery, shadow masters.
8. Impact: the substrate under MapReduce / Bigtable, inspiration for HDFS, establishing the "cheap machines + replication + software fault tolerance + scale-out" big-data paradigm.
9. Limits: single-master scaling bottleneck, small-file unfriendliness, relaxed consistency outsourcing complexity, non-general; Google itself succeeded it with Colossus.