Object Storage as the Source of TruthThe Zero-Disk Architecture: Outsourcing Consensus to S3

Scenario + Constraints

You are designing the storage layer for a Git hosting service in the agent era. The traffic shape has changed: hundreds of thousands of coding agents fetch concurrently and push in small increments across the same repositories. The read/write ratio is roughly 100:1, but write frequency is far above anything human developers produce.

• Scale: 100k repositories, up to 50 GB each; peak 300 pushes/s, read QPS in the tens of thousands.
• Durability: once a push returns success it cannot be lost (RPO = 0).
• Consistency: after a successful push, any node's subsequent fetch must see it (per-repo linearizability).
• Elasticity: nodes run on spot instances and can be reclaimed at any time; a new node must take traffic immediately.

The traditional answer is "stateful replica group + consensus": three replicas per repo, Raft leader election, pushes through the leader with majority acknowledgement. It works, but a replica group is stateful: scaling out means moving data, a dead node means rebuilding a replica, read scaling is capped by replica count, write throughput is pinned by leader round-trips — and uneven repo sizes make rebalancing a permanent chore.

High-Level Architecture

graph LR
    C["Git client
push / fetch"] --> LB["L7 routing
rendezvous hashing"]
    LB --> N1["Node A
NVMe warm cache"]
    LB --> N2["Node B
NVMe warm cache"]
    LB --> N3["New node C
cold / empty cache"]
    N1 -- "1. PUT data 2. CAS index" --> S3[("Object store
WAL = source of truth
immutable data + mutable index")]
    N2 -- "conditional GET
ETag hit then 304" --> S3
    N3 -- "materialize whole repo from WAL" --> S3
    N1 -. "UDP gossip: WAL advanced to seq=N" .-> N2
    N1 -. gossip .-> N3

Three things deserve calling out. There is exactly one source of truth: the WAL in object storage — immutable, content-addressed data objects plus one mutable index object that records order. Nodes are homogeneous and stateless: any node can accept a push for any repo, and the real git repository on local NVMe is only a warm cache. Gossip carries no correctness — it merely tells replicas earlier that there is new data to pull; correctness rests entirely on the index's conditional write, and a lagging replica always does a conditional GET to reconcile before serving a read.

Key Point 1: Move the Serialization Point from the Replica Group to a Conditional Write

Trade-off in one line: spend one PUT's latency, delete leader election, membership changes, and quorum coordination.

Principle

Consensus is fundamentally about imposing a total order on concurrent writes. Raft produces it with "a majority-acknowledged leader + monotonic terms", at the cost of maintaining a stateful replica group. Since 2024, S3 supports conditional writes: PutObject with If-Match: <etag> returns 412 Precondition Failed when the ETag does not match — a genuine atomic single-key compare-and-swap that is durable across AZs.

So the architecture inverts: data objects are written concurrently and unconditionally (immutable, safe to retry), and order is decided by one CAS on the index. The instant the CAS succeeds is both the moment "this push enters history" and the earliest moment you may ack the client. Whoever wins the CAS goes first; the loser re-reads, recomputes, retries — there is no leader, and none is needed.

sequenceDiagram
    participant A as Node A
    participant B as Node B
    participant S as Object store
    A->>S: PUT wal/0042.pack immutable data
    B->>S: PUT wal/0043.pack immutable data
    A->>S: PUT index If-Match etag_v7
    S-->>A: 200 OK new etag_v8
    A->>A: ack client: push succeeded
    B->>S: PUT index If-Match etag_v7
    S-->>B: 412 Precondition Failed
    B->>S: GET index sees v8 including A's record
    B->>S: PUT index If-Match etag_v8
    S-->>B: 200 OK new etag_v9
    Note over A,B: only the index CAS is serialized
data object PUTs stay fully concurrent

Trade-off

Pseudo-code

# Append a WAL record; the index CAS is the only serialization point (pseudo-code)
def append(repo, record):
    for _ in range(MAX_RETRY):
        idx, etag = s3.get_json(f"{repo}/index.json")     # read current WAL head
        key = f"{repo}/wal/{idx.next_seq:08d}.pack"
        s3.put(key, record)                              # 1) immutable: concurrent, retry-safe
        idx.entries.append({"seq": idx.next_seq, "key": key,
                            "sha": sha256(record)})
        idx.next_seq += 1
        try:
            s3.put_json(f"{repo}/index.json", idx, if_match=etag)   # 2) CAS
            return idx.next_seq - 1                      # only now may we ack the client
        except PreconditionFailed:                       # 412: someone advanced it first
            continue                                     # re-read, recompute, retry
    raise Contention(repo)
# A failed CAS leaves step 1 as an orphan object: unreferenced, left to GC, correctness intact

Key Point 2: Demote Local Disk to a Disposable Warm Cache

Trade-off in one line: drop the assumption that "local disk is the data" and get stateless nodes, linear read scaling, and painless spot reclamation.

Principle

Once truth lives in object storage, the identity of the git repository on local NVMe changes: it is not a replica, it is a cache. Three consequences follow: scaling out no longer moves data (bring a node up empty, materialize on demand); node failure no longer requires rebuilding a replica; and read scaling is no longer capped by replica count — Cursor reports read throughput scaling linearly to 100 replicas in synthetic tests.

The cost lands on the read path: a cache can lag, so you reconcile before serving — send a conditional GET carrying the local ETag, and an If-None-Match hit returns 304 with zero bytes downloaded (the steady-state hot path); only a genuinely stale cache pulls the seq delta. Gossip just announces "time to reconcile" a little earlier.

Trade-off

Treating the cache as a cache means accepting cold-read amplification: a new node's first fetch must materialize the repo from the WAL, potentially tens of times slower than the hot path. Two mitigations, each with a price: rendezvous hashing pins a repo's traffic to a few preferred nodes (higher hit rate, but you sacrifice load evenness and invite hot-repo problems); periodic snapshots turn materialization from "replay all history" into "snapshot + tail delta" (cheaper cold reads, paid for in storage and background CPU).

Pseudo-code

def serve_fetch(repo, want):
    local = cache.get(repo)                                  # a real git repo on NVMe
    head = s3.get_json(f"{repo}/index.json",
                       if_none_match=local.etag)             # conditional GET
    if head is NOT_MODIFIED:
        return git_upload_pack(local, want)                  # hot path: zero S3 data transfer
    for e in head.entries[local.seq + 1:]:                   # pull exactly the delta
        local.apply(s3.get(e.key))
    local.etag, local.seq = head.etag, head.last_seq
    return git_upload_pack(local, want)

def materialize(repo):                                       # when the cache is gone
    repo_dir = restore(s3.get(f"{repo}/snapshot/latest"))
    for e in wal_entries_after(snap.seq):                    # then the tail delta
        repo_dir.apply(s3.get(e.key))
    return repo_dir

Key Point 3: The Latency and Cost Curves Shift from Disk to Request Count

Trade-off in one line: the batching window is a dial — turn it right for throughput and cost, left for latency.

Principle

Write latency in a zero-disk architecture has a hard floor: the object store's PUT p99, because "nothing is acknowledged until it is in object storage" is the entire basis of the durability promise. WarpStream's published numbers are representative: with default S3 Standard, Produce p99 is around 400 ms (the p99 for writing a 4 MiB object to S3), dropping to 169 ms on S3 Express One Zone (low latency, single AZ). Cursor's counterpart figures are roughly 120 pushes/s on S3 Standard and 300+ pushes/s on Express One Zone — where the bottleneck is no longer S3 but git compacting on-disk data.

The cost model changes dimension too: you no longer pay for IOPS and capacity, you pay per request. Small-object storms are therefore the number one cost killer, and the only cure is batching: merge many writes (even across partitions or repos) into a single object, turning N PUTs into one.

Trade-off

The batching window behaves the same way: a longer window amortizes request cost and raises throughput, but adds the window length straight onto p99. Only the workload can set this dial: log ingestion will not notice 250 ms or more, but when an agent cannot take its next step until the push returns, every 100 ms is spent waiting.

Pseudo-code

// Batch many partitions into one shared object: billed per request, not per IOPS (pseudo-code)
for {
    batch := buf.Drain(maxWait: 250*time.Millisecond, maxBytes: 8<<20)
    key, err := objectStore.Put(encode(batch))     // one object, many partitions
    if err != nil { batch.NackAll(); continue }    // never ack what is not persisted
    offsets := coordinator.Commit(key, batch.Index())  // sequence and assign offsets
    batch.AckAll(offsets)                          // ack only after object + metadata commit
}

Key Point 4: You Are Betting Correctness on a Vendor's Conditional-Write Semantics

Trade-off in one line: skip an entire consensus implementation, and in exchange put the foundation of linearizability inside the semantics of an external API.

Principle

1. "Strongly consistent" does not mean "has CAS". S3 gave read-after-write strong consistency back in 2020, but that only guarantees you read the latest version; it offers no atomicity primitive. CAS arrived only with conditional writes in 2024 (AWS docs). That is why early Iceberg / Delta on S3 had to patch the hole with an external lock table.

2. CAS covers one key. Two conditional writes on two objects do not form a transaction, so all mutable state must funnel into a single key, with every other object immutable and content-addressed.

3. Implementation divergence is a real risk. Conditional-write semantics are not automatically identical across S3, MinIO, R2, and Ozone (Apache Ozone only implemented S3 conditional requests in 2026 to support this kind of optimistic concurrency). Claim multi-cloud, and you are actually betting on the weakest implementation.

Trade-off

Bet directly on object-store CAS: fewest components, but correctness is tied to one vendor. External consensus as a backstop (pointer in etcd/DynamoDB): buys back cross-key transactions and portability, at the price of inviting a stateful component back in. A dedicated coordinator (the KIP-1150 route): you own the ordering logic and can express complex offset semantics, but the coordinator is itself a stateful service that must be highly available — consensus was pushed away, not removed.

Pseudo-code

# Anti-pattern: two conditional writes are not one transaction
s3.put_json(f"{repo}/refs.json",  refs, if_match=e1)    # succeeds
s3.put_json(f"{repo}/index.json", idx,  if_match=e2)    # crash here, two keys diverge forever

# Correct shape: funnel into one key (Iceberg and Continuity share this skeleton)
mid = content_hash(full_state)
s3.put(f"{repo}/manifest/{mid}", full_state)            # immutable, retry-safe, unconditional
s3.put_json(f"{repo}/HEAD", {"manifest": mid},
            if_match=head_etag)                         # the only mutable pointer = the only CAS
# A stale node wakes up and retries? Its head_etag is necessarily stale, so 412 —
# the CAS is itself the fencing token.

Scaling and Optimization

Compaction is a first-class citizen: an unbounded WAL turns every cold read into a full-history replay, so the tail must be compacted into snapshots periodically — and that work contends for the same CAS point as the write path, which is exactly why Cursor's 120 pushes/s figure is measured "while compacting and replicating". Shard the CAS hot spot per repository: one global index object pins site-wide write throughput to contention on a single key; one mutable key per repo (or per table, per partition group) aligns contention with business granularity. Multi-region is the next wall: conditional writes are a within-region semantic and cross-region replication is asynchronous and non-atomic, so you either fall back to a single write region plus read-only replicas, or reintroduce cross-region consensus. Tail latency wants hedged requests: when p99 is set by an external PUT, about all you can do is issue a second copy for slow requests — and request count is the bill.

Pitfalls + Interview Questions

• "Ack from local disk, upload to S3 asynchronously." The latency chart looks great, but RPO is no longer 0 and the durability promise is broken. A classic interview trap: when you ack is the durability contract.
• Treating gossip as a correctness mechanism. It is UDP and it drops; it accelerates discovery but cannot replace the conditional-GET reconcile. Asked "what if gossip is lost?", the answer is "a replica lags a bit longer, but it always reconciles before serving a read, so correctness is unaffected".
• Getting the cost dimension wrong. After migrating from $/GB-month to $/request, thousands of small PUTs per second can cost more than the disks did. Batching is not an optimization here; it is a precondition.
• CAS contention without backoff. Naive retries amplify into livelock under load; you need exponential backoff with jitter, and the "412 rate from contention" belongs in your first-class metrics — it is the early signal that your sharding granularity is wrong.
• Follow-up list: 1) With no conditional write available, how do you patch the hole and at what cost? 2) Can two nodes both win a CAS — why not? 3) Which layer bottlenecks at 300 pushes/s, and how would you prove it? 4) Would you still choose this for a financial ledger that demands write p99 < 10 ms? 5) How do you GC orphan objects without deleting one that is still referenced?

Going Deeper

• Git at any scale · Cursor — the Continuity architecture first-hand: S3 WAL as source of truth, CAS ordering, rendezvous hashing, gossip, throughput numbers.
• KIP-1150 Accepted, and the Road Ahead · Aiven — what acceptance of Kafka Diskless Topics means and what remains unresolved.
• Kafka Is Dead, Long Live Kafka · WarpStream — the full argument for diskless agents writing straight to object storage, with the latency and cost arithmetic.
• AWS: S3 conditional writes — the precise semantics of If-Match / If-None-Match. Required reading before you design on them.
• Understanding Apache Iceberg's Consistency Model · Jack Vanlightly — the same skeleton analyzed on the data-lake side.
• Designing Data-Intensive Applications (Martin Kleppmann) — the replication and consensus chapters are the baseline for judging what this architecture actually removes.

Reflections