Multi-region — The Speed of Light Is the Final ConstraintData Residency, Active-Active, Cross-region Consistency, Latency

Scenario & Constraints

Design a social-commerce app with 100M global DAU, users spread across North America / EU / APAC. Goals: local-read p99 < 150ms (a trans-oceanic RTT alone is 120-180ms, so a single central region is guaranteed to blow the SLO); full-region outage RTO < 5 min; EU users' PII must stay inside the EU (GDPR Art. 44 cross-border transfer restrictions).

Why this is among the hardest System Design classes: the speed of light is non-negotiable. Beijing↔Virginia is ~90ms one way; a synchronous cross-region write costs 180ms RTT. You cannot engineer that away — you can only architect around it. The entire difficulty of multi-region is: which operations may cross regions, which must stay local, and who wins a data conflict.

High-level Architecture

Component roles: GeoDNS/Anycast routes each request to the nearest region by geography; every region is a self-contained full stack (edge→service→DB), so a regional failure doesn't drag down the others (Netflix's Isolation principle). Cross-region data defaults to async replication — synchronous replication turns every write into a trans-oceanic RTT. The EU primary does selective replication of PII: non-sensitive data (product catalog) syncs globally, PII stays in eu-west. Core tension: local = low latency but scattered data; centralized = strong consistency but high latency.

Key Technical Points

① Traffic routing & failover — locality vs session stickiness

Core trade-off: pinning a user to a "home region" is the simplest for consistency, but when that region dies you must fail over in seconds — and session/data consistency during the switch is the trap.

[Principle] Three routing layers: GeoDNS (returns different IPs by resolver location; coarse, lagged by DNS cache TTL); Anycast (one IP broadcast globally, BGP picks the nearest ingress; fast failover but a connection may mid-flight jump regions); application-layer routing (edge gateway looks up a "home region" table by user_id then forwards; most precise). Production usually combines them: Anycast into the edge, then L7 forwarding by the user's home region.

# Edge gateway: route by home region, degrade to backup on failure
def route(user_id):
    home = region_map.get(user_id) or geo_nearest(client_ip)
    if health[home].ok:
        return home
    # Failover: pick the next-best healthy region
    for r in sorted(regions, key=lambda r: rtt[client_ip][r]):
        if health[r].ok and r != home:
            return r  # note: after switching you may read replication-lagged data

Real cases: Netflix uses Zuul for edge routing — change routes at runtime, evacuate an entire region's traffic to another on failure (Active-Active blog). Cloudflare runs global Anycast, one IP landing on the nearest PoP. AWS Route 53 offers latency-based + geolocation routing.

② Active-active data consistency — who can write, who wins a conflict

Core trade-off: true active-active (every region writes the same data) has the lowest latency but must resolve conflicts; single-writer home-region has no conflicts but blocks writes when the owner is down; global strong consistency (Spanner) has no conflicts but pays cross-region latency on every commit.

[Principle] Three paradigms: (a) Active-Active + LWW (Last-Writer-Wins): every region writes, async bidirectional replication, conflicts resolved by timestamp — simple but it silently drops updates (two regions concurrently edit the same key; the loser vanishes). (b) Single-writer home region: each datum has one owner region; writes only happen at the owner, other regions read replicas and forward writes — no conflicts, at the cost of cross-region write latency and reduced availability when the owner fails. (c) Global consensus + TrueTime: Spanner uses GPS+atomic clocks to bound clock uncertainty to a few ms, commits via cross-region Paxos, giving external consistency — latency on the order of a cross-region RTT.

# LWW conflict resolution (DynamoDB Global Tables semantics)
def merge(local, remote):
    # each item carries a system timestamp; larger timestamp wins
    return remote if remote.ts > local.ts else local
    # danger: equal ts or clock skew => undefined => use logical clock/version vector

Selection heuristic: data that tolerates occasional lost updates (like counts, last-seen) → LWW; data that can't lose (balance, inventory) → single-writer home region or a strong-consistency store; cross-region counters → CRDT (a G-Counter increments per region independently, merges by sum — conflict-free by construction).

Real cases: DynamoDB Global Tables defaults to MREC mode = active-active + LWW, converging cross-region in 1-2s (AWS docs). Google Spanner uses TrueTime for globally externally-consistent transactions (OSDI 2012 paper). Netflix uses Cassandra's multi-directional async replication for cross-region eventual consistency.

③ Data residency & compliance — pinning data inside a geographic boundary

Core trade-off: global replication gives the best experience but violates GDPR; strict residency is compliant but an EU user traveling to the US can't reach their own data (needs locality + home-region fallback).

[Principle] Compliance requires PII to physically stay in a jurisdiction. The technique is geo-partitioning: make region part of the shard key so EU users' rows only land on EU nodes, and the replication topology is constrained within the jurisdiction. Key distinction: sensitive data (PII) is resident and never leaves; non-sensitive data (products, FX rates) replicates globally. The hard part is joins — the order lives in eu, the recommendation model in us, so you rely on ID references, not a cross-border JOIN.

-- CockroachDB: partition a table by region and pin its location
ALTER TABLE users SET LOCALITY REGIONAL BY ROW;
-- each row has a crdb_region column; EU users' rows physically stay on EU nodes
-- Super Region guarantees both: residency AND survival across zones within eu

Trap: backups and logs are PII too! Many teams make the primary resident but funnel binlog / audit logs / analytics pipelines into a single US warehouse — still a violation. The compliance boundary must cover the entire data lifecycle.

Real cases: CockroachDB's REGIONAL BY ROW + Super Regions exist specifically to solve "residency vs regional survival" (data domiciling docs). Stripe / Salesforce offer EU-only data processing options for EU customers.

④ Cross-region latency optimization — cut trans-oceanic round trips

Core trade-off: reading a local replica is fast but may be stale; confirming the latest cross-region every time is dominated by RTT. The essence of optimization is moving cross-region round trips from "per request" to "background async".

[Principle] Three moves: (1) read-local, write-home — reads always hit the local replica (0 cross-region), only writes may cross regions, since reads vastly outnumber writes. (2) batch/pipeline — collapse N cross-region calls into one, or use async replication instead of synchronous waiting. (3) edge compute — serve static/cacheable content at the PoP, only dynamic requests go to origin. The metric that matters is number of cross-region round trips, not bandwidth — one 180ms RTT is costlier than shipping 1MB.

# Anti-pattern: 3 serial cross-region calls in one request = 3×180ms = 540ms
inv = call_us(check_inventory)   # cross-region
price = call_us(get_price)       # cross-region
tax = call_us(calc_tax)          # cross-region
# Fix: local replicas + background async sync, request path 0 cross-region

Real cases: Cloudflare Workers execute logic at 300+ PoPs, avoiding trans-oceanic origin hops. Netflix deploys every user-facing-path service in all regions so requests never cross regions (Active-Active).

Scaling & Optimization

Evolution path: single region → active-passive (DR: cross-region async replication + manual failover) → read active-active (write-home, read global replicas) → full active-active (every region writes + conflict resolution). Most companies stop at "read active-active" — the conflict-governance cost of full active-active is very high. Next bottlenecks: cross-region replication bandwidth and lag (build the pipeline with CDC + Kafka MirrorMaker); failover drills (Netflix's Chaos Kong periodically really turns off a region to verify); global secondary indexes (keeping index consistency across regions is deep water, usually degrading to async eventual consistency).

Common Pitfalls & Interview Follow-ups

• "Active-active means every region can write" — wrong. True active-active pays a conflict-resolution cost; most "active-active" is really read-active-active / write-home. You must distinguish these in an interview.
• Read-your-writes under async replication lag: a user writes in eu, fails over to us to read, and may read a stale value. Remedies: stick to the home region, or force a home-region read for a short window after a write.
• Split-brain: during a network partition both regions think the other is dead and each takes over writes; after recovery the data conflicts. Rely on quorum/fencing tokens or a clear single-writer home.
• "Adding a region always improves availability" — not necessarily. If regions share one global strong-consistency component (a centralized ID allocator/config service), that component is a single point, and more regions amplify its blast radius.
• Cost blind spot: cross-region data transfer fees (AWS cross-region egress) + redundant per-region capacity — active-active is roughly 2-3× infra cost. Interviewers love "what SLA did the extra money buy?"
• Follow-up: how does Spanner give global strong consistency? TrueTime — GPS+atomic clocks bound clock error to a few ms; at commit it does "commit-wait", waiting out the uncertainty window to guarantee external consistency. It trades hardware for consistency.

Deep-dive Resources

• Designing Data-Intensive Applications (Kleppmann) — Ch. 5 Replication, Ch. 9 Consistency & Consensus; the theoretical bedrock for multi-region.
• Netflix TechBlog — Active-Active for Multi-Regional Resiliency, the classic industry breakdown of active-active.
• Spanner: Google's Globally-Distributed Database — OSDI 2012 paper, TrueTime and external consistency.
• AWS — How DynamoDB Global Tables work, a productized active-active + LWW.
• Cockroach Labs — Data Domiciling with CockroachDB, the engineering answer to residency vs regional survival.

Going Deeper