Networking Fundamentals — How Many Round Trips Does One Request Really TakeTCP/UDP, HTTP/1.1·2·3·QUIC, DNS, TLS Handshake, L4 vs L7 Load Balancing

Problem Scenario + Requirements

You're designing the ingress layer for a global API platform: 50M users across five continents, ~70% mobile, packet loss routinely 1–3%; API p99 target < 150ms, of which connection setup (DNS + TCP + TLS) often eats half of mobile time-to-first-byte. A request from Tokyo to a US-West data center already costs ~120ms of physical round-trip time (RTT); if the handshake takes 3 RTTs, that's 360ms before a single byte arrives.

The problem isn't "add servers"—it's that every RTT must be squeezed: how DNS steers a user to the nearest healthy DC, whether the transport is TCP or UDP, how to cut the handshake from 3 RTTs to 0, and whether load balancing happens at layer 4 or layer 7. These four decisions dominate first-packet latency, and in architect interviews they separate "can configure Nginx" from "understands the network."

High-Level Architecture (a request's network path)

graph LR
    U["Client
mobile/browser"]
    DNS["DNS resolve
GeoDNS + Anycast"]
    EDGE["Edge / L4 LB
Anycast VIP · DSR"]
    L7["L7 LB / Gateway
TLS terminate · route"]
    APP["App cluster"]

    U -->|"① name→IP"| DNS
    U -->|"② TCP/QUIC handshake"| EDGE
    EDGE -->|"③ TLS + HTTP"| L7
    L7 -->|"④ keep-alive pool"| APP

    classDef client fill:#1a2530,stroke:#64c8ff,color:#e8eef5
    classDef edge fill:#0e2030,stroke:#5eead4,color:#e8eef5
    classDef app fill:#2a1530,stroke:#ff7ab6,color:#e8eef5
    class U client
    class DNS,EDGE,L7 edge
    class APP app

DNS picks the DC; L4 picks the edge box; L7 terminates TLS and routes by path/tenant; origin reuses long-lived connections to skip handshakes

Each component owns a slice of latency: DNS steers users nearby (Anycast lets BGP decide "nearest"); L4 LB forwards by the connection 4-tuple—very high throughput but blind to content; the L7 LB/gateway terminates TLS and, seeing the HTTP message, does path routing, rate limiting, canary; the origin uses a keep-alive pool to amortize handshakes. Understanding this chain is how you locate "which RTT is slow."

Key Technical Points

1. TCP vs UDP: reliable & ordered vs connectionless & low-latency

Principle: TCP provides a reliable, ordered, connection-oriented byte stream—three-way handshake to connect, sequence numbers + ACK retransmit for loss, sliding window for flow control, congestion control (Cubic/BBR) to avoid overwhelming the network. The cost: 1 RTT to connect, and "ordered" delivery is a double-edged sword—if one packet is lost, already-arrived bytes must wait in the kernel buffer for the retransmit. That is the root of head-of-line (HOL) blocking. UDP is connectionless datagrams: fire-and-forget, no ordering, no congestion control—zero handshake, zero HOL blocking, but reliability/ordering/congestion are all the application's problem.

# Selection intuition: first ask "if one packet is lost, can the app tolerate it?"
lossy-OK, want latest   → UDP (voice: a dropped frame beats a stale one)
lossless, need ordering → TCP / QUIC (API, files, payments)
ordered but fear HOL blocking + frequent mobile network switch → QUIC (HTTP/3)

2. HTTP evolution: three sieges against head-of-line blocking

Principle: HTTP's performance history is essentially three wars against HOL blocking. HTTP/1.1 processes one request at a time per connection (pipelining was practically abandoned), so browsers open 6 parallel TCP connections—connection-level HOL blocking. HTTP/2 does application-layer multiplexing over one TCP connection: requests are sliced into frames tagged with stream IDs and interleaved, solving application-layer HOL blocking—but all streams still share one TCP byte stream, so a single lost TCP packet stalls every concurrent stream (transport-layer HOL blocking remains, and hurts more than HTTP/1.1 because the losses of 6 connections are compressed into 1). HTTP/3 drops TCP entirely and runs over QUIC: each stream has its own sequence space, so loss only stalls its own stream while the rest keep flowing—HOL blocking is fully solved.

Key insight: on high-loss mobile/transoceanic links, HTTP/2 may underperform HTTP/1.1's multiple connections—putting all eggs in one TCP basket means one jolt spills them all. That's exactly why you must descend to QUIC: HOL blocking lives in the transport layer, so bumping the HTTP version alone can't cure it—you have to change the transport.

3. TLS handshake: from 2-RTT to 0-RTT, and the replay trap

Principle: TLS layers encryption over plaintext TCP. TLS 1.2 needs 2 RTTs (ClientHello→ServerHello+cert→key exchange→Finished) before the first byte; stacked on TCP's 1 RTT that's 3 RTTs. TLS 1.3 cuts a negotiation round to just 1 RTT; for returning users a session ticket enables 0-RTT: the client puts application data (early data) in the very first packet, running handshake and data in parallel. QUIC goes further, merging the transport and TLS handshakes—1-RTT for new connections, 0-RTT for returning ones. Each saved RTT on a transoceanic link is tens to hundreds of milliseconds.

sequenceDiagram
    participant C as Client
    participant S as Server
    Note over C,S: TLS 1.2 (2-RTT)
    C->>S: ClientHello
    S->>C: ServerHello + Cert
    C->>S: KeyExchange + Finished
    S->>C: Finished ✓ only now can send data
    Note over C,S: TLS 1.3 0-RTT (resumption)
    C->>S: ClientHello + early data (request already here)
    S->>C: response + Finished

4. DNS + L4 vs L7 load balancing: getting traffic to the nearest healthy DC

Principle: before a request goes out it needs DNS resolution—a recursive resolver walks root→TLD→authoritative servers, relying on TTL caching to avoid the full walk each time. Going global uses two tricks: GeoDNS returns different IPs by resolver geography; Anycast announces the same IP from many locations, letting BGP route each user to the network-nearest node. Once traffic reaches a DC, load balancing splits into two layers: L4 (transport) only sees the IP:Port 4-tuple and forwards by connection—no decryption, extreme throughput, often with DSR (return path bypasses the LB); L7 (application) terminates TLS and parses HTTP, so it can route by URL path/header/cookie, do canary, rate limiting, WAF—at the cost of decrypting and parsing every connection.

# Common production combo: Anycast → L4 (volume+balance) → L7 (smart routing)
Client --DNS/Anycast--> [L4 LB: consistent hash picks edge, DSR return]
                         --> [L7 gateway: terminate TLS, /v2/* canary 5%]
                             --> App pool (keep-alive reuse, no re-handshake)
# L4 uses consistent hashing so existing connections aren't reshuffled when backends change (see Day 4)

Scaling & Optimization

• Connection reuse as the baseline: always use a keep-alive pool to the origin, amortizing the TCP+TLS handshake over hundreds/thousands of requests—the highest-ROI RTT saving.
• 0-RTT progression: TLS 1.3 → HTTP/3 0-RTT gives returning users near-zero-handshake first packets; hold the idempotency constraint strictly.
• Terminate TLS nearby: terminate TLS at the edge closest to the user (short handshake RTT), then ride a pre-established long connection from edge to origin (see Day 28).
• Congestion control upgrade: on long fat pipes (high bandwidth, high latency, e.g. transoceanic) switch Cubic → BBR, which models measured bandwidth/RTT for steadier throughput.
• Next steps: multi-region nearest-routing and failover (Day 42), edge caching to offload origin (Day 28).

Common Pitfalls + Interview Questions

Frequent follow-ups:

1. Draw an HTTPS request from typing the domain to receiving the first byte—which RTTs are involved, and how do you save each?
2. HTTP/2 already multiplexes—why HTTP/3? How exactly does TCP HOL blocking occur?
3. L4 vs L7 LB: which layer, what can each see, and when must you use L7?
4. How does Anycast achieve "nearest"? What's its hazard for long-lived TCP connections (route flap switches nodes)?
5. Why is TLS 1.3 0-RTT fast, and why dangerous? How would you enable it safely at the gateway?

Deeper Resources

• High Performance Browser Networking (Ilya Grigorik, free at hpbn.co): the authoritative text on latency sources and optimizations across TCP/UDP/TLS/HTTP.
• Cloudflare Blog: HTTP/3 past-present-future, TLS 1.3 overview, 0-RTT—first-hand accounts of HOL blocking and handshake optimization.
• Designing Data-Intensive Applications (Kleppmann), Ch. 8: unreliable networks, timeouts, and partitions—placing "the network" into distributed-systems context.
• RFC 9000 (QUIC) / RFC 9114 (HTTP/3) / RFC 8446 (TLS 1.3): first-hand specs; the handshake and 0-RTT anti-replay sections suffice.

Deep-Dive Questions (click to expand)