Your platform team inherits 500 bare-metal machines and 2000+ microservice instances: CPU utilization sits at <15% year-round (each service hogs a machine to avoid interference), releases go out via Ansible pushing box-by-box, and when a machine dies an on-call engineer gets up at 3am to migrate by hand. Goals: pack 2000 instances onto this fleet, push utilization past 40%, automate releases, and self-heal node failures within 30 seconds. This is exactly what Google's internal Borg — and open-source Kubernetes — were built to solve.
graph TD
U["User / CI
kubectl apply YAML"]
API["API Server
sole write entry · validate/authz"]
ETCD[("etcd
single source of truth
Raft strong consistency")]
SCHED["Scheduler
bind pod → node"]
CM["Controller Manager
reconcile loops"]
subgraph node["Worker Node ×N"]
KUBELET["kubelet
node agent"]
CRI["container runtime
containerd + runc"]
PROXY["kube-proxy / CNI
Service routing"]
POD["Pods
namespace+cgroup isolation"]
end
U --> API
API <--> ETCD
SCHED -.watch/bind.-> API
CM -.watch/reconcile.-> API
KUBELET -.watch local pods.-> API
KUBELET --> CRI --> POD
PROXY --> POD
classDef ctrl fill:#1a2530,stroke:#64c8ff,color:#e8eef5
classDef store fill:#2a1530,stroke:#ff7ab6,color:#e8eef5
classDef work fill:#0e2030,stroke:#5eead4,color:#e8eef5
class API,SCHED,CM ctrl
class ETCD store
class KUBELET,CRI,PROXY,POD work
The core is watch + reconcile: components never call each other directly — they watch the desired state on the API Server and each drives reality toward it
Responsibilities: the API Server is the sole writer to etcd, handling authz, validation, and optimistic concurrency (resourceVersion). etcd uses Raft to store the whole cluster's desired + actual state — the single source of truth. The Scheduler does one thing: pick a node for each unbound pod and write it back. The Controller Manager runs dozens of controllers (Deployment/ReplicaSet/Node…), each an infinite loop of "observe actual vs desired, act on the diff". The kubelet is the node agent — it watches pods assigned to its node and calls containerd to start them.
Core trade-off: kernel isolation buys startup speed and density, at the cost of a shared kernel = weaker isolation than a VM.
Principle: a container is not a "lightweight VM" — it's an ordinary Linux process, merely fenced in by two kernel features. Namespaces govern "what you can see" — pid (isolated process tree; PID 1 inside the container), net (own NIC/IP/ports), mnt (own filesystem view), uts (hostname), ipc, user (UID mapping). cgroups (control groups) govern "how much you can use" — CPU quota, memory ceiling, IO bandwidth, PID count. The image layers via overlayfs: read-only layers shared, copy-on-write on top. Together: the process thinks it owns a machine while sharing one kernel with dozens of neighbors.
| Isolation | Startup | Density/overhead | Fits | |
|---|---|---|---|---|
| Container (runc) | Weak (shared kernel; one kernel bug pierces all) | ~ ms | High (no guest OS) | Same-trust microservices |
| VM (KVM) | Strong (own kernel + hypervisor) | ~ seconds | Low (one OS each) | Multi-tenant hard isolation |
| gVisor/Kata | Medium-strong (userspace kernel / microVM) | Sub-second | Medium | Running untrusted code |
# cgroup v2 capping a container to 1.5 cores + 512MB (pseudo-shell)
mkdir /sys/fs/cgroup/mysvc
echo "150000 100000" > /sys/fs/cgroup/mysvc/cpu.max # 150ms CPU per 100ms window = 1.5 cores
echo 536870912 > /sys/fs/cgroup/mysvc/memory.max # exceed → OOM kill
echo $CONTAINER_PID > /sys/fs/cgroup/mysvc/cgroup.procs
# memory over limit → kernel OOM killer → kubelet records OOMKilled → restart per restartPolicyCore trade-off: declarative ("I want 3 replicas") self-heals better than imperative ("start a container"), at the cost of turning all logic into eventually-consistent async loops that are hard to debug.
Principle: in K8s you don't "start a container" — you declare intent: a Deployment says "nginx:1.25, 3 replicas". The Deployment controller then creates a ReplicaSet, whose controller ensures "actual pod count == 3". Every controller follows the same reconcile loop: read desired, read actual, compute diff, act, write status back, forever. Delete a pod → actual=2≠desired=3 → the controller immediately adds one. That is the essence of self-healing: there is no special "failure handling" code path — a failure is just an event that deviates actual from desired, and the loop naturally pulls it back. A Service provides a stable VIP + DNS, abstracting a set of ephemeral pod IPs into one unchanging entry point.
# Skeleton of every controller (pseudocode) — all of K8s runs on this loop
def reconcile(desired, actual):
if actual.replicas < desired.replicas:
create_pods(desired.replicas - actual.replicas) # short → add
elif actual.replicas > desired.replicas:
delete_pods(actual.replicas - desired.replicas) # over → remove
# key: idempotent + level-triggered (look at current state, not the event stream)
# missing an event is fine — next loop still sees the true current diffCore trade-off: the tighter you bin-pack for utilization, the less headroom is left for bursts and failover.
Principle: the scheduler runs two phases per pending pod. Filter (predicates): drop nodes that violate hard constraints — insufficient resources, node-selector mismatch, untolerated taint, anti-affinity conflict. Score (priorities): rank the survivors — most-idle node (LeastAllocated, spread) vs fullest node (MostAllocated, pack to save machines), affinity preferences, image already local. Bind to the top score. Self-heal via three probes: liveness (dead → restart), readiness (not ready → pull out of Service endpoints, but don't restart), startup (grace period for slow-starting services). At node level: node heartbeat times out → marked NotReady → its pods evicted → rebuilt elsewhere.
# liveness vs readiness: the most confused pair — misconfiguring = avalanche
livenessProbe: { httpGet: {path: /healthz}, periodSeconds: 10, failureThreshold: 3 }
# 3 consecutive failures → restart the container
readinessProbe: { httpGet: {path: /ready}, periodSeconds: 5 }
# failure → remove from Service, take no new traffic, but do NOT restart (wait for deps)
# ⚠️ pitfall: a liveness probe that checks a downstream dep (e.g. DB) → DB blips → every pod judged dead & restarted → self-inflicted avalancheCore trade-off: a per-pod Envoy sidecar buys "zero code change" retries/circuit-breaking/mTLS/observability, at the cost of an extra proxy hop and doubled resources.
Principle: with many microservices, network concerns — retries, timeouts, circuit breaking, mTLS, traffic splitting, tracing — become painful if baked into each service's business code: you'd maintain a library per language, and upgrades mean editing and redeploying every service. A Service Mesh extracts these into a sidecar proxy (Envoy), co-located in the pod, intercepting all inbound/outbound traffic. The business code just sends plain HTTP; retries/encryption/metrics happen transparently in the sidecar. Data plane (a fleet of Envoys) + control plane (Istio's istiod) are split: the control plane pushes routing rules and certs to all Envoys, which do the actual forwarding.
kubectl apply to a running pod, which components does it pass through? ② After a node dies, how and how fast is a pod rebuilt? ③ Why does declarative self-heal better than imperative? ④ What does a Service Mesh solve, at what cost, and when should you not adopt one? ⑤ liveness vs readiness, and what breaks when each is misconfigured?Edge-triggered means "pod deleted" is an event the controller must receive to act. Problem: network partitions, controller restarts, and dropped watch connections all lose events — miss "pod was deleted" once and the actual replica count is permanently short by 1 with nothing to fix it.
Level-triggered re-reads the current true state every loop and computes the diff: a missed event doesn't matter — next loop still lists "actual 2 ≠ desired 3" and adds one. After a controller restart, one full list re-aligns everything, no event replay needed.
The cost: actions must be idempotent (same diff applied many times yields the same result), plus there's polling/resync overhead. K8s balances both with informer caches + periodic resync: watch increments for efficiency, periodic full lists as a backstop against drops. This is the very root of self-healing — failure isn't a special case, just another deviation of the level.
etcd is a single Raft group; writes need a majority to fsync, so write throughput has a hard ceiling (order of tens of thousands/sec), and all API Server watch fan-out flows from it. Approaching thousands of nodes, symptoms appear: rising apply latency, watch events backing up, full lists crushing memory.
Mitigations: ① split the high-churn events objects into a separate etcd so they don't contend for IO; ② watch cache in the API Server so many clients share one watch instead of hitting etcd directly; ③ paginated list + resourceVersion to avoid full pulls; ④ control the count and update frequency of CRDs/objects (abusing high-frequency status writes is a common killer); ⑤ the ultimate move — split the cluster. A single cluster always has a scale ceiling; Google Borg long ago sharded by "cell", and K8s production commonly runs multi-cluster + federation, trading blast radius for scalability.
Packed to 90%, the cluster has almost no headroom. One node dies → its dozen-odd pods all go Pending needing reschedule → but the other nodes are also ~90% full, so they don't fit → pods stuck Pending, or the autoscaler adds machines (with minute-scale cold start). Meanwhile the surviving nodes must absorb the dead node's traffic while already near saturation → cascading overload → more nodes crushed. This is a correlated failure.
The core opposition: failover needs "empty slots elsewhere" to catch refugee pods, and empty slots are wasted utilization. High utilization = no slots = failover fails. The engineering answer isn't either/or but explicit reservation: keep N+1 or per-AZ headroom (e.g. enough per AZ to survive an entire AZ loss), use PodDisruptionBudget to cap simultaneous unavailability, and anti-affinity to spread replicas. Fundamentally this is the same as capacity planning's P99/queueing-theory headroom (Day 27).
Three bills: ① resources — each sidecar holds tens-to-hundreds of MB of memory + a slice of CPU, ×1000 pods = 1000 copies, possibly 10-20% of the cluster's resources spent purely on proxying; ② latency — every call now traverses "local Envoy out → peer Envoy in", two proxy hops, sub-ms each but cumulative and amplifying the P99 tail; ③ ops — a sidecar version upgrade means rolling-restarting all business pods, and injection failures / startup ordering (business sends traffic before the sidecar is up) are classic traps.
Why sidecar-less: this overhead is poor value for teams that just want "mTLS + basic metrics". Ambient mode sinks L4 (mTLS, TCP routing) into a shared node-level proxy (ztunnel), only routing through a shared waypoint proxy when L7 (HTTP routing, retries) is needed; Cilium uses eBPF to do L4 policy and load balancing directly in kernel space, skipping the proxy process entirely. The cost is re-weighing L7 capability and per-pod isolation granularity — a re-pick on the "full capability vs low overhead" spectrum, not a free lunch.
Container isolation rests entirely on the kernel's namespaces/cgroups. That means the attack surface is the entire Linux kernel — one kernel privilege-escalation bug (container escape) lets you break out of the container to the host, then peer into other tenants' containers on the same host. Fine for same-trust in-house microservices, but on public cloud a neighbor could be an attacker running malicious code, and one escape = cross-tenant data breach. Unacceptable.
AWS's answer: Lambda and Fargate run not on raw containers but on Firecracker microVMs — each function/task runs in a minimal KVM VM with its own guest kernel, isolating tenants via the hypervisor, a far narrower and more auditable boundary than kernel syscalls. It strips traditional VM device emulation, pushing startup to ~hundreds of ms with tiny memory overhead, reclaiming near-container density while keeping VM-grade isolation. Similar ideas: Google gVisor (a userspace re-implementation intercepting kernel syscalls) and Kata Containers (lightweight VMs). The takeaway: isolation strength and density can be partly reconciled, but never for free — running untrusted multi-tenant code means paying the isolation tax.