CS PAPERS DEEP-READ · PAPER 67
Agache et al. · Amazon Web Services · NSDI 2020
There is a kind of cloud service called "functions": you upload a small piece of code, never think about servers, and it runs only when someone calls it — billed by the millisecond. AWS Lambda is the biggest of them. In 2020 Amazon published the thing that actually runs your code underneath Lambda: Firecracker, a virtual machine shaved down to a skeleton. It lets one physical machine hold thousands of mutually distrusting customers' code at once, with each one booting in an eighth of a second.
Running a stranger's code on your own machine has always meant choosing between safe and fast.
A traditional virtual machine is safe: it hands each guest an entire fake computer, and whatever they do inside, they can't get out. The price is a boot that takes seconds and a few hundred megabytes of overhead — absurd for a function that runs for 20 milliseconds.
At the other end are containers: no fake computer, everyone shares one operating system and is merely shut into their own room. Light, and they start in milliseconds. But the walls of those rooms are that shared operating system, which keeps more than three hundred doors open to the outside — and those doors have been pried open before. Amazon was not willing to let mutually distrusting strangers share one operating system. The default assumption was: pick one.
Firecracker's idea is to interrogate that default: what exactly is slow about a virtual machine?
The answer: not the isolation, but the fact that it tries so hard to be a real computer. To be able to run any operating system at all, a general-purpose VM faithfully fakes an entire PC — power-on self test, floppy and optical drives, sound and graphics cards, USB ports, a whole expansion bus. On boot the guest has to walk that cabinet of fake hardware item by item, and that is where the seconds go. But a cloud function only needs to run a piece of code, not any antique PC. So shave the fake computer down to the few parts that are actually needed.
The "computer" Firecracker hands each guest has exactly four things on it: a network card, a disk, one wire for printing logs, and a keyboard with a single key — and that key is power-off. No self test, no expansion bus, not even a boot screen: it drops the operating system straight into memory and starts at the first instruction.
Think of a capsule hotel that stops fitting every room with a kitchen, a garage and a garden, and leaves only a bed, a lock and the utilities — the guest only wanted a night's sleep anyway. Small rooms mean thousands per floor; quick turnaround means the next guest checks in as fast as the last one leaves.
Two more things came along with it. First, new building material: the program that hands out the rooms was rewritten in a language where security bugs are much harder to write, in about fifty thousand lines — where the veteran equivalent runs to over a million. Less code, fewer seams to pry at. Second, a second lock outside the door: if someone really did climb out of a room, what they land in is not the corridor but a small cage that was stripped of nearly every privilege in advance.
Each of these "microVMs" costs under 5 MB of extra memory, takes about 125 milliseconds from creation to running customer code, and one machine can mint 150 of them per second. So a single physical machine can be packed tight with different customers' functions, none of them able to touch the others — safe and fast, no longer a choice of one. It was open-sourced in 2018, and both Lambda and Fargate run on it today.
One honest cost: this computer is deeply specialized — no graphics card, no expansion slots, so work that needs them simply cannot run here.
Virtual machines are slow and heavy not because isolation is expensive, but because they pretend to be a real computer with everything on it. Shave the fake hardware down to a network card, a disk, a log wire and a power-off key, rewrite the manager in a safer language, and wrap a cage around it — the isolation is still virtual-machine grade, yet boot is an eighth of a second and the overhead is a few megabytes. That is how cloud functions got to be safe and cheap at the same time.
Want the architecture diagrams, the rejected alternatives, and the measured numbers? → switch to the deep read
Firecracker is the minimal virtual machine monitor (VMM) AWS built for Lambda and Fargate: it throws away almost all of the hardware a general-purpose VMM emulates for the sake of generality, keeping only a virtio network device, a virtio block device, a serial port and a crippled keyboard controller whose sole job is to press "power off". Written in roughly 50,000 lines of Rust on top of Linux KVM, it gives each "microVM" a VMM memory overhead of under 5 MiB, about 125 ms from creation to user code, and up to 150 microVM creations per second per host — putting virtual-machine-grade isolation inside container-grade density and startup time for the first time.
The authors — Alexandru Agache, Marc Brooker and colleagues — are AWS engineers, and the paper appeared at NSDI 2020. Firecracker itself had been open source since late 2018; the code started as a fork of crosvm, Google's Rust VMM for Chrome OS, and the two diverged afterwards. It stands on KVM and virtio, the two pieces of infrastructure that made virtualization cheap, and it opened the way for the rust-vmm ecosystem and projects like Cloud Hypervisor that assemble a VMM out of Rust parts. Read it against the previous paper, the Berkeley View: that one argues about what serverless should be and what it lacks; this one shows how the brick underneath it was fired.
Early Lambda got its isolation from the EC2 instance boundary itself: one customer account's functions ran on virtual machines dedicated to that account, and different customers were separated by whole instances. Impeccable security, far too coarse a grain. A customer who only occasionally runs a 50 ms function still needs an instance standing by; instances boot in seconds, so capacity has to be reserved and kept warm. Low density means idle machines means cost — and cost is exactly what a per-millisecond billing model is selling against.
Raising density means packing different customers onto the same host, which forces the question: what serves as the isolation boundary? The paper lays the options out:
ioctl, /proc and /sys, which have produced escape vulnerabilities repeatedly. Stacking mutually distrusting tenants on one kernel was not an acceptable risk.From that the paper states six hard requirements: isolation, overhead and density, performance close to bare metal, compatibility with arbitrary Linux binaries, fast switching (create and destroy in milliseconds), and soft allocation (oversubscription must be possible). Nothing available met all six, so they wrote their own.
The pivotal judgement in the paper is that hardware virtualization itself is not slow. Since CPUs gained virtualization instructions and KVM moved that work into the kernel, the overhead of a guest executing instructions and touching memory has been small. The weight comes from the VMM's other job — faking an entire PC: BIOS self test, PCI bus enumeration, USB, floppy and optical drives, graphics, plug and play. The guest kernel probes that cabinet of fake hardware on boot, and most of those seconds go there — while the same emulation code is also the VMM's largest attack surface.
But a serverless guest is a Linux the platform itself supplies, running a customer's function. It never installs Windows, never needs a GPU, never hot-plugs USB. Once the compatibility requirement drops from "any operating system" to "any Linux binary", there is finally room to cut.
So Firecracker's device model retains four things: a virtio network device, a virtio block device, a serial port for logs, and a crippled keyboard controller whose only purpose is to let the host press "power off". No BIOS, no bootloader: Firecracker loads an uncompressed Linux kernel image directly into guest memory and jumps in according to the Linux boot protocol. The boot path is down to "load the kernel, find four devices".
The cost is explicit: the guest kernel has to be built for this stripped device set (virtio drivers on, most everything else off), and a customer who wants a GPU or PCI passthrough will not find one here. This is deliberate specialization, not an oversight.
If there is only one isolation boundary, that boundary has to be perfect. Firecracker assumes it is not, and stacks two.
The first is making the VMM small and sturdy: about 50,000 lines of Rust versus QEMU's million-plus lines of C — shrinking the attack surface along two axes at once, since Rust rules out buffer overflows and use-after-free at compile time, and those are precisely where VMM vulnerabilities come from. One microVM is one process, and each guest vCPU is a thread in that process, so a compromise or crash is contained to that one VM.
The second is the jailer: before Firecracker starts, it is locked in a cage — chroot for the filesystem root, namespaces for process and network views, cgroups for resources, dropped privileges so it does not run as root, and finally a seccomp allowlist permitting only the few dozen system calls a running microVM genuinely needs. The attack chain becomes: break the guest kernel, then break the KVM boundary, and even then land in a process that was stripped of privilege in advance.
The honest part: KVM and the host kernel remain in the TCB. This boundary is not zero-trust; it has merely been compressed onto a very narrow interface — the set of ioctls KVM exposes, far smaller than the full syscall surface.
Density is not only about fitting things in, it is about affording them. A function asks for 2 GB of memory and typically uses tens of megabytes. A microVM's memory comes from host pages allocated on demand: memory the guest never touches is never really handed over, so memory and CPU can be oversubscribed — what the paper calls soft allocation. That is the step that turns density into money.
The cost is neighbours competing for I/O. Firecracker therefore builds token-bucket rate limiters into the block and network devices (tokens drip into a bucket at a fixed rate and you wait when it is empty — capping the long-run average while still allowing short bursts), so the layer above can give each microVM a quota.
Its only external interface is a REST API over a Unix socket: configure the kernel, disk, network and vCPU count, then say "start". On each Lambda worker a component called the MicroManager drives that API, maintaining a pool of prepared microVM "slots" and assigning a slot to a function when a request arrives — a slot only ever serves one customer's one function, and is never reused across customers.
The paper's three headline numbers: under 5 MiB of VMM memory overhead per microVM, about 125 ms from creation to running user code, and up to 150 microVM creations per second per host. Those are the quantitative payoff for isolation, density and fast switching.
The evaluation puts Firecracker beside QEMU and gVisor on boot time, memory overhead and I/O throughput. Firecracker wins clearly on boot and memory — the firmware and device model it removed are exactly where QEMU spends time and bytes. On I/O the paper is candid: block and network throughput trailed QEMU's at the time; the trimmed virtio implementation gives up some performance, and the authors list it explicitly as known work remaining rather than papering over it.
For an industrial paper, though, the hardest evidence is not a benchmark: AWS Lambda runs entirely on Firecracker, and Fargate followed. Trillions of function invocations a day run on this thing — which says more about whether it holds up than any lab number.
① In one line: shave a VM's device model down to a virtio NIC, a block device, a serial port and a power-off key, rewrite the VMM in 50k lines of Rust, and virtual-machine-grade isolation arrives at container-grade density and startup.
② The old pain: early Lambda gave each customer account its own instance — clean isolation, far too coarse; low density, boots in seconds, machines idling, all in direct conflict with per-millisecond billing.
③ Rejected alternatives: containers (boundary is 300+ syscalls, too wide), language sandboxes (locks in a language, compatibility fails), user-space kernels (compatibility and performance both cost), general-purpose VMMs (a million lines, second-scale boots, tens of MB resident).
④ The pivotal judgement: virtualization is not what is slow, the device model is. KVM already hands CPU and memory to hardware; the seconds go to BIOS self test and PCI/USB enumeration — and serverless only needs "any Linux binary", which is what makes the cutting possible.
⑤ The mechanism: no BIOS, load an uncompressed kernel and jump in; four devices only; one process per microVM with vCPUs as threads; a jailer outside (chroot + namespaces + cgroups + seccomp allowlist + non-root) as a second boundary.
⑥ Operational knobs: demand-paged memory enables oversubscription; token-bucket rate limiters on block and network devices handle noisy neighbours; the control plane is a REST API on a Unix socket, with a MicroManager owning slots that are never reused across customers.
⑦ Results: < 5 MiB overhead, ~125 ms to user code, 150 creations per second per host; it beats QEMU on boot and memory, trailed it on I/O at the time, and the paper says so; the hardest evidence is Lambda and Fargate in production.
⑧ Impact: it dismantled the false "safe vs fast" dilemma, made the serverless cost model real, and seeded the rust-vmm family of "build your own VMM in Rust" projects.
⑨ Limits: no GPU or PCI passthrough, an I/O performance debt, cold starts still dominated by customer runtimes (later addressed by snapshot/restore), KVM still in the TCB, side channels unstopped, operational parameters undisclosed — and no change to serverless's ship-data-to-code structure.