Autopilot: Workload Autoscaling at Google

In one sentence

Autopilot is the autoscaling system Google runs on top of Borg. It sets each task's CPU and memory limits (vertical scaling) and each job's replica count (horizontal scaling) automatically: usage is accumulated into a histogram whose samples decay exponentially with age, a statistic is read off it (the peak for memory, a high percentile for serving CPU) and multiplied by a safety margin; in parallel, a pool of differently parameterised models replays history and votes, weighted by a "cost". By publication it covered more than 48% of Google's fleet-wide resource usage, cut slack — the gap between limit and actual usage — from 46% under human management to 23%, and reduced the number of jobs severely affected by OOMs by roughly an order of magnitude.

Glossary

Where it sits

The paper is led by Krzysztof Rzadca (University of Warsaw / Google) with a Google team that includes John Wilkes, an author of the Borg paper, and appeared at EuroSys 2020. It follows directly from Borg (2015): Borg solved "which machines should these containers be packed onto", while Autopilot fixes the half Borg never touched — the numbers those containers asked for were wrong to begin with. Downstream, it is the reference point for every "rightsizing" practice in the cloud today: the recommender in Kubernetes' Vertical Pod Autoscaler is built on exactly this decaying-histogram / percentile / safety-margin recipe.

The problem

Systems like Borg make you declare a limit at submission time. Declare too little and your tasks get OOM-killed or CPU-throttled; declare too much and the scheduler holds that capacity for you and gives it to nobody else. The incentives are completely asymmetric: underestimating costs you an outage, a pager and a postmortem; overestimating costs you a line on a bill nobody audits.

So everyone overshoots — and then never revisits the number. The load changes, the code gets optimised, the traffic moves away, and that figure still sits where somebody set it in a panic two years ago. The paper's name for the resulting gap is slack, and for manually managed jobs it runs to 46%: almost half of what was reserved was never used.

Why can't Borg's existing resource reclamation fix this? Borg does hand the unused portion to low-priority batch work, but that only hides the waste: a high-priority job's limit is still a hard reservation, still occupies schedulable capacity, and still drives how many machines capacity planning has to buy. Reclamation treats the symptom; a smaller limit treats the cause.

The core ideas

Turning "pick a limit" into something with a cost function

Autopilot's pivotal move is not an algorithm. It is admitting that the two errors cost different amounts, and writing that down explicitly. Each candidate limit gets a cost that is roughly the sum of three terms:

• The cost of overrunning: moments where usage exceeded the limit. Overrunning memory means the task dies, so it carries an enormous weight; overrunning CPU only means throttling, and weighs far less.
• The cost of waste: however much the limit sat above actual usage, measured as resources held for nothing.
• The cost of churn: changing a limit at all is penalised, because in Borg a changed limit often means the task is rescheduled and restarted — never free for a stateful or slow-starting service.

With that function in hand, "what number should I put here" stops being a judgement call requiring nerve and becomes an optimisation you can replay against history and score. That is the hinge of the whole paper.

Vertical scaling, part one: the sliding-window recommender

The first recommender is plain and effective. It samples each task's CPU and memory usage continuously and accumulates the samples into a histogram — not the raw time series, just the distribution, which is cheap to store and lets any percentile be read back at will.

The crucial twist is exponential decay: each sample's weight shrinks with age as w(τ) = 2^(−τ/t½), where τ is how old the sample is and t½ the half-life. In plain terms: the histogram fades, so a freak spike from a month ago does not hold today's quota hostage — but it isn't forgotten instantly either. The typical values the paper gives are a half-life of about 12 hours for CPU and about 48 hours for memory. Memory remembers longer, because forgetting too fast is paid for in dead tasks.

Which statistic you take depends on what happens when you exceed it:

• Memory: the peak over the decayed window. One overrun means death; there is nothing to haggle over.
• Serving CPU: a high percentile (p90 / p95). Being throttled for a few seconds now and then is a mild slowdown; paying for the all-time peak is not worth it.
• Batch CPU: the average is enough. Batch work never cared about momentary slowness, only about total throughput.

The resulting number is then multiplied by a safety margin — on the order of ten percent, and relatively wider for small tasks, whose relative fluctuations are larger — and rounded up to a tidy value. Not a line of it is difficult mathematics. All of its weight is in one judgement: which resource deserves which appetite for risk.

Then there is applying the new limit. Because a changed limit can trigger a restart, Autopilot changes them in a rolling fashion — never every replica of a job at once — and is especially cautious about lowering memory: raising is always allowed, lowering demands stronger evidence from history.

Vertical scaling, part two: the multi-model recommender

The sliding-window recommender has knobs: how long a half-life? p90 or p95? how much margin? The best setting is not the same for every job — a service with flat traffic and one with two daily spikes need entirely different fade rates. Tuning knobs per job by hand would just move the form-filling problem somewhere else.

The second recommender (whose approach the paper describes as inspired by reinforcement learning) answers this by not picking one parameter set, but keeping a stable of models. Each model carries fixed parameters: its own decay rate, its own safety margin, its own way of turning the histogram into a limit. For each job the system replays the last few weeks of real usage and asks each model: "if we had followed you, how often would you have overrun, how much would you have wasted, how many times would you have changed the limit?" The cost function above turns that into a score, models with lower cost get more say, and the recommendation is their score-weighted combination.

One more penalty matters: switching models costs something too. Otherwise a hair's-breadth difference in scores would have the system following A today and B tomorrow, yanking the limit back and forth — and each yank can restart tasks.

Horizontal scaling: fast up, slow down

Vertical scaling governs how big one replica is; horizontal scaling governs how many there are. The latter sets the replica count from a signal — CPU utilisation by default (total load divided by a target utilisation), or a custom metric the job owner supplies, such as queue depth or requests per second.

The instructive part is that it is deliberately asymmetric: scaling up follows recent load over a short window and reacts almost immediately, while scaling down looks at the peak over a multi-day window and descends one small step at a time. The reasoning is thoroughly practical — adding a replica you did not need costs a little money; removing one you did need leaves you caught out when the traffic returns. The same asymmetric-cost principle, applied a second time along the other axis.

Getting people to let go: the rollout is part of the design

Readers skip this section, and it is exactly what separates this paper from an algorithms paper. Taking quota control away from engineers is not technically hard; getting them to agree is. Autopilot's answer is a ladder of evidence: offline simulation on historical traces first; then a dry run in production that records what it would have set without applying anything, so owners can watch it against their own judgement; then controlled comparisons; then a phased handover with visible dashboards and a one-click way out at any moment. Small and new jobs are opted in by default, with opt-out available. This is engineering about how automation earns trust, not just about how a quota is computed.

Results

• Reach: at publication, autopiloted jobs already accounted for more than 48% of Google's fleet-wide resource usage. This is not a lab prototype; it is half of Google.
• Savings: autopiloted jobs run at about 23% slack against about 46% for manually managed ones — roughly half of it gone. In one comparison in the paper, absolute slack fell from the equivalent of about 12,000 machines to about 500; the authors estimate overall savings for Google in the tens of millions of dollars.
• Not paid for in stability: this is the counter-intuitive and most persuasive part. Tightening limits should mean more OOM kills — instead, the number of jobs severely affected by OOMs dropped by about an order of magnitude, and over 99.5% of job-days saw no OOMs at all. The machine's estimates were not just cheaper than the humans' guesses; they were more accurate.

Why it matters

First, it completes Borg's other half. Borg perfected the packing, but the input it packs — the limit each job declares — had always been wrong. No scheduler, however good, can rescue a systematically inflated order form. Autopilot fixes the input itself.

Second, it turns rightsizing from an operations craft into an optimisation problem. Once you can write a cost function and score candidates by replaying history, the work becomes automatable and continuous instead of depending on somebody's conscience one quarter. Kubernetes' VPA, cloud resource recommenders and FinOps tooling all walk this road — the VPA recommender is essentially the public implementation of this decaying-histogram / percentile / safety-margin recipe.

Third, and most underrated: it offers a template for how automation gets accepted. Simulate, dry-run, compare, phase in, allow exit — the path itself is reusable methodology.

Limitations and criticism

• It can only copy history. No history, no recommendation. Brand-new jobs, jobs just rewritten, and traffic that changes shape (a launch, a sale, failover spillover) all leave it lagging behind on a conservative default. The genuinely hard problem — predicting the future — it does not solve.
• Usage is itself distorted by the limit. What the system observes is not real demand but demand after being constrained: a throttled task simply looks like it uses less CPU. Every usage-driven quota system shares this feedback loop, and it can ratchet a starved service further down. High percentiles, safety margins and throttling metrics mitigate it; nothing removes it.
• Changing a limit is not free. Applying vertical scaling often means the task is rescheduled and restarted — a real cost for stateful services, services with caches to warm, services that take minutes to start. That is precisely why churn gets its own penalty term and why rollouts are gradual, and it means the approach fits stateless serving work best.
• The objective is the resource bill, not the user experience. Slack and OOMs are both proxies. Provisioning the CPU p95 correctly does not prove end-to-end tail latency was unharmed, especially near the throttling boundary. The paper does not — and could hardly — offer an SLO-level proof.
• Automation did not abolish human guessing; it relocated it. The relative weights of overrun versus waste, how the model pool is constructed, how much margin to allow — all remain global human judgements. It replaced "guess once per job" with "guess a handful of hyperparameters once". An enormous improvement, but not the disappearance of the problem.
• It is a production report, not a general algorithm. Every number comes from Borg and Google's comparatively homogeneous, thoroughly instrumented workloads. There is no public code or dataset, so it is hard to reproduce externally, and how much of the benefit survives in messier, less observable, more weakly isolated environments is open.

Takeaways