2007 to present · 839 posts · 16-year span (steady weekly cadence since September 2010) · One clear pivot: left tenured academia around 2020 for industry (three years at AWS, now MongoDB Research) · Still active, most recent post 12 September 2026 · Entirely free, no paywall · This page draws on 19 posts read in full, spanning January 2011 to September 2026, four of them from the earliest period, with samples on both sides of the pivot
To write this page I crawled the body text of all 839 posts on Murat Demirbas's blog and counted the internal links, wanting to know which posts are the load-bearing walls of a blog that has run for sixteen years.
The top result is not Paxos. It is not any paper review. It is a short piece published on New Year's Day 2018, half of which is about where the question mark came from and whether apes can ask questions. Forty-six inbound links.
Second place goes to his own invention, hybrid logical clocks (41). Third is the follow-up to that same New Year's piece (33). Six of the top ten are about how to ask questions, how to read a paper, how to write, how to show your thinking. Only four are technical.
A leading expert in distributed systems, and the foundation of his blog is a set of posts about how to think.
This is not an accident. At the end of 2017, after a decade as a professor and seven years of blogging, he sat down and gave himself a fairly unsparing diagnosis: his questions had gone blunt, they "often come out as shallow/predictable." Then came the line that really stings:
But yeah, I cannot be sure whether I have gone compliant/docile recently, can I?Mad questions, 2 January 2018
His New Year's resolution was not "ask more questions." He knew resolutions do not work. His move was to install a system: from then on, every blog post would end with a mandatory section called "MAD questions," plus a new blog tag to collect them.
A year later he reviewed the experiment. Across 70 posts, that section averaged 25% of the length of the main body and was frequently longer than the post itself. More surprising: a lot of half-finished drafts only became publishable because the section forced him to keep thinking. And his explanation for why something so cheap worked so well is the best move in the whole story — it works because
our default mode is to not question things. We have been implicitly taught this at our family, school, and work.Master your questioning skills, 17 December 2018
So "MAD" is not decoration, it is a permission slip: calling them mad questions granted him immunity from the usual assumptions. He also gives one unusually practical rule: do not be embarrassed by your first, stupid questions. "Bad questions lead to good questions." Exhaust them and the good ones surface. And how do you know a good one when it arrives? A good question is like a good joke — it has an unexpected element, it changes the perspective, and you feel it in your body.
All of this can read like self-help until you see the hard premise it hangs on, which is the real key to the man. He nailed it down in 2012:
We all have humble brains; don't let anybody tell you differently. Falling short of superpowers (genius/giftedness), we invent processes/systems to succeed, and the good thing about these processes are they are replicable (unlike being genius/gifted).Tell me about your thought-process, not just your results, 11 January 2012
Six years later he compressed the same axiom into five words, in an entirely different context: "Being smart does not scale." That was about why you cannot design a distributed protocol by reasoning it through in your head, and need exhaustive model checking instead.
Once you hold that key, his apparently unrelated subjects line up. They are all versions of one move: how does an ordinary mind get reliable understanding of a system too big to hold in the head? The answer is never "be smarter." It is always "build an instrument."
His most famous instrument is the hybrid logical clock, from 2014. Time in distributed systems was a stalemate then. Lamport's 1978 logical clocks record causal order but are divorced from real time, so you cannot ask what the database looked like last Tuesday. NTP-synchronized physical clocks have real time but cannot be tightly synchronized across regions, and come with leap seconds and non-monotonic jumps. Google's 2012 Spanner threw money at it — GPS receivers and atomic clocks to push the uncertainty down to 6ms — at the cost of building an entire timing infrastructure, and of having to explicitly wait out that uncertainty interval on every commit.
Three roads, each charging a toll. The question he and Sandeep Kulkarni asked ran sideways: why must we choose? Could one timestamp preserve causality, fit in constant space, and stay within a bound of the physical clock?
The naive construction fails: copy Lamport and take the max, and the logical part runs ahead of physical time without bound. The HLC trick is a small counter c plus a guarantee that it keeps getting reset — either you receive a larger logical time, and c goes to zero, or the physical clock catches up and pushes the logical value along, and c goes to zero again. Both roads lead to a reset, so the gap stays trivially bounded. Then the engineering flourish: let the logical part track only the high 48 bits of the physical clock (still microsecond granularity) and give the remaining 16 bits to c. The line that actually sells the design sits in a parenthesis:
(In our experiments c mostly stayed in single digits.)Hybrid Logical Clocks, 27 July 2014
Room for 65,536, single digits used. CockroachDB, MongoDB and YugabyteDB all run on it today.
Ten years later he wrote a five-part survey whose conclusion is worth more than the algorithm, because it redefines what time is for: synchronized clocks are not about knowing what time it is, they are a shared reference frame.
When nodes operate independently, they essentially exist in their own temporal universes.Use of Time in Distributed Databases (part 5), 14 January 2025
Align the frame and a great deal that used to require a round trip — consistent snapshots, conflict detection, leases — can be computed locally instead. But he holds one line, inherited from Barbara Liskov in 1991 and in his view unchanged since: use time for performance, not for correctness. When a clock misbehaves you should get slower, not wrong.
His sharpest recent judgment is from May 2026. He has taught TLA+ for years, always saying abstraction is the most important and hardest step. But a dissonance kept nagging: aren't computer scientists supposed to be good at abstraction already? Abstract data types, APIs, layering — staples of every CS curriculum. So why does every formal methods person report a gaping skill gap?
His answer: two opposite things hide under one word.
Modularity abstraction hides. Modeling abstraction reduces.The Two Abstractions of System Design: Hide or Reduce, 8 May 2026
Modularity abstraction draws boundaries and encapsulates, cutting vertically to hide the layer below so the module is easy to use — which means hiding concurrency and pretending operations are atomic. Joel Spolsky's famous complaint that all abstractions leak is entirely about this kind.
Modeling abstraction goes the other way. It cuts crosswise, reducing the system to the minimal behavioral skeleton for the one property you care about and discarding everything else. And — the counterintuitive part — it deliberately lets things leak: it exposes the fine-grained actions and every possible interleaving, then proves the invariants hold anyway. The payoff is not usability. It is harvesting the maximum safe concurrency the system has to offer.
The examples he reaches for are a potted history of the field, each readable as "what got thrown away": Lamport clocks discard wall-clock time and keep happens-before; HLC keeps both real time and causality and discards the rest; TrueTime turns time into a bounded-uncertainty interval; linearizability discards replication, caching and retries; log-is-the-database discards materialized state as the source of truth; MapReduce discards orchestration, parallelism, scheduling and fault tolerance.
If you want one sentence for the man, take the advice he gave distributed systems designers in 2023 — from someone who has spent half his career on consensus protocols:
Even as a distributed coordination researcher and Paxos expert, my advice is to reduce coordination.Hints for Distributed Systems Design, 2 October 2023
And his most effective instrument, in the end, is the blog itself. He holds a weekly quota, not for output but for "noticing" — staying in a frame of mind where things register. The quota, he says, lowers his standards for writing, which paradoxically raises them. Rereading his own posts from years ago "refires the same neurons," letting him reload the understanding he once strained to build.
The blog became his external memory. 839 posts, over five million views: one person's replicable, ungifted record of digging wells.
To write this page I did something laborious: I pulled the body text of all 839 posts on Murat Demirbas's blog and counted how they link to each other. The idea is simple — after sixteen years of writing, a few posts will be the ones the author keeps reaching back for. Those are the load-bearing walls.
The top result is not Paxos, not Spanner, not any paper review. It is a short piece published on New Year's Day 2018, whose first half is about how his mother, on family trips, would point at the plants outside the window and ask him why the landscape changed here, and whose second half seriously considers whether apes can ask questions and where the question mark came from. Forty-six inbound links from later posts.
Second, with 41, is his own technical invention: hybrid logical clocks. Third, with 33, is the one-year review of that same questioning experiment. Going down the list, six of the top ten are about how to ask questions, how to read papers, how to write, how to show your thinking. Only four are technical.
This is not a traffic ranking. He has one of those too, topped by "Learning about distributed systems: where to start?" and "What I'd do as a College Freshman in 2025." This is his own citation ranking: the bricks he reaches for when building something new.
Someone who has spent twenty-five years in distributed systems and databases and invented an algorithm now in wide industrial use, and the foundation of his blog is a set of posts about how to think. That is not a digression. That is the trunk.
Murat Demirbas is Turkish, took his PhD at Ohio State under Anish Arora, then did a postdoc at MIT with Nancy Lynch. From 2003 he taught computer science at the University at Buffalo (SUNY) for twenty-one years, winning an NSF CAREER Award in 2008. Around 2020 he left academia for industry: three years as a Principal Applied Scientist at AWS, and now Principal Research Scientist at MongoDB Research.
The blog is called Metadata. It became a regular habit in September 2010 for a very concrete reason: he wanted to move from distributed systems theory and wireless sensor networks into cloud computing, so he sat in on a colleague's cloud computing seminar and decided to write a summary of every paper he read, to force himself to digest it. The first one was on MapReduce, which he says "seems to have opened the floodgates." Sixteen years later: 839 posts, over five million views.
One detail is very him. The blog lives at muratbuffalo. In July 2026 he finally hauled a 12-by-6-foot U-Haul trailer across the country to the Bay Area, delayed a full day because his cat Pasha, sulking in the emptied house, ran off and did not come back until eleven at night. He wrote that Buffalo seemed to have "the escape velocity of a black hole," and that when he named the blog he had no idea the name would stick and "almost become a curse." The blog is still running; the most recent post is dated 12 September 2026.
The question he has not put down in sixteen years fits in one sentence: how does an ordinary mind get reliable understanding of a system too big to hold in the head?
Everything turns on "ordinary." That is not modesty, it is the axiom of his entire method, and he stated it without hedging:
We all have humble brains; don't let anybody tell you differently. Falling short of superpowers (genius/giftedness), we invent processes/systems to succeed, and the good thing about these processes are they are replicable (unlike being genius/gifted).Tell me about your thought-process, not just your results, 11 January 2012
He did not arrive at this abstractly. A class made him angry.
In that 2012 post he recalls his first networking course. The textbook opened by laying out the TCP and UDP packet formats field by field — lengths, flags, checksums — before introducing a single protocol idea or even the basic problem networking is trying to solve. The instructor followed it "blankly," reading from the publisher's slides, which, as he puts it, apparently did not target a human audience. He hated networking for years afterwards.
Then, characteristically, he did not stop at the complaint but followed the chain of responsibility upward. Why is the instructor like that? Laziness; he follows the textbook. Why is the textbook like that? Also lazy; it follows the style of technical papers. Why are papers written that way? He traces it to the Bourbaki school in twentieth-century France — the project that rewrote mathematics as an axiomatic edifice and moved the emphasis from concrete examples to abstract generality. He quotes Freeman Dyson: in the Bourbaki scheme, what is not in the textbooks is not mathematics, and concrete examples, not appearing there, are not mathematics.
His verdict is that rigor and generality are not the problem. The problem is that the paper deletes the thinking that produced the result. And he suspects a less flattering motive: deleting the process serves to "cover up the tracks of our 'lacking' thought-processes" and present a face of flawless conclusions. He says he has genuinely heard the logic in collaborations — it was hard for me to work out, so it should be hard for the reader.
The section lands on Haruki Murakami. He paraphrases What I Talk About When I Talk About Running: gifted writers touch the ground anywhere and water pours out; other writers have to learn to dig wells. And the day the water dries up comes for everyone — the gifted writer is stranded, having never trained for it, while the well-digger keeps digging.
This is the reason we should teach our students/audience how to dig wells, and not to rely on occasional strike of insights for success.Ibid.
Six years later he compressed the same axiom into five words for a completely different purpose — why you cannot design a distributed protocol by reasoning it through: "Being smart does not scale."
So everything he has done for sixteen years reads as one move: build instruments. Not think harder, not cultivate intuition, but construct concrete devices you can pick up every time, which do not depend on how you feel that day. Here are the four best ones.
At the end of 2017 he ran a check on himself, and the result was not flattering.
He already had a habit of stopping to interrogate himself. He works in pomodoros, and had customized his to spend the first and last four minutes forcing himself to "go meta": is this the right approach? What else could I try? Am I making good progress? Can this be made transformative?
The trouble was that the habit had become a routine. The questions "often come out as shallow/predictable." Still useful as a sanity check, but no longer able to move him anywhere. And then the genuinely uncomfortable admission:
But yeah, I cannot be sure whether I have gone compliant/docile recently, can I? Maybe I am losing my edge, because I have been in the system long enough. Maybe I start seeing myself as experienced now, and it is affecting me.Mad questions, 2 January 2018
That is a tenured professor stating in public that he might already have been domesticated.
What matters is what he does next. His resolution was to ask "more/better/crazier questions" — and he immediately notes that a resolution alone is worthless, so he had to install it as a system. The system: every blog post from now on ends with a mandatory section called MAD questions, and a new tag collects them so they stay searchable and retrievable.
Two things in this design are worth pausing on. First, it converts a matter of willpower into a matter of format — as long as you are still blogging, you have to write that section. Second, "MAD" is not a joke, it is a permission slip he issued to himself. He spelled this out a year later:
By calling these questions MAD questions, I gave myself the license/protection to go wild, uninhibited by traditional assumptions/constraints. This gave rise to reducing the bar, and approaching the topic with a beginner mind, as an outsider.Master your questioning skills, 17 December 2018
Seventy posts later he reported the numbers. The MAD section averaged 25% of the main body and was often longer than the post itself, costing perhaps fifteen or twenty extra minutes each time. But the real return was elsewhere: many pieces rotting in his drafts file (he keeps a blog.org file full of half-ideas) only became publishable because that section forced him to think a little further; and leads dug up in the questions often got folded back into the body, improving the post.
So why does something this cheap work? His explanation is the finest move in the whole business:
I think the reason for this is because our default mode is to not question things. We have been implicitly taught this at our family, school, and work. Asking too many questions, especially hard ones, is frowned upon in polite company.Ibid.
In other words this is not a problem requiring a new skill. It is a problem requiring disinhibition. The capacity was always there; it was being held down. He adds a physiological layer too: the brain is programmed to save energy, prefers shallow tasks, and avoids the intense sessions creative work demands. Asking questions primes the pump and drags it back online.
Two operational details are unusually usable. The first is do not be embarrassed by stupid questions: "Bad questions lead to good questions. So roll with them and exhaust them to get to the good questions." He stresses doing this in "a stress-free manner," like freewriting or brainstorming, trusting that going a little deeper will turn something up.
The second is how you recognize a good one, and it is his best image:
A good question is like a good joke. It has an unexpected element and provides a perspective change. After I get through some questions and I eventually arrive to a good question, I have a visceral reaction similar to that of hearing a good joke.Ibid.
That turns something vaporous into something decidable: you do not have to judge afterwards whether it was a good question, because you laugh on the spot.
And all of it points toward the (naturally MAD) question he raises at the end of that same post. Answers are teachable, so we teach answers — why do we not teach questioning? Kids in kindergarten are good at it; by middle school most have stopped. He knows of no course and no decent book that teaches it systematically; academia transmits it by apprenticeship, which amounts to "watch me ask questions and start imitating." Eight years on, his blog is the fullest answer he can give: a public, 839-post demonstration.
Before 2014, time in distributed systems was a stalemate everyone had accepted. Three roads, each with a toll.
The first is Lamport's 1978 logical clocks: each node keeps a counter, attaches it to messages, and on receipt takes the max and adds one. It faithfully records who came before whom, at the price of being completely divorced from real time — you cannot ask what the database snapshot looked like at 3pm last Tuesday, because there is no 3pm in a logical clock. It also assumes all communication stays inside the system; open a back channel and causality leaks.
The second is physical clocks: use the machine's own clock, synchronized over NTP. Now you have real time, but geo-distributed synchronization is never precise, leaving an irreducible uncertainty interval, plus leap seconds and non-monotonic jumps. When two events' intervals overlap you simply cannot order them, and you get inconsistent snapshots.
The third is Google's 2012 TrueTime in Spanner, which is blunt: buy GPS receivers and atomic clocks, push the uncertainty ε down to 6ms, and honestly report time as an interval rather than a point. The costs come in two parts. You must build the timing infrastructure in every datacenter, which he notes "can be prohibitive." And when intervals overlap TrueTime still cannot order them, so it has to explicitly wait them out — every commit burns that ε. Six milliseconds is not large, but it is certainly not zero.
So: causality without real time, or real time that costs money and still makes you wait.
The question he and Sandeep Kulkarni at Michigan State asked went sideways. Not "which of the three is the best deal," but why must we choose? They wrote the requirement as three conditions that must hold simultaneously:
Demanding all three at once looks greedy. This is exactly what he means by a good question: instead of picking among the options, it shifts perspective until the options themselves are void.
The naive construction fails, and he shows the failure honestly in the post: copy Lamport but take the max of logical and physical values, and with three nodes looping messages among themselves the logical part runs ahead of the physical clock without bound, destroying condition three.
HLC puts two numbers in the timestamp: l, tracking physical time, plus a small counter c for ordering within a single l. The real ingenuity is not adding c but guaranteeing it keeps getting reset: either you receive a larger l from someone else and c goes to zero, or l stays put while the physical clock catches up and pushes it, and c goes to zero again. Both paths lead to a reset, so c cannot climb and the gap between logical and physical time is trivially capped at ε. Under the weakest assumptions they prove c is less than N(ε+1).
Then the engineering move, the most skillful part of the post. NTP's 64-bit timestamp gives 32 bits to seconds and 32 to fractional seconds. They let l track only the high 48 bits — still microsecond granularity — and hand the remaining 16 bits to c, which can therefore grow to 65,536. The whole timestamp is still 64 bits and drops straight into anything that held an NTP timestamp.
And the line that actually convinces sits in a parenthesis:
(In our experiments c mostly stayed in single digits.)Hybrid Logical Clocks, 27 July 2014
Room for 65,536, single digits used. A theoretical bound, and an absurdly comfortable measured margin. CockroachDB, MongoDB and YugabyteDB all use HLC today.
The second half of the story arrives ten years later. From late 2024 into 2025 he wrote a five-part series surveying how distributed databases actually use time, and the conclusion is worth more than the algorithm, because it redefines what time is:
A key takeaway is that time serves as a shared reference frame that enables nodes to make consistent decisions without constant communication. ... When nodes operate independently, they essentially exist in their own temporal universes.Use of Time in Distributed Databases (part 5): Lessons learned, 14 January 2025
He even sets it beside a fashionable topic: while the AI community grapples with alignment, distributed systems have long confronted their own fundamental alignment problem. Align the frame and a great deal that used to require round trips — consistent snapshots under multi-version concurrency control, conflict detection, leases, fencing (using time to build a one-way door that shuts out a stale former leader) — can be computed locally, with no messages at all.
But he holds one line, which he considers unchanged for thirty-odd years:
Barbara Liskov's 1991 principle remains fundamental: leverage time for performance optimization, not correctness.Ibid.
He checks it case by case. Spanner uses TrueTime for performance while correctness rests on locking and consensus; DynamoDB uses timestamps to streamline transactions without depending on perfect synchronization. So when clocks misbehave, these systems get slower rather than wrong. Which is exactly where his own instrument sits: HLC supplies the monotonicity insurance that keeps NTP's misbehavior from tearing causality.
This is his sharpest recent judgment, and it starts from a dissonance that bothered him for years.
He has taught TLA+ for over a decade, always saying abstraction is the most important step and the hardest to learn. But the claim has a problem. Aren't computer scientists supposed to be good at abstraction? Abstract data types, APIs, layered design, encapsulation — staples of every CS curriculum for decades. So why do he and every other formal methods person he knows report that students cannot abstract, and flag it as the make-or-break skill?
In May 2026 he gave the answer: because two things with opposite goals hide under the same word.
Modularity abstraction hides. Modeling abstraction reduces.The Two Abstractions of System Design: Hide or Reduce, 8 May 2026
Modularity abstraction is the kind on the syllabus: draw boundaries, encapsulate, hide what is underneath. It cuts vertically, so whoever is above the interface need not know what happens below. Its goal is usability, and to be usable it works hard to hide concurrency and present operations as if they were atomic. He offers a persuasive piece of supporting evidence: Joel Spolsky's famous complaint that all abstractions leak names only this kind — TCP hiding IP, string libraries hiding character arrays, file systems hiding spinning disks, virtual memory hiding the MMU and paging, SQL hiding query plans, NFS hiding the network.
Modeling abstraction is the kind mathematicians and physicists use when building models to think with. It looks for the minimal, most elegant description that preserves the property you care about, and cuts away everything else. It does not slice vertically; it slices crosswise, along a behavioral plane, keeping only what is absolutely relevant to that property, and even then only the "what," never the "how." He points out that the resulting slice usually looks nothing like the system's actual organization.
Then the counterintuitive line. If modularity abstraction's life's work is sealing leaks, what about the other kind?
In stark contrast, the modeling abstraction is about identifying what should leak and leveraging it! It exposes the fine-grained actions and orderings, and proves that invariants hold despite the interleavings.Ibid.
Modularity abstraction hides concurrency; modeling abstraction spreads it all out. The first pays off in usability, the second, in his words, in harvesting the maximum safe concurrency from the system. You can only know which interleavings are safe to run loose after you have seen them all.
The examples he reaches for are a potted history of distributed systems, and each can be read as what it threw away: Lamport clocks discard wall-clock time and keep happens-before; HLC keeps real time and causality and discards the rest; TrueTime turns time into a bounded-uncertainty interval; consensus keeps only "agree on a single decision"; linearizability and every other consistency model discard replication, caching and retries; log-is-the-database discards materialized state as the source of truth, keeping only an ordered append-only sequence; MapReduce and Spark discard orchestration, parallelism, scheduling and fault tolerance, keeping a DAG of deterministic transforms and letting the framework reconstitute the rest from that skeleton.
He pre-empts one objection: some artifacts look like they belong to both categories (linearizability, consensus, log-is-the-database, MapReduce). That, he says, is not overlap but reuse — a well-designed artifact can serve simultaneously as a spec to refine against (modularity) and a skeleton to reason from (modeling). The two roles happen to land on one object; the roles remain distinct.
Worth noting how long this idea incubated. In January 2011, four months into the blog, his third paper review was of Butler Lampson's 1983 Hints for computer system design. In it he stops to emphasize a definition he expects will surprise many readers: an interface is not a function signature, it is the set of assumptions each programmer must make about the other program in order to demonstrate the correctness of their own. Twelve years later, in 2023, he wrote his own Hints for Distributed Systems Design, opening with "with apologies to Butler Lampson." The first hint on the list is "Apply abstraction."
This is his most sustained technical direction of the last five years, and the most immediately practical of the four.
Start with how strange the phenomenon is. A metastable failure is defined as:
Metastable failure is defined as permanent overload with low throughput even after the fault-trigger is removed.Metastable failures in the wild, 12 September 2023
Note the "even after." Your system gets shoved — a traffic spike, a bad config, a wiped cache — and then the hand withdraws, and the system does not come back. It is still running flat out, CPU pinned, producing almost nothing useful. In his phrase, it is "grinding but not doing useful work."
It is not a rare curiosity either. The OSDI'22 paper he discusses went through 600 public postmortem incident reports and identified 21 metastable outages across companies large and small. Around 35% were triggered by load spikes and 45% by buggy configuration or code deployments plus latent bugs; on the sustaining side, over half were retry-induced load amplification.
The structure fits in a two-by-two: one axis for what triggered it (load increase or capacity drop), the other for what sustains it (load increase or capacity drop).
Two of the lessons he draws are very much his.
The first is feel the pain:
Feel the pain! Don't mask the pain, feel the pain, and attribute the pain to the correct subsystem and shed load quickly so you do not trip over to the metastable state.Ibid.
This inverts the instinct behind graceful degradation. What you want to do is absorb the jitter and mask the pain so the layers above never see it; what he says is that masking pain is exactly what walks you into metastability, because by the time it hurts enough to notice, your only remaining option is a big reset. He also passes along a specific monitoring suggestion from Rebecca Isaacs: watch not the queue length but the rate of queue-length increase — a signal almost nobody monitors.
The second is shorter, and he labels it the meta lesson: "Don't DOS yourself!" Design so the system cannot launch a denial of service against itself. Concretely: beware any structure where one request begets more than one request, retries above all.
From that he generalizes to a principle about the danger of over-optimizing: do not optimize one part so hard that other cases produce disproportionate work amplification. Caching is the canonical case. Optimize to a 99.75% hit rate and you have built the system on the premise of a high hit rate; the day that premise fails, the system simply lacks the capacity to do the work it was always nominally doing. He cites DynamoDB as the positive model: it "prioritizes predictability over absolute efficiency," and its redesigned cache issues an asynchronous backend refresh even on a hit, deliberately keeping backend traffic constant regardless of hit ratio. It costs more; it buys away the bimodal behavior.
The best turn in this thread came in September 2026, days before I wrote this page. He remodeled metastability in TLA+, but this time as two separate components joined by a rely-guarantee contract: the retrier promises not to send retries as long as the queue stays under a threshold; the server promises to hold its stability basin as long as retries stay bounded.
Then he moved the initial backlog from 5 to 18 and ran the model checker. It blew up — a retry storm, the server at 100% utilization completing exactly one useful request per round, generating three times more load from its own retries than from its actual customers.
What matters is how it blew up:
In our runs, the storm sustained itself while every contract check kept passing: the system failed with no component at fault.Metastability as a failed conditional discharge of rely-guarantee composition, 3 September 2026
Nobody breached their contract and the system still collapsed, because both promises carried an "if." With a small backlog the two conditions hold each other up in a circular induction with a base case. With a large one, both implications remain logically true — their premises simply never apply, so they guarantee nothing. The base case had been knocked out.
That is why he insists on decomposing. A monolithic spec also shows the backlog growing, but only making the contracts explicit shows you which promise was too weak. And the fix is an edit to that promise: give the retrier a hard budget of B retries per round regardless of backlog, turning "at most B" from conditional into unconditional. He adds that a second, possibly better fix exists on the server side — serve fresh requests first instead of splitting capacity proportionally — which requires no client change at all, and where one server protects itself against any number of clients while the budget approach tightens as clients multiply.
Notice the echo. In January 2011, reading Lampson, he paused to stress that an interface is the set of assumptions each side must make about the other to prove itself correct. Fifteen years later he used that same concept to explain one of the nastiest failure modes in the cloud. That is not coincidence. That is an instrument he spent fifteen years sharpening, finally landing where it belongs.
Fight with the paper. His 2013 post How I read a research paper is his seventh most-linked, and the first answer sets the tone: "I fight with it." His stance: "I believe that the paper has to earn my trust, and I only accept the parts that I am obliged to accept." The process: print it, keep a highlighter and several coloured pens to hand, argue in the margins, mark anything opaque with WDYM (what do you mean), and hypothesize about what you do not understand, revising as you read on.
Then the endearing confession:
I am a bit ashamed of this, but I often write provocative things on the margin, like "This claim is stupid", "This is just bullshit". It is silly, but I do this to keep myself engaged with the paper and be emotionally involved as well. This way, I never fall asleep reading a paper.How I read a research paper, 25 July 2013
But the last step in the process is not winning. He says you must sleep on it and come back, because by then you have probably been too harsh and should put the paper back in context. He warns against confusing critical reading with being hypercritical and dismissing the paper's contributions. The line he quotes to explain this step is from Ender's Game: in the moment when I truly understand my enemy, well enough to defeat him, in that very moment I also love him.
He does not know what he thinks until he writes it. He says this repeatedly; it is the engine of the whole method. His weekly quota is not about output:
I set this goal not for the quantity of the output, but for keeping myself in the writing frame-of-mind. ... This also helps me lower my standards for writing, which paradoxically maintains and raises my standards.Why I blog, 8 March 2024
Sometimes, to satisfy the quota, he tells himself the topic is not interesting but he will attempt something anyway — and is then startled to find a fount of interesting things once he starts. His advice to new writers is anti-perfectionist in the same way: "mess up and tidy up later," keeping drafting strictly separate from editing, with a running file for dumping half-ideas and half-written text.
The blog as external memory. This use surprised him. He reports referring back to his own posts remarkably often, because "Because I strained my brain while writing them in the first place, reading them later refires the same neurons" — letting him quickly reconstruct the state of understanding he once built. A second brain with random access.
Exhaustion instead of confidence. This is the hardest landing of the humble-brains axiom. He asks: once the design is in pseudocode, can I not just check each action against my invariant myself? His answer: fine for toy examples, hopeless for real ones. He gives an example from his own teaching — the invariant talks about being in the critical section, while the actions talk about forks, per edge, over a dynamic priority graph, so doing that mapping in your head is nearly impossible. Hand it to a model checker doing breadth-first exhaustive search. What you save is not time; it is the opportunity to fool yourself.
After being scooped, take the blame back. In 2011 he coined "reverse-scooping": you solve a problem and publish first, then months later a better-known institution publishes the same result at a more visible venue, takes all the credit, and your work might as well not exist. He says the first few times he was very angry. How he eventually handled it is the man in miniature:
I take getting reverse-scooped to mean that I should have polished and published that work at the best venue I possibly could. ... Reverse-scooping is ugly, and I am not trying to rationalize it. But I find this new way of thinking to be more constructive than getting angry and complaining.Reverse-scooping, 19 December 2011
Note that he does not claim it is fair. He explicitly says it is not. He simply picked the attribution that lets him keep moving — which is how he treats everything else: do not try to repair the part you do not control, build an instrument for the part you do.
First, to be clear: I could not find any substantial, signed public criticism of him. That says something in itself — the paper-review genre rarely makes enemies. So what follows is my own judgment after reading, plus the criticisms he has written about himself, which are often harsher than mine.
1. The filter he distrusts supplies nearly all of his input. His opinion of peer review is very low — through Ponder Stibbons he calls it "ritualized academic hazing," and in his own voice he says papers "are written to satisfy three reviewers," turning defensive in some passages and pompous in others. His blog exists precisely to strip that costume off. But the inescapable fact is that what he reads is overwhelmingly published work that has already passed the process he distrusts. Ideas that failed to clear that filter mostly never reach him.
2. The fight happens in the margins; what gets published is the reconciliation. This one follows directly from his own described method. The final step of his reading process is to sleep on it, empathize, and "come to respect the paper and appreciate it more." That is a virtue as reading, but the consequence is that the fight is private and the public post is the post-reconciliation draft. Across 839 posts, the proportion saying "this paper is wrong" is strikingly low. "This is just bullshit" never became a blog post. For a reader this means his blog is an excellent translation service but not a reliable filter: he helps you understand the paper, and rarely tells you whether to believe it.
3. The biggest misjudgment he has admitted. This one is his, and he put it more bluntly than I could. When machine learning took off in 2016 he studied some, ran a few surveys, and then gave up, reasoning that the field sits too close to the application for principled algorithmic work. Reviewing the decade at the end of 2019 he wrote: "It turns out that I was judgmental, as several nice algorithmic and distributed systems started coming out recently." Worth noting that the misjudgment landed on the most consequential field of the following decade, and that its cause was exactly the taste he is proudest of. Taste cuts both ways: it filters noise, and it filters out things that have not yet taken shape.
4. He diagnosed the scaling problem of formal methods fifteen years ago, and his current fix leans on a tool he distrusts elsewhere. At the end of that 2011 Lampson review he had already written the fundamental difficulty with stepwise refinement: "these approaches cannot scale without some compiler support." Fifteen years on, the difficulty largely remains. His 2026 answer is to have an LLM write the TLAPS proofs — in the metastability post he states plainly that he "got Claude Code write TLAPS proofs" to lift the results to all parameter values. That is not necessarily wrong, but it is delicate, given what he argued the same year.
5. The most interesting tension, and he knows about it. In August 2026 he published The Safest Job from AI may be Writing, arguing that writing is the work least threatened: it is a wicked problem with no definitive formulation, no stopping rule, no objectively correct solution, where the only measure of success is "resonance inside another human mind," and where models "have zero skin in the game." He is not gentle about it, reporting a "visceral ick reaction" to AI prose.
But on the same blog, the widely shared satirical sketch from March 2025 — Ponder Stibbons announcing to the Dean that he is leaving the ivory tower — carries a byline he added himself: "(Written by a golem named ChatGPT with some prompting)." So his sharpest critique of AI writing and his most-circulated piece of AI-ghostwritten prose are seventeen months apart, by the same hand.
In fairness, he leaves his own door open. Having finished the argument against AI writing, the very next thing he says to the reader is: "Tell me where my logic does not hold up." Where most people would end on a conclusion, he ends on an invitation — which is precisely the system he installed for himself in 2018, still running eight years later.