The Incident-Fueled Detector FlywheelCompounding Reliability in Legacy Systems

Scenario + Requirements

You inherit a 15-year-old clearing and settlement system: roughly 2M lines of code, most original authors gone, documentation out of sync with reality. About 40 incidents and near-misses per month, 300 changes per week (a growing share submitted by coding agents). Plenty of postmortems get written, yet the rate of repeat incidents of the same class stays stubbornly high — the conclusions live in documents, not in anything a machine executes.

• Goal: every incident or near-miss is distilled into one executable detector in a judge library; every change must pass the full library before merge. Cut the repeat rate by an order of magnitude within 12 months.
• Scale: the library grows 30–50 detectors a month, nearing two thousand in three years; the full gate must stay under 20 minutes at p95 or developers will route around it.
• Constraints: change authors (human or agent) will optimize against the gate; the judges are code too, and they get edited.

High-Level Architecture

graph LR
    I["Incident / near-miss
postmortem"] --> G["AI generator
candidate detector + repro"]
    G --> O{"Independent oracle
red-green proof"}
    O -- "red on old build, green on fix" --> L[("Judge library
append-only
content-addressed + signed")]
    O -- "cannot reproduce" --> X["Back to humans"]
    L --> P["Public partition
failure detail visible"]
    L --> H["Holdout partition
returns one bit"]
    C["Change
human / agent"] --> GATE{"CI gate"}
    P --> GATE
    H --> GATE
    GATE -- "pass" --> M["Merge"]
    M -. "new incident" .-> I
    H -. "quarterly rotation
spent holdout goes public" .-> P

The generator (an LLM) reads postmortems, logs and diffs and produces candidate detectors plus minimal repro scripts — high volume, untrusted. The independent oracle handles admission: a candidate must fail on the incident build and pass on the fixed build, executed mechanically by CI, never judged by an LLM. The judge library is append-only, split into a public partition (failure details returned to authors for debugging) and a holdout partition (tells the gate only pass/fail). Rotation moves holdout detectors that have "leaked" through heavy querying into public, and backfills holdout with fresh incidents.

Key Point 1: Isolate the Judge from the Signals the Evaluated Party Sees

One-line trade-off: the more failure detail authors see, the faster they debug — and the easier the gate is to pass "on purpose".

Principle

Goodhart's law in CI: whatever signal authors can see is the signal that gets optimized. An author who sees the full assertion and can resubmit freely (especially a tireless agent) will find a change that makes this detector pass without fixing the root cause — an if-branch for the specific input, a swallowed exception, a reworded log line so the regex no longer matches. The fix is an old machine-learning rule: separate the training signal from the acceptance signal. The public partition serves debugging; the holdout serves acceptance. Holdout contents, names and even failing inputs stay hidden from authors, who get a single aggregate bit.

Trade-offs

Pseudo-code

# Gate: public partition returns detail, holdout returns one bit (pseudo-code)
def gate(change):
    pub = [d.run(change) for d in library.partition("public")]
    hold = [d.run(change) for d in library.partition("holdout")]
    report = {"public_failures": [r.detail for r in pub if not r.ok]}
    hold_ok = all(r.ok for r in hold)
    holdout_budget.charge(change.author)       # every query leaks information
    if not hold_ok:
        report["holdout"] = "FAIL"             # no names, no inputs
        page_owner_team(change, [r for r in hold if not r.ok])  # third party sees detail
    return report, (not report["public_failures"]) and hold_ok

Key Point 2: The Judge Library Is Append-Only

One-line trade-off: accept monotonic library growth and flaky-test upkeep in exchange for "nobody passes the exam by editing the examiner".

Principle

The cheapest shortcut for any evaluated party is editing the evaluator rather than itself. In an engineering org it looks mundane: "this test is flaky, comment it out", "the assertion is too strict, loosen it to ±5%". Each instance may be reasonable, but cumulatively the judge library silently degrades — while the gate stays green. Append-only closes that path: once admitted, a detector is content-addressed and cannot be edited in place; "modification" means appending a new version, and the old version keeps running until the new one proves it catches the bug on the same historical incident builds.

Trade-offs

Mutable test suite (the norm): flexible to maintain, but degradation is invisible. Strict append-only: degradation is impossible, but flaky detectors turn the gate randomly red and burn developer trust. The middle ground is a quarantine: a flaky detector moved to quarantine still runs and reports, it just stops blocking merges; quarantine has an SLA (say 14 days) after which it escalates to the owning team, and deletion requires two-person approval plus evidence (e.g. the code path is decommissioned). The core invariant: no single PR may modify both the code under test and the detector that judges it.

Pseudo-code

// Judge library: content-addressed, append-only; "edit" = new version alongside old (pseudo-code)
func (lib *Library) Supersede(oldID string, nd Detector, by Author) error {
    if lib.TouchesCodeUnderTest(by.PendingChange) {
        return ErrSameChangeEditsJudge // code and its judge can't change together
    }
    // the new version must go red on every incident build the old one caught
    for _, bad := range lib.Get(oldID).CaughtBuilds {
        if nd.Run(bad).OK { return ErrWeakerThanPredecessor }
    }
    lib.Append(nd.WithParent(oldID)) // old version kept until new one is stable N weeks
    return nil
}

Key Point 3: Rotate the Holdout Periodically

One-line trade-off: the longer a holdout is used, the less "passing it" is worth; rotate too often and you lose a baseline comparable across time.

Principle

A holdout does not stay clean forever. Every pass/fail bit the gate returns leaks a little information to the author; an agent resubmitting the same change dozens of times is performing adaptive queries against the holdout. Dwork et al.'s 2015 Science paper on the "reusable holdout" addresses exactly this: once the same holdout data is queried adaptively over and over, its validation conclusions become invalid unless queries are noised or budgeted. The takeaway: a holdout has a service life. So budget queries per author, and each quarter move "spent" holdout detectors to public and backfill with fresh incidents.

stateDiagram-v2
    state "Candidate" as cand
    state "Rejected" as rej
    state "Holdout" as hold
    state "Public" as pub
    state "Quarantine" as quar
    state "Retired" as ret
    [*] --> cand: AI generates
    cand --> rej: red-green proof fails
    cand --> hold: admitted (fresh incidents go to holdout)
    hold --> pub: query budget spent / quarterly rotation
    pub --> quar: repeatedly flaky
    hold --> quar: repeatedly flaky
    quar --> pub: fixed within SLA
    quar --> ret: two-person approval + code path removed

Trade-offs

Fixed holdout: comparable over time, but within months it has effectively become public. Full rotation every time: cleanest, but the new holdout comes only from recent incidents, coverage is narrow, and pass-rate curves show no trend. Rolling rotation + a small fixed anchor set (recommended): keep an anchor set that is rarely queried and runs only at release, to observe long-term drift; roll the rest by budget.

Pseudo-code

def rotate(lib, quarter):
    for d in lib.partition("holdout"):
        if d.queries_since_admit > QUERY_BUDGET or d.age > ONE_YEAR:
            lib.move(d, "public")             # spent holdout → public, still gating
    fresh = [d for d in lib.admitted_in(quarter) if d.partition == "public"]
    for d in sample(fresh, k=HOLDOUT_TARGET - lib.size("holdout")):
        lib.move(d, "holdout")                # backfill with incidents authors haven't seen
    # the anchor set never rotates and runs only at release: the long-term drift reference

Key Point 4: AI Sits in the Generator Seat; an Independent Oracle Judges

One-line trade-off: letting an LLM turn postmortems into candidate detectors in bulk raises throughput by an order of magnitude; but admission must be decided by a mechanically executable oracle, or errors compound in a correlated way.

Principle

Reading logs, finding a minimal repro, writing assertions — exactly the generation work LLMs are good at. But letting the same model decide "is this detector valid?" falls into the Day 51 trap: generator and judge share blind spots, and two layers of checking collapse into one. Admission therefore uses a red-green proof: the detector must fail on the build from the time of the incident (proving it really catches this bug), pass on the fixed build (proving it doesn't always alarm), and do both stably across N repeated runs (ruling out flakiness). All three are executed mechanically by CI; none requires a model's "opinion".

Trade-offs

The generator's target: each incident yields one precise regression + one candidate invariant, the latter reviewed by a human.

Pseudo-code

def admit(candidate, incident, runs=5):
    bad, good = incident.buggy_build, incident.fixed_build
    red   = [candidate.run(bad).ok  for _ in range(runs)]
    green = [candidate.run(good).ok for _ in range(runs)]
    if any(red):       return Reject("misses the original bug")  # must be stably red
    if not all(green): return Reject("false alarm or flaky")     # must be stably green
    if candidate.kind == "invariant":
        require_human_review(candidate)                # invariants = human seat
    return library.append(candidate, partition="holdout")

Scaling and Optimization

Impact-based selective execution: running two thousand detectors blows the 20-minute budget, so run only the affected subset by code dependency graph and historical failure correlation — but never prune the holdout or anchor set (otherwise authors infer holdout contents from "which ones ran"). Near-misses are worth more than incidents: higher base rate, no casualties, fresh details — they should be the generator's main fuel. Measure the flywheel itself: track the repeat-incident rate and "real bugs blocked pre-merge", not library size — library size is an activity metric, the first thing Goodhart comes for.

Common Pitfalls + Interview Questions

• Letting an LLM both write detectors and judge them valid — correlated failure, equivalent to no check. Asked "how do you ensure AI-generated tests are useful?", the expected answer is red-green proof + repeated-run stability, not "have another model review it".
• Treating the holdout as permanently clean. Agents retry far more than humans, so the holdout leaks faster; no query budget means no holdout.
• Staying green by "deleting flaky tests". The human version of editing the evaluator; flaky detectors must be quarantined, not deleted, with an SLA.
• One PR changes both code and tests. The stealthiest gate bypass; forbid it structurally in CI or force third-party approval.
• Follow-up list: ① At 10k detectors with a 5-minute gate, what do you trade off? ② Holdout fails but authors can't see detail — how do you keep productivity from collapsing? ③ What does a detector look like when the incident can't be reproduced in test (timing, data dependencies)? ④ How do you prove the flywheel works, rather than incident types simply shifting? ⑤ With agents filing 1,000 changes a day, how do you allocate query budgets?

Further Resources

• AIDE²: First Evidence of Recursive Self-Improvement · Weco — public/private score isolation, the 7/99 acceptance rate, external benchmark validation.
• The Darwin Gödel Machine · Sakana AI — the "objective hacking" case of a self-modifying agent removing hallucination-detection markers, and lineage logs.
• How SQLite Is Tested — the long-running discipline that a bug isn't fixed until it's in the regression suite.
• Site Reliability Engineering · Postmortem Culture: Learning from Failure (Google, 2016) — blameless postmortems and action-item culture.
• Dwork et al., The reusable holdout: Preserving validity in adaptive data analysis (Science, 2015) — why a holdout has a service life.

Deeper Thinking