Code Review as a Signal-Detection System — The FP Budget Decides Whether the Tool LivesFP Budget, Triage Layer, Signal-to-Noise Governance, Activity vs Outcome Metrics

Scenario + Constraints

You wire automated code review into a 5,000-engineer monorepo: a pile of linters, static analyzers, plus an LLM reviewer, all posting comments as bots on every diff. About 20,000 diffs/day, ~8 comments each — 160,000 comments daily. Week one everyone reads them. Week three everyone has learned one motion: scroll past every bot comment. The tool is still running, the metrics (comment count, coverage) still look great — but it's dead, because it has no signal, only noise.

This isn't fixable with "more rules." It's a signal-detection problem: binary classification of real bugs (signal) vs irrelevant noise, where the receiver's attention (engineers) is a scarce, system-wide shared resource. Once the false-positive rate is high enough to erode trust, all of the tool's recall is voided — a true positive nobody reads equals a miss.

• The core metric isn't throughput, it's precision: the fraction of shown comments actually worth acting on. QPS and latency are secondary.
• False-positive budget: Google's rule of thumb — code-review checks should have an effective false-positive rate below 10%, or developers start ignoring them wholesale.
• Actionability: every comment must be understandable, actionable, easy to fix; a vague "something might be wrong here" is itself a false positive.
• Timing: reports must appear while the author still has context in their head (at diff review), not in an offline scan three days later.
• Feedback loop: you must continuously measure each rule/analyzer's FP rate and govern accordingly.

High-Level Architecture (signal → triage → feedback)

graph TD
    D["Diff / PR
this change"]
    subgraph GEN["① Generation · multi-source signal"]
      L["Linter / Formatter
deterministic · cheap"]
      SA["Static analyzer
Infer-class · inter-procedural"]
      LLM["LLM Reviewer
semantic · costly · hallucinates"]
    end
    T["② Triage layer
severity × confidence gating
nitpick filter · dedup"]
    R["③ Shown in review
blocking / collapsed / dropped"]
    H["Engineer
NOT USEFUL button"]
    FB["④ Feedback loop
per-analyzer effective FP rate
over budget → auto-downgrade/disable"]

    D --> L & SA & LLM
    L & SA & LLM --> T --> R --> H
    H -.score.-> FB
    FB -.thresholds.-> T
    FB -.disable.-> GEN

    classDef src fill:#1a2530,stroke:#64c8ff,color:#e8eef5
    classDef mid fill:#1a1a30,stroke:#ffb450,color:#e8eef5
    classDef out fill:#0e2030,stroke:#5eead4,color:#e8eef5
    class L,SA,LLM src
    class T,FB mid
    class R,H out

Let the generation layer be prolific; all the value is in the triage layer and feedback loop — they trade recall for precision and make "which rules deserve to exist" a data-driven decision

Key Technical Points

1. Define metrics with signal-detection theory: effective, not theoretical, FPs

Principle: treat each comment as one binary decision falling into a confusion matrix — TP, FP, FN, TN. Tool authors love the technical FP definition ("the rule logically holds"); but to the user, a false positive is any report they didn't want to see — even if logically correct, if it's irrelevant and not worth fixing, it's noise. Google therefore defines an "effective false positive": the developer took no positive action after seeing it. This shift in vantage point is the foundation of the whole topic.

# Effective FP rate: derived from USER signal, not rule logic
def effective_fp_rate(analyzer):
    shown = analyzer.comments_shown            # comments displayed
    # "positive action" = changed code / clicked useful / marked will-fix
    acted = count(c for c in shown if c.got_positive_action)
    return 1 - acted / max(shown, 1)           # share nobody acted on = effective FP

# FP budget (Google's rule-of-thumb threshold)
FP_BUDGET = 0.10
def healthy(analyzer):
    return effective_fp_rate(analyzer) < FP_BUDGET

2. Triage layer: severity × confidence gating + nitpick filtering

Principle: the generation layer can be prolific, but not every true positive earns the right to interrupt a human. The triage layer scores each comment on two axes — severity (null deref vs variable naming) and confidence (how sure the analyzer is) — then decides its fate: high severity × high confidence → blocking; medium → collapsed suggestion; low → dropped. Style nitpicks (formatting, import order) shouldn't reach human eyes at all — hand them to a formatter's autofix, and reserve human attention for correctness and design.

# Triage: decouple "is it real" from "is it worth interrupting a human"
def triage(c):
    if c.category == "style" and c.autofixable:
        apply_autofix(c);  return DROP        # nitpick → machine fixes, don't ask
    score = c.severity * c.confidence
    if score >= BLOCK_TH:   return BLOCKING   # must resolve before merge
    if score >= SHOW_TH:    return COLLAPSED   # expandable suggestion
    return DROP                                # rather miss than make noise

# Same issue hit by multiple sources → dedup, keep the most actionable one
comments = dedup(comments, key=lambda c:(c.file, c.line, c.bug_class))

3. Diff-time vs batch: same report, timing decides signal-to-noise

Principle: half a report's value is its content, half is when it appears. The same bug report shown at diff review (author just wrote it, full context in head) gets fixed at a high rate; dumped as an offline batch scan — a list of thousands handed to the team — the fix rate approaches zero. Nobody wants to touch code someone wrote three months ago that's still running. Corollary: report only what's newly introduced by this change, never the standing backlog, or the first onboarding buries everyone under thousands of legacy findings.

4. Feedback loop: turn the FP budget into auto-enforced governance

Principle: triage thresholds can't be set by gut. Maintain an effective-FP-rate curve for each analyzer/rule (derived from NOT USEFUL clicks and whether code was actually changed), and manage it like an SLO error budget: a rule that stays over budget is auto-downgraded (blocking → collapsed) or disabled. This turns "which rules deserve to exist" from a political debate into a data verdict, and forces rule authors to own their signal quality.

# FP-budget gate: govern rule lifecycle like an SLO
def govern(rule):
    if rule.shown < MIN_SAMPLES:  return "canary"   # too few samples, don't sentence yet
    fp = effective_fp_rate(rule)
    budget = BUDGET_BY_SEVERITY[rule.severity]       # security is more tolerant
    if fp > budget * 1.5:  return "disable"          # badly over budget → retire
    if fp > budget:        return "downgrade"        # over budget → non-blocking
    return "active"

Scaling & Optimization

• Cost layering: an LLM reviewer is expensive and hallucinates. Run cheap deterministic linters/static analysis first and feed the LLM only the semantic issues they didn't cover; don't make a big model do a formatter's job.
• Govern LLM hallucination: LLMs are "confidently wrong" — poorly calibrated. Make them cite a specific line/rule, drop unsupported comments at triage, and sample-vote for consistency on the same issue.
• Free the humans: once machines absorb nitpicks and known patterns, redirect scarce human attention to what machines can't do: architecture, naming intent, boundary conditions, cross-system impact.
• Base rates: when real bugs are extremely rare, even high per-comment accuracy can let report-level FPs drown true positives — the base-rate fallacy applied to review (see Day 51, low-base-rate alerting).

Common Pitfalls + Interview Follow-ups

Deep Resources

• Sadowski et al., "Tricorder: Building a Program Analysis Ecosystem" (ICSE 2015): the NOT USEFUL button, user-vantage FPs, analyzer admission.
• Sadowski et al., "Lessons from Building Static Analysis Tools at Google" (CACM 2018): source of the effective FP < 10% line; the empirical foundation of this topic.
• Distefano et al., "Scaling Static Analyses at Facebook" (CACM 2019) + "Moving Fast with Software Verification": how diff-time deployment reverses fix rate.
• "Software Engineering at Google", Ch. 20 (Static Analysis): the full narrative of Tricorder in production.

Going Deeper (click to expand)