Semantic Caching

The Three-Tier Cache Stack: exact → prefix → semantic

Background & Principle

Treating "cache" as one thing is a rookie mistake. An LLM request has three cacheable points with wildly different cost and risk:

• L1 exact cache: hash the request text → answer. Lowest hit rate (one punctuation change and it misses), but zero semantic risk — a hit is guaranteed to be the same sentence.
• L2 prefix cache: Anthropic cache_control caches the stable prefix (system prompt, few-shot, long docs) server-side. It saves input tokens but still runs a full inference every time — the answer is recomputed, so it too is zero semantic risk. Note: Anthropic defaults to a 5-minute TTL (optional 1 hour), up to 4 breakpoints, and any byte change before a breakpoint invalidates the cache from that point onward — so place the breakpoint exactly on the boundary between the stable prefix and the volatile tail.
• L3 semantic cache: embed the request → nearest neighbor (ANN) → if distance is below threshold, return the old answer. Saves an entire inference (largest savings), but may hit the wrong old answer.

The key engineering decision: query the three tiers in order, with the semantic tier last, handling only what exact/prefix let through. An identical query should be caught by the zero-risk L1 and never reach L3 to gamble on a similarity check.

Hands-on

def answer(q, scope):
    qn = normalize(q)                          # normalize: collapse surface variants first
    if (a := L1.get(hash(qn, scope))): return a   # zero-risk hit
    hit = L3.search(embed(qn), scope, k=1)     # semantic neighbor
    if hit and hit.dist < TAU:  return hit.answer   # ⚠️ threshold gate
    a = llm(qn, scope)                         # L2 prefix cache applies inside this call
    L1.put(hash(qn, scope), a); L3.put(embed(qn), a, scope, ttl=...)
    return a

The Threshold Tunes Risk, Not Accuracy

Background & Principle

The two errors have grossly asymmetric costs. A miss costs one extra API call (a fraction of a cent plus a little latency) — harmless. A false hit costs a user a confidently wrong answer — broken trust, possibly legal fallout. Given the asymmetry, you don't pick the threshold at "max F1"; you pick it at "false hits pushed below tolerance" — which usually means a stricter threshold and more misses.

Redis uses cosine distance over [0,2], where 0 = identical and 2 = opposite; a smaller threshold = stricter = fewer false hits. The embedding choice matters even more than the threshold: a generic sentence encoder can't separate "return vs exchange" (near-synonyms, different answers), nor "can I return" vs "can't I return" (one word apart, opposite meaning, near-identical vectors). So don't guess the threshold — sweep it: take a batch of human-labeled (q1, q2, should-hit?) pairs, plot false-hit and false-miss curves, and pick the highest-hit-rate point subject to "false hits under the red line."

Hands-on

# Threshold sweep: pick tau under a false-hit cap, not at max F1
def pick_tau(pairs, embed, max_false_hit=0.01):
    best = None
    for tau in arange(0.05, 0.40, 0.01):
        fh = mean([dist(embed(a),embed(b)) < tau and not same
                   for a,b,same in pairs])     # false-hit rate
        hit = mean([dist(embed(a),embed(b)) < tau and same
                   for a,b,same in pairs])
        if fh <= max_false_hit: best = (tau, hit, fh)  # constraint first
    return best   # highest hit rate under the strict cap

Cache Poisoning: Correctness Isn't Just "Do the Questions Look Alike"

Background & Principle

Even with a perfect similarity judgment, a semantic cache has a subtler error class: the answer itself no longer holds. Three poison sources:

• Personalization: "Where is my order?" has a different answer per user — a cross-user hit is a disaster (and leaks others' privacy).
• Freshness: "latest price" or "today's stock" cached yesterday is poison today.
• Context dependence: the same "what about the second one?" points at entirely different things given different conversation histories.

The fix is a scope key: the cache key can't be just the query embedding — bolt on user_id / tenant / data_version / lang / context summary, and only allow hits within the same scope. Tier TTLs by freshness: static knowledge might get a day, prices/stock maybe 60 seconds or no cache at all. MeanCache's approach encodes a context chain per cached query to distinguish context-dependent queries from self-contained ones — only the latter are safe to reuse across sessions.

Hands-on

# scope key: similarity is only the necessary condition; scope match is the sufficient one
def scope_key(req):
    return {
      "tenant": req.tenant_id,          # isolate multi-tenant
      "user":   req.user_id if req.personalized else "*",
      "lang":   req.lang,
      "dv":     req.data_version,        # data changes → old cache auto-mismatches
    }
# hang invalidation hooks on the write path: source changes → clear cache by scope
def on_price_update(sku):
    cache.invalidate(scope={"dv": bump_version()})   # bump version → all old cache mismatches

Hit-Rate Engineering + Two-Metric Monitoring

Background & Principle

First, the safe ways to raise hit rate (no threshold change, no added risk):

• Canonicalization: lowercase, strip punctuation and extra whitespace, drop politeness noise ("pls / could you / would you mind"), so the L1 exact tier absorbs surface variants — less load on the semantic tier, a narrower false-hit surface.
• Negative cache: cache "no results / I don't know" too, so the same empty query doesn't keep hitting the LLM.
• Warmup: preload the store offline with high-frequency historical queries.

But hit rate is a dangerous single metric: loosen the threshold → hit rate rises → bill looks great → but false hits quietly rise too. So you need a shadow eval: sample some already-cache-hit requests, actually call the LLM, and compare whether old and new answers are equivalent (via LLM-as-judge or embedding similarity), estimating the live false-hit rate. This is the same idea as Day 42's shadow testing and Day 56's "stabilize eval with a distribution, not a single point" — use a cheap side channel to surface silent errors.

Hands-on

# shadow sampling: quietly recompute 1% of hits to quantify false-hit rate
def answer_monitored(q, scope):
    hit = semantic_lookup(q, scope)
    if hit and sample(0.01):              # 1% of hits go to shadow reconciliation
        truth = llm(q, scope)
        if not equivalent(hit.answer, truth):  # judge / similarity
            metrics.incr("cache.false_hit")   # alert on THIS, not hit rate
    return hit.answer if hit else llm(q, scope)