You own an AI coding assistant: 2M DAU, about 40M model calls per day, peaking around 2,000 RPS, with 75% of traffic on a single closed frontier-model vendor. Traditional dependencies (databases, clouds) fail by going down. This one has three extra failure modes: cutoff, repricing, and competing with you. All three have already happened:
Requirements: if any single vendor cuts you off completely within 72 hours, core features stay up and quality regression is measurable and bounded; each task has a hard cost budget and gross margin stays non-negative; p95 time-to-first-token (TTFT) stays under 1.5s through a switchover.
graph LR
C["Client / Agent"] --> GW["LLM Gateway
canonical request schema"]
GW --> R{"Router
capability tier · budget · health"}
R --> A1["Adapter: Vendor A
primary closed model"]
R --> A2["Adapter: Vendor B
warm-standby closed model"]
R --> A3["Adapter: open weights
hosted / self-run vLLM"]
R -.-> L[("Cost ledger
per-task budget")]
R -.-> H[("Health / breaker state")]
S["Shadow-traffic sampler"] --> EV["Eval harness
golden set + regression"]
A1 -.-> S
EV -->|"capability matrix"| R
CT["Contract & capacity registry
notice · quota · price"] --> R
The Gateway translates product requests into one internal schema (messages, tool definitions, cache hints); product code never imports a vendor SDK directly. The Router picks a model by asking: what capability tier does this task need, how much budget is left, and who is healthy right now. Adapters absorb vendor differences (tool-call formats, prompt-caching semantics, streaming events). The eval harness keeps scoring every candidate on shadow traffic and produces a capability matrix for the Router — without it, every switch is a blind switch. The contract registry turns commercial terms into data the router can read.
One-line trade-off: the thicker the abstraction, the faster you can switch — but it flattens each vendor's unique features (caching, long context, specific tool formats). You pay a performance tax for portability.
The goal is not "all models are the same" but making differences explicit. Each model declares a capability contract: context length, parallel tool calls, strict structured output, prompt-cache granularity and TTL, price. Each request declares what it needs. Routing is constraint matching, not swapping a model-name string. The most dangerous anti-pattern is the half abstraction: the interface is unified, but the prompts were tuned against one vendor — on switchover nothing errors, and quality quietly drops. So prompts are versioned per model family and stored alongside the adapters.
| Option | You gain | You give up |
|---|---|---|
| Direct vendor SDK | New features on day one, lowest latency | Switching cost measured in lines of code; a cutoff means editing product code |
| Lowest-common-denominator API | Switching is a config change | Differentiators like prompt caching and extended thinking go unused; you lose on both cost and quality |
| Capability contract + opt-in extensions (recommended) | Portable by default; unique features used explicitly with the dependency registered | You maintain a capability matrix and per-vendor prompt variants; eval cost grows linearly with model count |
# Capability contract: requests declare needs, the router matches constraints (pseudo-code)
@dataclass
class Capability:
max_context: int
parallel_tools: bool
strict_json: bool
cache_ttl_s: int # 0 = no prompt caching
usd_per_mtok_in: float
usd_per_mtok_out: float
def eligible(req, models):
out = []
for m in models:
cap = m.capability
if req.tokens_est > cap.max_context: continue
if req.needs_parallel_tools and not cap.parallel_tools: continue
if req.needs_strict_json and not cap.strict_json: continue
if m.quality[req.task_type] < req.min_quality: continue # from the eval matrix
out.append(m)
return out
def render_prompt(req, model):
return PROMPTS[req.task_type][model.family].format(**req.vars) # prompts versioned per model familymodels list, allow_fallbacks, sorting by price / throughput / latency) — "abstraction layer as a service", at the price of one more third-party dependency.One-line trade-off: running real traffic on several vendors (active-active) is expensive and inconsistent; switching only when something breaks (active-passive) is cheap, but the standby path silently rots.
Day 23 covered circuit breakers, retries, and bulkheads; the new problem here is keeping the standby path fresh. A fallback that never carries traffic has stale prompts, quotas, accounts, and rate limits — on cutoff day you discover the standby account's rate limit is 1/20 of the primary's. So the standby vendor should permanently carry 5–15% of real traffic, with capacity confirmed in writing. The second problem is correlated failure: a cutoff rarely looks like a 5xx. It looks like 403s, a sudden quota drop, or a model version being retired, and health checks must treat those as failures too. Third, degrade by capability tier: when the primary is gone, completion tasks can drop to a small open-weight model, but complex agent tasks should queue rather than fall to a model that will mangle code.
| Option | You gain | You give up |
|---|---|---|
| Single vendor + cold standby | Simplest; one set of prompt tuning | Standby path unproven; no capacity guarantee; quality unknown on switch day |
| Primary + warm standby (10% resident) | Standby continuously validated by real traffic; capacity and quota proven | Two prompt sets and two eval tracks; slight UX inconsistency |
| Vendor per task type | Best price-performance per task; naturally diversified | Any one cutoff takes out a whole feature class; most complex routing policy |
# Routing with breakers and tier-aware degradation (pseudo-code)
FATAL = {401, 403, 404} # auth failure / model retired: not jitter, a cutoff signal
def route(req):
for m in rank(eligible(req, MODELS), by=req.objective): # cost or latency
if breaker[m].open(): continue
try:
resp = m.adapter.call(req, timeout=req.ttft_budget)
breaker[m].success()
ledger.charge(req.task_id, m.cost(resp.usage))
return resp
except ProviderError as e:
if e.status in FATAL:
breaker[m].trip(hold_s=3600); alert("supplier-level", m, e)
else:
breaker[m].failure()
if req.tier == "agent":
return enqueue(req) # queue rather than drop to an under-capable model
return degrade(req) # completion tasks: fall to a small open-weight modelOne-line trade-off: open weights give you sovereignty and a cost floor, but you own post-training, inference ops, and the capability gap; budgeting by per-token list price systematically underestimates what reasoning models really cost.
Gundlach et al. at MIT FutureTech (2025) estimate that the price of reaching a fixed benchmark level falls about 5–10× per year, while the price of running frontier models themselves rises about 3–18× per year — bigger models and more reasoning tokens. The price sheet gets cheaper while your actual bill may grow: on the same task a reasoning model can emit an order of magnitude more tokens. So the budget unit must move from "dollars per million tokens" to "dollars per completed task", charged in real time per task, downgrading or stopping when over budget. Open weights are the floor of that ledger: their price cannot be raised unilaterally, and they cannot be taken back when you get acquired.
| Option | You gain | You give up |
|---|---|---|
| Closed frontier only | Strongest capability, zero ops | Price and supply decided by the other side; margins capped by upstream |
| Hosted open weights (third-party inference) | No GPU ops; switching host means switching an endpoint | Still a host dependency, but you can move between hosts |
| Self-run open weights + own post-training | Cost and roadmap under your control; RL on your own tasks | GPU capacity planning, inference optimization, a training team; the capability gap is closed by post-training |
# Per-task budget ledger: bill per completed task, not per token (pseudo-code)
class TaskBudget:
def __init__(self, task_id, usd_cap, tier_ladder):
self.spent, self.cap, self.ladder = 0.0, usd_cap, tier_ladder
def before_step(self, est_tokens, model):
projected = self.spent + model.cost_estimate(est_tokens)
if projected <= self.cap:
return model
cheaper = self.ladder.next_below(model) # step down: frontier → workhorse → open weights
if cheaper and self.spent + cheaper.cost_estimate(est_tokens) <= self.cap:
return cheaper
raise BudgetExceeded(self.task_id, self.spent) # product layer: tell the user or hand off
def after_step(self, usage, model):
self.spent += model.cost(usage) # reasoning tokens count too: the part the price sheet hides
metrics.histogram("usd_per_task", self.spent, tags={"model": model.name})One-line trade-off: long contracts and reserved capacity buy price and supply certainty but deepen lock-in; short contracts stay flexible but give you nothing in writing when a cutoff comes.
Technical redundancy only shortens recovery after a cutoff; contract terms decide how much warning you get. Terms to negotiate: termination notice period (Windsurf had under five days), change-of-control clause (can the vendor terminate unilaterally when you are acquired? — in the Cursor case OpenAI's stated reason was distrust that the new parent would honor its terms), model deprecation notice, reserved capacity and rate quotas, price-lock period, and data-use restrictions. The other half is an eval gate: before moving to a new model, compare task pass rates on a golden set plus shadow traffic. The regression must be measurable and bounded; otherwise the switch just ships a quality incident to users.
sequenceDiagram
participant V as Vendor A
participant Ops as Platform on-call
participant R as Router
participant E as Eval harness
V->>Ops: Termination notice / 403 spike
Ops->>R: Freeze A's share of new traffic
Ops->>E: Run golden set on candidate B and open weights
E-->>Ops: Capability matrix (pass-rate deltas per task)
Ops->>R: Ramp by task type 10%→50%→100%
R->>E: Continuous shadow comparison + user metrics
E-->>Ops: If regression exceeds threshold, queue / degrade that task type
# Switch gate: each task type's quality drop must stay within bounds (pseudo-code)
def promote(candidate, incumbent, task_types, max_drop=0.03):
plan = {}
for t in task_types:
base = eval_pass_rate(incumbent, GOLDEN[t]) if incumbent.alive else BASELINE[t]
cand = eval_pass_rate(candidate, GOLDEN[t])
shadow = shadow_agreement(candidate, t, sample=0.02) # online shadow comparison
if base - cand <= max_drop and shadow >= 0.9:
plan[t] = "ramp" # 10% → 50% → 100%
else:
plan[t] = "hold" # stay on standby / queue / tell users capability is limited
return planSay monthly model spend is $1M and the standby is 20% pricier per unit: 10% of traffic costs about $20K/month extra, plus a second prompt-and-eval track (about half an engineer). Compare: if a cutoff stops core features for 3 days at $3M monthly revenue, the direct loss is about $300K before churn. If the annual probability of a cutoff exceeds roughly 10%, the standby pays for itself — and the past 18 months already contain several public cutoffs and rule changes. Frame it as an insurance premium, not redundant waste.
Yes. Switching models mid-session breaks style, tool-use habits, even the model's grasp of the codebase. Design principles: session stickiness (no model change within a session unless a breaker trips), route by task type rather than randomly (so differences fall on explainable boundaries), and tell users explicitly about significant degradation. This mirrors the session guarantees from Day 6: read-your-writes guarantees users see their own writes; here you guarantee that within a session the user faces the same "reasoner".
Reason from the vendor's motives: being acquired, or about to be, by a competitor (Windsurf, Cursor); a product that competes head-on with the vendor's own apps; usage patterns that blow up unit cost (subscription limits drained by high-frequency agents). The implication is to include strategic risk in dependency assessment: thicken the standby during fundraising and M&A talks, reduce single-vendor reliance the more your product overlaps with the vendor's own, and never build the core path on usage rights that are not in a contract.
They turn "taken away" into "no longer updated". The weights are yours and old versions run forever, so there is no cutoff; but upstream may stop releasing, licenses may tighten for new versions, and geopolitics may make a given country's models unacceptable to some customers (enterprise procurement review). So the open-weight floor needs multiple sources too: at least two bases from different origins in the eval matrix, and a post-training pipeline that can move between them. The risk does not vanish; it becomes a slow variable, and you get time to respond.
Day 55 stakes it on a primitive like S3 conditional writes — documented, under an SLA, stable for years, with several object stores already compatible with the same semantics, so the replacement path is clear. An LLM dependency stakes it on capability, not an interface: the interface can be unified, but "this model gets this kind of task right" has no spec to be compatible with; only evals can prove it. So redundancy for the former lives at the protocol layer, and for the latter it must live at the eval layer — the eval matrix is the LLM era's compatibility spec.