Feature Platform & ML Infra — Engineering One Feature for Both Training and InferenceDual Store, Point-in-Time Correctness, Model Serving, Experimentation

Scenario + Requirements

Design a delivery / fraud-scale feature platform: 100+ models (delivery ETA, store ranking, fraud detection) sharing one feature set; online inference peaking at 10M feature reads/sec, 200 features per prediction, p99 < 10ms; while offline you must reconstruct the "as-of-that-moment" feature values for billions of training samples. This is the real scale at DoorDash and Uber.

The core tension: training uses offline batch (throughput-first), inference uses online low-latency (latency-first), yet both must be the same feature value—otherwise the model scores AUC 0.85 offline and drops to 0.79 in production. That is the infamous training-serving skew. The platform must solve three things: where to store, how to guarantee consistency, how to serve and validate.

• Feature freshness: balance-like features need second-level freshness, a user's 30-day spend can be hourly—different freshness, different cost.
• Read/write ratio: DoorDash publicly noted online writes are only 0.1% of reads—extremely read-heavy, which is what dictates Redis/KV over a general-purpose DB.
• Point-in-time correctness: a training label is produced at time T; features may only use values from before T, or you get label leakage.
• Reusable/discoverable: Uber Palette hosts 20,000+ features reusable company-wide, avoiding every team reinventing the wheel.

High-Level Architecture

graph LR
    SRC["Sources
event streams / warehouse / DB CDC"]
    subgraph Compute
      BATCH["Batch
Spark / SQL"]
      STREAM["Stream
Flink"]
    end
    REG["Feature Registry
definition + version + lineage"]
    OFF[("Offline Store
Parquet / warehouse")]
    ON[("Online Store
Redis / DynamoDB")]
    TRAIN["Training
PIT join → samples"]
    SERVE["Model Serving
fetch features + infer"]

    SRC --> BATCH & STREAM
    REG -.constrains.-> BATCH & STREAM
    BATCH --> OFF
    BATCH -->|materialize| ON
    STREAM --> ON
    OFF --> TRAIN
    ON --> SERVE

    classDef src fill:#1a2530,stroke:#64c8ff,color:#e8eef5
    classDef comp fill:#0e2030,stroke:#5eead4,color:#e8eef5
    classDef store fill:#2a1530,stroke:#ff7ab6,color:#e8eef5
    classDef sink fill:#1a1a30,stroke:#ffb450,color:#e8eef5
    class SRC src
    class BATCH,STREAM,REG comp
    class OFF,ON store
    class TRAIN,SERVE sink

One feature definition (registry) drives both batch and stream pipelines, landing in offline/online stores that feed training and inference

Component roles: the registry is the single source of truth, declaring each feature's entity key, type, transformation logic, and freshness SLA; batch handles heavy backfills and T+1 features, stream handles second-fresh features; the offline store keeps full history partitioned by time (for point-in-time reads), the online store keeps only each entity's latest value (for millisecond reads). This is the industry-standard dual-store architecture.

Key Techniques

1. Dual Store: offline warehouse vs online KV, one definition materialized twice

Principle: training needs "billions of samples × per-sample historical feature lookup"—a throughput-bound full scan, naturally a columnar warehouse (Parquet/Iceberg partitioned by event_ts). Inference needs "given a user, fetch 200 features within 10ms"—a point-lookup random read, naturally an in-memory KV. The same feature is materialized once in each engine, with the registry keeping semantics aligned.

# Online fetch: one pipeline pulling 200 keys, avoiding 200 RTTs
def fetch_online(entity_ids, feature_names):
    keys = [f"{fn}:{eid}" for eid in entity_ids for fn in feature_names]
    vals = redis.mget(keys)          # single round trip, 200 keys ~1ms
    return assemble(entity_ids, feature_names, vals)

# Materialization: batch pipeline flushes latest values into online store,
# storing only "latest per entity" (no history) → online store size stays bounded

2. Point-in-Time Correctness: the core mechanism against label leakage

Principle: a training sample is "(entity, label time T, label)"; when joining features to it, you may only use feature values with event_ts ≤ T. Naively joining the feature table's current value feeds "the future" into the model—inflated offline metrics, production collapse. The correct approach is a point-in-time join (as-of join): for each sample, find the "most recent feature before T" by entity key.

# Point-in-time join pseudocode (as-of join, prevents label leakage)
# labels: (entity_id, label_ts, y)   features: (entity_id, event_ts, value)
for row in labels:
    cand = features[entity_id == row.entity_id
                    and event_ts <= row.label_ts]   # strict ≤ label time
    row.feature = cand.sort_by(event_ts).last()      # latest value as-of then
# Engine: Spark partitions by entity_id + sorts by event_ts for a merge as-of join

3. Model Serving: splitting the latency budget between fetch and inference

Principle: an online prediction's p99 budget (say 30ms) must be split across "fetch features + preprocess + model forward + postprocess". Empirically, feature fetch often takes over half the latency—200 features scattered across multiple entities and stores. Optimizations: batched pipeline fetch, concurrent fetching within a request, and local L1 caching of hot entities.

4. A/B Experimentation: turning "shipping a model" into a measurable causal experiment

Principle: a new model can't ship just because offline AUC went up—offline metrics and business metrics (order rate, GMV) often diverge. Use randomized online experiments to establish causality: consistent hashing stably assigns users to control/treatment, and after enough sample size you compare statistical significance of business metrics, while watching guardrail metrics (latency, error rate, refund rate) to prevent "main metric up, side effect explodes".

# Stable bucketing: the same user always lands in the same group (reproducible, no flicker)
def bucket(user_id, exp_key, treat_ratio=0.5):
    h = hash_u64(f"{exp_key}:{user_id}")   # salt with experiment name → experiments stay orthogonal
    return "treatment" if (h % 10000) / 10000 < treat_ratio else "control"

Scaling & Optimization

• Tiered freshness: second-level via Flink stream, minute-level via micro-batch, day-level via Spark T+1—allocate compute by business value, don't make everything real-time (cost explodes).
• Feature monitoring & drift detection: continuously track each feature's distribution; alert when online distribution drifts from training distribution—skew often happens "silently".
• Cost governance: online store memory is the priciest—use protobuf/quantization compression, TTL-evict cold entities, and materialize only features actually referenced by models (prune via lineage).
• Embedding features: vector features (user/item embeddings) go to a dedicated vector store, managed separately from scalar features.

Pitfalls + Interview Questions

• Why do you need a dual store? Can one engine serve both training and inference? Where are the bottlenecks?
• Explain the three causes of training-serving skew and give one fix for each.
• A prediction fetches 200 features at p99 < 10ms—how do you split the latency budget and optimize the fetch?
• Why does plain user-randomized A/B fail in a two-sided market (delivery)? What experiment design replaces it?
• A hot entity (top store) has excessive single-point feature QPS—how do you hold it? (recall Day 2 caching hot keys)

Resources

• DoorDash: Building a Gigascale ML Feature Store with Redis—online store choice and compression in practice.
• Uber: Michelangelo Palette Meta Store Journey—feature registration, reuse, and metadata.
• Airbnb: Chronon: Airbnb's ML Feature Platform (Open Source)—one definition generating batch/stream pipelines + PIT backfill.
• Google: Rules of Machine Learning (Zinkevich)—the authoritative treatment of training-serving skew.
• Designing Data-Intensive Applications (Kleppmann): the batch/stream and derived-data chapters underpin how to reason about dual-store materialization.

Deeper Thinking (click to expand)