Building a Real-Time ML Feature Store: Stream Processing, Feature Pipelines, and Online Inference

Most ML systems fail not because the model is wrong, but because the data feeding the model is stale, inconsistent, or computed differently between training and inference. The feature store is the infrastructure layer that solves this. It is also the layer most teams build incorrectly the first time.

This article focuses on the infrastructure, not the models. Specifically: how features are computed, stored, served, and kept consistent between training-time and inference-time in a production system with real-time requirements.

The Dual-Compute Problem

Features need to exist in two places for different purposes:

Training: A model needs point-in-time correct feature values for thousands of historical examples. This is a batch job. You want feature values as they existed at the moment each label was generated, not current values. Querying live data for historical training sets produces label leakage.

Inference: A model making a prediction right now needs feature values with low latency. A fraud detection model needs user behavior features in under 10 milliseconds. Recomputing them on-the-fly during serving is too slow.

The naive solution is to compute features twice: once in a batch pipeline for training, once in application code for serving. This creates training-serving skew, the leading cause of silent model degradation. The feature values used during training are computed with different logic, different data ranges, or different aggregation windows than the values seen at inference time.

A feature store eliminates this by centralizing feature computation and making the same computed values available to both training jobs and online serving.

Feature Store Architecture

A production feature store has three components:

Offline store: A columnar data warehouse (Redshift, BigQuery, Snowflake) that holds historical feature values with timestamps. Used to generate training datasets. Reads are slow (seconds to minutes), but the dataset is complete and queryable.

Online store: A low-latency key-value store (Redis, DynamoDB, Cassandra) that holds the latest feature values per entity. Used at inference time. Reads must be under 10ms at p99.

Feature registry: A metadata catalog that defines features, their computation logic, and which store they live in. This is what allows training and serving code to reference the same logical feature definition.

The data flow runs in one direction: feature pipelines write to both stores. The offline store gets append-only writes with timestamps. The online store gets upserts keyed by entity ID.

// Feature registry entry -- TypeScript representation of metadata
interface FeatureView {
  name: string;
  entities: string[];           // e.g., ["user_id", "merchant_id"]
  features: FeatureDefinition[];
  source: FeatureSource;
  ttl?: number;                 // seconds before online store entry expires
  tags: Record<string, string>;
}

interface FeatureDefinition {
  name: string;
  dtype: "float" | "int64" | "string" | "bool";
  description: string;
  transformationRef?: string;   // reference to transformation logic
}

interface FeatureSource {
  type: "stream" | "batch";
  streamTopic?: string;         // Kafka topic for stream sources
  batchSchedule?: string;       // cron expression for batch sources
  offlineTable: string;         // destination table in offline store
  onlineStore: OnlineStoreConfig;
}

Stream Processing Pipelines

Real-time features require stream processing. The typical stack is Kafka as the message bus and Flink (or Spark Structured Streaming) for stateful computation.

User behavior features (clicks in last 5 minutes, transaction count in last hour) cannot be precomputed in batch. They require sliding window aggregations over a stream of events.

Here is a Python representation of a Flink job that computes a rolling transaction count feature:

from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import StreamTableEnvironment, EnvironmentSettings
from datetime import timedelta

env = StreamExecutionEnvironment.get_execution_environment()
settings = EnvironmentSettings.new_instance().in_streaming_mode().build()
t_env = StreamTableEnvironment.create(env, settings)

# Define the source: Kafka topic with transaction events
t_env.execute_sql("""
  CREATE TABLE transaction_events (
    user_id     STRING,
    amount      DECIMAL(10, 2),
    merchant_id STRING,
    event_time  TIMESTAMP(3),
    WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
  ) WITH (
    'connector' = 'kafka',
    'topic'     = 'transactions',
    'properties.bootstrap.servers' = 'kafka:9092',
    'format'    = 'json'
  )
""")

# Compute rolling 1-hour transaction count and sum per user
t_env.execute_sql("""
  CREATE TABLE user_transaction_features (
    user_id          STRING,
    txn_count_1h     BIGINT,
    txn_amount_sum_1h DECIMAL(12, 2),
    window_end       TIMESTAMP(3),
    PRIMARY KEY (user_id) NOT ENFORCED
  ) WITH (
    'connector' = 'redis',
    'host'      = 'redis:6379',
    'command'   = 'SET'
  )
""")

t_env.execute_sql("""
  INSERT INTO user_transaction_features
  SELECT
    user_id,
    COUNT(*)            AS txn_count_1h,
    SUM(amount)         AS txn_amount_sum_1h,
    window_end
  FROM TABLE(
    TUMBLE(TABLE transaction_events, DESCRIPTOR(event_time), INTERVAL '1' HOUR)
  )
  GROUP BY user_id, window_start, window_end
""")

The watermark on event_time is critical. Flink uses it to determine when a window is complete. Set it too tight and you drop late-arriving events. Set it too loose and your features are delayed. For transaction data, 5 seconds of watermark lag is typical. For IoT sensor data that may arrive minutes late, you need a longer watermark and a strategy for handling updates to already-emitted windows.

Feature Transformation Patterns

Features fall into three transformation categories, each with different pipeline requirements.

Identity features: Raw event fields stored as-is. No computation. User age from a profile table, merchant category code from an entity store. These are usually handled by a batch pipeline that syncs the entity store to the feature store on a schedule.

Aggregate features: Window functions over event streams. Transaction count, average order value over 7 days, click-through rate in the last hour. These require stateful stream processing or periodic batch aggregation.

Derived features: Combinations of other features computed at serving time. Ratio of transaction amount to 30-day average. These can be computed inline during serving but require the component features to already exist in the online store.

The rule for derived features: compute the components in the pipeline, not the derivative. Computing amount / avg_30d_amount at serving time is fine. Computing avg_30d_amount at serving time from raw transactions is not.

Point-in-Time Correctness for Training

This is where most teams get it wrong. When generating a training dataset, you need feature values as they existed at the time each training example occurred, not their current values.

Consider a fraud model trained on historical transactions. Each transaction has a label (fraud or not). The model needs feature values like user_txn_count_1h computed at the moment that specific transaction happened. If you join the current feature values instead of the historical ones, you introduce data leakage: the model sees future information that would not have been available at prediction time.

The offline store enables this through timestamped writes. Every feature update is written with a timestamp. A point-in-time join retrieves the most recent feature value at or before each event’s timestamp.

import pandas as pd

def point_in_time_join(
    entity_df: pd.DataFrame,      # columns: user_id, event_timestamp
    feature_history: pd.DataFrame # columns: user_id, value, created_at
) -> pd.DataFrame:
    """
    For each row in entity_df, find the most recent feature value
    where created_at <= event_timestamp.
    """
    entity_df = entity_df.sort_values("event_timestamp")
    feature_history = feature_history.sort_values("created_at")

    results = []
    for _, row in entity_df.iterrows():
        user_id = row["user_id"]
        cutoff = row["event_timestamp"]

        # Get all feature values for this entity before the cutoff
        candidates = feature_history[
            (feature_history["user_id"] == user_id) &
            (feature_history["created_at"] <= cutoff)
        ]

        if candidates.empty:
            feature_val = None
        else:
            feature_val = candidates.iloc[-1]["value"]

        results.append({**row, "feature_value": feature_val})

    return pd.DataFrame(results)

In practice you would not do this row-by-row in pandas for large datasets. BigQuery and Snowflake both have efficient AS OF join semantics, and the feature store platforms implement this at scale with sorted merge joins.

Online Serving Latency

Fraud detection models need feature retrieval in under 10ms. Recommendation models can afford 50-100ms. Real-time bidding requires under 5ms. These are not soft targets.

The online store is a read-through cache. The feature pipeline keeps it fresh. The serving layer reads from it.

import { createClient } from "redis";

interface FeatureVector {
  userId: string;
  features: Record<string, number | string | boolean>;
  fetchedAt: number;
}

const redis = createClient({ url: process.env.REDIS_URL });

async function getOnlineFeatures(
  userId: string,
  featureNames: string[]
): Promise<FeatureVector> {
  const pipeline = redis.multi();

  // Batch all feature reads into a single round-trip
  for (const featureName of featureNames) {
    const key = `feature:${featureName}:${userId}`;
    pipeline.hGetAll(key);
  }

  const results = await pipeline.exec();

  const features: Record<string, number | string | boolean> = {};
  for (let i = 0; i < featureNames.length; i++) {
    const raw = results[i] as Record<string, string> | null;
    if (raw && raw.value !== undefined) {
      // Type coercion based on declared dtype in registry
      features[featureNames[i]] = parseFeatureValue(raw.value, raw.dtype);
    }
  }

  return { userId, features, fetchedAt: Date.now() };
}

function parseFeatureValue(
  value: string,
  dtype: string
): number | string | boolean {
  switch (dtype) {
    case "float":
    case "int64":
      return Number(value);
    case "bool":
      return value === "true";
    default:
      return value;
  }
}

The multi/pipeline call batches all Redis reads into a single network round-trip. Without this, latency scales linearly with feature count. With 20 features and 0.5ms per Redis call, the naive approach adds 10ms of overhead before you even run the model.

Platform Comparison

Dimension	Feast (open source)	Tecton (managed)	Hopsworks (open source)	Custom build
Setup effort	High (infra on you)	Low (SaaS)	Medium (self-hosted or cloud)	Very high
Streaming support	Kafka via custom transform	Native, production-grade	Native Flink integration	Depends on your stack
Point-in-time joins	Supported	Supported, optimized	Supported	Must build
Online store options	Redis, DynamoDB, SQLite	Managed (DynamoDB-backed)	RonDB (MySQL-NDB)	Any
Offline store options	BigQuery, Snowflake, Redshift	Same + optimized	Hive, Delta, Iceberg	Any
Feature monitoring	Basic drift metrics	Built-in, alertable	Built-in	Must build
Cost model	Infra only	Per-feature or per-request	Infra + license	Infra only
When to choose	Control + budget	Speed + managed ops	ML-platform integration	Unique requirements

Feast is the most commonly used open-source option. It is a good choice if you already have Kubernetes and are comfortable managing the infrastructure. The Python SDK is mature and the BigQuery and Redis backends are well-tested.

The primary limitation of Feast is streaming support: it requires you to write and maintain your own stream transformation logic. Feast handles the registry, the serving layer, and the offline materialization, but the Flink or Spark job that computes real-time aggregations is your responsibility.

Custom builds make sense when your feature logic is tightly coupled to domain-specific computation that no off-the-shelf system handles, or when you have strict cost constraints and a small feature set. They consistently underestimate the operational burden of maintaining a feature catalog, handling schema evolution, and building monitoring.

Production Considerations

Freshness SLAs: Define a maximum acceptable age for every feature. A fraud feature that allows stale values 5 minutes old will behave differently from one that tolerates 1 hour of staleness. Track last_updated_at per feature per entity and alert when it exceeds the SLA. A freshness breach in the online store means the pipeline is lagging, not that the feature is wrong.

Backfill strategy: When you add a new feature to the registry, you need historical values to train on. Backfill jobs re-run feature computation over historical event data and populate the offline store. These jobs are expensive: a single feature with 2 years of history over 10 million entities requires significant compute. Budget for this explicitly.

Schema evolution: Features change. An aggregate window changes from 1 hour to 6 hours. A feature is deprecated. A new entity key is added. The registry must version feature definitions. Training datasets must pin the feature version used, not the latest. Serving code must handle missing features gracefully when a new feature version is not yet materialized for all entities.

Feature monitoring: Track three metrics per feature: drift (distribution shift vs. training baseline), freshness (how recent the online store values are), and null rate (fraction of serving requests where the feature is missing). Null rates above 5% typically indicate a pipeline failure, not a missing-entity edge case.

Cost: The online store is the most expensive component. Redis at production scale (millions of entities, hundreds of features) requires careful memory planning. A 64-byte value per feature times 100 features times 50 million users is 320 GB of Redis memory before replication. Use compressed serialization (MessagePack over JSON), set TTLs aggressively for features that do not need to persist across inactive periods, and partition hot vs. cold entity sets across separate Redis clusters.

Idempotency in pipelines: Stream processing jobs fail and restart. When your Flink job reprocesses events after a checkpoint, the feature values it writes must be deterministic. Write upserts, not inserts. Use event timestamps as the version key so replayed events do not corrupt the feature history.

Choosing Your Architecture

Start with a batch-only feature store if your latency requirements allow it. A nightly batch pipeline that populates Redis with precomputed features is far simpler to operate than a streaming pipeline. Many classification problems (churn prediction, credit scoring, next-best-offer) can tolerate features that are 1-24 hours stale.

Add streaming when you need features fresher than your batch cadence can provide. The operational cost is real: you now have a Kafka cluster, a Flink cluster, watermark tuning, and checkpoint recovery to manage.

Build custom only when the constraints of an off-the-shelf solution cannot be satisfied after a serious evaluation. The hidden cost of a custom feature store is the ongoing maintenance: you will rebuild monitoring, freshness tracking, backfill tooling, and schema management that every serious platform already provides.

The feature store is boring infrastructure. That is the point. When it works, training and serving code reference the same feature definitions and the model behaves exactly as trained. When it breaks, the model silently degrades because the data feeding it no longer matches what it learned from. Build it once, instrument it thoroughly, and do not touch it unless you have to.