The Problem Nobody Talks About in ML Courses
You train a model offline. It performs brilliantly on test data. You deploy it. A week later, predictions drift — not because the model is wrong, but because the feature
user_purchase_count_last_30dcomputed during training used a batch job that runs at 2 AM, while the production API computes the same feature on-the-fly using a different SQL query on a slightly different dataset. Same name. Different logic. Silent divergence.
This is training-serving skew, and it’s responsible for more production ML failures than model architecture ever will be. Feature stores exist to eliminate it — and that’s just the first problem they solve.
Core Concept: What a Feature Store Actually Is
A feature store is a data system purpose-built for one job: computing, storing, and serving ML features consistently across training and inference. It sits between your raw data sources and your models, acting as a shared registry and serving layer.
The architecture splits into three planes:
1. The Feature Registry (Metadata Layer) Every feature has a definition — the transformation logic, data source, owner, freshness requirement, and schema. The registry stores these definitions as code (not documentation). Tools like Feast use Python-defined FeatureView objects that describe how to compute a feature from a data source. Tecton adds a scheduling engine on top, treating features like managed pipelines rather than raw definitions.
2. The Offline Store (Historical Data) Training needs point-in-time correct data — the feature value as it existed at the moment of the label, not today’s value. This is harder than it sounds. If you’re predicting loan default at t=0, account_balance must reflect the value at t=0, not the current balance. Offline stores (typically Hive, BigQuery, Redshift, or Parquet on S3) store timestamped feature snapshots and support point-in-time joins — a specialized merge that avoids future data leakage.
3. The Online Store (Low-Latency Serving) Inference requires sub-10ms lookups. Batch snapshots are materialized into a key-value store (Redis, DynamoDB, Cassandra) and served via a gRPC or REST endpoint. The key is always an entity ID (user_id, order_id) and the value is the latest pre-computed feature vector.
The pipeline connecting offline to online is the materialization job — a scheduled or streaming process that reads from your data warehouse and writes current feature values to the online store. Feast supports batch materialization via Spark. Tecton adds native streaming materialization via Kafka/Flink, enabling features with freshness measured in seconds rather than hours.
