A search team at a mid-sized marketplace once re-embedded their entire product catalog — 4.2 million SKUs — through a real-time API built for query embedding. It took 11 days and cost roughly $3,800 in inference compute. The batch pipeline they built afterward did the same job in 6 hours for $140. Same model, same GPUs, wildly different economics. The difference wasn’t the hardware. It was whether the system was ever designed to batch.
The Core Split
An embedding generation pipeline turns raw content — text, images, product descriptions, code — into fixed-length vectors that downstream systems use for search, recommendation, deduplication, or RAG retrieval. Every such pipeline has to answer one architectural question before anything else gets built: does this workload tolerate delay, or does it need an answer in under 200ms?
Batch pipelines process large volumes of content on a schedule or trigger, with no single caller waiting on a response. Think nightly catalog re-indexing, backfilling a vector database after a schema migration, or embedding a new document corpus for RAG ingestion. Real-time pipelines sit in the request path — a user types a search query, and something has to turn that string into a vector in single-digit milliseconds before the vector search even starts.
The mechanism that makes batch cheap is dynamic batching at the GPU level. A transformer encoder like sentence-transformers/all-MiniLM-L6-v2 processes a batch of 64 sentences almost as fast as it processes one, because the bottleneck is kernel launch overhead and memory bandwidth, not raw FLOPs for small inputs. Padding sequences to a common length within a batch and running one forward pass amortizes that overhead across every item in the batch. Doubling batch size from 32 to 64 on an A10G typically adds under 15% to per-batch latency while doubling throughput — until you hit memory limits or the batch has such varied sequence lengths that padding waste eats the gains.
Real-time pipelines can’t wait to fill a batch, because the caller is blocked. So they either eat the full unbatched cost of a forward pass per request, or they use micro-batching with a tight time window — collect requests for 5-10ms, batch what arrived, then run inference. That window is a direct latency-vs-throughput knob, and getting it wrong in either direction is the single most common mistake in this space: too short and you never batch anything; too long and your p99 latency blows past your SLA.
There’s a third pattern worth naming even though it doesn’t get its own heading: pre-computed real-time. You batch-embed everything you can predict in advance (product catalogs, static documents) and reserve the real-time path only for genuinely unpredictable input, like a user’s search query. Most production RAG systems are actually this hybrid, not a pure batch or pure real-time system. Treating it as a binary choice is where a lot of teams waste engineering effort building real-time infrastructure they didn’t need for 95% of their embedding volume.
Critical Insights
Common knowledge: Batch throughput scales roughly linearly with GPU count up to the point where I/O — reading source documents, writing vectors to storage — becomes the bottleneck instead of compute. Most teams hit this around 4-8 GPUs on a single node before needing to shard the read/write path too.
Rare knowledge: Embedding drift is a batch-specific failure mode that catches people off guard. If you fine-tune or swap your embedding model, every previously embedded vector is now in a different semantic space than newly embedded ones, and cosine similarity between old and new vectors becomes meaningless. Real-time systems fail loudly (a timeout); this fails silently as slowly degrading search relevance that nobody notices until a customer complaint two weeks later.
Advanced insight: At high query volume, real-time embedding services benefit from request coalescing on near-duplicate queries — caching embeddings for the top N% most frequent queries (often power-law distributed, where 5% of unique queries account for 40%+ of traffic) eliminates a large fraction of real-time compute without touching the batch path at all.
Strategic impact: A batch pipeline failure is a data freshness problem — annoying, bounded, recoverable by rerunning the job. A real-time pipeline failure is an availability problem that takes down search or RAG retrieval entirely, because the vector never gets generated and the downstream query has nothing to search with. Design your fallback accordingly: real-time paths need circuit breakers and degraded-mode behavior (keyword search fallback, cached results); batch paths mostly need alerting and idempotent reruns.
Implementation nuance: Tokenizer mismatches between the batch and real-time paths are a bug class of their own. If your batch job runs a slightly different tokenizer version, max-length truncation setting, or normalization step than your real-time endpoint, you get vectors from the “same” model that aren’t actually comparable — and nothing in your stack will error on this. Pin tokenizer config identically across both paths and version it alongside the model weights.
