Semantic Search vs. Keyword Search: System Design Trade-offs

Introduction

A user types “something comfortable for standing all day in a hospital.” Your keyword index scans every product title and description for those exact tokens. Nothing meaningful surfaces. Your competitor, running a vector search pipeline, returns slip-resistant nursing clogs, anti-fatigue mats, and compression socks — none of which share a single word with the query. That gap — between what people type and what they mean — is the architectural problem at the center of this article, and the reason the two approaches have completely different infrastructure profiles.

---

Keyword Search: Inverted Indexes and BM25

Keyword search is built on the inverted index. For every unique term in your corpus, the index stores a posting list: the sorted set of document IDs containing that term, annotated with position and frequency metadata. At query time, the engine tokenizes the query, retrieves each term’s posting list, and scores matching documents using BM25 — a probabilistic ranking function derived from TF-IDF with two improvements: term frequency saturation and document length normalization.

The BM25 score for a document D against query Q is:

score(D,Q) = Σ IDF(qi) · [f(qi,D) · (k1+1)] / [f(qi,D) + k1·(1 − b + b·|D|/avgdl)]

The parameter k1 (default 1.2–2.0) controls how fast additional term occurrences stop adding relevance. The parameter b (default 0.75) penalizes long documents to prevent them from dominating purely by volume. These defaults work across most corpora without tuning.

Latency is predictable: O(log n) for the term lookup, linear in posting list size for scoring. Elasticsearch handles 10,000+ QPS on commodity hardware with p99 latency under 10ms for standard full-text workloads. The index is compact, deterministic, and easy to debug — you can reconstruct exactly why a document ranked where it did.

The hard limit is vocabulary mismatch. “Cardiac event” and “heart attack” are clinically synonymous but share zero tokens. “Python” is a language or a reptile. “Memory leak” and “heap overflow” describe overlapping problems. The index doesn’t model meaning, only token co-occurrence.

Semantic Search: Embeddings and ANN Retrieval

Semantic search encodes both documents and queries into dense vectors using transformer-based embedding models — typically producing 384- to 1536-dimensional float32 vectors. The model maps semantically similar text to nearby points in vector space, measured by cosine similarity. If the embedding model has learned that “heart attack” and “myocardial infarction” are synonymous, their vectors cluster together regardless of surface form.

The index structure is fundamentally different. Instead of an inverted posting list, you need an Approximate Nearest Neighbor (ANN) index. HNSW — Hierarchical Navigable Small World — is the dominant structure in production. It organizes vectors as a multi-layer proximity graph: coarse layers for fast long-range navigation, fine layers for precise local search. Query time navigates from coarse to fine, achieving O(log n) amortized complexity with recall configurable via the ef parameter.

The end-to-end pipeline: documents are chunked → encoded by the embedding model → stored in the vector index. Queries are encoded at query time before retrieval. Embedding latency is 5–15ms for compact models like all-MiniLM-L6-v2 and 30–80ms for larger 7B-parameter models. HNSW search adds 5–30ms depending on index size and ef. Total query time: 10–50ms typical, vs keyword’s 1–10ms.