A base model is a hypothesis. A fine-tuned model is a product. The gap between those two things isn’t mathematical — it’s operational. Teams that treat fine-tuning as “run a script in a notebook” eventually hit a wall: reproducibility breaks, models regress silently, datasets drift, and nobody can explain which checkpoint shipped to production last Tuesday. Designing a fine-tuning pipeline is fundamentally a distributed systems problem wearing an ML hat.

Core Concept: The Fine-Tuning Pipeline

Fine-tuning adapts a pre-trained foundation model to a specific task by continuing training on a curated domain dataset. The challenge isn’t the gradient math — it’s orchestrating the data, compute, validation, and versioning layers into a system that behaves correctly under failure, scales without bottlenecks, and produces artifacts you can actually trust.

A production fine-tuning pipeline has five distinct stages that must be independently restartable and observable.

Stage 1 — Data Ingestion & Versioning Raw training data arrives from object storage (S3, GCS), annotation pipelines, or RLHF feedback systems. Before anything trains, every dataset version must be fingerprinted. Tools like DVC or Delta Lake provide immutable dataset snapshots. The failure pattern here: teams that skip versioning can’t answer “what data produced this model?” six months later — which kills audit trails and compliance.

Stage 2 — Preprocessing & Tokenization Raw text gets cleaned, filtered for quality (perplexity filtering, dedup via MinHash LSH), and tokenized into packed sequences. This stage runs on CPU-heavy workers (separate from GPU training nodes) and produces binary shard files. A common mistake: running tokenization inside the training loop wastes GPU cycles on CPU-bound work and creates non-deterministic throughput under variable data shapes.

Stage 3 — Training Job Orchestration The training job itself is submitted to a job scheduler (Ray, Slurm, Kubernetes Job API) with explicit resource requests. Key decisions here: gradient checkpointing to trade compute for memory, mixed-precision (bf16/fp16) to reduce VRAM pressure, and gradient accumulation to simulate large batch sizes across limited hardware. For distributed runs, you choose between data parallelism (each node holds the full model), tensor parallelism (model sharded across nodes), or pipeline parallelism (model layers split across stages). Each has distinct failure modes — pipeline parallelism, for instance, introduces pipeline bubbles that reduce GPU utilization.

Stage 4 — Checkpoint Management Training jobs die. GPU nodes preempt. Cloud spot instances evict. Checkpoints must be written to durable storage at regular intervals and the pipeline must resume from the latest valid checkpoint without human intervention. A subtlety: corrupted checkpoints (written during a crash) need detection logic. Write to a temp path, validate, then atomic rename.

Stage 5 — Evaluation, Registry & Promotion After training completes, the model runs against a frozen eval harness — benchmark tasks that don’t change between runs, covering both target task performance and regression tests for general capabilities. Models that pass thresholds get registered in a model registry (MLflow, Weights & Biases Registry, or custom) with full lineage metadata: dataset version, hyperparameters, hardware config, eval scores. Promotion to staging or production requires explicit human approval or automated A/B gate logic.

Read more

This question is asked as a standalone (”design a distributed cache”) and as the deep-dive in almost every other system design question (”you mentioned adding a cache — walk me through how you’d build that”). It tests whether you understand cache eviction, consistent hashing, replication, and what happens when a cache node fails.

The Question

“Design a distributed in-memory cache like Redis. Support GET and SET operations. The cache should be fast, scalable, and handle node failures gracefully.”

Step 1 — Clarify

1. Capacity? Total data size to cache. This determines how many nodes you need. A single Redis instance handles ~10–25 GB of working memory effectively.

2. Eviction policy? LRU (least recently used) is the most common. Also: LFU (least frequently used), FIFO, TTL-based expiry. State which one unless the interviewer specifies.

3. Write policy? Write-through (update cache and DB simultaneously — strong consistency, higher write latency) vs write-around (bypass cache on writes, populate on read) vs write-back/write-behind (write to cache first, async persist to DB — lower latency, risk of data loss).

4. Read policy? Cache-aside (application checks cache first, populates on miss) vs read-through (cache automatically fetches from DB on miss). Cache-aside is the default; read-through requires the cache to know the DB schema.

5. Replication? Single leader + replicas (Redis Sentinel model) or leaderless (Redis Cluster model)?

Step 2 — Estimate

- 10 TB of data to cache across the cluster

- Target: < 1ms p99 for GET/SET

- Each node: 25 GB RAM → 400 nodes for 10 TB

- Throughput: 500K ops/sec per node → 200M ops/sec cluster capacity - Network: 1 Gbps per node, each GET returns avg 1 KB → 1M GETs/sec per node on bandwidth alone

The key insight: at 400 nodes, the routing layer (consistent hashing) must direct each request to the right node in < 0.1ms overhead, leaving < 0.9ms for the actual lookup.

Get Access to GitHub Repo

Step 3 — Core Data Structures

Hash table with LRU eviction:

The cache is essentially a hash map + a doubly-linked list for LRU tracking.

- Hash map: key → (value, pointer to LRU list node) — O(1) GET

- Doubly-linked list: most-recently-used at head, least-recently-used at tail - On GET: move the accessed node to the head of the list — O(1)

- On SET: add to head. If capacity exceeded: remove from tail (evict LRU) — O(1) - On eviction: remove from hash map and list simultaneously — O(1)

This O(1) for both GET and eviction is the data structure insight interviewers test. A naive LRU using a sorted list would be O(N) on every access.

Step 4 — Consistent Hashing (the probe)

With 400 nodes, how do you decide which node stores key K?

Naive modulo hashing: node = hash(key) % N. Simple. But when you add or remove a node (N changes), nearly every key remaps to a different node. A node failure or scale event causes a cache stampede — every miss hits the DB simultaneously.

Consistent hashing:

Place nodes on a circular ring (0 to 2³² - 1). Each node is assigned a position by hashing its identifier. To find the node for key K: hash K, find the nearest node clockwise on the ring.

Read more

This sentence appears in roughly 60% of system design interview answers involving a database. Sometimes it’s correct. Often it isn’t. Almost always it’s stated without the reasoning that determines whether it’s right or wrong.

The reasoning is everything. The partition key isn’t a preference — it’s a consequence of your read pattern.

The rule

Partition on the dimension you query by most frequently.

If the most common query is “give me all messages for user X” — partition by user_id. Every message for user X is on the same shard. One network hop.

If the most common query is “give me all messages in conversation C” — partition by conversation_id. Every message in conversation C is on the same shard.

Partition by sender_id in a messaging system where reads are by recipient? Every delivery requires querying every shard. You’ve distributed your data in a way that makes your hot read path maximally expensive.

How to get it right

Before naming a partition key, write down the two or three most frequent queries the system needs to serve. Then choose the partition key that makes those queries hit one shard.

For a payment system: the hot query is “give me all transactions for account X.” Partition by account_id.

For a ride-sharing system: the hot query is “give me all active drivers within 5km of this point.” Geohashing is the partition key — drivers in the same geohash cell land on the same shard.

For a social feed: the hot query is “give me all posts for user X’s followers, sorted by time.” Partition by user_id — each user’s feed is co-located.

Subscribe for Question Walkthrough

The celebrity problem

Even correct partition key choices have a failure mode: hot partitions.

If you partition a social graph by user_id and one of your users is a celebrity with 100 million followers, every follow/unfollow event writes to one shard. Every timeline read for those 100 million followers potentially reads from one shard. That shard is getting 100× the load of any other.

The fix depends on what’s hot. For write-heavy celebrities: scatter their data across multiple shards using a compound key (user_id + bucket_id), then merge at read time. For read-heavy content: replicate it across shards so reads can be served locally.

The L5 answer names the hot partition problem. The L6 answer names the specific fix and the conditions under which you’d apply it.

The mechanical keys worth memorizing

Some partition key decisions are standard enough that deviating from them signals a mistake:

Messages: by recipient_id (the hot read is “messages for user X”)

Notifications: by user_id (same reason)

Ride locations: by geohash (the hot read is proximity query)

Payment transactions: by account_id (the hot read is account history)

Web sessions: by session_id (the hot read is session lookup by ID)

Analytics events: by event_time or by (user_id, date) if per-user queries dominate

When you name a partition key in an interview, the follow-up is almost always: “what’s the failure mode?” The correct answer is almost always: “hot partitions when the distribution is uneven — here’s how I’d detect and address them.”

What “sharding” actually is

One last thing worth naming explicitly: sharding and partitioning are often used interchangeably in interviews, but they mean slightly different things. Partitioning is dividing data logically. Sharding is distributing those partitions across separate physical nodes. When interviewers ask “how would you shard the users table?” they mean both.

The answer format that works: state the partition key, explain why it fits the read pattern, name the hot partition risk, and describe the mitigation. Four sentences. Every sharding answer.

CTA: The distributed cache walkthrough (paid post #10) goes deep on consistent hashing and the hot partition problem specifically — including the virtual nodes fix and the thundering herd mitigation when a shard fails.

Interested in mastering this topic?

Unlock premium content with deeper explanations and real implementation patterns.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

Every time your application sends a 10,000-token system prompt to an LLM, the model recomputes attention over every single token — even if that system prompt hasn’t changed in weeks. For a product doing 50,000 requests per day, you’re paying full price 50,000 times to process the same instructions. Prompt caching eliminates that waste by storing the computed internal state of repeated token sequences and reusing it on subsequent requests.

How Transformer KV Caching Works

Before a transformer generates any output token, it processes your entire input through multiple attention layers. In each layer, every input token produces a Key and a Value vector — the KV pair. Attention then computes how much each token should attend to every other token by comparing queries against all keys and weighting the corresponding values. This operation is O(n²) in sequence length.

When you send the same prefix (e.g., a system prompt) across thousands of requests, you’re recomputing those same KV pairs from scratch each time. Prompt caching short-circuits this: after processing a token sequence, the provider stores the KV tensors for that prefix in high-bandwidth memory. On the next request with the identical prefix, the model skips recomputation and loads the cached KV state directly into the attention layers.

The critical constraint: the cache is keyed on an exact token-level prefix match. If you insert a user-specific sentence at position 50 of a 2,000-token system prompt, you’ve broken the prefix and busted the cache for everything after position 50. Cached content must always appear before dynamic content.

Anthropic’s implementation (as of late 2024) uses cache_control: {type: "ephemeral"} markers at specific content boundaries. Cache TTL is 5 minutes, reset on each cache hit, with a minimum cacheable block of 1,024 tokens. Cache write costs 25% more than a standard input token; cache reads cost 10% of the standard price — so a prefix hit after the first write is 90% cheaper per token on that segment.

Google’s Gemini context caching requires a minimum of 32,768 tokens and supports 1-hour TTLs with explicit expiry control. Their pricing model charges a per-hour storage fee for the cached context plus a reduced input price on cache hits — optimized for long-document workflows rather than high-frequency short exchanges.

OpenAI’s automatic caching (as of late 2024) requires no explicit markers — the API silently reuses KV state for repeated prefixes longer than 1,024 tokens, with cache hits appearing in the usage metadata. The tradeoff: less control over what gets cached.

Critical Insights

Structure your prompts as prefix stacks. Place static content first: system instructions → static examples → document context → dynamic user content. This isn’t stylistic — it’s the only way the prefix invariant holds. Teams that put user names or session IDs early in system prompts destroy their cache hit rate entirely.

Cache writes are not free. On Anthropic’s API, writing a new cache entry costs 1.25× the normal input price. If your TTL is 5 minutes and traffic is sparse enough that entries expire before being hit twice, you’re paying a premium with no benefit. Caching pays off when expected hits × (1 - 0.10) > expected misses × 0.25 — roughly when you expect more than one hit per write within the TTL window.

Few-shot examples are the highest-ROI caching target. A carefully crafted 50-example few-shot block can run 3,000–8,000 tokens. It’s static, deterministic, and processed identically on every call. Caching it drops that segment’s cost by 90% with no design constraints.

Document context caching changes RAG economics. Traditional RAG retrieves and injects 2,000–8,000 token chunks per query. With caching, you can inject a full 50,000-token document once, cache it, and run dozens of questions against it within the TTL — paying full price only on the first call. This is particularly effective for legal document Q&A, code review assistants, and report analysis pipelines.

Dynamic content length affects effective cache depth. If your user turn averages 200 tokens but your cached prefix is 4,000 tokens, 95% of input tokens are cached. If your average user turn is 3,000 tokens (e.g., pasting full code files), your effective cache fraction drops to 57%. Design your interface to keep dynamic content short and push static context into the cache.

Cache cold starts hurt p99 latency. The first request after a cache miss processes all tokens from scratch — often 2–4× slower than a cache hit. Under bursty traffic where cache entries expire during a quiet period, you’ll see latency spikes as entries are rewarmed. Build latency SLOs that account for this; don’t measure average latency alone.

Real-World Examples

Cursor (AI code editor) caches the entire workspace context — open files, project structure, and coding conventions — for each session. Their reported cache hit rates exceed 80% for active sessions, with per-request costs reduced 3–5× compared to naive injection. The architectural key: they keep file content structurally before the user’s active question, maintaining prefix integrity even as individual files change.

Notion AI uses document-level caching for their “Ask about this page” feature. A 20,000-word document gets cached once; all subsequent questions in that session hit the cache. They pad the document to 32,768 tokens when using Gemini to ensure eligibility. This reduced their per-query cost by roughly 60% on document-heavy workloads.

Harvey AI (legal) caches jurisdiction-specific prompt libraries — sometimes 40,000+ tokens of statutory references and case law summaries — as static prefix layers. Because legal workflows involve repetitive document review, a single cache write typically returns 15–30 hits before expiry.

Architectural Considerations

Prompt caching is invisible to your application’s logic but visible in cost and latency profiles. Build cache hit rate into your observability stack — most providers expose cache_read_input_tokens and cache_creation_input_tokens in usage metadata. Alert when hit rate drops below a threshold, since it signals a structural change in how prompts are being constructed.

Caching interacts with streaming: cache hits reduce time-to-first-token noticeably, which improves perceived responsiveness even when total generation time is similar. Monitor TTFT separately from generation throughput.

Avoid caching highly user-specific content (PII, session tokens, per-user preferences) in the shared prefix layer — this creates both cache poisoning risk and privacy surface area. Keep personalization in the dynamic suffix.

Practical Takeaway

GitHub Link

https://github.com/sysdr/sdir/tree/main/Prompt_caching_strategies/prompt-caching-demo

Audit your current prompt structure before any other optimization. Separate it into: (1) static system instructions, (2) static few-shot examples, (3) static document context, (4) dynamic user content. Add cache_control markers at each static boundary. Measure cache hit rate for one week.

For most production applications hitting the same model endpoint repeatedly, this single change reduces input token costs by 40–70% with no quality impact.

Run bash setup.sh to launch a local demo with a real Anthropic API backend. The dashboard sends identical system prompts with and without caching enabled, tracks cache hit/miss rates, measures latency distributions, and calculates live cost savings. Set your ANTHROPIC_API_KEY as an environment variable before running. Extend it by adding your own system prompt in config/prompts.js and watching how prompt structure (prefix vs. suffix placement) changes your cache hit rate in real time.

Youtube Demo Link:

The reason: it has a clear probe (the trie data structure and its distributed form), and it reveals whether candidates

understand read-heavy system optimization, pre-computation, and the latency constraints of real-time user interaction.

The Question

“Design the search autocomplete / typeahead system. As a user types in a search box, the system suggests the top 5–10 completions in real time.”

Variants: “Design Google search suggestions,” “Design Twitter’s search autocomplete,” “Design Amazon’s search bar.”

Step 1 — Clarify

1. What goes in suggestions? Historical search queries (Google-style), titles in a product catalog (Amazon-style), or usernames/topics (Twitter-style)? The data source changes the indexing strategy significantly.

2. How many queries is the system handling? Google processes ~100K searches/sec. For most interviews: 10M DAU × 10 searches/day = 100M searches/day = ~1,200/sec average, 10K/sec peak.

3. How fresh must suggestions be? Do new trending searches need to appear in suggestions within minutes, or is 24-hour staleness acceptable? Real-time freshness is much harder. Start by assuming daily updates — offer to discuss real-time as an extension.

4. Personalized or global? Global suggestions (same results for all users) are vastly simpler. Personalized (based on your history) requires per-user ranking on top of the global index.

5. Latency SLO? Suggestions must appear within 100ms of each keystroke to feel real-time. This is the hardest constraint in the system.

Step 2 — Estimate

- 10K search requests/sec at peak (one autocomplete request per keystroke) - Average query length: 20 characters → 20 requests per search session in the worst case

- Top-K queries to track: top 10 million unique queries globally

- Trie storage: each node is ~16 bytes (character + children pointers + metadata) — a trie of 10M queries averages ~50M nodes × 16 bytes = 800 MB. Fits in memory on a single machine.

- Response size: 5–10 query strings, avg 30 bytes each = 300 bytes per response

The 800 MB insight is critical. The entire suggestion index for most production systems fits in memory. This is what enables sub-millisecond lookups.

Get Access to GitHub Repo

Step 3 — Data Model and Core Data Structure

The trie

A trie (prefix tree) is the natural structure for autocomplete. Each node represents a character. A path from root to a node spells a prefix. Each node stores the top-K completions for that prefix.

The key design choice: Pre-compute and cache the top-K results at every node in the trie. This trades storage for latency — instead of traversing to all leaf nodes on every query, the answer is stored at the prefix node. Query time: O(len(prefix)) — just traverse down the trie.

Without pre-computation: query time is O(all_suffixes) — must find all strings below the current node and rank them.

query_frequency table (offline aggregation source)

query_text, frequency_count, last_updated_at

This is the source of truth for what goes in the trie. Updated daily via a MapReduce / Spark job over the day’s search logs. The job counts query occurrences, normalizes (lowercase, trim), and writes the top-N queries per prefix into the trie build pipeline.

Step 4 — Architecture

Three components with different update cadences.

Query service (read path — latency-critical)

1. Client types a character → sends GET /suggest?q=sea

2. Request hits a load balancer → routes to one of N Trie servers (in-memory trie, sharded by prefix range or consistent hash)

3. Trie server traverses the trie in O(len(query)) time → returns top-5 cached completions 4. Response cached at CDN edge (short TTL — 60 seconds) for common prefixes (typing “the” returns the same results for everyone)

5. Total latency: < 10ms (CDN hit) or < 50ms (trie server hit)

Trie build service (write path — offline, daily)

1. Raw search logs → Kafka → Spark/Flink streaming job

2. Aggregation job computes top-10M queries by frequency for the trailing 7 days

3. Trie builder constructs a new trie from scratch and pre-computes top-K at every node 4. New trie serialized to S3

5. Trie servers download and hot-swap the in-memory trie (blue-green style — new trie loaded in background, traffic cut over atomically)

The hot-swap is the L5+ signal. Naive implementations rebuild the trie in place and serve stale or partially-built data during the rebuild. The correct approach: build the new trie entirely before serving any requests from it.

Trending query ingestion (real-time extension)

For real-time freshness (new trending query appears within minutes):

1. Search events stream to Kafka

2. A windowed aggregation job (5-minute tumbling window) identifies queries surging in frequency

3. Trending queries are injected into the trie via a small patch update (not a full rebuild)

4. Patch propagates to all trie servers via pub-sub

This is the L6 extension — daily batch covers the base case, real-time streaming handles trending. Name both.

Read more

Here’s what those signals look like and what to do with each one.

Signals that you’re on track

“That’s interesting — tell me more about that.”

This is the best signal you can receive. The interviewer is engaged, they want to go deeper on something you said, and they’re inviting you to. Do it immediately. Whatever you were about to say next, pause it. Go deeper on the thing they asked about.

The mistake: treating this as confirmation that everything is fine and continuing your prepared flow. The interviewer told you where the interesting part of the question is. Follow them there.

Consistent nodding during a long explanation means you’re not losing them. Keep the same pace and depth.

A follow-up question that goes one level deeper on something you said — not redirecting to a new topic, but drilling into something you introduced — means the interviewer found your point substantive. This is different from a question that redirects you elsewhere

Signals that you need to change direction

“Let’s skip ahead to...” or “Actually, let’s focus on...”

The interviewer is redirecting you. This is not a signal that you did something wrong. It’s information: you’ve covered the previous section sufficiently, or you’re spending time on something that isn’t the probe they care about, or they have a specific component they want to evaluate and they’re efficiently navigating you there.

The correct response: stop mid-sentence if necessary. Don’t finish the paragraph. Say “Sure” and immediately go to where they’re pointing.

The wrong response: finishing your current point before the redirect. Taking 90 seconds to wrap up before pivoting. Explaining why you were going in the direction you were before changing direction.

None of these explanations are useful to the interviewer. The redirect supersedes everything else. Follow it immediately.

“What are the trade-offs?” immediately after you make a decision means: you stated the decision without justifying it. This is a probe, not a compliment. Answer it: what did you choose, what are you giving up, and what would change your mind.

Subscribe for Question Walkthrough

Signals that you’ve misunderstood the question

“What if this needs to work globally?”

“Let’s say the scale is 100× what you assumed.”

“Actually, multiple users can edit the file simultaneously.”

These are corrections. Not gentle suggestions — the premise you’ve been designing for is wrong and the interviewer is telling you.

The correct response: stop, acknowledge once (”Good point — let me revise”), immediately build forward on the corrected premise.

The wrong response: defending the original premise, explaining why the correction doesn’t change things significantly, or taking two minutes to re-explain your reasoning before accepting the correction.

One acknowledgment, then forward. The interviewer doesn’t need to understand why you made the original assumption. They need to see you incorporate new information and keep moving.

The silence probe

When you finish a section and the interviewer doesn’t immediately respond — they look at you, or look at the whiteboard, or just wait — this is almost always an invitation to go deeper.

The silence is not “that was fine, continue to the next section.” It’s “there’s more here, keep going.”

Most candidates treat silence as a signal to move on. The silence probe is actually the opposite: the interviewer has heard the surface answer and wants to see if you’ll surface the deeper one without being explicitly asked.

If you finish a section and get silence, don’t move to the next section. Go deeper on what you just said. Name a failure mode. Name a condition under which your design breaks. Name what you’d add with more time.

If you’ve genuinely exhausted the depth on that component, say so explicitly: “I think I’ve covered the core design here — should I move on to the data model?” This forces a response rather than leaving both of you in ambiguous silence.

Using the interviewer actively

The best candidate-interviewer dynamics aren’t interviews — they’re conversations. Candidates who treat the interviewer as a collaborator rather than an examiner consistently do better.

“I’m torn between two approaches here — let me think through both with you” is collaborative.

“I’m not sure which way this scales better — what factors would you weight more heavily?” is collaborative. It’s also information-gathering: if the interviewer says “I’d think about the write path specifically,” they’ve just told you where the probe is going.

This doesn’t mean delegating decisions to the interviewer. You still need to make choices and defend them. But using the interviewer as a thinking partner — bringing them into your reasoning process rather than presenting finished conclusions — produces a fundamentally different conversation.

That conversation is easier to score well on, because the interviewer has seen how you think, not just what you concluded.

CTA: The mock interview partners channel in the Discord is the fastest way to practice reading a real interviewer — not just a timer. Paid subscribers only.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

A user submits a prompt. The model generates 500 tokens. At 10 tokens/second, that’s 50 seconds of wall-clock time. At 100 tokens/second, it’s 5 seconds. The difference between those two numbers isn’t raw GPU power — it’s inference architecture. The same H100 cluster, configured differently, will produce radically different latency profiles. This article breaks down exactly why, and what production teams do about it.

Why LLM Inference Is Uniquely Hard

Unlike a CNN doing image classification, a large language model generating text is autoregressive — each output token depends on every token generated before it. You cannot parallelize generation across the output sequence. Token 50 must wait for token 49.

This creates two phases with completely different compute characteristics:

Prefill: Processing the input prompt. The entire prompt is available at once, so this is highly parallelizable. It saturates GPU compute (FLOP-bound). A 1,000-token prompt takes roughly the same wall-clock time as a 10-token one on modern hardware.

Decode: Generating output tokens one by one. Each step loads the entire model from GPU memory to compute a single token. This is memory-bandwidth-bound, not compute-bound. The GPU spends most of its time waiting on memory reads.

This asymmetry drives nearly every inference optimization.

KV Cache: The Non-Negotiable Foundation

During attention computation, every transformer layer produces Key (K) and Value (V) tensors for each token. In autoregressive decode, if you recompute these from scratch on every step, you’re doing quadratic redundant work.

The KV cache stores K and V tensors from all previous tokens. Each decode step only needs to compute K/V for the new token, then attend over the cached history. This reduces per-step compute from O(n²) to O(n).

The cost: memory. A single KV cache entry for one token across all layers in a 70B-parameter model consumes roughly 0.5–2 MB depending on precision and architecture. A 4,096-token sequence holds several GB of KV data per concurrent request. At 100 concurrent users, you’re looking at hundreds of gigabytes — before model weights.

PagedAttention (introduced by the vLLM project) solved the fragmentation problem that plagued early KV cache implementations. Instead of pre-allocating a contiguous memory block per request (wasting space when sequences are shorter than the maximum), PagedAttention stores KV cache in fixed-size non-contiguous “pages,” similar to virtual memory in an OS. This improved GPU memory utilization from ~20–40% to over 90% in benchmarks on real workloads as of late 2024.

Read more

The rate limiter question is compact, deceptively hard, and asked everywhere. It tests whether you understand distributed systems coordination problems — specifically, how to enforce a global limit across a cluster of servers when no single server sees all requests. At L5 the probe is the algorithm choice. At L6 the probe is distributed coordination without a single point of failure.

The Question

“Design a rate limiter. API clients are limited to N requests per second. Requests that exceed the limit return 429 Too Many Requests.”

Common variants: “Design Cloudflare’s rate limiting,” “Add rate limiting to a payment API,” “Design a per-user request quota system.”

Step 1 — Clarify

1. What are we rate limiting on? Per IP, per user/API key, per endpoint, or combinations? Answer: typically per API key at the application level, per IP at the network edge.

2. What granularity? Requests per second, per minute, per hour, per day? Multiple windows simultaneously (100 req/min AND 1,000 req/hour)?

3. Hard limit or soft? Hard: the 101st request this minute is always rejected. Soft: occasional burst allowed. Most production systems use hard limits.

4. Where does the limiter live? Middleware inside the API service, or a standalone sidecar/gateway? Standalone scales independently and is language-agnostic — correct answer for L5+.

5. What’s the SLO? The rate limiter must add < 1ms p99 latency to every request. It is in the hot path.

Step 2 — Estimate

- 50,000 API servers in a large deployment, each handling 10K req/sec - Total: 500M requests/sec — individual counters per API key must be updated atomically at this rate

- Redis cluster: 100K-500K ops/sec per node — need ~1,000-5,000 Redis nodes for 500M req/sec. In practice, shard by API key.

- Counter storage: 1M API keys × 4 bytes per counter × 60 windows = 240 MB — fits in memory

Get Access to GitHub Repo

Step 3 — API (internal interface)

Step 4 — Algorithm choice (the core probe)

Four real algorithms. The senior answer names all four, picks one with justification, and explains what changes at L6.

Fixed window counter

Divide time into fixed 1-minute windows. Count requests per window per API key. Reset at window boundary.

Win: Simple. O(1) time and space. One Redis INCR per request.

Lose: Boundary burst problem. A client can send 100 requests at 12:00:59 and 100 more at 12:01:01 — that’s 200 requests in 2 seconds while the “limit” is 100/minute. The window reset creates a seam they can exploit.

Sliding window log

Store the timestamp of every request in the last 60 seconds. On each request, count entries in the log and reject if > limit.

Win: Perfectly accurate. No boundary burst.

Lose: O(N) storage per API key (N = limit). For 1,000 req/min across 1M API keys, that’s 1 billion stored timestamps — expensive.

Sliding window counter (hybrid) ← correct answer for most interviews

Approximate the sliding window using two fixed-window counters: the current window and the previous window. Estimate the request count for the sliding window as:

rate = prev_count × (1 - elapsed_fraction) + curr_count

Where elapsed_fraction = how far into the current window we are. If we’re 30 seconds into a 60-second window, the weight of the previous window is 0.5.

Win: O(1) space per API key. Accurate within ~0.003% of the true sliding window (published by Cloudflare). No boundary burst exploit.

Lose: Approximate (but the error is tiny). Slightly more complex than fixed window.

This is what Cloudflare uses in production — they published a blog post on it. Saying “Cloudflare published their sliding window counter approach” is an L5+ signal.

Token bucket

Each API key has a bucket of capacity N tokens. Tokens refill at rate R per second. Each request consumes one token. If the bucket is empty, the request is rejected.

Win: Allows controlled bursting — a client can use saved-up tokens in a burst. Better user experience for legitimate burst traffic.

Lose: Two values to track per key (current tokens, last refill timestamp). Slightly harder to implement atomically.

When to use: When the product requirement explicitly allows bursting (e.g., “allow up to 50 requests in any 1-second burst as long as the 1-minute average is under 100/min”). For strict per-second limits, sliding window counter is simpler.

The senior recommendation: Default to sliding window counter (Cloudflare’s approach) for strict per-second/per-minute limits. Token bucket for burst-tolerant limits where the product explicitly needs it.

Read more

Learn System Design with System building, Subscribe Hands On coding course - LogStream

Every few weeks, someone posts in the Discord’s #offers-landed channel. L5 at Google. Senior at Stripe. Staff at DoorDash. L6 at Meta after two prior failed loops.

I read every one. After enough of them, patterns become visible. The engineers who pass consistently are not uniformly technically stronger than the ones who fail. They are systematically different in three specific ways.

They scope, then go deep. Not the other way around.

The most common failure mode I hear in post-mortems from engineers who got “No Hire” is some version of: “I designed a great system but ran out of time before the hard part.”

Every engineer who has passed at L5 or above in the last six months has told me some version of the same thing: they spent the first five minutes explicitly scoping — “I’m going to focus on the timeline and the fan-out architecture, and I’m going to acknowledge but not design the notifications, search, and ads” — and then went deep on the one component they chose.

The engineers who fail design everything shallowly. The engineers who pass design one thing deeply.

This is not a trivial shift. Most engineers feel obligated to cover everything. The pressure to show breadth is real. Resisting it and committing to depth in the first five minutes is a deliberate choice that requires practice.

Subscribe for Question Walkthrough

They say the hard thing out loud before the interviewer asks.

The standard pattern in a design interview is: candidate draws the happy path, interviewer probes the failure case, candidate scrambles to address it.

The pattern in successful candidates is different: candidate draws the happy path, then immediately says “the part of this design I’m most concerned about is the thundering herd when the cache node recovers — let me address that.” The probe never comes because the candidate already went there.

This signals something important: not just that you know the failure mode, but that you think about your own designs critically. That’s the operational maturity signal. It’s very hard to fake, and it’s very clear when it’s present.

They treat the interviewer as a collaborator, not an examiner.

This one is harder to describe but unmistakable when you see it.

Candidates who fail tend to treat the design round as a test — something being administered to them. They present, the interviewer asks, they defend. The dynamic is adversarial even when the interviewer is trying to be helpful.

Candidates who pass tend to use the interviewer. “I’m considering two approaches here — let me think through both with you.” “Does this constraint change which way you’d go?” “I’m not sure which way this scales better — what’s your intuition?”

This works for two reasons. First, it makes the interview feel collaborative, which influences the interviewer’s overall impression. Second, and more importantly, it gets you real information. When you say “I’m torn between these two approaches” and the interviewer says “let’s explore the second one,” they’ve just told you where the interesting part of the question is. That’s not cheating — that’s good communication.

What these three things have in common

None of them are technical. Scoping, self-critique, and collaboration are all communication skills. They’re skills you can practice in a mock interview. You cannot practice them by reading another walkthrough.

The most common mistake I see in prep is the ratio: 90% reading walkthroughs, 10% timed drills with feedback. The engineers who pass consistently tend to invert that ratio in the final 4 weeks before the interview.

The knowledge is necessary. It’s just not sufficient.

If you’ve been reading this newsletter and feel solid on the technical content, the thing to do next is not to read more. It’s to schedule a mock with someone who will score you honestly on scoping, critique, and collaboration — not just on whether you knew consistent hashing.

CTA:

Paid subscribers get one full system design walkthrough

every Tuesday — each mapped to its archetype so you build

pattern recognition, not answer memorization.

This week: Rate Limiter.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

Curious how this works in real systems?

The paid lessons cover scalability, optimization, and real-world engineering patterns.

You ship a customer-facing chatbot backed by GPT-4. It answers 80% of questions brilliantly. Then a user asks about a product SKU you added last week, a policy change from this morning, or a support ticket filed an hour ago. The model hallucinates confidently. The problem isn’t the model — it’s that the model’s weights are frozen in time and isolated from your operational data. Retrieval-Augmented Generation (RAG) is the architectural pattern that closes that gap: it lets a language model reason over your data, now, without retraining.

Core Concept: What RAG Actually Does

A language model is a frozen function. Its weights encode general world knowledge up to a training cutoff; it cannot access your database, your docs, your tickets. RAG separates the retrieval problem from the reasoning problem.

The pattern works in two phases:

Indexing phase (offline): Source documents — PDFs, markdown files, database rows, Confluence pages — are chunked into segments, typically 256–512 tokens. Each chunk is embedded into a high-dimensional vector using a model like text-embedding-3-small or bge-large-en. These vectors are stored in a vector database (Pinecone, Weaviate, pgvector, Qdrant). This phase runs continuously as data changes.

Query phase (online): When a user submits a query, that query is embedded using the same embedding model. The vector store performs approximate nearest-neighbor (ANN) search — cosine or dot-product similarity — returning the top-k most semantically similar chunks, typically k=3–10. Those chunks are injected into the LLM prompt as context. The model reasons over retrieved evidence rather than relying solely on its weights.

The critical architectural insight: the LLM is doing grounded reading comprehension, not recall. This is fundamentally different from how models are typically deployed.

Chunking strategy is load-bearing. If chunks are too large, you waste context tokens and dilute relevance. Too small, and you lose surrounding context needed to answer coherently. Semantic chunking — splitting at paragraph or section boundaries rather than fixed token counts — outperforms naive sliding windows. Hierarchical chunking (summary chunk + child detail chunk) handles both coarse-grained and fine-grained queries.

Embedding model selection affects recall ceiling. Retrieval quality is bounded by how well the embedding model captures semantic similarity for your domain. General-purpose models underperform on specialized corpora (legal, medical, code). Fine-tuning an embedding model on domain-specific positive/negative pairs — contrastive training — measurably improves recall. This step is skipped in most initial deployments and becomes the dominant bottleneck at scale.

Reranking separates relevant from merely similar. First-pass ANN retrieval optimizes for speed using approximate algorithms (HNSW, IVF). A reranker — typically a cross-encoder model like Cohere Rerank or a fine-tuned BERT — takes the top-50 ANN results and re-scores them with full cross-attention between query and document. Reranking improves precision substantially and adds 20–100ms latency. At production scale, you run the reranker only when the primary retrieval confidence is below a threshold.

Hybrid search outperforms pure vector search for most production workloads. Combining dense vector similarity with sparse BM25 retrieval (keyword matching) via reciprocal rank fusion (RRF) handles cases where exact-match terms — product IDs, error codes, names — would be missed by semantic search alone. Elasticsearch and OpenSearch support this natively; Weaviate and Qdrant have hybrid modes.

Read more

This is the question that defines the file sync and storage archetype. Eight questions in the question bank are variants — iCloud Photos, distributed file systems, cloud backup services, Git-style version control storage, CDN edge caching. They share a spine: chunked file storage, metadata management, multi-device consistency, and conflict resolution. Internalize this one and you can answer all eight.

The surface question sounds like object storage with a UI. The probe underneath is what separates candidates: what happens when two devices modify the same file simultaneously? That conflict resolution problem — deceptively simple on the surface, genuinely hard in distributed systems — is where every L5+ interview on this question goes.

For L4 / mid-level: upload/download flow and basic chunked storage. For L5 / senior: delta sync (only transfer what changed, not the whole file), deduplication, and conflict detection. For L6 / staff: vector clocks or last-writer-wins with operational transforms for true conflict resolution, the consistency model across devices, and what Dropbox actually does differently from naive S3.

The Question

“Design Google Drive or Dropbox. Users upload files from their devices. Files are accessible from any of their devices and sync automatically when changed.”

Common variants:

- “Design iCloud Drive / Apple Files.”

- “Design a cloud backup service.”

- “Design the file sync layer for a collaborative tool.”

- “Design a distributed file system.”

Step 1 — Clarify Before You Draw

1. Individual file storage or collaborative editing? Dropbox stores files — one writer at a time, sync on save. Google Docs is collaborative editing — multiple simultaneous writers with operational transforms. These are fundamentally different systems. Clarify and scope to Dropbox-style unless told otherwise.

2. What file types and size limits? Documents, images, video — or everything? Max file size matters: Dropbox supports files up to 50 GB. A naive single-upload approach collapses above a few hundred MB.

3. Multi-device sync — how many devices per user? Average user: 3–5 devices. Power user: 10+. Each device must receive all changes to synced folders. This drives the notification and sync architecture.

4. Versioning and deleted file recovery? Most cloud storage products retain file versions (Dropbox keeps 30–180 days of version history). Deleted files go to trash with a retention window. State the assumptions.

5. What scale? Dropbox at its peak: 700 million registered users, 500 million files uploaded per day. Working assumptions: 100M DAU, 1B files stored, average file size 1 MB.

6. What’s the SLO? File upload acknowledges within 5 seconds for files under 10 MB. Sync propagates to other devices within 30 seconds of the change. These two SLOs drive the architecture.

Step 2 — Estimate

- 100M DAU, each uploads/modifies ~5 files/day average → 500M file operations/day → ~5,800 ops/sec

- Average file size 1 MB → 5.8 TB/day of new or modified data ingested - Total files stored: 1B files × 1 MB average = 1 PB

- With deduplication (identical files stored once): real storage closer to 400–600 TB (60% dedup ratio is typical)

- Metadata per file: ~500 bytes (name, path, size, hash, version, owner, timestamps) → 500 bytes × 1B files = 500 GB of metadata — fits comfortably in a sharded relational database

- Change notifications: 500M operations/day / 86,400 seconds × 3 devices per user average = ~17,000 notification fanouts/sec

The deduplication estimate is the senior signal in estimation. Dropbox publicly states that deduplication dramatically reduces storage costs because many users store the same files (PDFs, installers, stock photos). Mentioning deduplication in the estimation shows you’ve thought about the system’s economics, not just its architecture.

Get Access to GitHub Repo

Step 3 — API Design

Chunked upload with presigned URLs. Files go directly from the client to S3 — the API server never handles the binary data. This eliminates the API tier as a bandwidth bottleneck. The upload_id ties the chunks together; the etags from each chunk PUT allow S3 to assemble the final object via S3 multipart upload completion.

The /sync/changes endpoint with cursors. This is the Dropbox API model (they call it list_folder/continue). Instead of polling “what changed since timestamp X,” the client maintains an opaque cursor that encodes its sync state. The server returns everything the client has missed since that cursor position. This handles offline sync correctly: a device that was offline for 2 weeks reconnects, passes its cursor, and gets all 2 weeks of changes in one call.

The /conflict endpoint. Most candidates forget this exists. Conflict resolution is a user-facing operation — the client detects a conflict, shows the user, and the user or the system decides. Having an explicit endpoint for conflict resolution signals you’ve thought about the user-facing side of the distributed systems problem.

Step 4 — Data Model

files table

The content_hash → storage_key mapping is the deduplication mechanism. Before storing a file, the server checks whether content_hash already exists in the storage index. If it does, it reuses the existing S3 object and just creates a new metadata row pointing to it. The same file can be “owned” by millions of users but stored in S3 once.

file_versions table

version_id, file_id (FK), version_number, content_hash, size_bytes,

storage_key, created_by_device_id, created_at, is_current

Every modification creates a new version row. The is_current flag moves to the newest version. Old versions are retained for 30 days (configurable), then the storage_key reference is eligible for deletion if no other version or file points to it.

sync_cursors table

cursor_id, device_id, user_id, last_processed_change_id, updated_at

Each device has a cursor. When the device calls /sync/changes, the server returns all changes after last_processed_change_id for that user, then updates the cursor. The cursor is device-specific, not user-specific — each device tracks its own sync position independently.

change_log table — the sync feed

change_id (SEQUENTIAL), user_id (INDEXED), file_id, change_type

(created/modified/deleted/moved),

version_id, changed_by_device_id, changed_at

This is the append-only event log that drives sync. Every file operation appends a row. When a device calls /sync/changes?cursor=X, the query is simply: SELECT * FROM change_log WHERE user_id=:uid AND change_id > :cursor_change_id ORDER BY change_id LIMIT :limit. Fast, simple, correct.

The change_id is a sequential integer (not UUID). Sequential IDs make the range query above maximally efficient. They can be generated by Postgres sequences on a per-user basis, or by a distributed ID sequence (Snowflake-style, with the user_id encoded in the ID).

Read more

Learn System Design with System building, Subscribe Hands On coding course - LogStream

Thirty days. One onsite loop. Here’s the plan I’d actually follow — not the plan that covers everything, because thirty days cannot cover everything, but the plan that prepares you for the specific interview in front of you.

The principle: targeted depth beats broad coverage

Thirty days of prep that covers six archetypes shallowly will lose to twenty days of prep that covers two archetypes deeply, plus ten days of mock interviews.

The engineers who pass consistently aren’t universally prepared. They’re specifically well-prepared for the questions they’re most likely to face. At most companies, a single loop involves one or two system design questions. If you know your target company’s interview patterns, you know which two archetypes to prioritize.

Research your target company before you start prep. Look at Glassdoor, Blind, and LinkedIn connections who’ve interviewed there recently. Identify the two most commonly asked archetypes. Those are your week 1 and week 2.

Subscribe for Question Walkthrough

Week 1 (Days 1–7): Deep drill, archetype #1

Pick the most likely archetype for your target company. Read two walkthroughs from that archetype fully. Don’t drill yet — read to understand the pattern.

Then do three timed cold drills from that archetype on days 3, 5, and 7:

• Set a 50-minute timer

• Design on paper with no notes, no searching

• When the timer ends, open the walkthrough and compare section by section

• Write down the top 3 specific gaps between your answer and the walkthrough

By day 7, you should produce a structurally correct answer — not perfect, but with the right components in the right places and the key failure modes named.

Week 2 (Days 8–14): Deep drill, archetype #2

Same structure. Different archetype — your second most likely one.

After two weeks, you have deep coverage of the two archetypes most likely to come up. That alone puts you in better position than most candidates who drilled 15 questions shallowly.

Week 3 (Days 15–21): Coverage and breadth

Spend week 3 doing shorter 30-minute drills across 4–5 archetypes you haven’t deeply covered. The goal is exposure, not mastery. Knowing the basic shape of an answer for a question you’re 20% likely to face is much better than having no answer.

The 30-minute format requires discipline: force yourself to reach the architecture step within 15 minutes. Don’t let estimation and API design consume your entire drill — practice moving through the framework efficiently.

Also in week 3: re-read the estimation numbers cheatsheet and the trade-offs cheatsheet. These apply to every question. Refreshing them mid-prep rather than at the very end gives them time to consolidate.

Week 4 (Days 22–28): Two mocks with real feedback

Stop reading. Stop drilling solo. Do two mock interviews with people who will give you specific, scored feedback.

The first mock: peer at your target level. Ask them to score you on the 8 dimensions of the take-home rubric. Ask them specifically whether your scoping, communication, and failure mode coverage were strong.

The second mock: someone above your target level. Their feedback will be harder to hear and more useful. What L6 candidates do that you’re not doing yet — that’s the gap you have four days to close before the interview.

After each mock: identify the single most impactful change. Spend two days on that one thing. One change made well is more valuable than ten changes made half-heartedly.

Days 29–30: Consolidation only

Day 29: one short drill (30 minutes), no new material. Re-read the framework cheatsheet. That’s it.

Day 30 (the day before): re-read the drill card for the archetype most likely to come up at your target company. This takes four minutes. Then stop.

Don’t read a new walkthrough the night before. New information the night before competes with everything you’ve consolidated and doesn’t have time to integrate. Rest is a legitimate and important part of preparation. Use it.

What this plan produces

After thirty days:

• Two archetypes with genuine depth and pattern recognition

• Coverage-level familiarity with four to five additional archetypes

• Two mocks with documented gaps and one specific improvement made to each

• The framework cheatsheet and estimation numbers fresh in working memory

You’ll encounter a question you’re less prepared for. When that happens, apply the framework. The six-step framework works on questions you haven’t drilled — that’s what it’s for. The warmup from four weeks of drilling makes the framework run faster and more automatically under pressure.

The thing most people get wrong

The ratio of reading to drilling in most engineers’ prep is about 80:20. Reading walkthroughs, reading concept explainers, reading other people’s summaries.

The ratio that produces results is closer to 40:60 — 40% reading and learning, 60% timed drills and mocks. The performance gap between knowing something and producing it under pressure in front of someone is real and only closes with practice under pressure.

If you have thirty days and you spend twenty-five of them reading, you’ve done the easy part and skipped the hard part. Do the hard part.

CTA:

Paid subscribers get one full system design walkthrough

every Tuesday — each mapped to its archetype so you build

pattern recognition, not answer memorization.

This week: Google Drive / Dropbox.

https://systemdr.systemdrd.com/subscribe

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

Want the complete learning path?

Unlock advanced modules, case studies, and guided exercises.

Your e-commerce search returns zero results for “running shoes for bad knees” because no product title contains that exact phrase. Your document store can’t surface the policy update most relevant to a user’s complaint, even though it’s clearly related. Your recommendation engine has no idea that a user who loves “Inception” might enjoy “Coherence.”

These failures share a root cause: traditional databases index exact values. A B-tree on a VARCHAR column matches “inception” or it doesn’t. It has no concept of semantic proximity. Vector databases exist to close that gap — storing and querying embeddings so that “similar meaning” becomes a first-class query operation.

Core Concept: What a Vector Database Actually Does

An embedding model (BERT, OpenAI’s text-embedding-3-large, CLIP, etc.) converts your raw data — text, images, audio — into a dense float array. A sentence like “my knee hurts when I run” becomes a 1,536-dimensional vector. Geometrically, semantically similar things cluster together in that high-dimensional space.

A vector database stores those arrays and answers one primary query: given a query vector Q, return the K nearest stored vectors by some distance metric (cosine similarity, Euclidean distance, or inner product).

The naïve approach is brute-force: compute the distance between Q and every stored vector. At 100M rows and 1,536 dimensions, that’s 153.6 billion float operations per query — too slow for interactive latency.

Approximate Nearest Neighbor (ANN) indexes trade a small amount of recall for orders-of-magnitude faster lookups. Both Pinecone and Milvus are fundamentally ANN index managers at their core.

HNSW: The Dominant Index Structure

Hierarchical Navigable Small World graphs are the current standard. Think of it as a multi-layer skip list, but the links between nodes are based on embedding similarity rather than sorted order.

• Layer 0 contains all nodes. Higher layers contain exponentially fewer nodes (probabilistic sampling).

• Insertion places the vector at the appropriate layers, connecting it to its M nearest neighbors at each layer (M is a tunable parameter — typically 16–64).

• Search enters at the top layer, greedily hops toward the query vector, then descends to lower layers for progressively finer traversal until Layer 0.

Search complexity is O(log N) in practice. The efConstruction parameter controls how many candidate neighbors are evaluated during index build (quality vs. build speed). The ef parameter at query time controls the dynamic candidate list size (recall vs. latency).

The non-obvious property: HNSW is not updatable without quality degradation. Deletions mark vectors as “deleted” but leave ghost nodes in the graph. After enough deletes, you need to compact — a full index rebuild.

IVF: The Alternative for Memory-Constrained Systems

Inverted File Index partitions the vector space into K Voronoi cells via k-means clustering. At query time, the search probes the nprobe nearest cluster centroids, then scans only those clusters. This dramatically reduces memory footprint compared to HNSW since you don’t store full graph adjacency.

Trade-off: if your query vector sits near a cluster boundary, the true nearest neighbor might be in an adjacent unprobed cluster. Recall degrades predictably as nprobe decreases.

IVF+PQ (Product Quantization) adds lossy compression of the stored vectors — splitting each vector into sub-vectors and replacing each with a codebook index. You can compress 1,536-dim float32 vectors (6KB each) to ~192 bytes with minimal recall loss. At 100M vectors, that’s the difference between 600GB and 19GB of index memory.

Read more

This is the question that defines:

The surface question sounds manageable. Netflix streams video to 270 million subscribers. How hard can it be to store a file and serve it? The probe underneath is what makes this question a senior filter. The interviewer is not testing whether you know what a CDN is. They are testing whether you understand why naive HTTP file serving collapses at scale, what adaptive bitrate encoding is and why it exists, how a CDN edge network makes cold-start latency irrelevant, and what happens when the most-watched show in the world premieres at 9 PM and 50 million people try to start the same episode simultaneously.

Those four problems — not the basic architecture — are where the round is decided.

For L4 / mid-level: the content storage and delivery pipeline at a high level. For L5 / senior: adaptive bitrate encoding with multiple renditions, CDN edge caching with pre-positioning, and the cold-start problem. For L6 / staff: the encoding pipeline at scale, multi-region active-active CDN strategy, the playback session architecture, quality-of-experience monitoring, and what Netflix actually does differently from a naive CDN deployment.

The Question

“Design Netflix. Users should be able to browse a catalog of movies and TV shows and stream them on demand with high quality and low buffering.”

Common variants:

- “Design YouTube’s video streaming system.”

- “Design Spotify’s audio streaming.”

- “Design Twitch’s live streaming platform.”

- “Design an adaptive bitrate streaming system.”

- “Design the video encoding pipeline for a platform like Netflix.”

The first two are nearly identical to Netflix. Twitch adds a real-time constraint (live, not on-demand) that changes the delivery model but not the encoding fundamentals. Spotify replaces video with audio — smaller files, same adaptive delivery architecture. The encoding pipeline variant focuses specifically on Phase 2 below.

Step 1 — Clarify Before You Draw

Six questions. The first three are load-bearing.

1. On-demand only, or also live streaming? On-demand (Netflix, Spotify) and live (Twitch, live sports) have meaningfully different architectures. On-demand content can be pre-encoded, pre-cached at CDN edges, and optimized offline. Live content has a latency requirement (viewers want near-real-time) that forces a different delivery model. Clarify and scope to on-demand unless told otherwise.

2. What devices must be supported? Smart TVs, mobile (iOS/Android), web browsers, game consoles. This matters because different devices support different codecs (H.264, H.265/HEVC, AV1, VP9) and different DRM systems (Widevine, FairPlay, PlayReady). Acknowledge the codec diversity; don’t try to design the full DRM system in the interview — it’s a separate subsystem.

3. What are the quality and latency SLOs? Netflix targets: playback starts within 2 seconds of pressing play. Buffering events under 0.5% of total play time. 4K HDR delivery available on supported devices. These numbers drive every architecture decision — state them explicitly before drawing anything.

4. What is the catalog size? Netflix has approximately 36,000 titles globally (varies by region). Each title is stored in multiple renditions and formats. State the working assumption: 36,000 titles × 20 renditions × ~4 GB per rendition = ~3 PB of encoded video. This estimate calibrates the storage and CDN design.

5. What scale? 270 million subscribers. Peak concurrent streams: approximately 15–20% of active users simultaneously — roughly 40–50 million concurrent streams. At peak (new season premiere, weekend evening), this spikes significantly. State the peak assumption explicitly.

6. Does the design include recommendations, search, and the catalog UI? These are separate systems (ML recommendation engine, search index, catalog metadata API). Acknowledge they exist; scope the interview to the streaming pipeline — content ingestion, encoding, storage, and delivery. Otherwise you’ll spend 30 minutes on recommendation and never get to the hard part.

Step 2 — Estimate

Working assumptions:

- 36,000 titles × 20 renditions per title = 720,000 encoded files

- Average rendition size: 4 GB (HD, 2-hour movie)

- Total encoded storage: 720,000 × 4 GB ≈ 3 PB of video content

- Metadata storage (titles, posters, descriptions, subtitles): ~5 TB — trivial by comparison - Peak concurrent streams: 50 million

- Average bitrate served: 5 Mbps (mix of HD and 4K streams)

- Peak egress bandwidth: 50M streams × 5 Mbps = 250 Tbps

That 250 Tbps peak egress number is the most important figure in the whole estimate. Netflix is one of the largest sources of internet traffic on the planet — at peak, it accounts for roughly 15% of global downstream internet bandwidth. No single origin infrastructure serves this. CDN edge servers distributed globally are the only viable answer. State this conclusion explicitly from the estimate.

Content ingestion rate: Netflix adds roughly 500–1,000 hours of new content per month. At 20 renditions per title, that’s ~10,000–20,000 encoding jobs per month — a background batch process, not a latency-sensitive path.

Get Access to GitHub Repo

Step 3 — API Design

Four APIs. The streaming API is the non-obvious one.

The senior move here: The manifest_url in the playback start response is the key design signal. You are not returning a video file URL — you are returning a manifest file URL. This distinction is the entire adaptive bitrate streaming architecture. Explain it before moving on.

The heartbeat endpoint is the second senior signal. Netflix does not stream and forget. Every 30 seconds, the player reports back: current position, current quality, buffer health, rebuffer events, estimated bandwidth. This telemetry is the basis for quality-of-experience monitoring, A/B testing new streaming algorithms, and detecting degraded CDN nodes before users notice. Candidates who include the heartbeat signal they’ve thought about the operational layer, not just the happy path.

Step 4 — Data Model

Five stores. Video content itself is not in any of these — it lives in object storage.

titles table — catalog metadata

title_id, title, description, genres[], cast[], director,

release_year, rating, available_regions[], duration_seconds,

thumbnail_url, created_at, updated_at

Sharded by title_id. Regional availability (available_regions[]) drives catalog filtering — a subscriber in Germany sees a different catalog than one in the US because Netflix licenses content per territory.

content_assets table — encoded renditions index

content_id, title_id, rendition_type (4K/1080p/720p/480p/360p),

codec (H264/H265/AV1), bitrate_kbps, file_size_bytes,

storage_uri (S3 path), manifest_uri, duration_seconds,

encoding_status (pending/processing/ready), created_at

The manifest_uri points to the HLS or DASH manifest file in S3 — the index of all segments for this rendition. When the player starts playback, it downloads the manifest first, then requests individual segments.

playback_sessions table — active and recent sessions session_id, user_id, content_id, device_type, started_at,

last_heartbeat_at, current_position_seconds, current_quality,

total_play_time_seconds, rebuffer_count, total_rebuffer_seconds

Not stored in your OLTP database. High-frequency writes (heartbeat every 30 seconds, 50M concurrent sessions = ~1.7M writes/sec). Use Cassandra or a time-series store. Partition by user_id so “resume where I left off” is a single-partition read. Short TTL — you only need the last 30 days of session data for analytics; older data goes to cold storage.

user_watch_history — resume position and completion user_id, title_id, content_id, position_seconds, completed (boolean), watched_at, device_type

This is what powers “continue watching.” Separate from the playback session — a session is live, watch history is the permanent record. Partition by user_id.

CDN content index — not a database table

The CDN’s internal index of what content is cached at which edge location. You don’t design this — it’s the CDN’s internal state. What you do design is the pre-positioning logic: before a major content release (new season of a hit show), Netflix pre-pushes the encoded files to CDN edge servers before playback demand arrives. This is called content pre-warming and it is one of the most important operational patterns in the whole system.

Read more

The standard failure mode: a candidate mentions CAP theorem as a vocabulary signal (”we need to think about the CAP trade-off here”), states the three letters (Consistency, Availability, Partition tolerance), and moves on without connecting it to a concrete decision.

The interviewer writes “mentioned CAP but couldn’t apply it.” The round continues.

Here’s what applying it actually looks like.

What CAP actually says

CAP theorem says: in the presence of a network partition — when some nodes in your distributed system can’t communicate with others — you must choose between consistency and availability. You cannot have both.

Consistency here means: every read returns the most recent write, or an error. Every node agrees on the current state.

Availability means: every request gets a response. It may not be the most recent data, but the system doesn’t error or time out.

Partition tolerance isn’t a choice — it’s a given. If you’re building a distributed system that runs over a real network, partitions happen. Network cables fail. Datacenters lose connectivity. You cannot opt out of partition tolerance. The real choice is always: when a partition happens, do you stay consistent or available?

Subscribe for Payments Walkthrough

Where it actually matters

The question is rarely “should I build a CP or AP system?” That’s too abstract to be useful. The useful version is: for this specific read, how stale can the data be before it causes a real problem?

Some reads tolerate stale data completely. Your Twitter follower count being one minute stale costs nothing. Your Netflix streaming recommendation being 24 hours stale costs nothing. These are candidates for eventual consistency — replicate async, serve locally, accept a brief window of inconsistency.

Some reads cannot tolerate stale data. Your bank account balance must be accurate before a payment is authorized. Your ticket reservation must be current before a seat is sold. These require strong consistency — at the cost of higher latency and potential availability loss during a partition.

The L5 answer isn’t “I’ll use eventual consistency.” It’s: “For the timeline read, I’ll accept eventual consistency — a post appearing 30 seconds late costs nothing. For the payment authorization, I need strong consistency — an incorrect balance check is a real financial loss. These two paths use different consistency models within the same system.”

PACELC: the extension worth knowing

CAP only talks about partition scenarios. PACELC extends the model to the normal case: even without a partition, there’s a trade-off between latency and consistency. If you want every read to reflect the latest write, you need to coordinate across replicas — which adds latency. If you serve from the nearest replica without coordination, you have lower latency but possible stale reads.

Knowing PACELC by name is a mild L6 signal. More importantly, understanding that the consistency/latency trade-off exists even on sunny days is what matters.

The three decisions CAP drives in practice

When you reach the data model step in a design interview, CAP should inform exactly three things:

Replication strategy. Synchronous replication (every write waits for all replicas to confirm) gives strong consistency at the cost of write latency. Asynchronous replication (write confirms when the leader writes, replicas catch up later) gives lower write latency but a brief consistency window.

Read routing. If you route reads to the leader only, you always get the latest data. If you route reads to any replica, you get lower latency but potentially stale data. Pick based on what each read is for.

Conflict resolution. When two nodes accept writes for the same key during a partition, you have a conflict. Last-writer-wins, vector clocks, CRDTs, operational transforms — each is a different answer to the same CAP-induced conflict problem.

The engineers who use CAP well in interviews don’t mention it as a signal. They use it as a decision tool. “For the payment authorization read, I need consistency over availability — I’ll route to the leader and accept higher latency.” That sentence is CAP applied correctly. It doesn’t require saying “CAP” at all.

CTA: The Stripe Payments walkthrough (paid post #2) covers the consistency decisions in a payment system end-to-end — exactly where strong consistency is required, exactly where eventual is acceptable, and why.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

Your three-node Elasticsearch cluster just survived a network blip. Thirty seconds of packet loss between your primary datacenter and a secondary AZ. When connectivity restored, you discovered two nodes had independently accepted writes during the partition. Both believe they are the authoritative master. Both have diverged. This is split-brain — and the next 60 seconds will determine whether your data converges gracefully or you spend the next six hours manually reconciling indexes.

What Split-Brain Actually Is

Split-brain occurs when a distributed cluster partitions into two or more segments that each achieve quorum independently, or when quorum rules are misconfigured to allow it. Each segment continues operating as if it were the whole cluster. Writes flow into both sides. Neither segment knows it is operating on a stale view of the world.

The core problem is divergent state: two leaders, two write paths, two versions of truth. When the partition heals, the cluster must decide which side’s state wins — or whether a merge is even possible.

Why quorum isn’t always enough. Strict majority quorum (N/2 + 1 nodes required) prevents split-brain in theory. In practice, operators often lower quorum thresholds under operational pressure (”we only have 2 of 3 nodes, let’s keep serving”), or they run even-numbered clusters, or they misconfigure minimum_master_nodes (Elasticsearch’s historical footgun). The gap between theoretical safety and deployed reality is where split-brain lives.

Three failure modes:

1. Network partition with symmetric quorum loss — a 2-node cluster loses its tiebreaker. Both nodes independently promote themselves.

2. GC pause masquerading as a failure — a node pauses for 8 seconds (JVM GC, system call blocking), the cluster assumes it’s dead and elects a replacement, then the original node wakes up believing it still holds the lease.

3. Asymmetric network failures — Node A can reach Node B, Node B can reach Node C, but A cannot reach C. Quorum calculations become topology-dependent.

Read more

This is the question that defines the real-time messaging archetype. Ten questions in the question bank are variants of it — Slack, Discord, Zoom, Twitter Spaces, iMessage, Facebook Messenger, Google Chat. They share a spine: a long-lived connection problem, a delivery semantics problem, and a presence problem. Get fluent with all three and you can answer any messaging interview you’re handed.

The surface question looks simple: two people want to send each other messages. The probe underneath is anything but. Interviewers use this question because it exposes how candidates think about distributed systems under failure — specifically what happens when the recipient is offline, when the network drops mid-message, when the same message arrives twice, and when a user has five devices. Those four failure modes, and your response to them, are where the round is decided.

For L4 / mid-level: the basic chat flow and message storage. For L5 / senior: delivery semantics under unreliable networks (at-most-once, at-least-once, exactly-once), presence at scale, and offline message handling. For L6 / staff: the multi-device consistency model, group message fan-out at WhatsApp’s scale (2 billion users), end-to-end encryption architecture, and what happens when a message is delivered to 256 group members across 12 time zones while half of them are offline.

The Question

“Design WhatsApp. Users should be able to send messages to each other in real time. Messages should be delivered even when the recipient is temporarily offline.”

Common variants:

• “Design Facebook Messenger.”

• “Design iMessage / Apple Messages.”

• “Design Slack’s messaging core.”

• “Design a real-time chat system.”

• “Design the messaging layer for [our product].”

All five are the same architecture. Slack adds channels and threads; the delivery model is identical. iMessage adds end-to-end encryption; the transport is identical. Discord adds voice and server-based channels; the one-to-one messaging core is identical.

Step 1 — Clarify Before You Draw

Six questions. Two of them are load-bearing for the entire design.

1. One-to-one only, or also group chats? One-to-one is the simpler problem. Group chats add fan-out — sending one message to N recipients — which at WhatsApp’s scale means up to 256 recipients per group. State which you’re designing for. For L5+: design for both, but start with one-to-one and explicitly extend.

2. What are the delivery guarantees? This is the load-bearing question. Three options:

• At-most-once: messages might be lost but never duplicated. Fine for ephemeral signals (typing indicators). Terrible for chat.

• At-least-once: messages are definitely delivered but might be delivered more than once. The receiver handles deduplication. Acceptable for most chat systems.

• Exactly-once: messages are delivered exactly once, regardless of retries. Theoretically ideal. In practice, you can approximate it with idempotency keys and deduplication at the receiver — but you cannot achieve it perfectly across an unreliable network. Saying “exactly-once is impossible in a distributed system” out loud is a senior signal.

WhatsApp and most production systems implement at-least-once delivery with client-side deduplication. State this explicitly.

3. What devices does each user have? Multiple devices per user (phone, tablet, web, desktop) massively complicates the model. A message sent to a user needs to land on all their active devices. On which device does the “read receipt” trigger? Clarify this early.

4. Offline handling — how long do you store messages for an offline recipient? If a recipient is offline for 30 days, do their messages accumulate on your servers? Most production systems store for 30 days and drop after that (with notification to sender). Clarify the retention window.

5. End-to-end encryption? E2EE (like WhatsApp’s Signal Protocol implementation) means the server cannot read message content — it only stores encrypted ciphertext and routing metadata. This changes the data model significantly. State whether you’re designing with E2EE or not. For L5: acknowledge it exists and where it fits. For L6+: explain the key exchange flow.

6. What scale? 2 billion monthly active users is WhatsApp’s actual scale. For the interview, pick a working number: 500M DAU, 100 billion messages per day (WhatsApp’s reported figure as of its scale). These numbers drive your storage and throughput estimates.

Step 2 — Estimate

Working assumptions:

• 500M daily active users

• 100 billion messages per day = ~1.16 million messages per second average, ~5M/sec peak

• Average message size: 100 bytes (text only), 500 bytes with metadata

• 30-day message retention for offline recipients

• 1 billion messages stored at any moment (roughly 1% of daily volume is “pending delivery”)

• Media (images, video, voice): handle separately via object storage (S3 / CDN pre-signed URLs); only metadata and a pointer lives in the chat system

Storage for messages: 100B messages/day × 500 bytes × 30 days retention ≈ 1.5 PB. This is your “hot” storage — messages waiting to be delivered or recently delivered. After 30 days, messages drop off the hot store. Delivered messages can be stored client-side and expunged from the server (this is exactly what WhatsApp does — server deletes after delivery confirmation).

Connection state: 500M users × ~30% concurrently connected = ~150M concurrent WebSocket connections. At 64KB per connection socket buffer that’s roughly 9.6 TB of connection memory across your fleet. This is one of the hardest scaling problems in messaging — the number of open connections.

Write throughput: 1.16M messages/sec average into a message store. This rules out single-node anything. You need a distributed message queue and a sharded persistence layer.

Get Access to GitHub Repo

Step 3 — API Design

Four endpoints. The real-time delivery is not REST — it’s a persistent connection.

WebSocket: wss://chat.whatsapp.com/connect

Auth: Bearer token in Upgrade request headers

Purpose: The long-lived connection for receiving messages and sending keep-alives. All message delivery happens here.

On connect: server sends any pending (undelivered) messages accumulated while the client was offline.

The senior move on the API step: Name the Idempotency-Key and message_id in the same breath. The message_id is client-generated before the send — if the client retries because it didn’t receive an ack, the server detects the duplicate via message_id and does not re-queue. The Idempotency-Key is the HTTP-level dedup. Both are necessary: the HTTP key handles transport-level retries, the message_id handles application-level deduplication across devices and reconnections.

Saying that distinction out loud — two levels of deduplication for two different failure modes — is the L6 signal on this question.

Read more

Learn System Design with System building, Subscribe Hands On coding course - LogStream

There’s a version of the system design interview where the candidate designs Twitter. The entire Twitter. Timeline, search, notifications, recommendations, ads, analytics, direct messages.

At the 40-minute mark they’re still drawing boxes. They haven’t gone deep on anything. The interviewer writes “breadth over depth, no clear prioritization” on the scorecard.

This is the most common failure mode among technically strong candidates, and it has nothing to do with technical knowledge. It’s a scoping problem.

Here’s the discipline that fixes it.

The negotiation in the first five minutes

Most candidates use the first five minutes to ask clarifying questions. How many users? What’s the read:write ratio? Does this need global availability?

These are fine questions. They’re also generic — they could be asked for any question. They don’t signal to the interviewer that you understand how to manage scope.

The candidates who score well on problem scoping don’t just ask questions. They propose scope.

“There are several interesting subsystems here: the timeline, search, notifications, the recommendation algorithm, and the ad serving layer. I’m going to focus on the timeline and the fan-out architecture. I’ll acknowledge the others exist and sketch where they’d connect, but I won’t design them in detail. Does that direction work for you?”

This is different from asking questions. You’re proposing what you’re going to design and explicitly not design, and you’re inviting the interviewer to redirect if they want something different.

Most interviewers will say yes. Some will redirect you: “Actually, I’m more interested in the fan-out problem for group chats specifically.” This is useful information you would not have gotten without proposing scope first.

Subscribe for WhatsApp Walkthrough

Why the pressure to cover everything exists

The instinct to design everything is understandable. The interview is evaluating your technical breadth, right? If you only design one component, won’t the interviewer think you don’t know the rest?

This misunderstands what the interview is evaluating.

Interviewers at L5+ levels are not checking whether you’ve memorized the architecture for every subsystem of Twitter. They know you’re familiar with notifications and search — you work as a software engineer. What they can’t infer from your resume is whether you can identify the hardest distributed systems problem in a question, scope down to it, and go deep enough to demonstrate real understanding.

Designing eight systems shallowly proves you’re familiar with eight systems. Designing one system deeply proves you can think through the hard parts of a complex problem. The second is what they’re scoring.

How to choose what to scope in

The right component to go deep on is the component with the most interesting distributed systems problem. Not the component you know best. Not the component that feels safest. The component where the answer is genuinely non-obvious.

For Twitter: the fan-out problem is more interesting than the user profile service. The timeline cache architecture is more interesting than the follow/unfollow API. Go where the hard problem lives.

A useful test: which component, if I described it at a surface level, would the interviewer immediately probe deeper on? That’s the component to start with.

What to do with the components you’re not designing

You don’t ignore them. You acknowledge them.

“Notifications would follow a standard fan-out-on-write pattern with push to APNs/FCM for mobile and WebSocket for web, similar to what we discussed for the timeline. I’m not going to design it in detail, but the interesting problem there is the delivery guarantee under failure — we’d need at-least-once delivery with client-side dedup.”

One sentence that shows you understand the problem, the standard solution, and the interesting edge case — without spending five minutes on a full design. This signals breadth without consuming depth time.

The specific framing that works

After the estimation section, before drawing anything, say this out loud:

“Before I draw the architecture, I want to lock scope. I’m going to go deep on [component X] because that’s where the interesting distributed systems problem is. I’ll acknowledge [components Y and Z] exist and show where they connect, but I won’t design them fully unless you’d like me to. Does that work?”

The explicit scope lock serves two purposes. It sets expectations so the interviewer isn’t waiting for you to cover components you’ve already decided to skip. And it gives the interviewer one last opportunity to redirect before you’ve committed 30 minutes to the wrong area.

After this moment, the scope is locked. Don’t drift. If you finish the deep dive early, go deeper on the component you chose — don’t add new components to fill time.

The uncomfortable truth about breadth

Demonstrating breadth is valuable. Demonstrating it through shallow coverage of many components is not the right way.

The right way: show breadth through the quality of your acknowledgments. When you mention a component you’re not designing and you correctly name the problem it has and the standard approach for solving it, you’ve shown breadth in one sentence. Compare that to spending five minutes on a shallow design of the same component — same signal, five minutes cheaper.

Scope discipline isn’t about knowing less. It’s about deploying what you know efficiently enough to demonstrate depth where it matters.

CTA: Tuesday’s paid post is the full WhatsApp walkthrough — including how to scope it correctly when the question says “design a messaging system” and you have 50 minutes.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

Curious how this works in real systems?

The paid lessons cover scalability, optimization, and real-world engineering patterns.

Sponsored

Build Distritubed Systems using System Design concepts with hands-on coding lessons every day - System Design Course

Preparing for a distributed systems interview?

→ Get the Complete System Design & DSA Framework

→ Deep-Dive: Troubleshooting Distributed Java Systems

Introduction

Your production system just went dark at 2:47 AM. Services are returning 503s, customers are screaming, and your on-call engineer is frantically searching through logs scattered across 200 containers on 40 nodes. Without a centralized log aggregation system, every second spent grepping individual machines is a second your system stays broken. Log aggregation isn’t just a nice-to-have — it’s the difference between a 4-minute MTTR and a 4-hour incident.

Core Concept: What Log Aggregation Actually Does

At its core, log aggregation solves a distributed systems problem: your logs are generated everywhere, but your engineers need to query them from one place. Every log aggregation system shares the same pipeline, regardless of branding: collect → ship → parse → index → query.

The architectural divergence happens at indexing and query layers, and those differences have enormous operational and cost consequences at scale.

The Collection Layer is where agents like Filebeat, Promtail, or Splunk Universal Forwarder tail log files or consume stdout/stderr from containers. These agents buffer locally (critical for handling backpressure), add metadata (hostname, container ID, Kubernetes labels), and ship upstream. The key failure mode here is the “silent drop” — when the downstream ingestion endpoint is overwhelmed, poorly configured agents discard logs rather than applying backpressure. You lose exactly the logs you need most: the ones generated during an incident when traffic spikes.

The Ingestion Layer (Logstash, Fluentd, or Splunk Indexers) performs parsing, enrichment, and routing. This is where unstructured log lines become structured JSON documents. At scale, this layer becomes a bottleneck. A single Logstash pipeline on modest hardware handles roughly 10,000-50,000 events/second. Facebook’s internal systems process petabytes of logs daily — they’ve replaced traditional parsing pipelines with stream processors that apply schema-on-read rather than schema-on-write, deferring expensive parsing until query time.

The Storage and Index Layer is where the three major architectures diverge fundamentally.

ELK Stack (Elasticsearch, Logstash, Kibana)

Elasticsearch uses an inverted index on every field by default. When you ingest a log line, Elasticsearch tokenizes every string value and creates a lookup from token → document IDs. This makes full-text search blazing fast — querying “ERROR database connection” across 100GB of logs takes milliseconds. The cost: indexing overhead is enormous. Elasticsearch uses approximately 3-10x the raw log size in index storage. A 1TB/day log pipeline requires 3-10TB of SSD storage — and Elasticsearch performs poorly on spinning disks.

The shard architecture adds operational complexity. An index is split into shards, and each shard is a self-contained Lucene index. Too few shards and you bottleneck on single-node write throughput. Too many shards (over-sharding) causes cluster state overhead that degrades all operations. The golden ratio that works in most production clusters: shard size between 10-50GB, with 1-2 primary shards per node’s CPU core count.

Splunk

Splunk is architecturally similar to ELK but adds a proprietary compressed index format and a more powerful query language (SPL). Its key differentiator is schema-on-read: Splunk extracts fields at search time rather than index time. This makes ingestion faster but search slower for first-time queries. Splunk’s SmartStore tiering (hot/warm/cold to S3) is genuinely excellent for cost management — organizations commonly achieve 70-80% storage cost reduction by moving 90-day+ data to object storage.

The hidden cost is licensing. Splunk’s volume-based pricing ($50-150/GB/day for enterprise licenses) makes it the most expensive option at scale. At 500GB/day ingestion, annual costs can exceed $10M. This is why large organizations often run Splunk for security/compliance workloads (where its SIEM capabilities justify cost) and migrate general application logs to ELK or Loki.

Grafana Loki

Loki takes a fundamentally different approach: it only indexes labels, not log content. A Loki stream is identified by a set of key-value labels ({app="payments", env="prod"}). The log lines themselves are stored compressed in chunks in object storage (S3, GCS) without content indexing.

This makes Loki dramatically cheaper — storage costs are close to raw log size, often 10-20x cheaper than Elasticsearch at equivalent volume. The trade-off is query performance. To search log content, Loki must decompress and scan chunks sequentially. A grep-style query across 24 hours of logs on a single service is fast (it finds the right chunks via label index). A query across all services for a specific string pattern requires scanning everything — and is orders of magnitude slower than Elasticsearch for that access pattern.

Loki’s architecture is intentionally Kubernetes-native. Its label model mirrors Prometheus labels, making it trivially easy to correlate metrics and logs on the same infrastructure. The logcli tool and LogQL query language will feel immediately familiar to anyone using PromQL.

Critical Insights

1. Cardinality Is the Killer (Loki): Loki’s index contains all unique label combinations. If you add user_id or request_id as a label, you explode cardinality — potentially millions of unique streams. This causes memory exhaustion in the ingester and dramatically degrades query performance. Keep label cardinality under 10,000 unique values per label dimension. User IDs belong in log content, not labels.

2. Elasticsearch Split-Brain Is Underestimated: When the cluster loses quorum (network partition, node failure during rolling restart), Elasticsearch can enter split-brain: two cluster halves both believe they are primary. Both accept writes. When the partition heals, one side’s data is lost. The mitigation is setting minimum_master_nodes = (N/2)+1 — but many teams run with defaults and discover this during their worst incident.

3. Logstash Memory Leaks in Pipeline Restarts: Logstash uses the JVM with a persistent queue on disk. During unclean restarts (OOM kill, SIGKILL), the queue can become corrupted. Events between the last checkpoint and the crash are reprocessed (duplicates) or lost (if queue corruption is severe). Switching to Filebeat + direct Elasticsearch ingest pipelines eliminates Logstash as a single point of failure — and reduces the pipeline’s memory footprint by 60-80%.

4. The Hot Shard Problem (Elasticsearch): If all your services use time-based index names (logs-2025.01.15), all write traffic hits today’s index. The shards for today’s index receive 100% of writes while yesterday’s shards sit idle. Index lifecycle management (ILM) with rollover policies (roll when shard hits 30GB or 24h) distributes writes more evenly and prevents the “hot shard” bottleneck.

5. Retention Policy Mismatch: Most teams set a single retention policy. Production reality: security logs need 1-year retention (compliance), application debug logs need 7 days (cost), and business metrics need 90 days (analytics). ELK’s index-per-service-per-retention-tier architecture and Loki’s label-based bucket policies both support this — but require explicit configuration. Defaulting to uniform retention is a significant hidden cost.

6. Structured Logging Multiplier Effect: Teams that enforce structured JSON logging (every log line is valid JSON with consistent field names) get dramatically better results from any aggregation system. Field extraction at parse time is zero-cost. Ad-hoc grok pattern matching on unstructured text adds 30-50% CPU overhead to Logstash/Fluentd pipelines and produces inconsistent results when log formats change.

Real-World Examples

Uber’s Migration Away from Elasticsearch: Uber ran one of the largest ELK deployments in the industry — 600TB+ of log data. They hit fundamental limits on reindexing speed during schema migrations and cluster-wide query performance degradation during peak traffic. Their solution was a hybrid: ELK for recent 7-day hot data (fast search), Apache Hive on HDFS for cold storage (cheap retention). This pattern — tiered storage with different query semantics per tier — is now standard at hyperscale.

Cloudflare’s Loki Adoption: Cloudflare handles ~50 million HTTP requests/second across their edge. Their logging pipeline using Loki with label-based routing reduced storage costs by 85% compared to their previous Elasticsearch deployment, at the cost of accepting slower ad-hoc content searches. For their use case (known-structure queries, metric correlation, alerting), this trade-off was acceptable.

Splunk for Security, Not Application Logs: JPMorgan Chase and similar financial institutions maintain Splunk exclusively for SIEM workloads — security event correlation, compliance reporting, and anomaly detection where SPL’s power and Splunk’s security content library justify the cost. Application performance logs run on ELK with cold-tier archival to S3 via Searchable Snapshots.

Architectural Considerations

Log aggregation integrates directly with your alerting layer — Kibana/Grafana can trigger PagerDuty alerts on log patterns, but this creates a hidden coupling between storage availability and incident response. If your log cluster is unhealthy during an incident, you lose both visibility and alerting simultaneously. The mitigation: route critical structured events (errors, exceptions) to a separate, lightweight stream (Kafka + minimal consumer) that feeds alerting independently of the full log pipeline.

Cost modeling matters here. Elasticsearch at 100GB/day ingestion runs approximately $15,000-25,000/month on cloud infrastructure when accounting for multi-replica storage, hot node compute, and snapshot costs. Loki on the same workload runs $2,000-5,000/month. The break-even analysis depends entirely on your query access patterns — if your team runs full-text searches across all services constantly, Elasticsearch’s performance advantage may justify the cost.

Practical Takeaway

GitHub Link

https://github.com/sysdr/sdir/tree/main/Log_Aggregation_at_scale/log-aggregation-demo

The right log aggregation architecture depends on your query patterns, not your log volume. Full-text search across diverse log formats → Elasticsearch. Metric-correlated log tailing on Kubernetes → Loki. Compliance and security analytics with budget → Splunk.

Run the demo:

bash

bash setup.sh

The demo spins up a local Loki + Grafana stack alongside a simulated Elasticsearch cluster, with a synthetic load generator producing structured log streams. You’ll see:

• Label cardinality impact on Loki query performance in real time

• Elasticsearch index hot-shard behavior under write load

• Side-by-side query latency comparison for identical workloads

Extend the demo by modifying load-generator/config.json to increase cardinality (add user_id as a Loki label) and watch query performance degrade. This is the single most important failure pattern to understand before deploying Loki in production.

Youtube Demo Link:

This is the question that defines the marketplace and matching archetype. Ten questions in the question bank are variants of it — DoorDash delivery, Airbnb booking, Tinder matching, parking reservation, surge pricing, ride-share fraud. They share a spine: a supply side, a demand side, a location component, and a real-time matching requirement. The architecture that works for Uber dispatch, with minimal modification, works for all ten.

The surface question — “design Uber” — is a 4-hour answer. Interviewers narrow it deliberately: “design the dispatch system,” “design how a driver gets matched to a rider,” “walk me through what happens between the rider hitting ‘Request’ and the driver getting a notification.” All three collapse to the same probe: how do you find the nearest available supply in real time, at scale, without burning down your database on every request?

The answer involves a data structure most candidates haven’t touched since their algorithms course: geospatial indexing. Specifically, quadtrees or geohashing — and knowing which one and why is the senior signal.

For L4 / mid-level: describe the matching flow and a naive approach. For L5 / senior: articulate the geospatial indexing choice with its trade-offs. For L6 / staff: the consistency model across supply/demand state, the surge pricing data flow, multi-city scaling, and what happens when a driver’s GPS signal drops.

The Question

“Design the Uber dispatch system. A rider requests a ride. The system finds nearby available drivers and assigns one to the rider.”

Common variants:

• “Design DoorDash / UberEats food delivery dispatch.”

• “Design Lyft’s matching system.”

• “Design a system to match drivers to delivery orders in real time.”

• “Design the driver-to-rider matching at Uber.”

All four are the same architecture. DoorDash has one additional dimension — the restaurant (a fixed supply point instead of a moving one) — but the matching logic is identical.

Step 1 — Clarify Before You Draw

Six questions. Two of them are load-bearing for the whole design.

1. What does “dispatch” include — match only, or also route and surge? Dispatch in the narrow sense is “find a driver and assign them to a rider.” Routing (which road to take) is a separate system (Google Maps API or in-house). Surge pricing (dynamic fare) is another separate system. Clarify scope explicitly. Design the matching layer; acknowledge the others exist.

2. How many active drivers? What’s the geography — one city or global? Active drivers in a single city: tens of thousands. Globally: millions. This shapes whether your geo-index fits in memory on one machine or needs distribution. For a US metro at peak, 10,000–50,000 active drivers is a reasonable working number. Global scale: 5–10M.

3. How frequently do drivers update their location? Every 4–5 seconds is typical. This is a high-frequency write stream — 50,000 drivers × 1 write/4s = ~12,500 writes/sec in one busy metro. Globally: millions of writes/sec. State this number; it drives the design of the location ingestion layer.

4. What’s the matching latency SLO? A rider hits “request” and expects a driver to accept within 30–60 seconds. The matching decision itself should happen in milliseconds — sub-100ms for the initial candidate selection. If it takes 5 seconds to find a candidate, the experience breaks.

5. Can a driver be matched to multiple requests simultaneously? No — a driver has one state: available or not. This is the consistency problem at the heart of the question. Two riders cannot be dispatched to the same driver. The system that manages this state is load-bearing.

6. What failure modes matter most? Driver goes offline mid-trip. GPS signal drops. Driver declines the match. Rider cancels after match. Each one requires a state machine transition. State the three most important ones out loud.

Step 2 — Estimate

Working assumptions for a single large metro deployment:

• 50,000 active drivers at peak

• 200,000 ride requests per hour at peak = ~55 ride requests/sec

• Driver location update: every 4 seconds, so 50,000 / 4 = 12,500 location writes/sec

• Matching query: for every ride request, find all drivers within ~2 km radius

• Average matching candidate set: 20–50 drivers

• End-to-end dispatch latency target: < 100ms for candidate selection, < 30s for driver acceptance

Location data per driver per update: ~100 bytes (driver_id, lat/lng, timestamp, status).

Writes: 12,500/s × 100 bytes ≈ 1.25 MB/s. Trivial for a single machine.

If global (5M drivers): 5M / 4 = 1.25M writes/sec. Serious — needs distributed location ingestion. But for this walkthrough, start with the single-metro case and note the global extension.

The geo-index in memory for 50,000 drivers: 50,000 × 100 bytes = 5 MB. Fits entirely in memory on a single machine. This insight — that the active driver set for one city is tiny — is what enables the in-memory geo-index approach.

Get Access to GitHub Repo

Step 3 — API Design

Three endpoints. The first is rider-facing, the second is driver-facing, the third is the internal matching call.

The senior move on location updates: note that the driver app batches and sends at 4-second intervals, not on every GPS tick (which might be 1-second intervals). Batching reduces write load by 4x and extends driver battery life. Saying this out loud — that the client has a role in write amplification — is an insight most candidates miss.

heading and speed_mps in the location update. Junior candidates send only lat/lng. Senior candidates send velocity vector too. Why: ETA estimation to a pickup point requires knowing not just where the driver is but which way they’re going and how fast. A driver 500m away facing the wrong direction on a one-way street is further than a driver 700m away pointed at you. The heading/speed fields feed the ETA model.

Step 4 — Data Model

Four stores. The geo-index is the unusual one.

drivers table — source of truth for driver state

driver_id, name, vehicle_type, license_plate, rating_avg, total_trips,

current_status (ENUM: available/on_trip/offline/suspended),

current_lat, current_lng, last_location_update, created_at

Sharded by driver_id. The current_status and current_lat/lng columns are updated frequently — this is the consistency-critical state. More on that in Step 6.

rides table — the ride state machine

Ride state machine: searching → matched (driver found) → accepted (driver acknowledged) → driver_en_route → arrived → in_progress → completed | cancelled

Any state can transition to cancelled. The driver_id is only set at the matched transition. If a driver declines, status reverts to searching and the matching process re-runs.

location_history — time series store

Not your OLTP database. Location events are time-series data: (driver_id, lat, lng, timestamp, heading, speed). At 12,500 events/sec, that’s 1 billion events/day in one metro. You don’t store this in Postgres. You use a time-series database (InfluxDB, TimescaleDB, Cassandra with a time-ordered key) or push to a data lake for analytics.

The hot query (”where is this driver right now?”) hits the geo-index, not this table. This table is for analytics, replay, and dispute resolution (”I was at X at time T — prove it”).

driver_geo_index — the in-memory geo-index

This is the heart of the system. Not a SQL table. An in-memory spatial index that answers “which drivers are within radius R of point (lat, lng) in sub-millisecond time.”

Two real options: quadtrees or geohashing. This is Step 6’s deep dive.

Read more