A user opens the app, watches a video for 1.4 seconds, and swipes away. That single swipe has to update what they see next — not in the next session, not in the next hour, in the next scroll. Most recommendation systems built in the 2010s couldn’t do that. They retrained overnight, served a static ranked list, and called it personalization. TikTok’s “For You” page broke that model by treating every watch, pause, replay, and skip as a live signal that reshapes the next candidate pool within seconds. That’s the part worth understanding — not the neural net architecture, which nobody outside the company has verified, but the systems problem of making feedback loops that fast without falling over.

The Core Problem: Candidate Generation Under a Latency Budget

A “For You” feed request has roughly 100–200 milliseconds to go from “user opened app” to “here are 10 ranked videos.” In that window, the system has to narrow a catalog of hundreds of millions of videos down to a few hundred candidates…

Read more

Two-sided marketplace matching with asymmetric information (you don’t know who liked you until you match). The interesting system design problem is the “like” storage — when user A likes user B, how do you efficiently detect that B has also liked A (a mutual match)?

Step 1 — Data Model

profiles: user_id, age, gender, location (geohash), photos[], bio, preferences{...} likes: liker_id, liked_id, created_at

UNIQUE INDEX: (liker_id, liked_id)

matches: match_id, user_id_1, user_id_2, created_at

Get Access to Github Repo

Step 2 — Match detection

At scale: the mutual-like check is a hot query path. Cache it in Redis: SET likes:{B}:{A} 1 on every like. Match detection = GET likes:{B}:{A} — O(1), < 1ms.

Read more

Name two failure modes you haven’t fixed and the conditions under which you’d revisit the design.

Not generic failure modes. Specific ones from your design.

“The two things I’d revisit: first, if the celebrity threshold for switching from fanout-on-write to fanout-on-read turns out to be lower than 1M followers in practice — I’d tune it based on actual write amplification metrics after the first week in production. Second, if the change_log grows faster than anticipated — I’d add a compaction job earlier than the 30-day window I assumed.”

Subscribe for Question Walkthrough

This demonstrates:

1. You understand your design is an approximation, not a solution.

2. You’ve already thought about the monitoring signals that would tell you it’s failing.

3. You know how to respond when it fails.

That’s the entire difference between a candidate who has read about systems and a candidate

who has been on-call for them.

CTA: The cohort curriculum goes beyond textbook designs — training you to proactively identify specific failure modes, establish clear production monitoring thresholds, and articulate exact trigger conditions for revisiting architecture before an interviewer even asks.

Subscription link

https://systemdr.systemdrd.com/subscribe

This free lesson is designed to get you started.

The paid content helps you master the topic with structured deep dives.

A legal-tech startup shipped a contract-summarization feature in late 2024. It cited “Section 8.3(b)” of an NDA that had no Section 8.3(b) — the document only ran to Section 6. Nobody caught it before a paying customer did, in front of their own lawyer. The model wasn’t broken. It was doing exactly what autoregressive generation does: producing the most statistically plausible next token, with no built-in mechanism that checks whether “8.3(b)” actually exists in the source text. That gap between fluent and true is the entire reason guardrail and verification layers exist.

What a Verification Layer Actually Does

A hallucination isn’t a bug in the traditional sense — there’s no stack trace, no exception, no failed assertion. The model produces well-formed, confident-sounding text that happens to be false, fabricated, or unsupported by the input it was given. Guardrails don’t fix the model. They wrap it, checking outputs (and sometimes inputs) against something more reliable than the model’s own confidence.

There are three broad layers, and most production systems use some combination of all three.

Input-side guardrails constrain what the model is allowed to reason about in the first place. Retrieval scoping, strict system prompts, and schema-constrained generation (forcing JSON output against a defined schema) all reduce the surface area for fabrication before generation even starts.

Output-side verification checks what came out. The simplest version is a grounding check: does every factual claim in the response trace back to a span in the retrieved context? This is usually implemented as a second LLM call — a smaller, cheaper model reads the source documents and the generated answer, then flags claims it can’t locate. Some teams use natural language inference (NLI) models instead, scoring entailment between source and claim rather than doing a full generative re-read, which is faster and cheaper per call but less able to catch subtle misattributions.

Structural constraints enforce format and citation requirements at generation time rather than after the fact. Forcing every factual sentence to end with a bracketed source reference, then programmatically verifying that reference resolves to a real chunk ID, catches the NDA problem directly — an unresolvable citation is a hard signal, not a probabilistic guess.

The trade-off nobody likes to say out loud: every layer you add increases latency and cost, and none of them get you to zero. A verification layer running on a second model call typically adds 300–900ms and roughly doubles your per-request token cost. You’re buying a lower hallucination rate, not immunity from it.

Subscribe to Access Private Github Repo

Read more

Logs are high-volume, write-once, read-rarely data. The system must ingest millions of log lines per second, store them durably, and make them searchable within seconds. The tension: write throughput (sequential append) vs search (inverted index, which requires knowing content in advance).

Step 1 — Clarify

- Ingestion rate: 1M log lines/sec (large deployment)

- Average log size: 500 bytes → 500 MB/sec ingestion bandwidth

- Search SLO: search across last 24h of logs within 5 seconds

- Retention: 30 days hot (searchable), 1 year cold (archive)

- Structured (JSON) logs or unstructured (free text)?

Get Access to Github Repo

Read more

Here’s the complete framework for talking about indexes in an interview — specific enough to signal expertise, concise enough to not derail the conversation.

B-tree index (the default)

A B-tree index stores values in sorted order as a balanced tree. Lookups, range queries, and ORDER BY operations all use it efficiently.

Use for: equality queries (WHERE user_id = 123), range queries (WHERE created_at BETWEEN x AND y), sorting (ORDER BY created_at DESC).

The B-tree is the default index type in Postgres, MySQL, and most relational databases. When you say “I’d add an index” without specifying a type, this is what you’re getting.

Hash index

A hash index maps each value to a hash. Lookups by exact equality are O(1). Range queries are impossible — the hash has no ordering.

Use for: pure equality lookups where range queries will never be needed. Session ID lookups. Token validation. Cache key lookups.

In practice: Redis and most key-value stores use hash indexes internally. Postgres supports hash indexes but they’re rarely the right choice over B-tree because B-tree handles both equality and range queries while hash handles only equality.

Subscribe for Question Walkthrough

Composite index

A composite index covers multiple columns: CREATE INDEX ON orders (user_id, status, created_at).

The critical rule: composite indexes follow the left-prefix rule. This index serves queries on (user_id), (user_id, status), and (user_id, status, created_at). It does NOT serve queries on (status) alone or (created_at) alone.

Use for: queries that always filter by a leading column and sometimes filter by subsequent columns. “Give me all pending orders for user X from the last 7 days” — (user_id, status, created_at) is the right index.

The order of columns in a composite index matters: put the most selective columns first if you want to cover a range of queries, or put the equality predicates before the range predicates.

Partial index

A partial index covers only a subset of rows, defined by a WHERE clause.

CREATE INDEX ON orders (created_at) WHERE status = 'pending'

This index only stores rows where status is ‘pending’. It’s much smaller than a full index on created_at. Queries that filter by status = 'pending' and then sort by created_at will use this index. Queries that don’t filter by status won’t.

Use for: high-cardinality tables where most queries filter on the same condition. “Show pending orders” is a common query; “show completed orders” is rare. Index the pending subset specifically.

Covering index

A covering index includes all columns needed to satisfy a query, so the database can answer the query entirely from the index without touching the underlying table.

If your query is SELECT user_id, status FROM orders WHERE user_id = 123, an index on (user_id, status) is a covering index — it has both columns. The query engine never touches the orders table rows.

The performance difference is significant: a covering index means one I/O operation (read the index), not two (read the index, then follow the pointer to the table row for each result).

Use for: frequently-run queries on large tables where you want to eliminate the table access entirely. Add the extra columns at the cost of a slightly larger index.

How to talk about indexes in an interview

The right pattern: identify the hot queries first, then derive the indexes.

“The hot query is: all pending orders for a given user, sorted by creation time. I’d create a composite index on (user_id, status, created_at). This serves the user_id equality, the status equality, and the created_at range all from one index. I’d make it a partial index filtered to status=’pending’ to reduce the index size since orders spend very little time in pending state at scale.”

Four sentences. Names the query. Names the index type. Names the column order reasoning. Names the optimization.

That’s the L5 format for any index discussion. L6 adds: name the trade-off (write overhead — every INSERT and UPDATE on the indexed columns must also update the index) and the conditions under which you’d remove it (if write performance degrades and the query is no longer hot).

CTA: The cohort curriculum goes beyond surface-level statements — walking through exact index selection for high-throughput queries, ordering composite keys, leveraging partial and covering indexes to drop table read I/O, and evaluating the write-amplification trade-offs at scale.

Subscription link

https://systemdr.systemdrd.com/subscribe

Interested in mastering this topic? Unlock premium content with deeper explanations and real implementation patterns.

A team at a mid-size fintech shipped a support-ticket summarizer in March. It worked fine in staging. Three days after launch, one customer’s integration retried a failed webhook every 400 milliseconds, each retry firing a fresh call to Claude. The bill for that one afternoon came out to $4,200. Nobody had built a rate limiter in front of the model calls, because in staging nobody needed one. That’s the whole argument for an AI gateway in six sentences.

What a gateway actually does

An AI gateway is a proxy layer sitting between your application code and the model providers — Anthropic, OpenAI, whoever. It’s not a new idea; it’s the same reverse-proxy pattern API teams have used for REST services since the 2000s, aimed at a workload that happens to be metered by tokens instead of requests. The gateway does three jobs: it limits how fast callers can hit the model, it tracks what every call costs in near-real time, and it decides what happens when a caller exceeds their budget or the provider throttles you back.

The rate-limiting piece almost always uses a token bucket, not a fixed window counter. Fixed windows have a well-known edge case: a caller can burn their entire quota in the last second of one window and the first second of the next, doubling their effective rate at the boundary. A token bucket avoids that. Each API key gets a bucket that holds up to N tokens (say, 50) and refills at a steady rate (say, 10 per second). Every request costs some number of tokens — often the actual LLM token count, not a flat “1 request = 1 token” charge, since a 50-token prompt and a 4,000-token prompt do very different amounts of work downstream. Request comes in, gateway checks the bucket, if there’s enough capacity it deducts and forwards the call; if not, it returns a 429 immediately, before the request ever reaches the model provider.

That last detail matters more than it sounds. Rejecting locally, in Redis, costs you a few milliseconds. Rejecting after the provider call has already gone out costs you the full latency of that call — and, worse, you’ve already been billed for the tokens. A gateway that rate-limits after the fact isn’t a rate limiter, it’s a cost report.

Cost tracking runs in parallel. Every response from the provider comes back with usage metadata — input tokens, output tokens, sometimes cached tokens billed at a different rate. The gateway multiplies those against a price table (which you have to keep updated by hand or by scraping the provider’s pricing page, because there’s no standard API for “tell me your current prices”) and writes a ledger entry: which API key, which model, how many tokens, what it cost. Aggregate that ledger by key, by team, by hour, and you get a live spend dashboard instead of a surprise on the monthly invoice.

Where this gets subtle

Common knowledge: everyone building on LLM APIs eventually adds some rate limiting, usually a naive per-minute counter, usually after a bill scare.

Rare knowledge: token buckets need per-key buckets, not one global bucket, or your best customer gets throttled because a different customer is misbehaving. Getting this wrong is the single most common gateway bug — I’ve seen it ship to production at three different companies.

Advanced insight: at real scale, the bucket state itself becomes a bottleneck. A single Redis instance doing a check-and-decrement on every request for thousands of keys per second will show up as your p99 latency floor before the model call even starts. Sharding buckets by key hash, or using Redis Cluster with client-side hashing, fixes it — but now you’ve got a distributed rate limiter, which has its own consistency headaches (two gateway nodes racing to decrement the same bucket).

Strategic impact: cost control isn’t just finance hygiene. A gateway that tracks spend per feature lets you kill a runaway feature without killing the whole product — set a hard budget cap on the “auto-summarize” endpoint specifically, and when it hits zero, that endpoint degrades to a cached or non-LLM fallback while checkout, search, and everything else keeps working. Without per-feature attribution, your only lever is an all-or-nothing kill switch on the whole gateway.

Implementation nuance: decide up front whether rate limits apply to requests or tokens per minute. Most providers rate-limit you on both, separately, and your gateway’s limits should mirror that shape — otherwise you’ll pass your own gateway’s check and get throttled by the provider anyway, which is a confusing failure mode to debug at 2 a.m.

Read more

A task queue decouples work submission from work execution. The key systems design problems: at-least-once vs exactly-once delivery, task visibility timeout (what happens if a worker dies mid-task), retry with backoff, and dead-letter queues.

Step 1 — Clarify

- At-least-once or exactly-once task execution? (At-least-once is standard — tasks must be idempotent)

- Priority queues? (Yes — critical tasks jump the queue)

- Task timeout (maximum execution time before considering a worker dead)? - Retry policy (max attempts, backoff strategy)?

- Dead-letter queue for permanently failed tasks?

Step 2 — Data Model

tasks table:

task_id (UUID), queue_name, payload (JSON), status

(queued/in_progress/completed/failed/dead),

priority (int), attempt_count, max_attempts, visibility_timeout_at, next_attempt_at, created_at, completed_at, error_message

Get a Access to Github Repo

Read more

Step 1: Buy 90 seconds with clarifying questions.

You don’t need to know the answer before you ask the questions. “Let me make sure I understand the scope — are we designing for real-time delivery or batch? How many users at peak?” You’re not stalling — you’re genuinely narrowing the problem while your brain maps the question to the nearest archetype.

Subscribe for Question Walkthrough

Step 2: Name the archetype out loud.

“This sounds like it has the same core tension as a feed fanout system — lots of writes amplifying into lots of reads, with a latency constraint.” If you’re right, the interviewer will nod. If you’re wrong, they’ll redirect. Either way, you’ve signaled you have a framework.

Step 3: Run the 6-step framework on the archetype, not the specific question.

If you’ve identified it as a “real-time messaging” problem, run the messaging framework — WebSocket, delivery semantics, offline handling — and adapt the names to fit the new question.

Step 4: Go deep on the hard part, not the familiar parts.

If you’ve never seen the question before, the interviewer knows it. They’re not expecting a polished answer. They’re watching whether you can reason under uncertainty. Go to the hardest distributed systems problem in the question and show that you can think through it, even imperfectly.

What not to do: Don’t say “I haven’t seen this question before.” It signals that you were expecting to pattern-match, not think. Even if the question is new, treat it as a normal design round and work through it.

CTA: The cohort curriculum is built around recovering when the question is unfamiliar — clarifying the scope, identifying the right system archetype, applying a proven framework, and going deep on the hardest distributed-systems trade-off instead of relying on pattern-matching.

Subscription link

https://systemdr.systemdrd.com/subscribe

Want the full picture?

The premium lessons include advanced insights, practical exercises, and comprehensive walkthroughs.

A customer uploads a 40-second phone video of a leaking pipe and types “is this covered under my policy?” One request, three payloads: a text question, an audio track, and roughly 1,200 video frames. Most teams building this feature start by wiring the video into a vision model, the audio into a transcription service, and the text straight to the LLM, then stitching the three outputs together in a prompt. It works in the demo. It falls apart at 200 requests per second, when the video encoder queue backs up, the transcription service returns four seconds after the vision model already timed out, and the LLM ends up reasoning over a transcript-shaped hole. Multi-modal system design is mostly about not letting that happen.

The Core Problem: Modalities Don’t Move at the Same Speed

A multi-modal system ingests inputs of fundamentally different sizes, formats, and processing costs, then produces a unified response. The naive architecture treats this as three parallel single-modal pipelines that converge at the end. The problem is that text tokenizes in microseconds, a 512×512 image embeds in 30–80ms on a decent GPU, and a 40-second audio clip takes 1–3 seconds to transcribe even with a fast ASR model. If your orchestrator waits for all three before calling the LLM, your p99 latency is set by your slowest modality, every time, regardless of how fast the other two are.

The fix isn’t a faster audio model. It’s decoupling ingestion from fusion. Each modality gets its own preprocessing pipeline — resize and encode for images, chunk and transcribe for audio, tokenize for text — running independently and writing normalized output to a shared store (commonly Redis or a lightweight message queue keyed by request ID). A fusion layer watches for all required pieces to arrive, or a timeout to expire, before assembling the final prompt.

That timeout decision is where most of the design tension lives. Do you wait for the slow modality, or degrade gracefully and answer with partial context? Insurance claims can’t guess at what’s in a video. A customer support chatbot answering “what’s this error mean” from a screenshot plus text can often proceed on text alone if the vision pipeline is backed up, flagging the response as partial. The right answer is domain-specific, but the architecture has to support both paths — most teams only build the happy path and discover the gap during an incident.

Fusion itself has two common strategies. Early fusion projects every modality into a shared embedding space (think CLIP-style joint embeddings) before any reasoning happens — this is what lets a model compare “a photo of a dog” against an actual photo directly. Late fusion keeps modalities separate through their own specialized models and only combines outputs at the decision layer, e.g., concatenating a transcript, an image caption, and the user’s text into a single LLM prompt. Late fusion is easier to build and debug, since each component fails independently and visibly. Early fusion captures cross-modal relationships that late fusion misses entirely — a joint embedding model can tell you the audio tone doesn’t match the words, while three separate pipelines never make that comparison at all.

Read more

Reddit’s front page is community-aggregated content ranked by the “hot” algorithm. The interesting system design question is: how do you maintain a ranked list across thousands of subreddits with millions of concurrent voters, without recomputing the full ranking on every vote?

The Hot Algorithm

This decays old posts naturally without a cron job — the time term shrinks the relative contribution of votes as a post ages. An hour-old post needs 10× more upvotes to rank above a brand-new post.

Architecture

- Vote events:

User upvotes → Kafka → vote aggregation worker → update posts.score in the posts table

- Front page:

Precomputed per-subreddit sorted lists in Redis ZSETs, score = hot algorithm value. On vote: re-score the post and ZADD with new score. Front page read = ZREVRANGE /r/all 0 24 — O(log N)

- The r/all problem:

r/all aggregates top posts from all subreddits. At peak, millions of posts compete. Maintain a separate ZSET for r/all, updated when any subreddit post’s score enters the top-1000 of its subreddit

- New posts:

Use a separate new ZSET per subreddit sorted by created_at. New tab = ZREVRANGE on this set. No algorithm needed.

- Award/controversy:

Track controversiality score (high upvotes AND high downvotes) in a separate sorted set for the controversial sort

Get Access to GitHub Repo

Read more

It’s the Google Docs question. Design a real-time collaborative editor.

Here’s why it’s the hardest — and why that makes it the most valuable one to understand deeply even if you never get asked it directly.

Why it’s different from every other question

Most system design questions have a “store data, serve data” structure underneath. Messages are stored and delivered. Files are stored and synced. Videos are stored and streamed. The interesting problems are about doing these operations reliably at scale.

Collaborative editing is different. The problem isn’t storage or delivery. It’s merging — two users type simultaneously into the same document, and both changes must survive in a consistent order on both clients, in real time, without either user’s edits being lost.

This is a fundamentally different problem. Storage and delivery can be solved with good distributed systems primitives. Merging concurrent edits requires a different class of solution: Operational Transformation or CRDTs.

Subscribe for Question Walkthrough

The naive approaches fail interestingly

Most candidates start with one of two intuitions.

The first: “the server is the source of truth — last write wins.” This silently destroys data. User A types “Hello” and user B types “World” simultaneously. Whichever write arrives at the server last overwrites the first. One user loses their edit invisibly. Not acceptable.

The second: “the server merges conflicting writes.” This sounds right until you try to implement it. What does “merge” mean for concurrent text edits? If A inserts a character at position 5 and B deletes a character at position 3 simultaneously, what’s the correct merged result? The positions have shifted. The server cannot naively apply both operations in sequence.

This is where OT comes in.

What Operational Transformation actually does

Each edit is represented as an operation: insert(position, character) or delete(position). When two operations are generated concurrently — before either user has seen the other’s edit — they must be transformed against each other before both can be applied.

The classic example: document is “Hello.” User A inserts “ World” at position 5. User B simultaneously inserts “!” at position 5.

After A’s operation, position 5 has shifted — “!” should now be inserted at position 11. The transform function adjusts B’s operation before applying it. Result: “Hello World!”

The server maintains a history of all operations. When it receives an operation, it transforms that operation against all operations the client hasn’t yet seen, then broadcasts the transformed version to all clients. The document converges to the same state on all clients.

Why this matters for questions you will get asked

Understanding OT deeply unlocks several adjacent concepts:

The operation log as a data model. An OT-based system stores operations, not document state. The document is the result of replaying all operations from the beginning. This is event sourcing — the same pattern used in payment ledgers, audit trails, and any system where history matters more than current state.

The difference between mutable and immutable data models. Most system design candidates default to mutable state (update the row when something changes). The operation log is immutable — you only ever append. Understanding why immutable append-only logs are often better than mutable state is one of the higher-order concepts in distributed systems.

CRDTs as an alternative. Conflict-free Replicated Data Types solve the same merging problem differently — by designing data structures where all operations commute (can be applied in any order and produce the same result). Notion uses CRDTs. Google Docs uses OT. Both work.

If you understand one of these deeply, you understand a design principle that appears in message delivery systems, distributed databases, version control systems, and collaborative tools.

The Google Docs question is hard because it requires understanding a class of problem — concurrent state merging — that doesn’t appear in most backend engineering work. But once you understand it, you see it everywhere.

The full walkthrough is in the Vault under Real-Time Messaging. The drill card is the most complex card in the set. Both are there when you’re ready to go deep.

CTA: The cohort curriculum is built around closing exactly these three gaps — state merging, immutable operation logs, and real-time OT mechanics

Subscription link

https://systemdr.systemdrd.com/subscribe

The deeper concepts start where this lesson ends.

Explore the premium content for advanced techniques and system-level thinking.

A team at a 30-person fintech startup wires up a LangChain agent to answer support tickets: check account balance, look up a transaction, escalate if fraud-flagged. Day two in production, one agent run calls the “check balance” tool 41 times in a row before the request times out at 60 seconds. Nobody touched the code. The model just got stuck restating the same thought and re-invoking the same tool, and the framework had no built-in ceiling to stop it. That’s not a bug report you’ll find in the docs. It’s the default behavior of an orchestration loop with no guardrails, and it’s why understanding what’s actually happening under AgentExecutor.invoke() matters more than knowing the API surface.

The Core Mechanism

Strip away the abstractions and both frameworks are doing the same basic thing: turning a sequence of LLM calls, tool calls, and data transformations into a graph, then walking that graph. LangChain’s LCEL (LangChain Expression Language) builds this as a static DAG — you pipe a prompt into a model into an output parser using the | operator, and the shape of that graph is fixed at definition time. A chain built this way runs the same three or four nodes every time, in the same order, regardless of input. That’s a chain, and it’s the easy 80% case.

Agents are different, and this is where most of the confusion (and most of the production incidents) live. An agent doesn’t have a fixed graph. Instead, it runs a loop: the LLM receives the user’s goal plus a scratchpad of everything done so far, decides what to do next, and the framework parses that decision, executes it, appends the result to the scratchpad, and calls the LLM again. This is the ReAct pattern (Reason + Act) that both LangChain’s AgentExecutor and LlamaIndex’s ReActAgent implement almost identically under different class names. The graph is decided at runtime, one node at a time, by the model itself.

LlamaIndex frames this slightly differently at the retrieval layer. Its QueryEngine abstraction splits the pipeline into a retriever (fetch candidate nodes from a vector index), a node postprocessor chain (rerank, filter by similarity threshold, deduplicate), and a response synthesizer (combine retrieved chunks into a final answer, using strategies like compact — stuff everything into one prompt — or tree_summarize — recursively summarize in a binary-tree pattern when context won’t fit). Each stage is swappable, which is the actual value proposition: you’re not buying “RAG,” you’re buying a slot machine of interchangeable retrieval and synthesis strategies you can A/B test without rewriting glue code.

The part vendors don’t advertise: the output parser sitting between “LLM emits text” and “framework executes an action” is doing enormous, fragile work. The LLM outputs a string. Something has to turn Action: search_transactions\nAction Input: {"account_id": "4471"} into an actual Python function call with actual arguments. That parser is a regex or a JSON-mode contract, and it breaks constantly — trailing commas, an extra sentence before the JSON blob, a tool name the model slightly misremembers. When it breaks, the default behavior in most setups isn’t to fail loudly. It’s to catch the exception, shove “Invalid format, try again” back into the scratchpad, and loop. That’s your 41-call incident.

Read more

The API gateway is a cross-cutting infrastructure component that every FAANG-tier system uses. The question tests whether you understand authentication, rate limiting, routing, and load balancing as composable middleware layers — and what the operational challenges are when this component is in the critical path of every API call.

Step 1 — Clarify

- What features: auth, rate limiting, routing, load balancing, caching, request transformation, observability?

- Scale: 1M req/sec aggregate across all upstream services

- Multi-tenant (different customers, different routing rules) or single-tenant? - SLO: gateway latency overhead < 5ms p99 (adds < 5ms to every request)

Step 2 — Architecture

The middleware chain (executed in order for every request):

1. TLS termination — decrypt HTTPS at the gateway; upstream services communicate in plaintext within the datacenter

2. Authentication — validate API key or JWT token. Token validation: symmetric JWT (verify signature with shared secret, ~0.5ms) or asymmetric (fetch public key from auth service — cache aggressively, 24h TTL)

3. Rate limiting — per API key, per endpoint, sliding window counter in Redis (see Rate Limiter walkthrough)

4. Request routing — match URL pattern to upstream service. Stored in a routing table (Redis or in-memory hash map, updated via config push)

5. Load balancing — pick a healthy upstream instance (round-robin, least-connections, or consistent hash for sticky sessions)

6. Request forwarding — proxy the request to the upstream; handle upstream timeout and retry

7. Response transformation — add CORS headers, strip internal headers, compress response

8. Observability — emit request log, latency metric, status code metric to monitoring pipeline

Configuration plane: Routing rules, rate limits, auth policies are stored in a config database (etcd or Postgres) and pushed to gateway instances via pub-sub. Gateways cache config in-memory; config staleness of 30 seconds is acceptable.

Health checking: Gateway maintains a health map of all upstream instances. Active health checks (HTTP probe every 5s) + passive health checks (circuit breaker on error rate). Unhealthy instances removed from the load balancing pool.

Get Access to GitHub Repo

Read more

But the path to the interview room, the logistics surrounding it, and the context after an offer are different in ways that matter. Here’s what actually differs.

What’s genuinely different

The remote-first hiring pipeline. Most FAANG-tier companies now conduct all interviews remotely. This is an advantage for India-based engineers — you’re no longer at a geographic disadvantage for the initial loop. The downside: time zones. A loop with a US-based company typically means some interviews between 7:30 PM and midnight IST. This is worth planning around — don’t schedule a round at midnight if you have a critical work deadline the next morning.

Level calibration. A “Senior Engineer” at a well-regarded Indian tech company (Flipkart, Meesho, Swiggy, Juspay) maps to L4–L5 at Google/Meta depending on tenure and scope. An “architect” or “lead” at a larger Indian IT firm (Infosys, Wipro, TCS) typically maps to L4. This isn’t a judgment — it’s a calibration issue. Know your actual level going into the interview, not just your title.

The PPP pricing reality. FAANG-tier base salaries for India-based roles are significantly lower than their US equivalents — typically ₹40L–₹1.2Cr TC for L4–L6, versus $200K–$600K TC in the US for equivalent levels. This is expected and not negotiable in the same way US offers are. The negotiating leverage for India-based roles is typically in joining bonus and ESOP vesting schedules, not base.

Visa and relocation. If you’re targeting a US relocation, the current H-1B backlog for Indian nationals is 10–15 years. This is a real constraint. Some engineers time their loop specifically to target companies with large India offices or a clear path to a Canada/Singapore transfer as an intermediate step. Worth being explicit about your relocation preferences early in the recruiter conversation, not after the offer.

What isn’t different

The technical bar. Identical. The system design questions are the same. The rubric is the same. The expected depth for L5 in Bangalore is the expected depth for L5 in Seattle.

The preparation path. Identical. Cold drills, walkthroughs, mock interviews with peers. The prep that works in the US works in India. This newsletter covers both.

The failure modes. Identical. Scoping too broadly, communicating at the wrong level, not handling failure modes — the same mistakes fail the round everywhere.

The timeline pressure. Identical. Most engineers underestimate how long a loop takes. Initial screen, technical screen, onsite loop, offer, negotiation, background check — 6–12 weeks is typical. Plan accordingly.

One practical difference in preparation

The pool of practice partners in India is large but unevenly distributed. Engineers at Flipkart, Meesho, PhonePe, Razorpay, and similar companies have usually seen these interviews and can give calibrated mock feedback. Engineers at service companies may not have. Be deliberate about who you practice with — the value of a mock is entirely in the quality of the feedback.

The Discord has a regional channel specifically for India-based members. The #india channel is where timezone coordination, India-specific company intel, and INR pricing questions go. Post there if you’re looking for mock partners in IST.

The INR pricing

This newsletter’s paid content is available at ₹399/month or ₹2,999/year. The Vault is ₹9,999. The Cohort is ₹49,999. These are PPP-adjusted prices — the same content, different checkout link. Email with subject “INR pricing” and I’ll send the local currency link.

The technical content is identical regardless of which pricing you use.

CTA: The cohort curriculum is built around closing exactly these three gaps — each module targets one of them directly.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

The free lesson gives you the overview.

The premium version shows you exactly how to apply it step by step

Date Notice: Today is September 25, 2026. This article reflects knowledge verified through January 2026. Specific benchmark numbers, chip specs, and framework version details should be checked against current vendor documentation before you cite them anywhere serious.

A Pixel phone running a 7B parameter model on-device sounds impossible until you check the math twice. Full precision, that model needs 28GB of memory just to hold the weights — more RAM than the phone has, period. Ship the same model at 4-bit precision and it drops to roughly 3.5GB. Suddenly it fits, with room left for the KV cache and the rest of the OS. Nothing about the model’s architecture changed. The numbers representing its weights got smaller. That’s quantization, and it’s the reason on-device AI stopped being a demo and started being a product category.

Two Different Knives

Quantization and pruning both shrink a model, but they cut in different places, and conflating them causes real design mistakes.

Quantization reduces the numerical precision used to store and compute weights and activations — typically FP32 or BF16 down to INT8, INT4, or even lower. A weight of 0.7328491 stored in FP32 takes 32 bits. Quantized to INT8 with a scale factor, it might round to a value representable in 8 bits, reconstructed at inference as scale × int_value. You lose precision, not parameters. The model keeps every neuron, every connection — they just get expressed more coarsely.

Pruning removes parameters entirely. Unstructured pruning zeroes out individual weights below some magnitude threshold, producing a sparse matrix. Structured pruning removes whole channels, attention heads, or layers, which is the version that actually speeds things up on standard hardware, because unstructured sparsity needs specialized kernels (NVIDIA’s 2:4 sparsity on Ampere+ GPUs being the main mainstream exception) to translate into real latency wins. Without that hardware support, an unstructured-pruned model still runs the same dense matmuls, just with a lot of the multiplications wasted on zeros.

The two compose. A common pipeline: prune a model structurally to cut parameter count 30-40%, fine-tune to recover accuracy, then quantize the survivors to INT8 or INT4 for the final size and latency win. Doing it in the other order — quantize then prune — usually produces worse results, because pruning decisions made on a lower-precision model are less reliable; small weight magnitudes get harder to distinguish from quantization noise.

Read more

Time-series data at massive scale. The interesting problem is write throughput (millions of metrics per second from thousands of servers) vs query throughput (interactive dashboards needing sub-second response over months of data). These two requirements pull the architecture in opposite directions — you optimize writes by sequential append, you optimize range queries by pre-aggregation.

Step 1 — Clarify

- What types of data: metrics (numeric time-series), logs (text events), traces (distributed request spans)? Scope to metrics.

- Ingestion rate: 1M metrics/sec (single large deployment)

- Query SLO: dashboard queries over last 24h should return in < 2 seconds - Retention: 15-second resolution for 24h, 1-minute resolution for 30 days, 1-hour resolution for 2 years

- Alerting required? (Yes — threshold-based and anomaly-detection alerts)

Step 2 — Estimate

- 1M metrics/sec × 8 bytes/datapoint = 8 MB/sec write throughput

- At 15-second resolution: 1M × 4 datapoints/min × 60min × 24h = 5.8B datapoints/day - At 8 bytes/point: 46 GB/day raw. Over 30 days: 1.4 TB

- With rollup (15s → 1min → 1h): ~50GB/month per compression tier

Get Access to GitHub Repo

Step 3 — Data Model

Time-series storage: Each metric is identified by a metric name + tag set (e.g., cpu.usage{host=web-01, region=us-east}). Stored as:

- Metric metadata: metric_id → {name, tags, retention_policy}

- Datapoints: (metric_id, timestamp) → value — ordered by timestamp, stored in a write-optimized time-series DB (InfluxDB, Prometheus TSDB, or custom LSM-tree)

Rollup tables: Three separate tables — raw (15s resolution), hourly (pre-aggregated to 1-minute), daily (pre-aggregated to 1-hour). Background jobs roll raw data up periodically, dropping the raw data after the retention window.

Read more

Here’s how to tell them apart and when to reach for each one.

What they have in common

All three decouple producers from consumers. Instead of Service A calling Service B directly, A puts a message somewhere and B reads it when it’s ready. This makes A and B independently scalable, independently deployable, and independently failure-tolerant.

The decoupling is the shared property. Everything else differs.

Kafka: streaming and fan-out

Kafka is a distributed log. Producers append messages to topics. Consumers read from topics at their own pace. Messages are retained for a configurable period (days or weeks), regardless of whether anyone has read them.

The key properties that distinguish Kafka:

Replay. A consumer can re-read old messages. If you deploy a new service and need to process all historical events, Kafka lets you rewind to the beginning. SQS and task queues don’t.

Fan-out. Multiple independent consumer groups can read the same topic simultaneously. Each group maintains its own offset. Service A and Service B can both consume the same stream of events without interfering with each other. SQS and task queues don’t support this natively.

Ordering. Messages within a partition are strictly ordered. If order matters — event sourcing, audit logs, ledger updates — Kafka preserves it.

Use Kafka when: you have multiple consumers that need to read the same events, you need replay capability, you’re building a streaming pipeline where order matters, or you need to decouple high-throughput event producers from slower consumers.

Don’t use Kafka when: you need guaranteed exactly-once delivery of individual work items to exactly one worker, or when you have a small team that can’t operationally support a Kafka cluster.

Subscribe for Question Walkthroughs

SQS: simple at-least-once work distribution

SQS is a managed queue. A producer enqueues a message. A consumer polls, processes, and deletes it. The message is invisible to other consumers during processing (the visibility timeout). If the consumer crashes, the message reappears.

The key properties:

Simplicity. Fully managed by AWS. No cluster to operate. You pay per message.

At-least-once delivery. Each message is delivered to at least one consumer. Duplicates are possible (rare but possible). Your consumer must be idempotent.

One consumer per message. Each message is processed by exactly one consumer group. No fan-out. If you need multiple services to process the same message, you need SNS in front of SQS (one topic, multiple queues).

Use SQS when: you need a simple, reliable queue for distributing work to a pool of workers, you’re on AWS, and you don’t need replay or fan-out.

Task queues (Celery, Redis Queue, Postgres FOR UPDATE SKIP LOCKED)

Task queues are purpose-built for job execution. A task has a type, a payload, a status, a retry count, and a deadline. Workers claim tasks atomically, process them, and mark them complete. Failed tasks retry with backoff.

The key properties:

Task lifecycle management. Task queues track the status of individual work items. You can query “which tasks are in progress right now” or “which tasks have failed more than 3 times.” Kafka and SQS don’t natively track per-item status.

Retry with backoff. If a task fails, it’s retried after an exponential delay. Dead-letter queuing for tasks that fail past the max retry count. This is the production pattern for background job processing.

Priority. Tasks can have priority levels. High-priority tasks jump the queue. Standard queues don’t support this natively.

Use a task queue when: you’re running background jobs (email sending, image processing, report generation), you need per-task status visibility, you need retry with backoff, or you need priority queuing.

The interview decision

In a system design interview, the choice should follow from the requirement:

Need multiple services to react to the same events? → Kafka
Need to offload work from a web request to a background process? → SQS or task queue
Need to build a streaming analytics pipeline? → Kafka
Need to retry failed jobs with backoff and track their status? → Task queue
Need a managed, simple work queue with no infra overhead? → SQS
Need replay capability for new consumers or debugging? → Kafka

The mistake to avoid: defaulting to Kafka for everything. Kafka is operationally heavy (unless you’re using Confluent Cloud or a managed service). For a small-scale job queue, Postgres with FOR UPDATE SKIP LOCKED is simpler, cheaper, and perfectly adequate up to millions of jobs per day.

State your choice, name the alternative you’re not using, and explain why. That’s the format for any technology decision in this category.

CTA: The full walkthroughs go beyond knowing what Kafka, SQS, or task queues are. They show you how to make the decision in an interview — what requirement should drive the choice, which alternative to rule out, and how to explain the tradeoff like a Senior vs Staff engineer.

If you found this valuable, the full library includes 52 walkthroughs, bonus lessons, exercises, and detailed interview breakdowns.

https://systemdr.systemdrd.com/subscribe

In March 2023, a fraud-scoring team at a mid-size payments processor pushed a retrained model straight to 100% of transaction traffic on a Friday afternoon. The new model had a better offline AUC — 0.94 versus 0.91. Nobody checked what that 3-point gain did to the false-positive rate on transactions over $2,000. By Monday, the support queue had 1,400 tickets from customers whose legitimate purchases got blocked, and the on-call engineer spent six hours rolling back a change that took four minutes to deploy. This is the exact failure canary releases and shadow mode exist to prevent.

Core Concept

Canary releases route a small, controlled slice of live traffic to a new model version while the rest keeps hitting the current one. You start at 1% or 5%, watch the metrics that matter — latency, error rate, business KPIs like approval rate or click-through — and ramp up in steps: 5% → 25% → 50% → 100%, usually over hours or days, not minutes. If anything degrades past a threshold, you roll back the percentage to zero instead of rolling back a deployment.

Shadow mode is a different animal. Every request still goes to the production model and that response is what the user gets. But the same request also gets duplicated — usually via an async fan-out — and sent to the candidate model in parallel. The candidate’s output never reaches the user. It gets logged, compared against the production model’s output, and scored on agreement rate, latency, and resource cost. Shadow mode carries zero user-facing risk, which is exactly why teams underuse canary and overuse shadow when they should combine both.

Read more

The web crawler question tests distributed systems breadth: politeness (don’t hammer a single server), deduplication (don’t re-crawl the same URL twice), scheduling (recrawl priority), and parsing at scale. The interesting distributed systems problem is managing a frontier of billions of URLs across a cluster of crawlers without duplication.

Step 1 — Clarify

Step 2 — Estimate

Get Access to GitHub Repo

Step 3 — Key Components

URL Frontier: A distributed priority queue of URLs to crawl. Priority based on: page importance (PageRank estimate), content freshness, recrawl deadline. Implemented as a two-level structure: an in-memory heap of high-priority URLs, backed by a disk-based queue (Kafka or a custom LSM-tree store) for the full frontier.

Deduplication: Before adding a URL to the frontier, check if it’s already been crawled. Use a Bloom filter (fast, probabilistic, O(1) membership check) to catch already-seen URLs. False positives acceptable (skip a valid URL) but false negatives not (avoid re-crawling). Secondary deduplication: content hash of the fetched page to detect mirror pages.

Politeness policy: Per-domain rate limiting. Maintain a domain_last_crawled map. Enforce minimum interval between requests to the same domain (e.g., 1 request/sec per domain). Respect robots.txt — fetch and cache per domain on first visit.

DNS pre-resolution: DNS lookup adds latency to every fetch. Pre-resolve and cache DNS for all domains in the frontier. TTL-matched to DNS record TTL.

Read more