The web crawler question tests distributed systems breadth: politeness (don’t hammer a single server), deduplication (don’t re-crawl the same URL twice), scheduling (recrawl priority), and parsing at scale. The interesting distributed systems problem is managing a frontier of billions of URLs across a cluster of crawlers without duplication.

Step 1 — Clarify

Step 2 — Estimate

Get Access to GitHub Repo

Step 3 — Key Components

URL Frontier: A distributed priority queue of URLs to crawl. Priority based on: page importance (PageRank estimate), content freshness, recrawl deadline. Implemented as a two-level structure: an in-memory heap of high-priority URLs, backed by a disk-based queue (Kafka or a custom LSM-tree store) for the full frontier.

Deduplication: Before adding a URL to the frontier, check if it’s already been crawled. Use a Bloom filter (fast, probabilistic, O(1) membership check) to catch already-seen URLs. False positives acceptable (skip a valid URL) but false negatives not (avoid re-crawling). Secondary deduplication: content hash of the fetched page to detect mirror pages.

Politeness policy: Per-domain rate limiting. Maintain a domain_last_crawled map. Enforce minimum interval between requests to the same domain (e.g., 1 request/sec per domain). Respect robots.txt — fetch and cache per domain on first visit.

DNS pre-resolution: DNS lookup adds latency to every fetch. Pre-resolve and cache DNS for all domains in the frontier. TTL-matched to DNS record TTL.

Read more

Name two failure modes you haven’t fixed and the conditions under which you’d revisit the design.

Not generic failure modes. Specific ones from your design.

“The two things I’d revisit: first, if the celebrity threshold for switching from fanout-on-write to fanout-on-read turns out to be lower than 1M followers in practice — I’d tune it based on actual write amplification metrics after the first week in production. Second, if the change_log grows faster than anticipated — I’d add a compaction job earlier than the 30-day window I assumed.”

This demonstrates:

1. You understand your design is an approximation, not a solution.

2. You’ve already thought about the monitoring signals that would tell you it’s failing.

3. You know how to respond when it fails.

That’s the entire difference between a candidate who has read about systems and a candidate who has been on-call for them.

This is just the beginning. The premium content includes practical projects, frameworks, and deeper technical insights.

CTA: The Question Vault is organized by company and archetype, so you can focus your prep where it matters. Each drill card shows the specific questions, patterns, and probes to prioritize for your target company. All 52 walkthroughs are in the Vault.

https://systemdr.systemdrd.com/subscribe

—Sumedh

Subscribe Now

(Knowledge cutoff note: registry tooling in this space — MLflow, Vertex AI Model Registry, SageMaker Model Registry — moves fast. Version numbers and feature sets below reflect general architecture, not a specific release; check current docs before treating any tool-specific detail as current.)

What a Registry Actually Does

A model registry is a system of record for trained models: it tracks every version of every model, the metadata that describes it (framework, training data hash, evaluation metrics, who trained it and when), and — the part people underestimate — its current lifecycle stage. Stages are typically something like None → Staging → Production → Archived, and the registry enforces transitions between them rather than just labeling them.

Read more

Unlike most system design questions, fraud detection has two hard constraints that conflict: latency (fraud checks must complete in < 100ms inline with the payment flow) and accuracy (false positives block legitimate transactions — costly; false negatives allow fraud — also costly). The architecture must serve both.

Step 1 — Clarify

- Inline (synchronous, blocks payment) or async (payment proceeds, flag for review later)? - Correct answer: both. Hard blocks for obvious fraud, async review for suspicious. - What signals are available: card data, user history, device fingerprint, IP, merchant category?

- False positive tolerance: rejecting 1% of legitimate transactions costs real revenue - Target: catch > 99% of fraud while blocking < 0.1% of legitimate transactions

Step 2 — Data Model

Get a Access to Github Repo

Step 3 — Architecture (two-layer)

Layer 1 — Inline rules engine (< 10ms): Hard rules that block obviously fraudulent transactions:

Implemented as: rules stored in Redis (fast read), evaluated in-memory per transaction. No ML — pure boolean rules. Fast enough to be fully synchronous.

Read more

After building the 52-question bank, I went back and tagged each question by which company most commonly asks it and at what frequency. The pattern is clearer than I expected.

Here’s what it shows — and what it means for your prep if you have a specific company in your loop.

Amazon

The most-asked question at Amazon is a variant of marketplace matching or recommendation. Design the product search system. Design the recommendation engine. Design the shopping cart and inventory reservation.

The probe Amazon reaches for most consistently: what happens at scale? The interviewers are looking for whether you’ve thought about the system failing under load, not just the happy path. Estimation with consequences is weighted heavily at Amazon because of the Well-Architected Framework culture internally.

If you have an Amazon loop: prioritize Uber Dispatch (geospatial matching), Amazon Product Search, and the Distributed Task Queue. The infrastructure archetype is also heavily represented at Amazon — URL shortener, API gateway, rate limiter.

Google

Google asks the hardest technical questions in the bank. The most-asked questions are in the search and infrastructure archetypes: web crawler, search autocomplete/typeahead, Google Docs (collaborative editing), distributed file systems.

The probe Google reaches for: can you go deep on the hard distributed systems problem? Google interviewers typically don’t probe breadth — they probe depth on one component until you hit the edge of your knowledge.

If you have a Google loop: prioritize the hardest question in each archetype (CRDT/OT for collaborative editing, consistent hashing for caches and sharding, bloom filters for crawlers). You won’t need to cover 10 questions in depth — you need to be able to go very deep on whichever question they ask.

Meta

Meta asks social feed questions at a much higher rate than other companies, which makes sense given the products: Facebook feed, Instagram feed, Stories, Reels. The fan-out architecture in all its variants is the most commonly asked pattern.

The probe Meta reaches for: the celebrity problem and how you handle it. Large-account fan-out is a real operational challenge at Meta’s scale, and every interviewer there has direct experience with the tradeoffs.

If you have a Meta loop: be extremely solid on Twitter Timeline (the canonical fanout question), Instagram Feed (adds ML ranking on top), TikTok-style recommendation (adds ANN search), and WhatsApp/Messenger (their messaging infrastructure).

Stripe

Stripe’s questions are almost all in the money movement archetype. Payments, fraud detection, ledger design, rate limiting for their API.

The probe Stripe reaches for: correctness under failure. Idempotency, double-entry ledger design, exactly-once-or-at-least-once semantics, what happens when a webhook fails and retries. Stripe engineers have to think hard about these problems in their actual work, and the interview reflects that.

If you have a Stripe loop: be very solid on Stripe Payments (especially idempotency keys), payment fraud detection, and distributed rate limiting.

Apple

Apple asks a lot of file sync and storage questions — makes sense given iCloud. Also streaming media (Apple Music, Apple TV+) and messaging (iMessage).

If you have an Apple loop: prioritize Dropbox/Google Drive, iCloud Photos, Spotify Audio Streaming (same patterns as Apple Music), and WhatsApp (same core as iMessage).

The pattern across all companies

Every company clusters around the products they actually build. The hardest preparation mistake is treating all 52 questions as equally likely.

If you have a specific company in your loop and 3 weeks to prepare, the most efficient path is: identify the 2–3 archetypes most relevant to that company, go deep on those archetypes, and do one solid drill per archetype. Then use the remaining time for your weakest archetype as a hedge.

Breadth matters less than targeted depth at this stage of prep.

CTA: The Question Vault is organized by company and archetype, so you can focus your prep where it matters. Each drill card shows the specific questions, patterns, and probes to prioritize for your target company. All 52 walkthroughs are in the Vault.

https://systemdr.systemdrd.com/subscribe

—Sumedh

Ready to level up?

Subscribe now

You train a model offline. It performs brilliantly on test data. You deploy it. A week later, predictions drift — not because the model is wrong, but because the feature user_purchase_count_last_30d computed during training used a batch job that runs at 2 AM, while the production API computes the same feature on-the-fly using a different SQL query on a slightly different dataset. Same name. Different logic. Silent divergence.

This is training-serving skew, and it’s responsible for more production ML failures than model architecture ever will be. Feature stores exist to eliminate it — and that’s just the first problem they solve.

Core Concept: What a Feature Store Actually Is

A feature store is a data system purpose-built for one job: computing, storing, and serving ML features consistently across training and inference. It sits between your raw data sources and your models, acting as a shared registry and serving layer.

The architecture splits into three planes:

1. The Feature Registry (Metadata Layer) Every feature has a definition — the transformation logic, data source, owner, freshness requirement, and schema. The registry stores these definitions as code (not documentation). Tools like Feast use Python-defined FeatureView objects that describe how to compute a feature from a data source. Tecton adds a scheduling engine on top, treating features like managed pipelines rather than raw definitions.

2. The Offline Store (Historical Data) Training needs point-in-time correct data — the feature value as it existed at the moment of the label, not today’s value. This is harder than it sounds. If you’re predicting loan default at t=0, account_balance must reflect the value at t=0, not the current balance. Offline stores (typically Hive, BigQuery, Redshift, or Parquet on S3) store timestamped feature snapshots and support point-in-time joins — a specialized merge that avoids future data leakage.

3. The Online Store (Low-Latency Serving) Inference requires sub-10ms lookups. Batch snapshots are materialized into a key-value store (Redis, DynamoDB, Cassandra) and served via a gRPC or REST endpoint. The key is always an entity ID (user_id, order_id) and the value is the latest pre-computed feature vector.

The pipeline connecting offline to online is the materialization job — a scheduled or streaming process that reads from your data warehouse and writes current feature values to the online store. Feast supports batch materialization via Spark. Tecton adds native streaming materialization via Kafka/Flink, enabling features with freshness measured in seconds rather than hours.

Read more

How to use this: Read your own submission first. Score it against the rubric. Then read this. Find the specific gap. That gap is what you drill this week. Don’t read this before submitting your own answer — the cold attempt is the point.

The question (recap)

Design a notification service. Users have devices; events generate notifications; notifications need to be delivered via push, email, SMS, or in-app depending on user preferences. Failures should retry. The system handles 100M users and 10M notifications per hour at peak.

Step 1 — Clarify

Before drawing anything, three questions matter most:

1. What triggers a notification? Internal events (new message, order shipped, payment received) vs external events (third-party webhooks). The answer shapes the ingestion layer — do producers call an API, or does the service subscribe to an internal event bus?

2. What delivery channels? Push (APNs/FCM), email (SendGrid/SES), SMS (Twilio), in-app (WebSocket). Each has different latency tolerance and reliability guarantees. Push notifications: best-effort, can drop. Emails: must deliver, queued. SMS: must deliver, expensive. In-app: only when the user is connected.

3. What are the SLOs per channel? Push notifications should fire within 5 seconds of the triggering event. Emails within 30 seconds. SMS within 60 seconds. In-app real-time (< 1 second). These numbers drive the queue and worker design.

Step 2 — Estimate

- 100M users, 10M notifications/hour peak = ~2,800 notifications/sec

- Channel distribution (typical): 70% push, 20% email, 8% in-app, 2% SMS - Push: 2,000/sec. Email: 560/sec. In-app: 224/sec. SMS: 56/sec

- Notification record size: ~500 bytes (user_id, channel, template, payload, status) - Storage: 10M/hour × 24h × 30d × 500 bytes ≈ 3.6 TB/month of notification history - User preferences: 100M users × 200 bytes per preference record ≈ 20 GB — fits in a single replicated RDBMS

The 70/20/8/2 channel split drives the worker pool sizing. Push workers handle 70% of the load and need the most horizontal scale.

Get Access to GitHub Repo

Step 3 — API Design

The dedup_key is the idempotency mechanism. If a producer calls /send twice for the same logical event (retry after a timeout), the duplicate is detected by dedup_key and the second call returns the original notification_id without re-queuing. Same pattern as Stripe’s idempotency key.

The template_id separates content from delivery. Producers don’t write notification text — they reference a pre-defined template and pass variables. This allows marketing/product teams to update notification copy without engineering deploys, and enables A/B testing of notification content independently of the delivery system.

Step 4 — Data Model

notifications table

One user has many device tokens. When sending push, the service fetches all active tokens for the user and sends to each. If a token comes back as invalid (APNs/FCM returns “unregistered”), mark is_active = false immediately.

Read more

Most advice about levelling up from L5 to L6 sounds like this: go broader, go deeper, show more impact, lead across teams.

All of this is true. None of it tells you what to do differently on Tuesday morning.

This post is the concrete version. What L6 actually adds in a system design interview, what experience you need before the interview will show it, and what to do in the next 90 days if you’re targeting L6 and currently at L5.

First: understand what L6 is actually assessing

The L5 interview asks: can you design a system that works?

The L6 interview asks: can you design a system that works, that you could operate at 3 AM when it’s failing, and that a team of 10 engineers could build and maintain without constant guidance from you?

Those are different questions. The first is technical correctness. The second is operational maturity plus organizational awareness. Preparing for L6 without understanding this distinction means you’ll produce an excellent L5 answer and wonder why it didn’t land.

Subscribe for Question Walkthrough

The three things L6 adds in the room

I’ve written about Senior vs Staff breakdowns in the paid walkthroughs, but let me be explicit here about what the additions actually are.

SLOs with monitoring signals. An L5 says “I’d target p99 < 100ms.” An L6 says “I’d target p99 < 100ms, and the signal I’d alert on is cache hit ratio dropping below 90% — that’s the leading indicator that Postgres load is about to spike. By the time the latency exceeds 100ms it’s already a problem, but the cache hit ratio tells me it’s about to be a problem while I still have time to act.”

That addition — the leading indicator, not just the threshold — takes 20 seconds to say and is the clearest operational maturity signal in a system design interview. L5 candidates almost never include it without being asked. L6 candidates include it naturally because they’ve been on-call and learned, usually painfully, that the metric you alert on is not the metric you care about.

The revisit condition. An L5 says “I’d partition by user_id because the hot read pattern is per-user.” An L6 says all of that and then adds: “The failure mode here is hot partitions if one user has disproportionate data volume. I’d monitor partition size variance in the first week and set up automatic re-partitioning if any shard exceeds 3× the average. That number is a starting hypothesis — I’d tune it based on actual data.”

The revisit condition shows that you’ve designed systems that later needed to be changed, and that you build in the instrumentation to know when to change them. This is genuinely hard to fake if you haven’t done it. It’s not a vocabulary signal — it’s an experience signal.

Organizational implications. This is the most L6-specific addition and the one most candidates skip entirely. After designing a payment notification system: “One thing I’d flag: this architecture creates a dependency between the payments team and the notifications team at the API contract level. If the notifications schema changes, payments has to coordinate. I’d define the notification contract as a stable interface owned by a platform team so that each product team can deploy independently.”

This sentence is completely invisible to an L5 candidate because L5 engineers generally work within one team and haven’t felt the cost of cross-team coupling. L6 engineers have — they’ve been the person blocked for two weeks because another team changed a shared interface without warning. That experience makes the organizational implication feel obvious to name.

What experience you need before the interview will show it

Here’s the uncomfortable truth: if you’ve spent your entire L5 tenure as an individual contributor on a single product team, building new features, the L6 interview will be hard regardless of how much you study.

L6 evidence is experiential. The monitoring signal addition requires having been on-call and learned what you actually alert on vs what you care about. The revisit condition requires having designed something that later broke in production and had to be changed. The organizational implication requires having felt the friction of cross-team coupling firsthand.

The 90 days before a targeted L6 loop should not just be interview prep. It should be deliberate experience accumulation:

Get on-call rotation, if you’re not already. Even 4 weeks of on-call for a production service will teach you more about operational maturity than any amount of walkthrough reading. You’ll learn what the leading indicators are, what the lagging indicators are, and what it feels like to have the wrong alert fire at 2 AM. That experience will show in the interview automatically.

Own a cross-team project, however small. Propose a shared API, a platform improvement, or a migration that affects two teams. Go through the coordination overhead. Feel the friction of interface ownership. When the interview asks about organizational implications you’ll have something real to draw on.

Write one design doc that gets scrutinized. Not a quick proposal — a full design doc reviewed by engineers senior to you who push back on your decisions. The experience of defending architectural choices against experienced criticism is directly transferable to the interview room.

The 90-day interview prep plan for L6

Given the above, the split should be different from an L5 prep plan.

Weeks 1–4: Get the experience if you don’t have it. See above. No amount of interview prep replaces the operational instincts that on-call builds.

Weeks 5–8: Drill specifically on L6 signals. For every question you drill, force yourself to add three things after the architecture: the monitoring signal for the most critical component, the revisit condition for the riskiest design decision, and the organizational implication of the biggest architectural boundary you introduced. These three additions are almost mechanical once you know to look for them.

Weeks 9–10: Two mock interviews. Ask the mock interviewer to specifically probe the L6 signals: “What would you alert on?” “What would make you reconsider this architecture?” “What does this mean for team autonomy?” If you can answer those three probes naturally in a mock, you’ll answer them naturally in the real interview.

Weeks 11–12: Consolidation. Same plan as any loop prep — drill cards, framework cheatsheet, one cold drill per day. Target the archetypes most relevant to your company.

What to do if your current role isn’t giving you L6 experience

Some engineers are targeting L6 at a new company because their current role doesn’t have the scope to build that experience. That’s a legitimate situation and worth being honest about.

The interview prep in that case should lean more heavily on vicarious experience: read incident post-mortems from engineering blogs (Cloudflare, Stripe, Datadog, and PagerDuty all publish detailed post-mortems). Read design documents from open source projects. Follow engineering blogs from companies operating at the scale of your target company.

The monitoring signal and revisit condition habits can be built by asking “what would tell me this design is failing?” after every drill, even if you’ve never been on-call. It’s not the same as lived experience, but it builds the reflex of thinking that way.

The organizational implication is harder to simulate. The closest substitute is: for every architectural boundary you draw in a design (this service vs that service, this team’s responsibility vs that team’s), explicitly name who owns the interface, what happens if either side changes it, and whether there’s a better way to isolate the two.

The question worth asking yourself

Before you start interview prep, sit with this for a few minutes: in the last six months, have you made architectural decisions that affected engineers outside your immediate team? Have you been woken up at 2 AM by a broken system and had to diagnose it? Have you owned an interface that another team depended on?

If yes to two or three: you probably have the experiential foundation and the interview prep is about learning to articulate it explicitly.

If no: the 90 days before your loop are more valuable spent accumulating those experiences than reading walkthroughs. The interview is a test of whether you think like an L6 engineer. That thinking comes from doing L6-level work, even briefly, more than from any amount of preparation.

The walkthroughs tell you what to say. The experience gives you something true to say.

CTA: The Senior vs Staff section in every paid walkthrough breaks down exactly what L6 adds for that specific question — not in the abstract, but with the exact phrases and decisions that separate Hire from Strong Hire at that level.

If you found this valuable, the full content library includes bonus lessons, exercises, and detailed breakdowns.

https://systemdr.systemdrd.com/subscribe

—Sumedh

The FYF is not a follow-graph feed — it’s a recommendation feed. The candidate pool is “every video ever uploaded” filtered by ML, not “videos from accounts you follow.” Same fan-out and ranking architecture as Twitter Timeline, but the candidate generation layer is completely different.

Step 1 — Clarify

- Follow-graph feed or pure recommendation? (FYF = recommendation — no follow required)

- Short video only (≤60s) or mixed? This drives storage and encoding decisions - Real-time engagement signals (likes, shares, completion rate) must feed back into ranking within minutes — how fresh?

- DAU? (TikTok: ~1B DAU globally)

- SLO: next video must load in < 500ms — the “infinite scroll” UX depends on it

Step 2 — Estimate

- 1B DAU × 40 videos watched/day = 40B video plays/day = ~463K plays/sec - Each user needs a pre-fetched queue of 10–20 recommended videos to enable instant scroll

- Candidate generation per user: rank top 500 from a pool of billions → return top 20 - Engagement events (play, like, share, skip): ~5 events per video = 200B events/day = 2.3M events/sec feeding the ranking model

Get Access to GitHub Repo

Step 3 — API Design

The prefetch_next_10 in the response is the key design signal — the client pre-fetches the next 10 videos while the user watches the current one, achieving the illusion of instant load.

Step 4 — Data Model

- videos table: video_id, creator_id, s3_key, duration_ms, transcript, embedding (vector), tags[], upload_at

- engagement_events (Kafka → time-series store): user_id, video_id, event_type, watch_pct, timestamp

- user_interest_profile: user_id, topic_vectors[], creator_affinity[], updated_at — updated continuously from engagement stream

- video_recommendation_cache (Redis): user_id → [ordered list of 50 pre-ranked video_ids, TTL=10min]

Read more

This phrase ends more system design answers than any other. It sounds thoughtful. It signals nuance. And it is the single most effective way to tell an interviewer you don’t know how to make a decision.

Every technology choice depends on the use case. That’s not an answer — it’s a preamble. The answer is what it depends on, and how you decide.

Here’s the decision framework for choosing between relational and non-relational storage that works in an interview setting.

Step 1: Write down your two hottest queries

Before naming any storage technology, write down the two or three queries the system will run most frequently. Specifically: what are the inputs, what are the outputs, and what constraints do they have?

“Give me all messages for user X, ordered by time” — this is a range query on a partition key. Wide-column store handles this well.

“Give me all orders for account X with status=pending, joined to the product table” — this has a join. Relational database handles this better.

“Give me the 10 nearest restaurants within 5km of this coordinate” — this is a spatial query. PostGIS or a geohash index.

The storage decision follows from the query. The query never follows from the storage decision.

Subscribe for Payments Walkthrough

Step 2: Apply the decision rules

Use a relational database when:

• Your data is structured and schema is stable

• You need ACID transactions across multiple rows or tables

• Your queries involve joins, aggregations, or complex filtering

• Your write volume is under roughly 5–10K writes per second per node

• Your total data per table is under roughly 100M rows before sharding gets painful

Use a wide-column store (Cassandra, HBase) when:

• You need high write throughput (100K+ writes per second)

• Your queries are predictable and partition-key-based — no joins, no ad-hoc filtering

• Your data is naturally time-ordered (events, messages, metrics)

• You need built-in TTL on rows (messages that expire, session data)

Use a key-value store (Redis, DynamoDB) when:

• Your reads are pure point lookups by a single key

• You need sub-millisecond latency

• You’re storing simple values or small objects

• You need atomic counters or sorted sets

Use object storage (S3) when:

• You’re storing large binary objects (files, images, video, backups)

• You need very high read throughput via CDN

• Latency of 50–200ms per request is acceptable

Step 3: Name the anti-pattern you’re avoiding

The most powerful version of a storage decision isn’t “I’d use Cassandra.” It’s “I’d use Cassandra instead of Postgres here because the write volume — 50,000 events per second — will exceed what a single Postgres instance can handle, and the queries are all partition-key-based so I don’t need the join capability I’d be giving up.”

The comparison forces you to show you considered the alternative. The reason forces you to show you understand the trade-off.

The one rule that overrides everything

If you need a multi-row ACID transaction — an operation where either all rows change or none do — you need a relational database or a relational-compatible distributed database (CockroachDB, Spanner). No amount of Cassandra tuning gives you true ACID across multiple partition keys.

Payment systems almost always need ACID. Account balances must be updated atomically. Inventory reservation must be atomic. Order creation and payment capture must be atomic.

“I’d use Cassandra for the payment ledger” is a common answer that fails almost every payment system design question. The interviewer will ask “what happens if the debit succeeds but the credit fails?” and there is no good answer with Cassandra.

Know what you’re trading away before you trade it.

CTA: The Stripe Payments walkthrough goes deep on exactly this decision — why the ledger needs ACID, how double-entry accounting makes the atomicity requirement explicit, and what happens at the infrastructure level when you enforce it. Paid post #2.

Interested in mastering this topic?

Unlock premium content with deeper explanations and real implementation patterns.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

A customer support system at a fintech startup gets this message: “My transfer failed yesterday, can you check the status and also tell me if my tier qualifies for waived fees?”

A dumb LLM call answers one question and halts. A well-built agent understands this requires two tool calls (transaction lookup, account tier check), sequencing them based on partial results, remembering context from earlier in the conversation, and producing a single coherent reply.

The difference between these two outcomes is not the model — it’s the agent framework wrapped around it. Understanding how memory, tools, and planning modules fit together is the foundational skill for anyone building production AI systems in 2025 and beyond.

Core Concept: The Three-Module Architecture

An agent framework is a runtime loop that connects an LLM to the outside world through three distinct subsystems.

Read more

The inventory reservation problem. A listing is available on specific dates, and two users trying to book the same listing for overlapping dates cannot both succeed. The hard problem: prevent double-booking without making the availability check and reservation a single-threaded bottleneck.

Step 1 — Clarify

- Single room per listing (Airbnb) or multiple rooms (hotel)?

- Search: by location + date range + filters?

- Booking flow: instant book (reserve immediately) or request-to-book (host approves)? - How to handle the “someone else just booked it” race condition?

Step 2 — Data Model

listings: listing_id, host_id, title, location, price_per_night, instant_book (bool) availability: listing_id, date, status (available/blocked/reserved), booking_id (nullable) PRIMARY KEY: (listing_id, date) — one row per listing per night

bookings: booking_id, listing_id, guest_id, check_in, check_out, status

(pending/confirmed/cancelled),

total_price, idempotency_key, created_at

The availability table with one row per listing per night is the key data model decision. Date range availability check = SELECT COUNT(*) FROM availability WHERE listing_id=X AND date BETWEEN check_in AND check_out AND status=’available’. If count = (check_out - check_in) in days, all nights are available.

Get Access to GitHub Repo

Step 3 — Reservation flow (preventing double booking) BEGIN;

-- Check all nights available and lock them

The FOR UPDATE lock on the availability rows means only one transaction can proceed for any given listing+date combination. The second concurrent booking attempt blocks, then fails the count check when it finally runs.

Read more

The most common reason technically strong candidates get a “Hire” instead of “Strong Hire” is not technical depth. It’s running out of time before demonstrating it.

The deep dive section — the hard distributed systems problem at the center of the question — is where L5 separates from L4, and where L6 separates from L5. It’s also the section that gets cut when the interview runs short. You spent 20 minutes on estimation and API design and now you have 10 minutes left for the architecture and zero for the deep dive.

The fix is specific time checkpoints that you self-enforce in the room. Not approximate — specific.

The 50-minute allocation

Here is the allocation that consistently produces complete answers with time for depth:

Total: 45–47 minutes, leaving a 3–5 minute buffer.

Subscribe for Question Walkthrough

The checkpoints to self-enforce

Minute 10: API must be drafted and scope must be locked. If you’re still asking clarifying questions at minute 10, you’ve spent your estimation time and you’ll have to compress the data model.

Minute 22: Schema and key flows should be on the board. At this point you should have: at least two tables sketched with primary keys and indexes, and at least one paragraph of the architecture explained. If you’re still on API design at minute 22, move immediately.

Minute 35: The deep dive must be in progress. Not about to start — actually underway. If you’re at minute 35 with only the high-level architecture sketched and no deep dive started, you have a problem. Stop the high-level walkthrough and pivot to the deep dive now. An incomplete high-level architecture plus a solid deep dive is worth more than a complete architecture with no deep dive.

Minute 48: Trade-offs and failure modes should be underway. If you’re still deep-diving at minute 48, spend the last two minutes naming the top three things that break in your current design and what you’d do about them. Even naming them without a full answer is better than silence.

The discipline required

The hardest part of time management is stopping something that’s going well to move on.

You’re mid-explanation of the data model. It’s going well. You understand this part. The interviewer is nodding. You want to finish the thought.

Move anyway. The next section matters more than this one continued.

The data model section’s job is to demonstrate that you can design appropriate schemas with the right indexes and sharding strategy. Once that’s demonstrated — once you’ve covered the main table, the partition key, and at least one non-obvious design decision — the section is done. More detail doesn’t add signal; it steals time from sections where signal is still available.

What to do when you fall behind

If you realize you’re behind the checkpoints, the correct response is compression, not panic.

For the architecture: “Let me give you the key components and the main flows rather than a complete walkthrough.” Five minutes instead of ten. Names the components, traces one read path and one write path, stops.

For the data model: “The critical table here is the messages table — let me focus on that one rather than all five tables.” Two minutes instead of seven. Names the partition key, names the TTL strategy, stops.

For the API: “Three endpoints — the core create, read, and webhook. Let me sketch those quickly.” Two minutes. Moves on.

Compression is always better than omission. A compressed section signals that you know what you’re doing and are managing time deliberately. An omitted section is a missing signal.

The permission to be incomplete

Most engineers feel they should finish each section before moving to the next. This intuition works in normal engineering work. In a timed interview, it’s wrong.

The interview is scored as a whole, not section by section. A round with complete clarification, estimation, API, data model, architecture, deep dive, and trade-offs — each at 70% coverage — scores better than a round with perfect clarification, estimation, and API at 100% coverage with no deep dive.

The deep dive is the section with the highest score differential. It’s where the round is decided. Everything before it is the setup. Don’t let the setup consume so much time that the payoff never arrives.

CTA: The drill cards have a time-box bar at the bottom of every card — the exact minute allocation for that specific question, weighted toward where the probe lands. All 37 are in the Vault.

Curious how this works in real systems?

The paid lessons cover scalability, optimization, and real-world engineering patterns.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

A user types “something comfortable for standing all day in a hospital.” Your keyword index scans every product title and description for those exact tokens. Nothing meaningful surfaces. Your competitor, running a vector search pipeline, returns slip-resistant nursing clogs, anti-fatigue mats, and compression socks — none of which share a single word with the query. That gap — between what people type and what they mean — is the architectural problem at the center of this article, and the reason the two approaches have completely different infrastructure profiles.

Keyword Search: Inverted Indexes and BM25

Keyword search is built on the inverted index. For every unique term in your corpus, the index stores a posting list: the sorted set of document IDs containing that term, annotated with position and frequency metadata. At query time, the engine tokenizes the query, retrieves each term’s posting list, and scores matching documents using BM25 — a probabilistic ranking function derived from TF-IDF with two improvements: term frequency saturation and document length normalization.

The BM25 score for a document D against query Q is:

score(D,Q) = Σ IDF(qi) · [f(qi,D) · (k1+1)] / [f(qi,D) + k1·(1 − b + b·|D|/avgdl)]

The parameter k1 (default 1.2–2.0) controls how fast additional term occurrences stop adding relevance. The parameter b (default 0.75) penalizes long documents to prevent them from dominating purely by volume. These defaults work across most corpora without tuning.

Latency is predictable: O(log n) for the term lookup, linear in posting list size for scoring. Elasticsearch handles 10,000+ QPS on commodity hardware with p99 latency under 10ms for standard full-text workloads. The index is compact, deterministic, and easy to debug — you can reconstruct exactly why a document ranked where it did.

The hard limit is vocabulary mismatch. “Cardiac event” and “heart attack” are clinically synonymous but share zero tokens. “Python” is a language or a reptile. “Memory leak” and “heap overflow” describe overlapping problems. The index doesn’t model meaning, only token co-occurrence.

Semantic Search: Embeddings and ANN Retrieval

Semantic search encodes both documents and queries into dense vectors using transformer-based embedding models — typically producing 384- to 1536-dimensional float32 vectors. The model maps semantically similar text to nearby points in vector space, measured by cosine similarity. If the embedding model has learned that “heart attack” and “myocardial infarction” are synonymous, their vectors cluster together regardless of surface form.

The index structure is fundamentally different. Instead of an inverted posting list, you need an Approximate Nearest Neighbor (ANN) index. HNSW — Hierarchical Navigable Small World — is the dominant structure in production. It organizes vectors as a multi-layer proximity graph: coarse layers for fast long-range navigation, fine layers for precise local search. Query time navigates from coarse to fine, achieving O(log n) amortized complexity with recall configurable via the ef parameter.

The end-to-end pipeline: documents are chunked → encoded by the embedding model → stored in the vector index. Queries are encoded at query time before retrieval. Embedding latency is 5–15ms for compact models like all-MiniLM-L6-v2 and 30–80ms for larger 7B-parameter models. HNSW search adds 5–30ms depending on index size and ef. Total query time: 10–50ms typical, vs keyword’s 1–10ms.

Read more

Same fan-out architecture as Twitter Timeline, but the ranking model is more complex — Instagram ranks not just by recency but by predicted engagement (like, comment, share probability) for each specific user-post pair. The probe is the ranking pipeline and how features are assembled at query time without violating the 500ms SLO.

Architecture delta from Twitter Timeline

Read more

They’re not exotic. They’re not about knowing obscure algorithms. They’re about the difference between an answer that sounds right and an answer that demonstrates you understand what you’re designing.

Gap 1: Numbers that don’t drive anything

Most cohort members can estimate. They get DAU, they calculate requests per second, they calculate storage. The numbers are usually in the right order of magnitude.

Then the next section of their answer proceeds as if the numbers never existed.

“We’ll have approximately 50,000 writes per second” followed by “I’d store user data in a Postgres database.” No mention of whether Postgres handles 50,000 writes per second. No acknowledgment that this number might matter for the technology choice.

The numbers should be the reason for the architectural decisions. Not a box checked before the real answer begins.

The fix is simple: after every estimation number, add one sentence starting with “this means.” 50,000 writes per second — this means a single Postgres instance will be the bottleneck; I’ll need to either shard or move to a write-optimized store like Cassandra. Now the number is doing work.

Gap 2: The happy path only

Most cohort submissions design a system that works when everything goes right. The API is called, the database responds, the message is delivered. The walkthrough is coherent.

Then nothing goes wrong. There are no failure modes. There’s no offline user, no database timeout, no cache failure, no duplicate message delivery.

The simplest version of this is in the messaging question: a candidate designs the delivery path for an online user but never addresses what happens when the recipient is offline. The question literally asks about this. The omission is not subtle.

The fix is one explicit section at the end of every design: “three things that break.” Name them, name the conditions, and describe how you detect and handle each. Even if the answer is “we’d accept degraded behavior and alert on it” — that’s a real answer.

Subscribe for Question Walkthrough

Gap 3: Decisions without reasons

“I’d use Kafka for the event stream.”

Why Kafka? This sentence appears in dozens of submissions every cohort. Sometimes Kafka is the right answer. Sometimes a simple SQS queue would be better. Sometimes a database table with FOR UPDATE SKIP LOCKED is better. The choice depends entirely on the requirements.

When you state a technology choice without a reason, you’re signaling that you know the word, not the trade-off. The interviewer probes exactly here. “Why Kafka over something simpler?” is one of the most common follow-up questions in design interviews.

The fix is to never name a technology without one of these two phrases following it: “because” or “instead of.”

“I’d use Kafka because the producers and consumers have different throughput — decoupling them lets each scale independently.” Now the choice has a reason.

“I’d use Redis instead of Memcached because we need persistence and sorted sets for the leaderboard.” Now the choice has a comparison.

Why these gaps matter more than knowledge gaps

The three things above — numbers with consequences, failure modes, and reasoned decisions — are what experienced engineers do automatically when they design systems. Not because they’re trying to impress anyone. Because they’ve been on-call for systems that failed when the failure mode wasn’t anticipated, and they’ve had to defend technology choices to skeptical colleagues.

The interview is asking: does this person think like someone who has built and operated real systems?

The three gaps signal no, even when the technical knowledge is there.

If your answers have all three gaps, the fix is not more studying. It’s one mock interview where you discipline yourself to add “this means,” “three things that break,” and “instead of” to every section of your answer.

CTA: The cohort curriculum is built around closing exactly these three gaps — each module targets one of them directly.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

Curious how this works in real systems?

The paid lessons cover scalability, optimization, and real-world engineering patterns.

A training job lands in your cluster. It requests 4 GPUs. Your cluster has 8 GPUs across 4 nodes — 2 per node. The job sits Pending for 11 minutes before the scheduler finds a fit. Meanwhile, a researcher watching a progress bar assumes their cloud bill is accumulating. It is. The cluster is doing nothing useful. This is not a capacity problem. It is a scheduling problem, and it is more common than most platform teams admit.

How Kubernetes Sees a GPU

Kubernetes has no native GPU awareness. The kernel does. The gap is bridged by a device plugin — a DaemonSet running on each node that advertises resources to the kubelet using a gRPC interface. NVIDIA’s device plugin, for instance, registers nvidia.com/gpu as an extended resource and tells the kubelet exactly how many are available.

When a pod requests nvidia.com/gpu: 2, the scheduler treats this resource identically to how it treats CPU or memory: as an integer that must be satisfied on a single node. The GPU is not a shared resource in the default model. A GPU requested by one pod is exclusively allocated to it until the pod terminates. This matters enormously for utilization math.

The scheduler’s default behavior is bin-packing when LeastAllocated priority is off, or spreading when it’s on. For GPU workloads, bin-packing tends to produce better utilization — it concentrates GPU jobs onto fewer nodes, leaving entire GPU nodes free for large multi-GPU jobs rather than leaving each node half-occupied and unusable.

What Happens at Allocation

1. The scheduler scores candidate nodes using configured priority functions.

2. A node passes if its free nvidia.com/gpu count is ≥ the pod’s request.

3. The kubelet on the winning node calls the device plugin’s Allocate() RPC.

4. The device plugin returns a set of environment variables (CUDA_VISIBLE_DEVICES) and optional mounts.

5. The container runtime injects these into the container, making specific GPU indices visible.

The pod never directly names a GPU. It gets whatever the plugin assigns. This is by design — the plugin handles NUMA affinity, topology constraints, and MIG configuration below the Kubernetes layer.

MIG and Time-Slicing Change the Unit of Allocation

On NVIDIA A100 and H100 GPUs, Multi-Instance GPU (MIG) partitions a physical GPU into isolated instances with dedicated memory and compute slices. A100 80GB can yield up to 7 MIG instances (1g.10gb profile), each behaving like an independent smaller GPU. The device plugin can advertise these as nvidia.com/mig-1g.10gb resources. A pod requesting one instance gets guaranteed isolation — no memory bandwidth contention with other tenants on the same card.

Time-slicing is the alternative for GPUs that don’t support MIG (or when you need more than 7 partitions). The plugin oversubscribes the GPU, advertising N logical GPUs where N = physical × replicas. Multiple pods get CUDA_VISIBLE_DEVICES pointing at the same physical device. There is no memory isolation. If your LLM inference pod OOMs, it takes every co-located pod with it.

Read more

Slack is WhatsApp + channels + threads + search.

The messaging core is identical to WhatsApp (WebSocket, Cassandra, delivery semantics). The probes unique to Slack: channel message fan-out (a channel with 10,000 members means one message = 10,000 deliveries), message threading (replies live in a separate namespace from channel messages), and full-text search over message history.

Step 1 — Clarify

- Direct messages only, or channels + DMs + threads? (Scope to channels + DMs) - Search over message history? (Yes — search is a major Slack differentiator) - Max members per channel? (Slack allows ~10K for standard, unlimited for Enterprise Grid)

- File sharing? (Yes — same object storage pattern as WhatsApp media) - SLO: message delivery < 200ms p99 within same workspace

Step 2 — Estimate

- 20M DAU × 200 messages/day = 4B messages/day = ~46K messages/sec - Average channel size: 50 members → fan-out 50 deliveries per message - Large channels (10K members): 1 message = 10K WebSocket pushes - Message retention: Slack keeps all history (unlimited on paid plans) → ~500 bytes/msg × 4B msgs/day × 365 days = ~730 TB/year

Get Access to GitHub Repo

Step 3 — API Design

WebSocket: wss://slack.com/connect (per workspace per device)

ts (timestamp) is Slack’s actual message ID — a Unix timestamp with microsecond precision, unique per channel.

Step 4 — Data Model

- messages (Cassandra): PARTITION BY channel_id, CLUSTER BY ts DESC — identical pattern to WhatsApp but scoped to channel

- threads: thread_ts (FK to parent message ts), reply messages in same table with thread_ts as additional cluster key

- channel_members: channel_id, user_id, joined_at — indexed both ways (channel→members for fan-out, user→channels for sidebar)

- search_index (Elasticsearch): message ts, channel_id, text, user_id, timestamp — updated async via Kafka consumer

- unread_counts (Redis): user_id:channel_id → {unread_count, last_read_ts} — updated on every message and read event

Read more

The system design interview round has a failure rate that surprises most engineers when they first encounter it. Strong engineers — people who are genuinely good at their jobs, who have built real systems, who know distributed systems concepts — fail this round at a rate that doesn’t correlate with their actual technical competence.

After reading enough post-mortems, the failure pattern is consistent. It’s almost never the technical content.

Failure mode 1: designing instead of scoping

The question is “design Twitter.” The candidate starts designing Twitter. The entire Twitter.

Fifty minutes is not enough time to design a system with the surface area of Twitter. Everyone who attempts to design all of it produces a shallow sketch of everything. The interviewer wanted a deep design of one or two components. The candidate gave a shallow design of twelve.

The fix is the first five minutes of the interview. Use them to scope explicitly: “There are several interesting subsystems here. For this interview, I’m going to focus on the timeline and the fan-out architecture. I’ll acknowledge notifications, search, and ads exist but won’t design them — does that work for you?”

Most interviewers say yes. If they don’t, they redirect you to what they actually want to probe. Either way you’ve avoided spending 45 minutes designing the wrong thing.

Failure mode 2: communicating at the wrong level

“I’d use a database for user data and a cache for performance.”

This sentence tells an interviewer almost nothing. Which database? Which cache? What’s being cached? Why? What’s the eviction strategy? What happens on cache miss?

The problem isn’t that the candidate doesn’t know the answers. They often do. The problem is they’re communicating at the level of “I’ve heard of these technologies” rather than “I understand the trade-offs.”

The fix is the three-sentence pattern: the decision, the reason, and the condition under which you’d change your mind. “I’d use Redis for the timeline cache with LRU eviction and a 24-hour TTL. The hot read pattern is point lookups by user_id, and Redis’s sub-millisecond gets absorb the read load that would otherwise hit Postgres on every feed request. If the working set grows beyond 20% of total DAU being simultaneously active, I’d evaluate a tiered cache.”

That’s the same knowledge communicated at the right level.

Subscribe for Question Walkthrough

Failure mode 3: treating the interviewer as an adversary

The design round is not an oral exam. The interviewer is not trying to catch you out.

Candidates who treat it like an exam defend their choices when challenged, interpret probing questions as criticism, and become less flexible as the round goes on. The interviewer asks “what if the scale was 10× what you assumed?” and the candidate hears “your estimation was wrong” and gets defensive.

The more useful interpretation: the interviewer is helping you get to the interesting part of the question. When they push back, follow their lead. “Good point — at 10× scale, the component that breaks first is the Postgres write path. Let me address that.”

Failure mode 4: running out of time before the deep dive

The deep dive — the hard distributed systems problem at the center of the question — is where the round is decided. It’s where L5 candidates separate from L4, and where L6 candidates separate from L5.

It’s also the section that gets cut when time runs out.

The fix is self-enforcing time checkpoints. Minute 10: API must be drafted, scope locked. Minute 20: schema and key flows sketched. Minute 35: deep dive in progress. Minute 48: trade-offs and failure modes underway.

If you’re at minute 25 with no architecture on the board, the correct move is to stop wherever you are, skip to the architecture, and do it roughly. A rough architecture with a deep dive is more valuable than a complete schema with no architecture.

The pattern

All four failure modes are communication problems, not knowledge problems.

If you’ve done the technical prep and you’re still not passing, the problem is almost certainly one of these four. The fix is not more reading. It’s a timed mock interview with someone who will score you specifically on scope management, communication depth, interviewer collaboration, and time allocation.

Knowledge gets you to the room. Communication gets you the offer.

CTA: If you recognise one of these four failure modes in your own prep — reply and tell me which one. I answer every reply and if you describe your specific situation I’ll point you to the right resource.

Subscription link

https://systemdr.systemdrd.com/subscribe

—Sumedh

A base model is a hypothesis. A fine-tuned model is a product. The gap between those two things isn’t mathematical — it’s operational. Teams that treat fine-tuning as “run a script in a notebook” eventually hit a wall: reproducibility breaks, models regress silently, datasets drift, and nobody can explain which checkpoint shipped to production last Tuesday. Designing a fine-tuning pipeline is fundamentally a distributed systems problem wearing an ML hat.

Core Concept: The Fine-Tuning Pipeline

Fine-tuning adapts a pre-trained foundation model to a specific task by continuing training on a curated domain dataset. The challenge isn’t the gradient math — it’s orchestrating the data, compute, validation, and versioning layers into a system that behaves correctly under failure, scales without bottlenecks, and produces artifacts you can actually trust.

A production fine-tuning pipeline has five distinct stages that must be independently restartable and observable.

Stage 1 — Data Ingestion & Versioning Raw training data arrives from object storage (S3, GCS), annotation pipelines, or RLHF feedback systems. Before anything trains, every dataset version must be fingerprinted. Tools like DVC or Delta Lake provide immutable dataset snapshots. The failure pattern here: teams that skip versioning can’t answer “what data produced this model?” six months later — which kills audit trails and compliance.

Stage 2 — Preprocessing & Tokenization Raw text gets cleaned, filtered for quality (perplexity filtering, dedup via MinHash LSH), and tokenized into packed sequences. This stage runs on CPU-heavy workers (separate from GPU training nodes) and produces binary shard files. A common mistake: running tokenization inside the training loop wastes GPU cycles on CPU-bound work and creates non-deterministic throughput under variable data shapes.

Stage 3 — Training Job Orchestration The training job itself is submitted to a job scheduler (Ray, Slurm, Kubernetes Job API) with explicit resource requests. Key decisions here: gradient checkpointing to trade compute for memory, mixed-precision (bf16/fp16) to reduce VRAM pressure, and gradient accumulation to simulate large batch sizes across limited hardware. For distributed runs, you choose between data parallelism (each node holds the full model), tensor parallelism (model sharded across nodes), or pipeline parallelism (model layers split across stages). Each has distinct failure modes — pipeline parallelism, for instance, introduces pipeline bubbles that reduce GPU utilization.

Stage 4 — Checkpoint Management Training jobs die. GPU nodes preempt. Cloud spot instances evict. Checkpoints must be written to durable storage at regular intervals and the pipeline must resume from the latest valid checkpoint without human intervention. A subtlety: corrupted checkpoints (written during a crash) need detection logic. Write to a temp path, validate, then atomic rename.

Stage 5 — Evaluation, Registry & Promotion After training completes, the model runs against a frozen eval harness — benchmark tasks that don’t change between runs, covering both target task performance and regression tests for general capabilities. Models that pass thresholds get registered in a model registry (MLflow, Weights & Biases Registry, or custom) with full lineage metadata: dataset version, hyperparameters, hardware config, eval scores. Promotion to staging or production requires explicit human approval or automated A/B gate logic.

Read more