Three technologies for moving work between services. Frequently confused. Used interchangeably in interview answers. Rarely the same answer in production.
Here’s how to tell them apart and when to reach for each one.
What they have in common
All three decouple producers from consumers. Instead of Service A calling Service B directly, A puts a message somewhere and B reads it when it’s ready. This makes A and B independently scalable, independently deployable, and independently failure-tolerant.
The decoupling is the shared property. Everything else differs.
Kafka: streaming and fan-out
Kafka is a distributed log. Producers append messages to topics. Consumers read from topics at their own pace. Messages are retained for a configurable period (days or weeks), regardless of whether anyone has read them.
The key properties that distinguish Kafka:
Replay. A consumer can re-read old messages. If you deploy a new service and need to process all historical events, Kafka lets you rewind to the beginning. SQS and task queues don’t.
Fan-out. Multiple independent consumer groups can read the same topic simultaneously. Each group maintains its own offset. Service A and Service B can both consume the same stream of events without interfering with each other. SQS and task queues don’t support this natively.
Ordering. Messages within a partition are strictly ordered. If order matters — event sourcing, audit logs, ledger updates — Kafka preserves it.
Use Kafka when: you have multiple consumers that need to read the same events, you need replay capability, you’re building a streaming pipeline where order matters, or you need to decouple high-throughput event producers from slower consumers.
Don’t use Kafka when: you need guaranteed exactly-once delivery of individual work items to exactly one worker, or when you have a small team that can’t operationally support a Kafka cluster.
SQS: simple at-least-once work distribution
SQS is a managed queue. A producer enqueues a message. A consumer polls, processes, and deletes it. The message is invisible to other consumers during processing (the visibility timeout). If the consumer crashes, the message reappears.
The key properties:
Simplicity. Fully managed by AWS. No cluster to operate. You pay per message.
At-least-once delivery. Each message is delivered to at least one consumer. Duplicates are possible (rare but possible). Your consumer must be idempotent.
One consumer per message. Each message is processed by exactly one consumer group. No fan-out. If you need multiple services to process the same message, you need SNS in front of SQS (one topic, multiple queues).
Use SQS when: you need a simple, reliable queue for distributing work to a pool of workers, you’re on AWS, and you don’t need replay or fan-out.
Task queues (Celery, Redis Queue, Postgres FOR UPDATE SKIP LOCKED)
Task queues are purpose-built for job execution. A task has a type, a payload, a status, a retry count, and a deadline. Workers claim tasks atomically, process them, and mark them complete. Failed tasks retry with backoff.
The key properties:
Task lifecycle management. Task queues track the status of individual work items. You can query “which tasks are in progress right now” or “which tasks have failed more than 3 times.” Kafka and SQS don’t natively track per-item status.
Retry with backoff. If a task fails, it’s retried after an exponential delay. Dead-letter queuing for tasks that fail past the max retry count. This is the production pattern for background job processing.
Priority. Tasks can have priority levels. High-priority tasks jump the queue. Standard queues don’t support this natively.
Use a task queue when: you’re running background jobs (email sending, image processing, report generation), you need per-task status visibility, you need retry with backoff, or you need priority queuing.
The interview decision
In a system design interview, the choice should follow from the requirement:
Need multiple services to react to the same events? → Kafka
Need to offload work from a web request to a background process? → SQS or task queue
Need to build a streaming analytics pipeline? → Kafka
Need to retry failed jobs with backoff and track their status? → Task queue
Need a managed, simple work queue with no infra overhead? → SQS
Need replay capability for new consumers or debugging? → Kafka
The mistake to avoid: defaulting to Kafka for everything. Kafka is operationally heavy (unless you’re using Confluent Cloud or a managed service). For a small-scale job queue, Postgres with FOR UPDATE SKIP LOCKED is simpler, cheaper, and perfectly adequate up to millions of jobs per day.
State your choice, name the alternative you’re not using, and explain why. That’s the format for any technology decision in this category.
CTA: The full walkthroughs go beyond knowing what Kafka, SQS, or task queues are. They show you how to make the decision in an interview — what requirement should drive the choice, which alternative to rule out, and how to explain the tradeoff like a Senior vs Staff engineer.
If you found this valuable, the full library includes 52 walkthroughs, bonus lessons, exercises, and detailed interview breakdowns.
https://systemdr.systemdrd.com/subscribe
The Question Vault has all 52 walkthroughs organized by archetype — so you can see the pattern across questions, not just the surface answer.
Access all 52 Questions here