Queues, Kafka and event-driven systems cheat sheet

The 14 Queues, Kafka and event-driven systems interview answers in 30 seconds each, plus quick facts. Printable, free.

One page per topic: print it or save it as PDF from your browser's print menu. Practise these questions

Rules of thumb

Key by the entity whose events must stay in order.
Commit offsets after processing; make handlers idempotent.
Dedup with an inbox table in the same transaction as the effect.
Bounded retries, then dead-letter with context; alert on DLQ depth.
Additive event changes only; registry with a compatibility mode.
Alert on consumer lag in seconds and on growth.
One scheduler, a lock and idempotency per period for cron jobs.

Work queues vs publish/subscribe

In a work queue each message goes to one of several competing workers, which spreads load: resize these images, send this email. In publish/subscribe each message goes to every subscribing group, which fans out one event to independent consumers: OrderPlaced goes to billing, shipping and analytics. Kafka does both at once: within a consumer group, partitions are shared like a queue; each separate group receives every message like pub/sub.

Background job queues

Put a job on a durable queue (a broker, or a PostgreSQL table claimed with SKIP LOCKED) and let workers process it outside the request. Reliability comes from acknowledging only after the work is done, retrying failures with backoff, capping attempts and moving poison jobs to a dead-letter queue, and making each job idempotent because it may run more than once. Enqueue in the same transaction as the business change, or through an outbox, so a rollback doesn't send the email anyway.

Kafka partitions, keys and ordering

A topic is split into partitions, each an append-only log. With a key, the producer hashes it to choose a partition, so every message for one key lands in the same partition; without a key, messages are spread for balance. Kafka guarantees order only within a partition, so key by the entity whose events must stay in order, such as the order id. In our Kafka 4 run, every key went to the same partition each time.

Consumer groups and rebalancing

The group coordinator assigns each partition to exactly one consumer in the group. When membership changes, or a consumer stops polling for longer than max.poll.interval.ms, the group rebalances and partitions move. The new owner resumes from the last committed offset, so anything processed but not committed is processed again. Cooperative rebalancing moves only the partitions that need to move instead of pausing everyone.

Committing offsets: at-most-once vs at-least-once

Commit after processing for at-least-once: if the consumer crashes mid-batch, the messages are read again, so the handler must be idempotent. Committing before processing gives at-most-once: a crash loses the messages that were committed but not processed. For read-process-write pipelines inside Kafka, transactions can commit the output and the input offsets atomically, which Kafka calls exactly-once.

Keeping messages in order

Route all messages for one entity to the same ordered channel (same Kafka key, or SQS FIFO message group) and process each channel with one consumer at a time. Then avoid things that reorder inside it: parallel processing of one partition, and retries that push a failed message to the back. Where you can't guarantee order, carry a version or sequence number and have consumers ignore or buffer events older than what they've applied.

Dead-letter queues and poison messages

Classify failures. Transient errors (timeouts, a database restart) get retries with backoff. Permanent ones (bad payload, failed validation) won't succeed on retry, so after a small number of attempts move the message to a dead-letter queue with the error and metadata, and continue with the next. Alert on DLQ depth, fix the cause, then replay dead-lettered messages deliberately.

Idempotent consumers and deduplication

Either make the effect naturally idempotent, for example 'set status to paid' or an upsert keyed by the event's entity, or record each processed message id in a table with a unique constraint inside the same transaction as the effect. A redelivered message then hits the constraint, and you skip it. Keep processed ids at least as long as the broker can redeliver.

Backpressure when consumers fall behind

A durable queue absorbs bursts, but if the input rate stays above processing capacity the backlog grows without limit. Options: scale consumers (up to the partition count), make processing cheaper (batching, fewer round trips), slow the producers (bounded queues, 429 or 503 to callers), or shed low-value load. Inside a process, bounded queues make producers wait instead of using unbounded memory.

Event sourcing

Instead of storing the current state, you store every change as an immutable event (AccountOpened, MoneyDeposited) and derive state by replaying them. You get a complete audit trail, the ability to rebuild read models or answer 'what was the balance on 1 March', and natural integration events. The costs: queries need projections, replaying long streams needs snapshots, and changing event formats needs careful versioning because history is never rewritten.

Evolving event schemas

Treat events as a public contract. Make only compatible changes: add optional fields with defaults, never remove, rename or retype a field consumers use. Enforce it with a schema registry (Avro, Protobuf or JSON Schema) set to a compatibility mode such as BACKWARD, so incompatible schemas are rejected at publish time. For a real break, publish a new event type or version alongside the old one until consumers move.

Kafka or RabbitMQ?

Kafka is a replicated, partitioned log: messages stay for a retention period whether or not they're consumed, consumers track their own offsets and can replay, and throughput is very high. It suits event streams, many independent consumers and reprocessing. RabbitMQ is a broker that routes messages to queues and deletes them once acknowledged, with flexible routing, per-message acknowledgements, priorities and delays. It suits task queues and request routing.

Scheduled jobs that run once

Cron in every replica runs the job once per replica. Run scheduling in one place (a Kubernetes CronJob, a cloud scheduler, or one scheduler process) and still guard the job itself: take a lock (PostgreSQL advisory lock or a lease) so overlapping runs can't happen, and make it idempotent by recording which periods are done. Schedules in UTC avoid daylight-saving surprises.

Monitoring consumer lag

Measure lag: for each partition, the latest offset minus the group's committed offset, and better, how old the oldest unprocessed message is. Alert when lag in time crosses what the business tolerates and when it keeps growing, not on a fixed message count. Pair it with processing rate and error rate per consumer, so you can tell 'more traffic' from 'consumers stuck'.

Quick facts

Kafka ordering: Guaranteed only within a partition, never across a whole topic.
Kafka 4: Runs in KRaft mode only; ZooKeeper support was removed in 4.0.
Idempotent producer: Enabled by default since Kafka 3.0; stops producer retries from writing duplicates.
Consumers vs partitions: A group can't have more active consumers than partitions; extras sit idle (seen in our Kafka 4.1.2 run).
Partitioner difference: The Java client hashes keys with murmur2; librdkafka defaults to a CRC32-based partitioner.
Redis pub/sub: Fire-and-forget: subscribers that aren't connected miss messages. Streams keep them.