Dead-letter queues and poison messages

Queues, Kafka and event-driven systems Mid-level Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Pick the fix

A consumer fails on a message whose JSON is invalid. Best handling?

Options
Hint 1 of 1

Will a retry change the payload?

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A Dead-letter it with the error after one or two attempts and keep consuming

Correct. Retries can't fix a malformed payload.

Retries help with temporary failures. A message that can't be parsed will fail the same way every time, so move it aside with its error after one or two tries and let the partition move on.

Why not the others?
  • B Retry with backoff forever. It will never succeed and blocks the partition.
  • C Skip it silently. Data is lost with no trace.
  • D Crash the service to get attention. Restarts hit the same message again.

30-second answer

Classify failures. Transient errors (timeouts, a database restart) get retries with backoff. Permanent ones (bad payload, failed validation) won't succeed on retry, so after a small number of attempts move the message to a dead-letter queue with the error and metadata, and continue with the next. Alert on DLQ depth, fix the cause, then replay dead-lettered messages deliberately.

Practise saying it · 30 s. No recording, just a timer

Handle poison messages.

Say your answer out loud, then check which of these points you covered:

  • Transient vs permanent
  • Bounded retries
  • DLQ with context
  • Alert on depth
  • Replay after fixing

Deep dive

SQS and RabbitMQ have built-in DLQ redrive; with Kafka you write to a DLQ topic yourself.

Keep the original payload, headers, error text, attempt count and source offset.

For ordered streams, dead-lettering skips a message; decide whether later messages for that key must wait.

Validate at the producer too, so bad messages are rarer.

Common mistakes

  • Infinite retries on a parse error.
  • A DLQ nobody watches.
  • Replaying the DLQ without fixing the cause.

Senior take & follow-ups

Transient vs permanent, bounded retries, DLQ with context, alerting and replay.

Follow-up questions

Think of your answer first, then open the hint.

  1. What should a DLQ record contain?

    Show hint

    Payload, headers, error, attempts, source position, time.

  2. Retry topic or DLQ?

    Show hint

    Retry topics for transient errors with delays; DLQ for permanent ones.

Related questions

Suggest a fix if something here is wrong or out of date.