Idempotent consumers and deduplication

Queues, Kafka and event-driven systems Senior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Predict the output

The same event arrives twice. What does the final SELECT print?

SQL · PostgreSQL 17
CREATE TABLE inbox (event_id text PRIMARY KEY);
CREATE TABLE points (user_id int PRIMARY KEY, total int);
INSERT INTO points VALUES (1, 0);
WITH new AS (
  INSERT INTO inbox VALUES ('evt-9') ON CONFLICT DO NOTHING
  RETURNING 1
)
UPDATE points SET total = total + 10
WHERE user_id = 1 AND EXISTS (SELECT 1 FROM new);
WITH new AS (
  INSERT INTO inbox VALUES ('evt-9') ON CONFLICT DO NOTHING
  RETURNING 1
)
UPDATE points SET total = total + 10
WHERE user_id = 1 AND EXISTS (SELECT 1 FROM new);
SELECT total FROM points;

Options
Hint 1 of 1

What does the CTE return the second time?

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A 10

Correct. The second insert conflicts and returns no row, so the update matches nothing.

The inbox table records which event ids were already handled. The second INSERT hits the primary key and DO NOTHING returns no row, so EXISTS is false and the second +10 is skipped.

Why not the others?
  • B 20. That's what happens without the inbox check.
  • C 0. The first delivery is new and applies.
  • D A unique violation. ON CONFLICT DO NOTHING absorbs it.
Result on PostgreSQL 17 (psql)
total
10

(1 row)

30-second answer

Either make the effect naturally idempotent, for example 'set status to paid' or an upsert keyed by the event's entity, or record each processed message id in a table with a unique constraint inside the same transaction as the effect. A redelivered message then hits the constraint, and you skip it. Keep processed ids at least as long as the broker can redeliver.

Practise saying it · 30 s. No recording, just a timer

Make a consumer idempotent.

Say your answer out loud, then check which of these points you covered:

  • Natural idempotency
  • Inbox table, unique id
  • Same transaction as effect
  • Stable message ids

Deep dive

The dedup insert and the business write must commit together, or a crash between them breaks it.

Producers must give retried messages the same id.

Effects outside your database (emails, payments) need their own idempotency keys.

Expire old dedup rows by time or partition, not by count.

Common mistakes

  • Checking an in-memory set that's lost on restart.
  • Recording the id after the effect, in a separate transaction.
  • Using the broker's delivery tag, which changes on redelivery, as the id.

Senior take & follow-ups

Natural idempotency or an inbox table with a unique id, written in the same transaction.

Follow-up questions

Think of your answer first, then open the hint.

  1. What makes 'add 10 points' idempotent?

    Show hint

    A unique ledger row per event id instead of an in-place increment.

  2. How long keep processed ids?

    Show hint

    Longer than the maximum redelivery window.

Suggest a fix if something here is wrong or out of date.