Committing offsets: at-most-once vs at-least-once

Queues, Kafka and event-driven systems Senior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Spot the bug

This consumer can lose messages on a crash. Which line causes it?

  1. def handle(batch: list[str], db: list[str],
  2. commits: list[int], offset: int) -> None:
  3. commits.append(offset + len(batch))
  4. for msg in batch:
  5. db.append(msg.upper())
  6. ​
  7. ​
  8. db: list[str] = []
  9. commits: list[int] = []
  10. handle(["a", "b"], db, commits, 0)
  11. print(commits, db)
Hint 1 of 1

What if the process dies inside the loop?

Choose an answer, then open Show answer below to check it.

Show answer
  • Answer: line 3 commits.append(offset + len(batch))

The offset is committed before the messages are processed. A crash during the loop leaves them committed but never written: at-most-once. Commit after the loop.

30-second answer

Commit after processing for at-least-once: if the consumer crashes mid-batch, the messages are read again, so the handler must be idempotent. Committing before processing gives at-most-once: a crash loses the messages that were committed but not processed. For read-process-write pipelines inside Kafka, transactions can commit the output and the input offsets atomically, which Kafka calls exactly-once.

Practise saying it · 30 s. No recording, just a timer

When do you commit offsets?

Say your answer out loud, then check which of these points you covered:

  • After processing: at-least-once
  • Before: at-most-once
  • Idempotent handlers
  • Offset with result in one transaction

Deep dive

enable.auto.commit=true commits on a timer, which can commit messages your async code hasn't finished.

Store the offset in your database in the same transaction as the result, and seek to it on startup, for effectively-once with an external store.

Idempotent producers (default since Kafka 3.0) stop producer retries from writing duplicates.

Commit in batches to limit overhead, accepting more replay after a crash.

Common mistakes

  • Auto-commit with work handed to a thread pool.
  • Committing before writing to the database.
  • Assuming Kafka transactions make an HTTP call exactly-once.

Senior take & follow-ups

Map commit timing to semantics, require idempotent handlers, and mention offsets stored with results.

Follow-up questions

Think of your answer first, then open the hint.

  1. How do you get effectively-once into PostgreSQL?

    Show hint

    Write the result and the offset in one database transaction.

  2. What does the idempotent producer prevent?

    Show hint

    Duplicates from the producer's own retries.

Related questions

Suggest a fix if something here is wrong or out of date.