Leader election, leases and fencing tokens

Distributed systems Senior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Spot the bug

Workers write reports to storage. Which line lets a stale leader overwrite newer data?

  1. class Storage:
  2. def __init__(self) -> None:
  3. self.highest = 0
  4. self.data = ""
  5. ​
  6. def write(self, token: int, value: str) -> None:
  7. self.highest = max(self.highest, token)
  8. self.data = value
  9. ​
  10. ​
  11. s = Storage()
  12. s.write(34, "from new leader")
  13. s.write(33, "from paused old leader")
  14. print(s.data)
Hint 1 of 1

Which line should depend on the token?

Choose an answer, then open Show answer below to check it.

Show answer
  • Answer: line 8 self.data = value

The write is accepted no matter the token. It should reject token < self.highest before writing, so the old leader's write with token 33 fails.

30-second answer

Use a coordination service (etcd, ZooKeeper, Consul) or a database row to grant a lease: a lock with an expiry that the holder must renew. The danger is a leader that stalls, for example in a long GC pause, while its lease expires and a new leader takes over; when it wakes it still thinks it leads. Fencing tokens fix this: each lease grant comes with a number that only goes up, and the storage rejects writes carrying an older number.

Practise saying it · 35 s. No recording, just a timer

Explain fencing tokens.

Say your answer out loud, then check which of these points you covered:

  • Lease with expiry
  • Pauses outlive leases
  • Monotonic token per grant
  • Storage rejects older tokens

Deep dive

Lease expiry relies on bounded clock drift; keep lease durations well above expected pauses.

PostgreSQL advisory locks tie leadership to a session, which ends when the connection drops.

Kubernetes Lease objects are used for leader election by controllers.

Redis-based locks without fencing are fine for efficiency (avoid duplicate work) but not for correctness.

A store that checks fencing tokens Ran on Python 3.13.5 · mypy --strict
Python 3.13
class FencedStorage:
    def __init__(self) -> None:
        self.highest = 0
        self.data = ""

    def write(self, token: int, value: str) -> bool:
        if token < self.highest:
            return False
        self.highest, self.data = token, value
        return True


s = FencedStorage()
print(s.write(34, "new"), s.write(33, "stale"), s.data)

Common mistakes

  • Trusting a lease without fencing for correctness-critical writes.
  • Renewing the lease on a timer thread that keeps running while the work thread is stuck.
  • Treating 'I acquired the lock' as permanent.

Senior take & follow-ups

Explain leases, the pause scenario step by step, and fencing tokens as the fix.

Follow-up questions

Think of your answer first, then open the hint.

  1. Where do fencing tokens come from?

    Show hint

    The coordination service's version or revision number for the lock (etcd revision, ZooKeeper zxid).

  2. What if the storage can't check tokens?

    Show hint

    Then you can't make it safe with leases alone; make the operation idempotent.

Related questions

Suggest a fix if something here is wrong or out of date.