Detecting failures and avoiding split brain

Distributed systems Senior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Pick the fix

A two-node database pair keeps failing over and sometimes both nodes accept writes. Best fix?

Options
Hint 1 of 1

With two nodes, who breaks a tie?

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A Add a third voter (witness) so promotion needs a majority, and fence the old primary

Correct. A majority stops both sides from deciding alone; fencing stops the old one writing.

With two nodes, neither can tell a dead peer from a broken link, so both may promote themselves. A third voter means promotion needs 2 of 3 votes, and fencing stops the old primary writing.

Why not the others?
  • B Shorten heartbeat timeouts. More false failovers.
  • C Add a load balancer health check. The balancer can't tell partitioned from dead either.
  • D Make replication asynchronous. Doesn't address who's primary.

30-second answer

A node is presumed dead when it misses heartbeats for a timeout, but a slow node, a long pause and a network cut all look the same. Short timeouts give false failovers; long ones give slow recovery. Split brain happens when the old primary is cut off but alive and keeps accepting writes. Prevent it by requiring a majority (quorum) to act as primary, fencing the old one (revoking its access or powering it off), and fencing tokens on writes.

Practise saying it · 35 s. No recording, just a timer

How do you avoid split brain?

Say your answer out loud, then check which of these points you covered:

  • Slow vs dead is ambiguous
  • Timeouts trade speed vs false alarms
  • Majority to promote
  • Fence the old primary

Deep dive

A primary that can't reach a majority should stop accepting writes on its own.

Two-node clusters need a third voter (witness) to break ties.

Phi-accrual failure detectors output a suspicion level instead of a yes or no, adapting to the network.

Patroni, a common PostgreSQL HA manager, uses a DCS (etcd, Consul) as the arbiter.

Common mistakes

  • Automatic failover in a 2-node cluster with no witness.
  • Failover timeouts shorter than routine GC or I/O pauses.
  • Not fencing the old primary after promotion.

Senior take & follow-ups

Admit you can't distinguish slow from dead, then cover quorum, fencing and timeout tuning.

Follow-up questions

Think of your answer first, then open the hint.

  1. What is STONITH?

    Show hint

    'Shoot the other node in the head': forcibly power off the old primary.

  2. Why do false failovers hurt?

    Show hint

    Failover itself causes errors and can lose async-replicated writes.

Related questions

Suggest a fix if something here is wrong or out of date.