Circuit breakers and bulkheads

Distributed systems Mid-level Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Predict the output

Threshold 3, cooldown 30 s. What does the trace show at t=5?

Python 3.13
import enum


class State(enum.Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half-open"


class Breaker:
    def __init__(self, threshold: int, cooldown: float) -> None:
        self.threshold, self.cooldown = threshold, cooldown
        self.state, self.failures, self.opened_at = State.CLOSED, 0, 0.0

    def allow(self, now: float) -> bool:
        if (
            self.state is State.OPEN
            and now - self.opened_at >= self.cooldown
        ):
            self.state = State.HALF_OPEN
        return self.state is not State.OPEN

    def record(self, ok: bool, now: float) -> None:
        if ok:
            self.state, self.failures = State.CLOSED, 0
            return
        self.failures += 1
        if (
            self.state is State.HALF_OPEN
            or self.failures >= self.threshold
        ):
            self.state, self.opened_at = State.OPEN, now


b = Breaker(threshold=3, cooldown=30)
trace = []
for t, ok in [
    (0, False),
    (1, False),
    (2, False),
    (5, True),
    (31, False),
    (62, True),
]:
    if b.allow(t):
        b.record(ok, t)
        trace.append(f"t={t} called -> {b.state.value}")
    else:
        trace.append(f"t={t} rejected fast")
print("\n".join(trace))

Options
Hint 1 of 1

When did it open, and for how long?

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A t=5 rejected fast

Correct. Three failures opened the breaker at t=2; the cooldown hasn't passed, so the call isn't made.

The third failure at t=2 trips the breaker, and opened_at becomes 2. allow() only switches to half-open once 30 s have passed, so at t=5 it refuses straight away without calling the dependency.

Why not the others?
  • B t=5 called -> closed. The breaker is open until t=32, so the call never happens.
  • C t=5 called -> half-open. Half-open only starts after the 30 s cooldown.
  • D t=5 called -> open. Open breakers don't call the dependency at all.

30-second answer

A circuit breaker watches calls to a dependency and, after too many failures, opens: calls fail fast for a cooldown period instead of waiting on a service that's down. Then it lets a trial request through (half-open); success closes it, failure opens it again. Retries try harder; a breaker stops trying, which protects both the caller's threads and the struggling dependency. Bulkheads add separate pools per dependency so one can't exhaust them all.

Practise saying it · 35 s. No recording, just a timer

Explain circuit breakers.

Say your answer out loud, then check which of these points you covered:

  • Closed, open, half-open
  • Fail fast while open
  • Different from retries
  • Fallbacks
  • Bulkheads per dependency

Deep dive

Trip on error rate over a window plus a minimum request count, not on one failure.

Return a fallback while open: cached data, a default, or a clear 503.

Expose breaker state as a metric; an open breaker is an alert.

Service meshes (Envoy outlier detection) can do this outside application code.

Common mistakes

  • Retrying inside an open breaker.
  • A single breaker for all dependencies.
  • No half-open state, so recovery is all-or-nothing.

Senior take & follow-ups

Describe the three states, contrast with retries, and pair it with bulkheads and fallbacks.

Follow-up questions

Think of your answer first, then open the hint.

  1. What's a bulkhead?

    Show hint

    Separate connection or thread pools per dependency, so one slow one can't take them all.

  2. Where should the breaker live?

    Show hint

    In the caller, per dependency (and sometimes per endpoint).

Suggest a fix if something here is wrong or out of date.