Design a distributed rate limiter

System design Senior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Pick the fix

Each of 50 servers enforces 100 requests/min in memory. A client sees about 5,000/min accepted. Fix?

Options
Hint 1 of 1

Where does each server keep its count?

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A Keep the counters in a shared store like Redis and update them atomically

Correct. One limit across all servers.

Each server only sees its own share of the traffic, so 50 separate limits add up to 50 × 100. A shared counter updated atomically gives one limit no matter which server gets the request.

Why not the others?
  • B Lower each server's limit to 2. Uneven load balancing makes this wrong for most clients.
  • C Add more servers. Raises the effective limit further.
  • D Use sticky sessions. Clients can still spread across connections.

30-second answer

Keep counters in a shared fast store, usually Redis, keyed by client and window, and update them atomically with a Lua script or INCR with expiry. A token bucket allows bursts up to its capacity; a sliding-window counter is smoother than fixed windows. Put the check at the gateway or in middleware, return 429 with Retry-After, and decide what happens if Redis is down: usually fail open for availability, with a local in-memory limit as a backstop.

Practise saying it · 35 s. No recording, just a timer

Distributed rate limiter.

Say your answer out loud, then check which of these points you covered:

  • Shared Redis counters
  • Atomic Lua/INCR
  • Token bucket / sliding window
  • 429 + Retry-After
  • Fail-open policy

Deep dive

Atomicity matters: read-then-write from many servers races.

Shard Redis by client key for scale; each key lives on one node.

Local token buckets with periodic sync trade precision for latency.

Different limits per plan and per endpoint.

Common mistakes

  • Per-server in-memory limits, so the real limit is 50× higher.
  • Non-atomic read-modify-write in Redis.
  • Failing closed and taking the API down when Redis blips.

Senior take & follow-ups

Shared atomic counters, algorithm choice, placement, 429 semantics, failure mode.

Follow-up questions

Think of your answer first, then open the hint.

  1. Fail open or closed?

    Show hint

    Usually open with a local backstop; closed for abuse-sensitive endpoints like login.

  2. Why Lua?

    Show hint

    The check and update run atomically on the Redis server.

Related questions

Suggest a fix if something here is wrong or out of date.