Why percentiles, not averages

Performance and scaling Junior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Predict the output

What do the two lines print?

Python 3.13
import statistics

latencies_ms = [20] * 95 + [900] * 5
cuts = statistics.quantiles(latencies_ms, n=100, method="inclusive")
print("mean", statistics.fmean(latencies_ms))
print("p50", cuts[49], "p99", cuts[98])

Options
Hint 1 of 1

95 × 20 + 5 × 900 = ?

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A mean 64.0 then p50 20.0 p99 900.0

Correct. The mean sits between the two groups; the percentiles show both.

The mean, (95 × 20 + 5 × 900) / 100 = 64, describes no actual request. The median is 20 and the 99th percentile is 900, which together show the slow tail.

Why not the others?
  • B mean 20.0 then p50 20.0 p99 20.0. Five requests take 900 ms.
  • C mean 460.0 then p50 460.0 p99 900.0. That would be the midpoint of two values, not the mean of 100.
  • D mean 64.0 then p50 64.0 p99 64.0. Percentiles come from the sorted values.

30-second answer

Averages hide the slow tail. If 95% of requests take 20 ms and 5% take 900 ms, the mean is 64 ms, which describes nobody, while p99 shows the 900 ms users actually feel. Tails also compound: a page that makes many backend calls hits the slow tail of at least one far more often than any single call does. Track p50 for typical experience and p95/p99 for the tail, per endpoint.

Practise saying it · 30 s. No recording, just a timer

Why percentiles?

Say your answer out loud, then check which of these points you covered:

  • Mean hides the tail
  • Fan-out amplifies tails
  • Aggregate histograms
  • p50 and p99 per endpoint

Deep dive

You can't average percentiles across servers; aggregate histograms instead.

Coordinated omission: load tools that wait for slow responses under-report the tail.

Hedged requests (send a backup after the p95 time) cut tail latency for idempotent reads.

Set SLOs on percentiles, for example p99 under 300 ms.

Mean vs percentiles Ran on Python 3.13.5 · mypy --strict
Python 3.13
import statistics

latencies_ms = [20] * 95 + [900] * 5
cuts = statistics.quantiles(latencies_ms, n=100, method="inclusive")
print("mean", statistics.fmean(latencies_ms))
print("p50", cuts[49], "p99", cuts[98])

Common mistakes

  • Reporting only the mean.
  • Averaging p99s from different hosts.
  • Measuring only inside the server, missing queueing before it.

Senior take & follow-ups

Show how the mean misleads, explain fan-out amplifying tails, and how to aggregate properly.

Follow-up questions

Think of your answer first, then open the hint.

  1. Why can't you average p99s?

    Show hint

    Percentiles aren't linear; combine the underlying histograms.

  2. What's a hedged request?

    Show hint

    A second copy sent after a delay; use whichever answers first.

Related questions

Suggest a fix if something here is wrong or out of date.