Performance and scaling cheat sheet

Load balancing strategies

Round robin sends requests to servers in turn, so a slow server still gets its full share and its queue grows. Least-connections (or least outstanding requests) sends new work to the server with the fewest in flight, so a slow server naturally gets less. Health checks remove servers that fail outright, but a slow server that still answers usually passes them. Layer 4 routes TCP connections; layer 7 sees HTTP and can route by path or header.

from itertools import cycle

servers = cycle(["A", "B", "C"])
sent = {"A": 0, "B": 0, "C": 0}
for _ in range(9):
    sent[next(servers)] += 1
print(sent)

Why percentiles, not averages

Averages hide the slow tail. If 95% of requests take 20 ms and 5% take 900 ms, the mean is 64 ms, which describes nobody, while p99 shows the 900 ms users actually feel. Tails also compound: a page that makes many backend calls hits the slow tail of at least one far more often than any single call does. Track p50 for typical experience and p95/p99 for the tail, per endpoint.

import statistics

latencies_ms = [20] * 95 + [900] * 5
cuts = statistics.quantiles(latencies_ms, n=100, method="inclusive")
print("mean", statistics.fmean(latencies_ms))
print("p50", cuts[49], "p99", cuts[98])

Capacity with Little's law

Little's law says the average number of requests in the system equals arrival rate times average time in the system: L = λW. At 200 requests per second taking 0.25 s each, about 50 are in flight at any moment, so you need at least 50 workers (threads, connections or async slots) just to keep up, plus headroom because servers run badly near 100% utilisation and queues grow sharply as load approaches capacity.

arrival_rps = 200
time_in_system_s = 0.25
in_flight = arrival_rps * time_in_system_s
print(in_flight)  # requests being served at any moment

per_worker_rps = 1 / time_in_system_s
print(arrival_rps / per_worker_rps)  # workers needed at 100% busy

Finding where time goes

Measure before guessing. Start from traces: they show whether time is in your code, the database, or another service. If it's the database, look at the slow query log and EXPLAIN ANALYZE. If it's your code, profile: cProfile or a sampling profiler like py-spy on the live process shows which functions burn CPU, and a flame graph makes the hot path obvious. Fix the biggest cost, then measure again.

import cProfile
import pstats


def slow_sum(n: int) -> int:
    return sum(i * i for i in range(n))


prof = cProfile.Profile()
prof.enable()
slow_sum(200_000)
prof.disable()
stats = pstats.Stats(prof)
print(stats.total_calls > 0)  # type: ignore[attr-defined]

Scaling up vs scaling out

Vertical scaling is simplest: no code changes, no distribution problems, and modern machines are huge, so it's often right for databases until it isn't. Its limits are a hard ceiling, cost that rises steeply at the top, and one machine as a failure point. Horizontal scaling adds capacity and redundancy, but needs stateless services, a load balancer, and for data, replication or sharding with all its consistency trade-offs. Usually: scale stateless tiers out, scale the database up first, then add replicas, then shard.

Scaling reads with replicas

Replicas stream the primary's changes and serve read-only queries, moving read load off the primary and giving failover candidates. The catch is replication lag: a user who just wrote may read an old value from a replica. Handle it by sending a user's reads to the primary for a short time after they write (read-your-writes), routing reads that need fresh data to the primary, or waiting until the replica has replayed the write's log position.

Batching database writes

Each separate INSERT costs a network round trip, statement parsing, and, if autocommitted, a WAL flush at every commit. Batching removes most of that: wrap many inserts in one transaction, send multi-row INSERTs or executemany (psycopg 3 pipelines it), or use COPY, PostgreSQL's bulk-load path, which is the fastest for large loads. Keep batches moderate (thousands of rows) so locks and memory stay bounded.

CREATE TEMP TABLE events (id int, kind text);
INSERT INTO events
SELECT g, 'view' FROM generate_series(1, 10000) AS g;
SELECT count(*) FROM events;

Compressing API responses

Usually yes for text like JSON: repetitive keys compress very well, cutting transfer time on slow mobile networks and egress cost. Use gzip or Brotli negotiated via Accept-Encoding, often at the proxy or CDN rather than in app code. Skip tiny responses (the overhead can make them bigger) and already-compressed content like images. Avoid compressing secrets alongside attacker-controlled text on the same response, the BREACH attack pattern.

import json
import zlib

rows = [{"id": i, "status": "shipped", "country": "IN"}
        for i in range(500)]
raw = json.dumps(rows).encode()
packed = zlib.compress(raw, 6)
print(len(raw) > 10 * len(packed))

Autoscaling services

Scale on a signal that tracks load and reacts early: CPU for CPU-bound services, in-flight requests or queue depth for I/O-bound ones and workers. Keep a minimum that handles normal peaks, since new instances take time to boot and warm caches. Add cooldowns so it doesn't flap, cap the maximum so you don't overwhelm the database, and pre-scale for known events. Autoscaling doesn't help when the bottleneck is a shared dependency.

Load testing a backend

Decide the question first: maximum throughput at an SLO, behaviour at expected peak, or how it fails past capacity. Use a production-like environment and data volume, realistic request mixes and think times, and an open-model generator (fixed arrival rate) so slow responses don't quietly reduce load. Ramp up in steps, watch latency percentiles, errors and saturation of every dependency, and find the first thing that breaks.

Hot keys and hot partitions

A hot key defeats even partitioning, because all its traffic hashes to one place. For reads, cache it close to callers (in-process or replicated cache entries with a suffix such as key#1..key#8), and collapse duplicate requests. For writes, split the key: write to N sub-keys (counter#0..counter#7) and sum on read, or buffer and batch updates. Detect hot keys with per-key metrics or sampling before they hurt.

Memory growth in Python services

First confirm it's a leak and not a cache warming up or fragmentation: watch RSS over hours under steady load. Common causes are unbounded caches and module-level dicts, objects kept alive by references in globals or closures, growing per-request lists, and C extensions. Use tracemalloc to compare snapshots and see which lines allocate the growth, then bound the structure (an LRU with a max size) or fix the reference.