Observability and operations cheat sheet

Structured logging

Structured logs (usually JSON, one event per line) let you filter and aggregate by field instead of grepping text: order_id = 42 AND level = ERROR. Every line should carry a timestamp, level, service and version, a request or trace id to tie lines together, and the identifiers of the thing being worked on. Keep messages constant ('payment captured') and put variable data in fields, so identical events group together.

Metrics and label cardinality

Each unique combination of a metric's labels is a separate time series that the system stores and indexes. Labels like endpoint, status and region multiply into a manageable number, but adding an unbounded label such as user id, request id or full URL multiplies series by the number of distinct values, blowing up memory, cost and query time. Keep labels bounded and low-cardinality; put per-user detail in logs or traces instead.

endpoints = 40
status_codes = 8
regions = 4
series = endpoints * status_codes * regions
print(series)
users = 100_000
print(series * users)  # after adding a user_id label

Distributed tracing

Each request gets a trace id; each unit of work (an HTTP handler, a database query, a call to another service) is a span with a start, duration, parent and attributes. Services pass the trace context along in headers (W3C traceparent), so the backend can assemble the whole tree. A trace shows where time went across services for one request, which metrics (aggregates) and logs (separate lines) can't show on their own.

SLOs, error budgets and alerts

An SLI is a measured ratio of good events, like the share of requests that succeed under 300 ms. An SLO is the target for it over a window, say 99.9% over 30 days. The error budget is what's left: 0.1% of requests, or about 43 minutes of full downtime in 30 days. Alert on burn rate, how fast you're spending the budget, using a fast window for sharp outages and a slow one for steady leaks, so pages mean users are hurting.

slo = 0.999
minutes_30_days = 30 * 24 * 60
budget = (1 - slo) * minutes_30_days
print(round(budget, 1))

error_rate = 0.0144  # observed over the last hour
burn = error_rate / (1 - slo)
print(round(burn, 1))

Liveness and readiness checks

Liveness asks 'is this process stuck beyond repair?' and a failure restarts it, so it should check only the process itself (it can serve a trivial request). Readiness asks 'should this instance get traffic now?' and a failure just removes it from the load balancer, so it can check warm-up and critical dependencies. Putting dependency checks in liveness is dangerous: a database blip makes every instance restart at once.

RED and USE dashboards

For each request-serving service, RED: Rate (requests per second), Errors (failed requests per second or share), and Duration (latency histogram, shown as p50/p95/p99). For each resource (CPU, memory, disk, connection pools, queues), USE: Utilisation, Saturation (queued work) and Errors. RED tells you users are affected; USE helps find which resource is the cause.

Keeping secrets and personal data out of logs

Logs are copied widely, kept long and read by many people, so treat them as semi-public. Log ids, not personal values; never log request bodies or headers wholesale; redact known sensitive fields (Authorization, cookies, password, card numbers) in the logging layer with an allow-list of what may be logged; and scrub query strings. Set retention limits and access controls, and scan logs for leaks.

Running an incident

Declare it and name an incident commander who coordinates rather than debugs. Open one channel and post short status updates on a schedule. Mitigate first: roll back the latest deploy, flip the feature flag, fail over or shed load, before hunting the root cause. Check the impact with your SLO dashboards, communicate to users through the status page, and keep a timeline. Afterwards, run a blameless postmortem.

Blameless postmortems

It explains what happened and why the system allowed it, without blaming people: people act sensibly given what they knew, so the fix is in tools, processes and defaults. A good one has the impact (with numbers), a timeline, contributing factors (usually several), what went well, and a short list of action items with owners and dates that make the failure less likely or less harmful. Share it widely and track the actions to done.

Audit logs

Application logs help engineers debug and can be sampled or dropped. Audit logs are a record of who did what to which resource and when, kept for security, compliance and customer trust, so they must be complete, tamper-evident and retained for a defined period. Write an event for every security-relevant action (logins, permission changes, data exports, admin actions) with actor, action, target, time, source IP and result, to append-only storage.