Chaos experiments and failure testing

Testing backends Senior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Order the steps

Order a chaos experiment.

  • Stop and roll back if the abort condition is met
  • Record findings and fix the weaknesses
  • Write the hypothesis and the abort condition
  • Pick a small blast radius
  • Inject the fault and watch SLO metrics

Choose an answer, then open Show answer below to check it.

Show answer

Answer: C D E A B

  1. Write the hypothesis and the abort condition Defines success and when to stop.
  2. Pick a small blast radius One instance or 1% of traffic.
  3. Inject the fault and watch SLO metrics And whether alerts fire.
  4. Stop and roll back if the abort condition is met Safety first.
  5. Record findings and fix the weaknesses Then repeat.

30-second answer

Run controlled experiments: state a hypothesis ('if one Redis replica dies, p99 stays under 300 ms'), inject the failure in a limited blast radius (kill an instance, add latency, drop a dependency), watch the SLO metrics, and stop immediately if they degrade beyond a limit. Start in staging with game days, then move carefully to production. The findings are usually missing timeouts, bad retries and untested failovers.

Practise saying it · 25 s. No recording, just a timer

Chaos testing.

Say your answer out loud, then check which of these points you covered:

  • Hypothesis
  • Small blast radius
  • Watch SLOs, abort
  • Latency faults too
  • Start in staging

Deep dive

Fault injection at the network layer (toxiproxy) works well in integration tests.

Test timeouts and retries by adding latency, not only hard failures.

Verify alerts fire during the experiment.

Netflix's Chaos Monkey randomly terminated production instances.

Common mistakes

  • Chaos without a hypothesis or abort switch.
  • Starting in production with the whole fleet.
  • Never testing database failover.

Senior take & follow-ups

Hypothesis, small blast radius, observe SLOs, abort, learn; start in staging.

Follow-up questions

Think of your answer first, then open the hint.

  1. What's the blast radius?

    Show hint

    How much traffic or how many users the experiment can affect.

  2. What do experiments usually find?

    Show hint

    Missing timeouts, retry storms, untested failover.

Related questions

Suggest a fix if something here is wrong or out of date.