Flaky tests

Testing backends Junior Python 3.13+ PostgreSQL 17+ Reviewed By Best Latest Apps

Pick the fix

A test asserts rows == [a, b] and fails 1 run in 50. The query has no ORDER BY. Fix?

Options
Hint 1 of 1

What guarantees row order?

Choose an answer, then open Show answer below to check it.

Show answer

Answer: A Add ORDER BY to the query (or compare as sets if order doesn't matter)

Correct. Order isn't guaranteed without it.

SQL returns rows in no particular order unless ORDER BY says so, and the order can change with plans, vacuums or parallel scans. Make the order explicit, or compare as a set.

Why not the others?
  • B Retry the test three times. Hides the bug.
  • C Add a sleep before the query. Order isn't about timing.
  • D Run tests serially. Order still isn't guaranteed.

30-second answer

Treat flakiness as a bug. Track which tests flake (re-run failures automatically to detect, not to hide), quarantine the worst so they don't block everyone, and fix the cause. Common backend causes: shared state between tests, ordering assumptions (SQL without ORDER BY), real time and sleeps, timeouts too tight for CI, network calls to real third parties, and races in async code.

Practise saying it · 25 s. No recording, just a timer

Flaky tests.

Say your answer out loud, then check which of these points you covered:

  • A bug
  • Detect and quarantine
  • Shared state, order, time
  • Wait on conditions

Deep dive

Run tests in random order to expose hidden dependencies.

Replace sleeps with waiting on a condition with a deadline.

Give each test unique data (ids, emails).

Quarantine needs an owner and a deadline.

Common mistakes

  • Retrying until green and moving on.
  • Asserting row order without ORDER BY.
  • Calling real external APIs in tests.

Senior take & follow-ups

Track, quarantine, fix; list the usual causes.

Follow-up questions

Think of your answer first, then open the hint.

  1. Why does SQL order flake?

    Show hint

    Without ORDER BY, row order isn't guaranteed and can change.

  2. Is auto-retry OK?

    Show hint

    For detection and reporting, not as the fix.

Related questions

Suggest a fix if something here is wrong or out of date.