Flaky Tests Diagnosis & Stabilization (Advanced)

Theory

Flaky tests waste engineering time and erode trust in CI.

Advanced flakiness work means:

• reproducing locally,
• identifying non-determinism sources,
• applying stabilizing techniques,
• measuring reduction in flake rate.

1) Common flakiness sources

• Race conditions (missing await, callbacks not awaited)
• Shared mutable state across tests
• Real time usage (Date.now(), timers) without control
• Randomness without a seed
• External IO (network, filesystem, real DB) without isolation
• Parallel test execution causing interference

2) Async correctness beats “waiting”

Waiting with timeouts is fragile.

Prefer:

• await the actual promise that represents completion
• use act() / proper lifecycle waits (framework-dependent)
• coordinate async dependencies deterministically

3) Determinism checklist

To stabilize a test, ensure:

• fixed clock (fake timers)
• fixed random seed
• isolated data (fresh DB schema, transactions, cleanup)
• no dependency on execution order

Code Example (TypeScript/Jest: fake timers + seeded randomness)

// notification.ts
export function sendWelcomeEmail(
  user: { email: string },
  clock: () => number,
  send: (email: string, subject: string) => void,
) {
  const timestamp = clock();
  send(user.email, `Welcome! (${timestamp})`);
}

// notification.test.ts
import { sendWelcomeEmail } from "./notification";

test("should send deterministic welcome email", () => {
  // Fake clock
  const fixedNow = 1710000000000;
  const clock = () => fixedNow;

  const sent: any[] = [];
  const send = (email: string, subject: string) => sent.push({ email, subject });

  sendWelcomeEmail({ email: "[email protected]" }, clock, send);

  expect(sent).toEqual([
    { email: "[email protected]", subject: "Welcome! (1710000000000)" }
  ]);
});

Practice

1. Pick a flaky test you’ve seen.
2. Record:
   • when it fails (timing, environment, parallelism)
   • logs around failure (state + timestamps)
3. Apply exactly one stabilization technique:
   • control time/random
   • isolate shared state
   • replace “sleep/wait” with proper awaits
4. Re-run with CI-like settings (parallelism) to verify.

Common pitfalls

• Retrying everything (hides real problems)
• Using delays/timeouts instead of waiting for real completion
• Shared singletons (global caches, in-memory maps) not reset between tests
• Tests that depend on DB state not cleaned between runs