Flaky Tests Diagnosis & Stabilization (Advanced)

Theory

Flaky tests waste engineering time and erode trust in CI.

Advanced flakiness work means:

• reproducing locally,
• identifying non-determinism sources,
• applying stabilizing techniques,
• measuring reduction in flake rate.

1) Common flakiness sources

• Race conditions (missing await, callbacks not awaited)
• Shared mutable state across tests
• Real time usage (Date.now(), timers) without control
• Randomness without a seed
• External IO (network, filesystem, real DB) without isolation
• Parallel test execution causing interference

2) Async correctness beats “waiting”

Waiting with timeouts is fragile.

Prefer:

• await the actual promise that represents completion
• use act() / proper lifecycle waits (framework-dependent)
• coordinate async dependencies deterministically

3) Determinism checklist

To stabilize a test, ensure:

• fixed clock (fake timers)
• fixed random seed
• isolated data (fresh DB schema, transactions, cleanup)
• no dependency on execution order

Code Example (TypeScript/Jest: an injected clock)

Every example runs under Jest with ts-jest, with a jest.config.js containing module.exports = { preset: "ts-jest", testEnvironment: "node" }; in the same directory.

// notification.ts
export function sendWelcomeEmail(
  user: { email: string },
  clock: () => number,
  send: (email: string, subject: string) => void,
) {
  const timestamp = clock();
  send(user.email, `Welcome! (${timestamp})`);
}

// notification.test.ts
import { sendWelcomeEmail } from "./notification";

test("should send deterministic welcome email", () => {
  // Fake clock
  const fixedNow = 1710000000000;
  const clock = () => fixedNow;

  const sent: any[] = [];
  const send = (email: string, subject: string) => sent.push({ email, subject });

  sendWelcomeEmail({ email: "a@example.com" }, clock, send);

  expect(sent).toEqual([
    { email: "a@example.com", subject: "Welcome! (1710000000000)" }
  ]);
});

$ npx jest notification.test.ts
PASS ./notification.test.ts
  √ should send deterministic welcome email (4 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.882 s
Ran all test suites matching notification.test.ts.

Had sendWelcomeEmail called Date.now() itself, the subject would differ on every run and the test could only assert a pattern. With the clock passed in, the test pins it and asserts the exact subject. send is captured into an array rather than performing real IO, so the test cannot be affected by a mail server either.

Code Example (TypeScript/Jest: freezing the system clock)

Injecting a clock means changing the function’s signature. When the code already calls Date.now() or new Date() directly, Jest can fake the system clock itself.

// welcome.ts
export function welcomeSubject(): string {
  return `Welcome! (${Date.now()}, ${new Date().toISOString()})`;
}

// welcome.test.ts
import { welcomeSubject } from "./welcome";

beforeEach(() => {
  jest.useFakeTimers();
  jest.setSystemTime(new Date("2024-03-09T16:00:00Z"));
});
afterEach(() => jest.useRealTimers());

test("Date.now() and new Date() both read the fake clock", () => {
  expect(welcomeSubject()).toBe("Welcome! (1710000000000, 2024-03-09T16:00:00.000Z)");
});

test("the fake clock only moves when the test moves it", () => {
  jest.advanceTimersByTime(1_000);
  expect(Date.now()).toBe(1710000001000);
});

$ npx jest welcome.test.ts
PASS ./welcome.test.ts
  √ Date.now() and new Date() both read the fake clock (8 ms)
  √ the fake clock only moves when the test moves it (2 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        2.698 s
Ran all test suites matching welcome.test.ts.

welcomeSubject takes no clock parameter, yet the test asserts its exact output. jest.useFakeTimers() replaces Date along with the timer functions, and setSystemTime sets where the fake clock starts. The 8 ms the first test took in real time didn’t move the fake clock. Only advanceTimersByTime(1_000) did, by exactly 1,000 ms. beforeEach resets the clock for every test, so neither test depends on the other. Use an injected clock when you control the code’s design, and setSystemTime when you are testing code that reads the time directly.

Code Example (TypeScript/Jest: unseeded vs seeded randomness)

A single green run proves little when a test depends on randomness. Run it many times and count.

// shuffle.ts
export function shuffle<T>(items: T[], rand: () => number = Math.random): T[] {
  const out = [...items];
  for (let i = out.length - 1; i > 0; i--) {
    const j = Math.floor(rand() * (i + 1));
    [out[i], out[j]] = [out[j], out[i]];
  }
  return out;
}

// shuffle.test.ts
import { shuffle } from "./shuffle";

test("shuffle changes the order", () => {
  expect(shuffle([1, 2, 3])).not.toEqual([1, 2, 3]);
});

$ for i in $(seq 20); do npx jest shuffle.test.ts >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
      6 FAIL
     14 pass
$ for i in $(seq 20); do npx jest shuffle.test.ts >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
      2 FAIL
     18 pass

The same unchanged test failed 6 times in 20 runs, then 2 times in 20. A correct Fisher-Yates shuffle of three items returns the original order one time in six, about 3.3 failures per 20 runs, and real tallies scatter around that. The test is wrong, not the shuffle: “the order changes” is not something a shuffle guarantees. Two small batches also gave noticeably different flake rates, so measure over many runs before you decide a fix has worked.

The fix has two parts. Pin the randomness where the test needs an exact answer, and assert only real invariants where it doesn’t.

// seeded.ts: mulberry32, a tiny seeded PRNG. Same seed, same sequence.
export function seeded(seed: number): () => number {
  let a = seed >>> 0;
  return () => {
    a = (a + 0x6d2b79f5) >>> 0;
    let t = a;
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

// shuffle-seeded.test.ts
import { shuffle } from "./shuffle";
import { seeded } from "./seeded";

test("seed 42 always gives the same order", () => {
  expect(shuffle([1, 2, 3], seeded(42))).toEqual([3, 1, 2]);
});

test("any shuffle keeps exactly the same items", () => {
  const out = shuffle([1, 2, 3]);
  expect([...out].sort()).toEqual([1, 2, 3]);
});

$ npx jest shuffle-seeded.test.ts
PASS ./shuffle-seeded.test.ts
  √ seed 42 always gives the same order (2 ms)
  √ any shuffle keeps exactly the same items

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        1.867 s
Ran all test suites matching shuffle-seeded.test.ts.
$ for i in $(seq 20); do npx jest shuffle-seeded.test.ts >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
     20 pass

Twenty out of twenty. The first test gets an exact, reproducible order from a fixed seed. When it fails, it fails every time, so it can be debugged. The second still uses Math.random, but it asserts something true of every possible shuffle, so it can’t flake.

Code Example (TypeScript/Jest: fake timers)

Code that waits on a real timer makes its test wait too, and slow tests get “fixed” with shorter delays that then flake. Jest’s fake timers move time forward on command.

// reminder.ts
export function scheduleReminder(send: () => void, delayMs: number) {
  setTimeout(send, delayMs);
}

// reminder.test.ts
import { scheduleReminder } from "./reminder";

beforeEach(() => jest.useFakeTimers());
afterEach(() => jest.useRealTimers());

test("sends the reminder after exactly one minute", () => {
  const send = jest.fn();
  scheduleReminder(send, 60_000);

  jest.advanceTimersByTime(59_999);
  expect(send).not.toHaveBeenCalled();

  jest.advanceTimersByTime(1);
  expect(send).toHaveBeenCalledTimes(1);
});

$ npx jest reminder.test.ts
PASS ./reminder.test.ts
  √ sends the reminder after exactly one minute (19 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        4.612 s
Ran all test suites matching reminder.test.ts.

A one-minute timer was tested in 19 ms, and the test checks the boundary to the millisecond: nothing at 59,999 ms, one call at 60,000 ms. No real time passes, so a slow CI machine can’t change the result. afterEach restores real timers so the fake clock doesn’t leak into other tests in the same file.

Code Example (TypeScript/Jest: a fixed sleep vs waiting for completion)

The job below takes a variable 5-25 ms, standing in for work whose duration depends on load. The first test guesses how long to wait.

// job.ts
export function runJob(results: string[]): Promise<void> {
  const workMs = 5 + Math.random() * 20;
  return new Promise((resolve) =>
    setTimeout(() => {
      results.push("done");
      resolve();
    }, workMs),
  );
}

// job-sleep.test.ts
import { runJob } from "./job";

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

test("job finishes (checked after a fixed sleep)", async () => {
  const results: string[] = [];
  runJob(results); // not awaited
  await sleep(15);
  expect(results).toEqual(["done"]);
});

$ for i in $(seq 20); do npx jest job-sleep.test.ts >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
     12 FAIL
      8 pass
$ npx jest job-sleep.test.ts
FAIL ./job-sleep.test.ts
  × job finishes (checked after a fixed sleep) (34 ms)

  ● job finishes (checked after a fixed sleep)

    expect(received).toEqual(expected) // deep equality

    - Expected  - 3
    + Received  + 1

    - Array [
    -   "done",
    - ]
    + Array []

       7 |   runJob(results); // not awaited
       8 |   await sleep(15);
    >  9 |   expect(results).toEqual(["done"]);
         |                   ^
      10 | });
      11 |

      at Object.<anonymous> (job-sleep.test.ts:9:19)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        3.27 s
Ran all test suites matching job-sleep.test.ts.

Twelve failures in twenty runs. The failure message is the classic flaky-test signature: the assertion ran while the job was still working, so results was empty. The tempting fix is a longer sleep. Change sleep(15) to sleep(30) and tally again:

$ for i in $(seq 20); do npx jest job-sleep.test.ts >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
     20 pass

Twenty passes on this machine, because 30 ms is longer than the slowest job here. That doesn’t make the test correct. The sleep is still a guess about how long the job takes, and a slower or busier CI runner can push the job past it. Every run also now pays the full 30 ms whether the job took 5 ms or 25. The fix is to wait for the event itself.

// job-await.test.ts
import { runJob } from "./job";

async function waitFor(condition: () => boolean, timeoutMs = 1000) {
  const start = Date.now();
  while (!condition()) {
    if (Date.now() - start > timeoutMs) throw new Error("condition not met in time");
    await new Promise((r) => setTimeout(r, 5));
  }
}

test("job finishes (await its promise)", async () => {
  const results: string[] = [];
  await runJob(results);
  expect(results).toEqual(["done"]);
});

test("job finishes (poll the observable condition)", async () => {
  const results: string[] = [];
  runJob(results); // imagine no promise is exposed
  await waitFor(() => results.length === 1);
  expect(results).toEqual(["done"]);
});

$ npx jest job-await.test.ts
PASS ./job-await.test.ts
  √ job finishes (await its promise) (34 ms)
  √ job finishes (poll the observable condition) (36 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        3.603 s
Ran all test suites matching job-await.test.ts.
$ for i in $(seq 20); do npx jest job-await.test.ts >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
     20 pass

Same job, same random durations, twenty passes. Awaiting the promise is the best fix because the test resumes exactly when the work completes. When the code under test exposes no promise, polling the condition you actually care about is the next best fix. waitFor returns as soon as results has its entry, and its 1-second timeout is only a ceiling for a job that never finishes. Unlike a fixed sleep, it doesn’t make the test wait that long when the job is fast.

Practice

1. Find an order dependency. cart.ts keeps its items in a module-level array, and cart.test.ts has a “new cart is empty” test followed by an “add one item” test. Both pass in file order. Run the file with npx jest --randomize --seed=N for seeds 1-6, then fix it so every seed passes.
2. Watch a retry hide the flake. Add jest.retryTimes(3, { logErrorsBeforeRetry: true }) to a copy of the original shuffle test, saved as shuffle-retry.test.ts, and tally 20 runs. Is the test fixed?
3. Test the poller’s ceiling. Point waitFor at a job that never records completion. How long does the test take, and what does the failure say?
4. Replace the sleep with fake timers. Rewrite job-sleep.test.ts with jest.useFakeTimers() and jest.runAllTimers() instead of awaiting anything, and tally 20 runs.

// cart.ts
const items: string[] = []; // module-level: shared by every test in the file

export function addItem(name: string) {
  items.push(name);
}

export function itemCount() {
  return items.length;
}

export function clearCart() {
  items.length = 0;
}

// cart.test.ts
import { addItem, itemCount } from "./cart";

test("a new cart is empty", () => {
  expect(itemCount()).toBe(0);
});

test("adding one item gives a count of 1", () => {
  addItem("apple");
  expect(itemCount()).toBe(1);
});

$ npx jest cart.test.ts >/dev/null 2>&1 && echo "default order: pass" || echo "default order: FAIL"
default order: pass
$ for s in 1 2 3 4 5 6; do echo "seed $s: $(npx jest --randomize --seed=$s cart.test.ts >/dev/null 2>&1 && echo pass || echo FAIL)"; done
seed 1: pass
seed 2: FAIL
seed 3: pass
seed 4: FAIL
seed 5: pass
seed 6: FAIL
$ npx jest --randomize --seed=2 cart.test.ts
FAIL ./cart.test.ts
  √ adding one item gives a count of 1 (6 ms)
  × a new cart is empty (7 ms)

  ● a new cart is empty

    expect(received).toBe(expected) // Object.is equality

    Expected: 0
    Received: 1

      2 |
      3 | test("a new cart is empty", () => {
    > 4 |   expect(itemCount()).toBe(0);
        |                       ^
      5 | });
      6 |
      7 | test("adding one item gives a count of 1", () => {

      at Object.<anonymous> (cart.test.ts:4:23)

Seed:        2
Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 passed, 2 total
Snapshots:   0 total
Time:        3.556 s, estimated 4 s
Ran all test suites matching cart.test.ts.

// cart.test.ts
import { addItem, itemCount, clearCart } from "./cart";

beforeEach(() => clearCart());

test("a new cart is empty", () => {
  expect(itemCount()).toBe(0);
});

test("adding one item gives a count of 1", () => {
  addItem("apple");
  expect(itemCount()).toBe(1);
});

$ for s in 1 2 3 4 5 6; do echo "seed $s: $(npx jest --randomize --seed=$s cart.test.ts >/dev/null 2>&1 && echo pass || echo FAIL)"; done
seed 1: pass
seed 2: pass
seed 3: pass
seed 4: pass
seed 5: pass
seed 6: pass

// shuffle-retry.test.ts
import { shuffle } from "./shuffle";

jest.retryTimes(3, { logErrorsBeforeRetry: true });

test("shuffle changes the order", () => {
  expect(shuffle([1, 2, 3])).not.toEqual([1, 2, 3]);
});

$ for i in $(seq 20); do npx jest shuffle-retry.test.ts >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
     20 pass

$ npx jest shuffle-retry.test.ts
 LOGGING RETRY ERRORS  shuffle changes the order
 RETRY 1 

    expect(received).not.toEqual(expected) // deep equality

    Expected: not [1, 2, 3]

      4 |
      5 | test("shuffle changes the order", () => {
    > 6 |   expect(shuffle([1, 2, 3])).not.toEqual([1, 2, 3]);
        |                                  ^
      7 | });
      8 |

      at Object.<anonymous> (shuffle-retry.test.ts:6:34)

PASS ./shuffle-retry.test.ts
  √ shuffle changes the order (1 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        3.289 s
Ran all test suites matching shuffle-retry.test.ts.

// job-stuck.test.ts
async function waitFor(condition: () => boolean, timeoutMs = 1000) {
  const start = Date.now();
  while (!condition()) {
    if (Date.now() - start > timeoutMs) throw new Error("condition not met in time");
    await new Promise((r) => setTimeout(r, 5));
  }
}

function runStuckJob(results: string[]) {
  // bug: the job never records completion
}

test("a job that never finishes", async () => {
  const results: string[] = [];
  runStuckJob(results);
  await waitFor(() => results.length === 1);
});

$ npx jest job-stuck.test.ts
FAIL ./job-stuck.test.ts
  × a job that never finishes (1005 ms)

  ● a job that never finishes

    condition not met in time

      2 |   const start = Date.now();
      3 |   while (!condition()) {
    > 4 |     if (Date.now() - start > timeoutMs) throw new Error("condition not met in time");
        |                                               ^
      5 |     await new Promise((r) => setTimeout(r, 5));
      6 |   }
      7 | }

      at waitFor (job-stuck.test.ts:4:47)
      at Object.<anonymous> (job-stuck.test.ts:16:3)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        2.514 s
Ran all test suites matching job-stuck.test.ts.

// job-fake.test.ts
import { runJob } from "./job";

beforeEach(() => jest.useFakeTimers());
afterEach(() => jest.useRealTimers());

test("job finishes (fake timers, no real waiting)", () => {
  const results: string[] = [];
  runJob(results);
  jest.runAllTimers();
  expect(results).toEqual(["done"]);
});

$ npx jest job-fake.test.ts
PASS ./job-fake.test.ts
  √ job finishes (fake timers, no real waiting) (3 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.405 s
Ran all test suites matching job-fake.test.ts.
$ for i in $(seq 20); do npx jest job-fake.test.ts >/dev/null 2>&1 && echo pass || echo FAIL; done | sort | uniq -c
     20 pass

Common pitfalls

• Retrying everything (hides real problems)
• Using delays/timeouts instead of waiting for real completion
• Shared singletons (global caches, in-memory maps) not reset between tests
• Tests that depend on DB state not cleaned between runs