Reducing Noise in Automated Memory Tests

Your CI memory check fails one run in five, the team has started re-running it until it goes green, and a real leak shipped last month because nobody believed the red build. This guide from Automated Memory Leak Detection in CI, in the Browser DevTools & Performance Profiling Workflows section, shows where the noise in memory measurements comes from and the specific techniques that make a memory test fail only when something actually leaks.

Symptom Root Cause Immediate Action Measurable Impact
Heap delta varies ±5 MB between identical runs Measuring before GC settles; lazy compilation Force GC several times with pauses before sampling Variance drops to well under 1 MB
First iteration always “leaks” One-time initialisation counted as growth Warm up, then measure from a post-warm-up baseline First-run artefacts excluded
Small leaks are indistinguishable from noise Too few iterations Scale iterations until expected leak ≫ noise Leaks of a few KB per action detectable
Test fails only on busy CI runners Timers and network timing differ under load Mock time and network; wait for explicit UI states Same result on laptop and CI
Single outlier run fails the build One sample decides pass/fail Take several samples and compare medians or slopes Outliers stop failing builds

Root Cause: Memory Readings Are Snapshots of a Moving System

A JavaScript heap is never still. Objects are allocated continuously, and the collector reclaims them on its own schedule — minor collections every few megabytes of allocation, major collections when the old generation crosses a dynamic limit. A reading of heapUsed or JSHeapUsedSize taken at an arbitrary moment includes an unknown amount of garbage. Two identical runs can differ by megabytes simply because one sample landed just before a collection and the other just after.

Beyond garbage, several one-time effects add growth that is not a leak: code compiled on first use, inline caches and feedback vectors that fill as functions warm up, lazily loaded modules and chunks, font and style caches, and framework internals such as a router’s route table. All of these happen on the first iteration of a flow, which is why naïve “before vs after one run” tests fail consistently on correct code.

Environmental noise adds more: timers that fire at different points in slower CI runs, network responses that arrive before or after a component unmounts, parallel tests sharing the same process, and different Node or Chrome versions with different heap sizing. And finally there is statistical noise — any threshold set close to the natural variation of a single reading will fail a fraction of the time.

Each source has a matching countermeasure: force collection so readings measure live objects only; warm up so one-time growth happens before the baseline; repeat enough that the leak signal outgrows the noise; control time and network; and decide pass/fail on robust statistics rather than a single reading. The analysis-side equivalent — taking three snapshots rather than two — is described in the three-snapshot technique.

Raw readings versus stabilised readings The left panel shows heap readings taken without forced GC at random moments, scattered over a range of about 8 megabytes so no trend is visible. The right panel shows readings taken after warm-up and forced GC at each checkpoint; they fall on a tight line with a small but clear upward slope of about 40 kilobytes per iteration, revealing a real leak. Raw readings (no GC, no warm-up) spread ≈ 8 MB — leak invisible Post-GC after warm-up slope ≈ 40 KB per iteration — a real leak

Step-by-Step Fix

  1. Measure the noise floor first. Run the test ten times on a known-good build and record the reading each time. Verification: you know the natural spread, for example “post-GC heap after the flow varies by ±300 KB”.
  2. Force collection before every reading. In Node, run with --expose-gc and call gc() two or three times with a macrotask between calls; in the browser, use the DevTools Protocol’s HeapProfiler.collectGarbage before reading JSHeapUsedSize. Verification: the spread measured in step 1 shrinks substantially.
  3. Warm up before the baseline. Run the flow several times before taking the baseline reading, so compilation, caches and lazy chunks are already in place. Verification: the difference between the first and second measured iteration is no larger than between any later pair.
  4. Choose iterations so leaks dominate noise. If the smallest leak you care about is 20 KB per iteration and your noise floor is 300 KB, run at least 50 iterations so the leak adds 1 MB — several times the noise. Verification: a deliberately introduced 20 KB-per-iteration leak fails the test every time.
  5. Decide on slopes or medians, not single readings. Take readings at several checkpoints (every 10 iterations) and fit a slope, or repeat the whole measurement three times and compare medians. Verification: injecting one outlier reading does not change the verdict.
  6. Control the environment. Mock timers and network, wait for explicit UI states, run memory tests in isolation on consistent runners, and pin Node and Chrome versions. Verification: twenty consecutive CI runs on an unchanged commit all pass.
False failures per 100 runs by test design A naive single before and after reading fails 22 times in 100 runs of correct code. Adding forced GC reduces that to 9. Adding warm-up reduces it to 3. Using a slope across ten post-GC checkpoints reduces it to 0, while still catching a 20 kilobyte per iteration leak every time. False failures on a known-good build (per 100 runs) Single reading 22 + forced GC 9 + warm-up 3 + slope over 10 checkpoints 0 — and still catches a 20 KB/iteration leak

Command and Code Reference

Use case: a browser-side measurement helper with forced GC and checkpoints. Uses the DevTools Protocol so it works in Puppeteer and Playwright (Chromium).

// measure.mjs — stable heap readings for page flows
export async function stableHeap(cdp, page) {
  for (let i = 0; i < 3; i++) {
    await cdp.send('HeapProfiler.collectGarbage');     // full GC in the renderer
    await new Promise((r) => setTimeout(r, 50));       // let finalizers and idle tasks run
  }
  const { JSHeapUsedSize } = await page.metrics();
  return JSHeapUsedSize;
}

export async function heapSlope(cdp, page, flow, { warmup = 5, checkpoints = 10, every = 10 } = {}) {
  for (let i = 0; i < warmup; i++) await flow();       // one-time growth happens here
  const xs = [], ys = [];
  for (let c = 0; c < checkpoints; c++) {
    for (let i = 0; i < every; i++) await flow();
    xs.push((c + 1) * every);
    ys.push(await stableHeap(cdp, page));
  }
  // Least-squares slope in bytes per iteration: robust to a single outlier
  const n = xs.length, mx = xs.reduce((a, b) => a + b) / n, my = ys.reduce((a, b) => a + b) / n;
  const num = xs.reduce((s, x, i) => s + (x - mx) * (ys[i] - my), 0);
  const den = xs.reduce((s, x) => s + (x - mx) ** 2, 0);
  return num / den;
}

Use case: a Playwright test that asserts on the slope.

// memory.spec.mjs
import { test, expect } from '@playwright/test';
import { heapSlope } from './measure.mjs';

test('opening and closing the inspector does not leak', async ({ page, context }) => {
  const cdp = await context.newCDPSession(page);
  await page.clock.install();                          // deterministic timers
  await page.route('**/api/**', (r) => r.fulfill({ json: { ok: true } })); // no live network
  await page.goto('/projects/42');

  const flow = async () => {
    await page.getByRole('button', { name: 'Inspector' }).click();
    await page.getByRole('dialog').waitFor();
    await page.keyboard.press('Escape');
    await page.getByRole('dialog').waitFor({ state: 'hidden' });
  };

  const bytesPerIteration = await heapSlope(cdp, page, flow);
  expect(bytesPerIteration).toBeLessThan(5 * 1024);    // < 5 KB per open/close
});

Verification and Regression Prevention

A memory test is trustworthy when it has passed twenty or more consecutive runs on an unchanged build and fails every time on a build with a deliberately injected small leak. Record both results when you introduce the test, so reviewers can see its sensitivity and its false-positive rate. Re-check them when you upgrade Chrome, Node or your framework, since heap sizing and internal caching change between versions.

Treat a memory test failure like any other test failure: investigate before re-running. If the test is noisy, fix the test; do not add retries, which teach the team that red builds are meaningless. Keep the noise-reduction helpers in one shared module so every memory test — including Playwright memory testing for single-page apps and heap budgets — benefits from the same discipline.

Sensitivity versus false positives Across twenty consecutive runs on an unchanged build, the measured growth stays below the threshold line. On a build with a small deliberately injected leak, every run exceeds the threshold. A trustworthy test shows both behaviours. growth consecutive runs injected small leak: fails every run unchanged build: 20 passes in a row Record both results when the test is introduced and re-check after Chrome, Node or framework upgrades.

Edge Cases and Gotchas

GC can make things look better than they are

Forcing GC removes garbage, which is what you want — but it also hides churn that users experience as jank. Memory tests answer “does it leak?”; pair them with performance tests if you also care about allocation rate.

Heap readings miss external memory

JSHeapUsedSize does not include ArrayBuffer backing stores, canvas pixels or decoded images. A leak of 50 MB of image data can pass a JS-heap test untouched. For those flows, also track DOM node counts, document counts or process-level metrics.

Headless versus headed Chrome

Headless Chrome has slightly different memory behaviour — no GPU process in some configurations, different image decoding paths. Keep thresholds calibrated for the mode your CI actually uses.

Test order matters

When several memory tests share one browser page, a leak in the first inflates the baselines of the rest and can hide their own leaks. Give each memory test a fresh page or context.

Frequently Asked Questions

How many iterations should a memory test run?

Enough that the smallest leak you care about produces growth several times larger than the noise floor. Measure the noise first, then choose the count; for typical UI flows, 50 to 200 iterations is common.

Is comparing heap sizes better than using WeakRefs?

They answer different questions. A WeakRef test checks whether one specific object is released and is almost noise-free. A heap-growth test catches accumulation anywhere in the flow, including objects you did not think to track. Use WeakRef tests for known leak sites and slope tests for flows.

Should memory tests run on every pull request?

Fast, focused ones should. Longer soak tests — thousands of iterations or long sessions — are better on a nightly schedule. What matters most is that failures are trusted, so only gate merges on tests with a proven zero false-positive rate.