Using --heap-prof Sampling in Production

Heap snapshots tell you what is retained, but your production problem is different: GC time is high and RSS churns, and you need to know which code allocates under real traffic — without the pause of a snapshot or the overhead of an allocation timeline. This guide from Diagnosing Node Memory with Heapdump and Clinic, part of Node.js Server-Side Memory Management, shows how to collect V8’s sampling heap profiles from Node.js — at startup with --heap-prof or on demand through the inspector module — and how to read them.

Symptom Root Cause Immediate Action Measurable Impact
High GC share, no obvious leak Heavy allocation (churn) in request handling Record a sampling heap profile under real traffic Top allocating functions identified
Snapshot pauses unacceptable in production Snapshots stop the world Use sampling: continuous, low overhead No user-visible pauses
--heap-prof file never appears Profile is written at normal process exit Exit gracefully, or use on-demand sampling via inspector Profiles captured reliably
Profile dominated by startup allocations Sampling from process start Start sampling after warm-up with the inspector API Profile reflects steady state
Allocation attributed to generic helpers Bottom-up view stops at utility functions Expand callers in Heavy view Actionable call sites

Root Cause: Sampling Trades Precision for Always-On Safety

V8’s sampling heap profiler records a stack trace for allocations chosen at random intervals of allocated bytes — by default in Node.js about every 512 KB (--heap-prof-interval) — rather than for every object. Aggregated over minutes of traffic, the samples give a statistically sound picture of where bytes are allocated, with an overhead low enough for production (typically a few percent). It is the server-side counterpart of the allocation sampling profiler in Chrome, and the output format (.heapprofile) is the same, so Chrome DevTools can load it.

By default a sampling profile reports only the sampled objects that are still alive when you stop; that answers “which code allocates what is retained” and complements snapshots. For churn investigations you want objects that were already collected too: the inspector protocol’s HeapProfiler.startSampling accepts flags to include objects collected by minor and major GCs, which turns the profile into an allocation-rate profile — exactly what you need when GC pauses or promotion rather than retention are the problem.

There are two ways to collect. node --heap-prof starts sampling at process start and writes a .heapprofile file when the process exits normally (configurable with --heap-prof-dir, --heap-prof-name and --heap-prof-interval). It is simple but has two drawbacks for servers: the profile includes startup, and a process killed with SIGKILL (for example by an out-of-memory kill or a hard container stop) never writes it. On-demand sampling uses the built-in inspector module inside the process to call HeapProfiler.startSampling and stopSampling via an admin endpoint, capturing a window of steady-state traffic without restarting — the more practical option for long-running services.

Choosing a Node.js memory capture for production Heap snapshot: stop-the-world pause proportional to heap, reveals retained objects and retainers, use on a drained instance. Allocation timeline: high overhead instrumentation, reveals exact allocations, not for production traffic. Sampling heap profile: a few percent overhead and no long pauses, reveals allocating stacks statistically, suitable for live traffic. Heap snapshot pause ∝ heap size retained objects + retainers drained instance only Allocation timeline high overhead every allocation tracked staging / local only Sampling heap profile a few percent overhead allocating stacks (statistical) optional: include GC'd objects safe under live traffic

Step-by-Step Fix

  1. Add an on-demand sampling endpoint. Ship the admin handler shown below on an internal, authenticated port. Verification: in staging, starting and stopping returns a .heapprofile JSON document.
  2. Warm up, then sample steady state. After the instance has served traffic for a while, start sampling with GC’d objects included (for churn) or excluded (for retention), and let it run for 2–10 minutes. Verification: CPU and latency metrics show no significant change during sampling.
  3. Stop and store the profile. Stop sampling and upload the profile to storage. Verification: a file of a few hundred KB to a few MB is stored.
  4. Load it in DevTools. Open Chrome DevTools (any page), Memory → Load, and select the .heapprofile. Verification: the profile appears under Sampling profiles with Heavy, Tree and Chart views.
  5. Rank allocators and walk up to callers. Sort Heavy (Bottom Up) by Self Size, then expand the top rows to find the application function that drives them (a serializer, a per-request mapper, a logger). Verification: you can name the call site to change.
  6. Fix and compare. Deploy the change and repeat steps 2–5 under similar traffic. Verification: the function’s share of sampled bytes falls, and GC share in production metrics drops.
Where a busy API allocates (sampled, GC'd objects included) Over a five minute sample: logRequest accounts for 34 percent of allocated bytes because it stringifies full request bodies, JSON serialisation of responses 22 percent, the ORM row mapper 18 percent, validation 9 percent, and everything else 17 percent. Share of sampled bytes (5 min, live traffic) logRequest 34% — stringifies full bodies JSON.stringify (responses) 22% mapRow (ORM) 18% validate 9% other 17%

Command and Code Reference

Use case: on-demand sampling through the inspector module.

// admin-heap-sampling.js — internal admin port only
const inspector = require('node:inspector/promises');
let session = null;

adminApp.post('/admin/heap-sampling/start', requireAdmin, async (req, res) => {
  session = new inspector.Session();
  session.connect();                                   // in-process, no network port
  await session.post('HeapProfiler.enable');
  await session.post('HeapProfiler.startSampling', {
    samplingInterval: 256 * 1024,                      // ~one sample per 256 KB allocated
    includeObjectsCollectedByMajorGC: req.query.churn === '1',
    includeObjectsCollectedByMinorGC: req.query.churn === '1',
  });
  res.sendStatus(204);
});

adminApp.post('/admin/heap-sampling/stop', requireAdmin, async (req, res) => {
  const { profile } = await session.post('HeapProfiler.stopSampling');
  session.disconnect();
  session = null;
  const key = `heapprof/${process.env.HOSTNAME}-${Date.now()}.heapprofile`;
  await storage.put(key, JSON.stringify(profile));     // load later in DevTools → Memory
  res.json({ key });
});

Use case: startup-to-exit sampling for batch jobs. For short-lived processes, the flag is simplest.

# Writes ./profiles/<name>.heapprofile when the job exits normally
node --heap-prof --heap-prof-dir=./profiles --heap-prof-interval=262144 jobs/rebuild-index.js

Verification and Regression Prevention

Verify that sampling is safe in your environment by running it on one production instance while watching latency, CPU and GC metrics; the difference should be within normal variance. Verify the fix by comparing two profiles captured under similar traffic: the function you changed should have a much smaller share, and production GC share or allocation rate (from perf_hooks GC tracking) should fall.

Make sampling part of your runbook for “GC time is high” and “memory churns” alerts, and keep a baseline profile per service after each major release, so a regression can be diffed against a known-good profile. For retention problems, follow up with snapshots as in taking heap snapshots from a live Node.js process.

Two verifications for sampling in production Run sampling on one production instance while watching latency, CPU and GC metrics, and confirm the difference is within normal variance. Then compare two profiles captured under similar traffic: the function you changed should have a much smaller share, and production GC share or allocation rate should fall. On one production instance Safe Latency, CPU and GC within normal variance while sampling. Effective The changed function’s share is much smaller. Visible in prod GC share or allocation rate falls after deploy.

Edge Cases and Gotchas

Profiles are statistical

Small, infrequent allocations may not appear, and byte totals are estimates. Compare shares between profiles taken with the same interval and duration rather than reading exact numbers.

Anonymous functions and minified code

Anonymous callbacks and bundled server code reduce readability. Name important functions and, if you bundle server code, keep source maps available for mapping positions back to sources.

Workers need their own sessions

The in-process inspector session covers the main isolate. Worker threads need sampling started inside each worker (the inspector module works there too) if they are part of the investigation.

Graceful exit for --heap-prof

--heap-prof writes on normal exit. Make sure your shutdown path lets the process exit on its own after SIGTERM, rather than being killed after a timeout, or the profile is lost.

Frequently Asked Questions

What does node --heap-prof do?

It starts V8’s sampling heap profiler when the process starts and writes the resulting .heapprofile file when the process exits normally. The file shows which call stacks allocated sampled bytes and can be opened in Chrome DevTools’ Memory panel.

Is heap sampling safe to run in production?

Generally yes. It samples allocations at byte intervals rather than tracking every object, so overhead is typically a few percent and there is no long pause. Validate on one instance first and use an authenticated, internal-only trigger.

How is a sampling profile different from a heap snapshot?

A snapshot records every live object and its retainers at one moment, pausing the process. A sampling profile records where allocations happen over a period, with low overhead, but has no retainer information. Use sampling to find allocators and snapshots to find retainers.

How do I include garbage-collected allocations?

Start sampling through the inspector protocol with includeObjectsCollectedByMajorGC and includeObjectsCollectedByMinorGC set to true. The profile then attributes allocation volume, including churn, rather than only objects still alive at the end.

What sampling interval should I use?

The default of about 512 KB is a good start for busy services. Lower intervals such as 128–256 KB give finer detail for shorter windows at slightly higher overhead; much lower values approach instrumentation costs. Keep the interval constant between runs you intend to compare.

How long should a production sampling window be?

Long enough to cover representative traffic — typically two to ten minutes at normal load. Very short windows are dominated by whatever happened to run; very long windows add little new information and make the profile harder to relate to specific incidents.

Can sampling run continuously?

Some teams keep sampling on permanently and upload a profile every few minutes, which builds a history of allocation behaviour per release. The overhead is usually acceptable, but measure it for your service first and keep the profile storage bounded.

Can I view .heapprofile files outside Chrome?

Yes. Tools such as speedscope can display them as flame graphs, and the format is JSON you can process with scripts. Chrome DevTools remains the most direct viewer, with Heavy, Tree and Chart views.