Exposing Node.js Memory Metrics with prom-client

Instrumentation is the part of memory work that has to be done before the incident, and it is small enough to finish in an afternoon. This page belongs to production memory monitoring and container limits in Node.js Server-Side Memory Management, and gives the complete exporter plus the alert rules that use it.

Symptom Root Cause Immediate Action
Dashboard shows only rss Default metrics registered, nothing added Add the four memoryUsage gauges
Cannot tell a heap leak from a buffer leak external and arrayBuffers not published Publish both alongside heapUsed
Latency degrades before any memory alert No GC timing exported Add a major-GC millisecond counter
Worker pool memory invisible memoryUsage() reports one isolate Have workers report with a thread label
Alerts fire on every deploy Rules written on absolute bytes Rewrite as slope and ratio rules

Root Cause: Default Metrics Answer the Wrong Half

prom-client’s collectDefaultMetrics gives you a solid baseline — resident set size, total heap, CPU, handles — and stops exactly where memory diagnosis begins. It tells you that the process is using more memory. It cannot tell you whether the growth is in the JavaScript heap, in buffers, or in a native module, and that classification is the first decision in every investigation.

Three additions close the gap. The memoryUsage() gauges published together let the derived relationships be charted: heapTotal − heapUsed is slack, rss − heapTotal is everything outside the heap. GC timing turns pressure into a number with no units of its own — the fraction of wall-clock time spent in major collections — which is comparable across instance sizes and unaffected by deploys. Event-loop delay is the bridge to user impact: it is the series that shows memory pressure becoming latency.

The alerting side matters as much as the collection. Absolute thresholds break on every release that legitimately adds memory, and a rule that pages falsely three times gets silenced. Slopes and ratios survive deploys because a new baseline changes the intercept, not the gradient.

What each instrumentation layer can answer Default metrics answer whether memory is growing. Added memoryUsage gauges answer where the growth is. GC timing answers how close the process is to collapse. Event-loop delay answers whether users are feeling it yet. Each layer is a few lines of code on top of the previous one. Four layers, each a few lines, each answering a different question collectDefaultMetrics() answers: is memory growing? — rss and heap size, no breakdown + heapUsed · heapTotal · external · arrayBuffers answers: WHERE is it growing? — heap, buffers or native + major GC milliseconds counter answers: how close to collapse? — the ratio rises before the heap does + event-loop delay p99 answers: are users feeling it? — the series that justifies the page

Step-by-Step Fix

  1. Register defaults with a prefix. Command: collectDefaultMetrics({ prefix: 'app_' }). Verification: /metrics returns app_process_resident_memory_bytes.
  2. Add the four gauges on a 10 s interval. Verification: all four appear and move together under load.
  3. Attach the GC observer. Verification: the major-GC counter increases during a load test and stays flat when idle.
  4. Add the event-loop histogram. Verification: p99 rises visibly when you deliberately block the loop for 200 ms.
  5. Write the two rules. Heap-floor slope and GC ratio. Verification: replay a historical leak through the rules and confirm both would have fired before the abort.

Command & Code Reference

The complete exporter. This is the whole of the collection side.

const client = require('prom-client');
const { PerformanceObserver, constants, monitorEventLoopDelay } = require('node:perf_hooks');

client.collectDefaultMetrics({ prefix: 'app_' });

const g = (name, help) => new client.Gauge({ name, help });
const heapUsed = g('app_heap_used_bytes', 'live JS objects after the last collection');
const heapTotal = g('app_heap_total_bytes', 'heap reserved by V8');
const external = g('app_external_bytes', 'memory held by C++ objects bound to JS');
const arrayBuffers = g('app_array_buffers_bytes', 'subset of external owned by ArrayBuffers');
const loopP99 = g('app_event_loop_delay_p99_ms', 'event-loop delay, 99th percentile');

// A counter, not a gauge: the ratio is derived in the query, so a scrape gap
// never loses time that was actually spent collecting.
const gcMajorMs = new client.Counter({
  name: 'app_gc_major_milliseconds_total',
  help: 'cumulative milliseconds spent in major collections',
});

new PerformanceObserver((list) => {
  for (const e of list.getEntries()) {
    if (e.detail?.kind === constants.NODE_PERFORMANCE_GC_MAJOR) gcMajorMs.inc(e.duration);
  }
}).observe({ entryTypes: ['gc'] });

const loop = monitorEventLoopDelay({ resolution: 10 });
loop.enable();

setInterval(() => {
  const m = process.memoryUsage();
  heapUsed.set(m.heapUsed);
  heapTotal.set(m.heapTotal);
  external.set(m.external);
  arrayBuffers.set(m.arrayBuffers);
  loopP99.set(loop.percentile(99) / 1e6);   // nanoseconds → milliseconds
  loop.reset();
}, 10_000).unref();

Worker threads report their own isolates, because the main thread cannot see them.

// worker.js — post the numbers to the parent every ten seconds.
const { parentPort, threadId } = require('node:worker_threads');
setInterval(() => {
  const m = process.memoryUsage();          // this worker's isolate only
  parentPort.postMessage({ type: 'mem', threadId, heapUsed: m.heapUsed, rss: m.rss });
}, 10_000).unref();

// parent.js — publish them as a labelled series.
const workerHeap = new client.Gauge({
  name: 'app_worker_heap_used_bytes',
  help: 'per-worker isolate heap',
  labelNames: ['thread'],
});
worker.on('message', (msg) => {
  if (msg?.type === 'mem') workerHeap.set({ thread: String(msg.threadId) }, msg.heapUsed);
});

The two alert rules. Both are dimensionless or slope-based, so a deploy that legitimately raises the baseline does not fire them.

groups:
  - name: nodejs-memory
    rules:
      # 1. Retention: the post-collection floor is climbing.
      #    min_over_time approximates the floor; the slope is what matters.
      - alert: HeapFloorRising
        expr: |
          deriv(min_over_time(app_heap_used_bytes[5m])[2h:5m]) > 1.5e6
        for: 30m
        annotations:
          summary: "heap floor rising >1.5 MB/h for 2h on "

      # 2. Pressure: more than 5% of wall clock spent in major collections.
      - alert: GcPressureHigh
        expr: |
          rate(app_gc_major_milliseconds_total[5m]) / 1000 > 0.05
        for: 10m
        annotations:
          summary: " spending >5% of wall clock in major GC"
Which alert fires first Over ninety minutes the heap floor climbs slowly toward the ceiling. The GC pressure ratio crosses five per cent at minute forty, half an hour before the heap floor alert would fire and nearly an hour before the process would abort. One incident, two rules — the ratio rule buys 50 minutes elapsed (0 → 90 min) heap floor time in major GC GC ratio > 5% — alert fires at 40 min heap slope alert at 70 min abort Both rules are correct; the difference is whether the on-call engineer gets a coffee or an outage.

Verification & Regression Prevention

Verify each metric by making it move on purpose. Allocate and retain 200 MB and watch app_heap_used_bytes step up; allocate 200 MB of Buffer and watch app_external_bytes step up instead. Block the loop with a synchronous 300 ms loop and watch p99 spike. A metric that has never been observed to move is a metric nobody should build an alert on.

Then replay a historical incident through the alert expressions — most Prometheus setups can evaluate a rule against recorded data — and confirm both would have fired with enough lead time. For prevention, add a /metrics smoke test to the deployment pipeline that asserts each expected series is present and non-zero; the most common regression is not a broken rule but a metric that silently stopped being registered after a refactor.

How to prove each metric works Five metrics with a deliberate action for each: retain two hundred megabytes of objects for heap used, allocate buffers for external, block the loop for event-loop delay, run an allocation storm for the GC counter, and start a worker for the per-thread gauge. Make each series move on purpose before trusting it Metric Action that must move it app_heap_used_bytes retain 200 MB of objects in a module-scope array app_external_bytes allocate 200 MB of Buffers and keep them app_event_loop_delay_p99_ms block the loop synchronously for 300 ms app_gc_major_milliseconds_total allocate and discard in a tight loop for 30 s app_worker_heap_used_bytes start a worker and allocate inside it only

FAQ

Does collectDefaultMetrics already cover memory?

Partly. It publishes resident set size and total heap, which is enough to see the shape of a problem but not to classify it. The external and arrayBuffers gauges, and the GC timing counter, are what separate a JavaScript retention bug from a buffer or native-memory one, and those need to be added explicitly.

How expensive is this instrumentation?

Negligible. process.memoryUsage() costs on the order of tens of microseconds, the GC observer is passive, and the event-loop histogram is implemented natively. Sampling every ten seconds costs far less than one part in a hundred thousand of a service’s CPU.

Should each worker thread publish its own metrics?

Yes, with a thread label. process.memoryUsage() reports only the calling isolate, so a worker pool’s memory is invisible from the main thread. Have each worker post its numbers to the parent on an interval and publish them as labelled series.