Production Memory Monitoring and Container Limits

A leak you can reproduce locally is an afternoon’s work; a leak you can only see in production is an outage that repeats until you have the right instrumentation, and this guide within Node.js Server-Side Memory Management is about installing that instrumentation before you need it. It covers the metrics worth exporting, how the V8 heap ceiling relates to a container limit, and how to separate a genuine leak from a service that was simply sized too small — the distinction that decides whether you go looking for a retainer chain or raise a limit.

Conceptual Grounding: Four Numbers and Two Ceilings

process.memoryUsage() returns four numbers, and each answers a different question. heapUsed is live JavaScript objects after the most recent collection — the number that reveals retention. heapTotal is how much heap V8 has reserved, so the gap between the two is fragmentation and slack. external counts memory held by C++ objects bound to JavaScript ones, which is where Buffer, ArrayBuffer and most stream chunk data live. rss is the whole process as the operating system sees it, including all of the above plus code, thread stacks and native-module allocations.

Above those sit two independent ceilings. V8’s own limit, set by --max-old-space-size, governs the old space only; hitting it produces a JavaScript-level fatal error with a stack trace. The container limit governs the entire process; hitting it produces a SIGKILL from the kernel, no stack trace, and an OOMKilled status. Because the second ceiling counts everything and the first counts only part, a process configured with a heap ceiling equal to its container limit will always die the silent way — which is why the ceiling belongs at roughly three quarters of the limit.

Four numbers, two ceilings Concentric regions: heapUsed sits inside heapTotal, which sits inside resident set size along with external buffers, native allocations and thread stacks. The V8 heap ceiling applies only to the old space region; the container limit applies to the whole resident set. Which threshold you hit first decides whether you get a stack trace rss — everything the OS counts heapTotal — reserved by V8 heapUsed — live objects the only number that shows retention slack + fragmentation external Buffers, ArrayBuffers native addons, stacks, code segment --max-old-space-size stops here → FATAL ERROR + stack trace container limit → SIGKILL, OOMKilled, no trace Set the green line to about 75% of the red one and every out-of-memory event becomes debuggable.

Diagnostic Workflow

  1. Export the four gauges on a timer. Command: a setInterval of 10 s calling process.memoryUsage(). Expected metric: four series per instance; sampling cost under 0.1 ms.
  2. Add GC timing. Command: new PerformanceObserver(...).observe({ entryTypes: ['gc'] }) from node:perf_hooks. Expected: major collections should account for well under 2% of wall-clock time on a healthy service.
  3. Record the post-collection floor, not the raw series. Expected: take the minimum heapUsed in each five-minute window; that floor is flat on a healthy service and monotonically rising on a leaking one.
  4. Compare rss growth against heapUsed growth. Expected: they should move together. rss rising while heapUsed is flat means external buffers or a native module, not a JavaScript object leak — a completely different investigation.
  5. Check the two ceilings against each other. Command: node -e "console.log(v8.getHeapStatistics().heap_size_limit / 1048576)" inside the container, and compare with the container’s own limit. Expected: the heap limit should be around 75% of the container limit; equal values mean every failure will be an OOMKilled with no diagnostics.
  6. Wire the alert to an artefact. Expected: when the floor slope alert fires, the instance drains and writes a .heapsnapshot to object storage, so the next step is reading a retainer chain rather than waiting for a recurrence.

Code Patterns & Signatures

The minimal exporter: four gauges plus event-loop delay, in a form any Prometheus-compatible scraper can read.

const client = require('prom-client');
const { monitorEventLoopDelay } = require('node:perf_hooks');

const heapUsed = new client.Gauge({ name: 'node_heap_used_bytes', help: 'live JS objects' });
const heapTotal = new client.Gauge({ name: 'node_heap_total_bytes', help: 'heap reserved by V8' });
const rss = new client.Gauge({ name: 'node_rss_bytes', help: 'whole-process footprint' });
const external = new client.Gauge({ name: 'node_external_bytes', help: 'buffers outside the heap' });
const loopP99 = new client.Gauge({ name: 'node_loop_delay_p99_ms', help: 'event-loop delay' });

// A histogram is cheaper than sampling the loop by hand and never misses a stall.
const loop = monitorEventLoopDelay({ resolution: 10 });
loop.enable();

setInterval(() => {
  const m = process.memoryUsage();
  heapUsed.set(m.heapUsed);
  heapTotal.set(m.heapTotal);
  rss.set(m.rss);
  external.set(m.external);
  loopP99.set(loop.percentile(99) / 1e6);   // nanoseconds → milliseconds
  loop.reset();
}, 10_000).unref();                          // unref so it never holds the process open

Measuring the share of wall-clock time spent collecting — the earliest reliable signal that a process is heading for an abort.

const { PerformanceObserver, constants } = require('node:perf_hooks');

let majorMs = 0;
let windowStart = Date.now();

const obs = new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) {
    // kind is a bit flag; MAJOR is the mark-compact collection that matters.
    if (entry.detail?.kind === constants.NODE_PERFORMANCE_GC_MAJOR) {
      majorMs += entry.duration;
    }
  }
});
obs.observe({ entryTypes: ['gc'] });

setInterval(() => {
  const elapsed = Date.now() - windowStart;
  const ratio = majorMs / elapsed;           // 0.02 = 2% of time in major GC
  if (ratio > 0.05) {
    console.warn(`gc pressure: ${(ratio * 100).toFixed(1)}% of wall clock in major GC`);
  }
  majorMs = 0;
  windowStart = Date.now();
}, 60_000).unref();

Sizing the heap ceiling from the container limit at start-up, so the same image behaves correctly under any limit.

# cgroup v2 exposes the container's memory limit to the process itself.
LIMIT_BYTES=$(cat /sys/fs/cgroup/memory.max 2>/dev/null || echo max)

if [ "$LIMIT_BYTES" = "max" ]; then
  HEAP_MB=1024                                  # unconstrained: pick a sane default
else
  # 75% of the limit, leaving room for buffers, native memory and stacks.
  HEAP_MB=$(( LIMIT_BYTES / 1048576 * 75 / 100 ))
fi

echo "container limit: $((LIMIT_BYTES / 1048576)) MB → heap ceiling: ${HEAP_MB} MB"
exec node --max-old-space-size=${HEAP_MB} server.js
Same leak, two failure modes In the first timeline the heap ceiling equals the container limit, so the kernel kills the process at the limit with no stack trace and no artefact. In the second the ceiling is set to seventy-five per cent, so V8 aborts first with a fatal error, a stack trace and a heap snapshot written by the alert handler. The ceiling you choose decides what you have to work with afterwards Ceiling = container limit memory climbs → kernel SIGKILL at the limit OOMKilled no stack trace · no snapshot · restart counter is the only evidence Ceiling = 75% of the limit GC pressure alert fires first snapshot written V8 abort stack trace · heap snapshot in object storage · retainer chain available immediately Both processes died. Only the second one told you why.

Alerting Rules That Survive a Deploy

The reason memory alerting has a reputation for noise is that most rules are written against an absolute number, and absolute numbers move every time the application changes. A rule that fires above 700 MB is really a rule that fires whenever someone adds a feature, and after the third false page it gets silenced. Three rule shapes avoid that.

Alert on the slope of the post-collection floor. Take the minimum heapUsed in each five-minute bucket, fit a line over the last two hours, and alert when the slope exceeds a few megabytes per hour sustained across the window. This survives deploys because a new baseline shifts the intercept, not the slope, and it catches the slow leaks that absolute thresholds only notice once they are catastrophic.

Alert on the ratio of time spent in major collections. Above roughly five per cent of wall-clock time, latency is already degrading and the process is on the path described earlier: pressure, thrash, stall, abort. The ratio is dimensionless, so it needs no recalibration when the heap ceiling or the instance size changes, and it fires minutes before any size-based rule would.

Alert on the gap between rss and heapUsed. A widening gap at a flat heap means external or native memory, which is a different investigation with different fixes. Making it a separate alert stops the on-call engineer from opening a heap snapshot that will show them nothing.

Two supporting practices make all three work. Label every series with the release version, so a step change at a deploy boundary is visible rather than mysterious. And attach an automatic action to the first rule: when the slope alert fires, drain one instance, write a heap snapshot to object storage and page with a link to it. The difference between an alert that says “memory is rising” and one that says “memory is rising, here is a snapshot with the retainer chain” is the difference between an investigation that starts tomorrow and one that starts now.

Finally, treat the restart counter as a first-class memory metric. A service configured to restart above a threshold will look perfectly healthy on every gauge while restarting forty times a day, and the restart rate is the only series that shows it.

Symptom-to-Fix Reference Table

Symptom Root Cause Immediate Action Measurable Impact
Pods restart with OOMKilled and no logs Heap ceiling at or above the container limit Set --max-old-space-size to ~75% of the limit Failures become V8 aborts with a stack trace
rss climbs while heapUsed is flat External buffers or a native module, not a JS leak Chart external alongside rss; audit stream and image code Points the investigation at the right layer on day one
Heap looks fine, latency collapses High GC pressure from allocation rate Export the major-GC time ratio and alert above 5% Detects the problem minutes before the abort
Memory graph is a noisy sawtooth with no trend Sampling raw heapUsed instead of the post-collection floor Chart the five-minute minimum The slope becomes visible immediately
Alerts fire on every deploy Threshold set on absolute bytes, and the baseline moved Alert on the slope of the floor over a rolling window Removes almost all false positives
Leak reproduces only in production Different concurrency and data volume than staging Capture a snapshot from a drained production instance Turns an unreproducible bug into a retainer chain
From alert to artefact, automatically When the slope alert fires, the instance is drained from the load balancer, a heap snapshot is written to object storage, the on-call engineer is paged with a link to it, and the instance is restarted. Every step runs without human involvement, so the investigation starts with evidence already in hand. The alert is worth far more when it arrives with a snapshot attached 1 · alert fires heap floor slope or GC ratio 2 · drain remove one instance from the pool 3 · capture writeHeapSnapshot to object storage 4 · page with a link to the snapshot, not a graph 5 · restart return the instance to the pool Steps 2 and 5 exist because a snapshot freezes the event loop for seconds — never capture from a serving instance. Step 4 is the difference between an on-call engineer opening a dashboard and one opening a retainer chain. The whole path is perhaps forty lines of automation, written once, and it changes every future incident. Keep the artefact for the life of the incident ticket; the two metric values keep forever as a time series.

Edge Cases & Gotchas

heapUsed immediately after a request looks like a leak. Anything measured before the next collection includes short-lived garbage. Chart the floor, and if you need a point measurement, take it after global.gc() on an instance started with --expose-gc.

Worker threads are invisible in the main thread’s numbers. process.memoryUsage() reports the calling isolate. A pool of four workers can be consuming three times the main thread’s heap while the main thread’s gauge stays flat, so worker isolate budgets need their own reporting via worker.performance or a message from each worker.

cgroup v1 and v2 expose the limit at different paths. The start-up script above reads memory.max (v2); on v1 the file is memory/memory.limit_in_bytes and an unconstrained container reports an enormous sentinel value rather than max. Handle both, or the ceiling calculation silently produces a nonsense number.

A restart policy masks the slope. Automatic restarts above a memory threshold are a reasonable bridge, but they make the memory graph look healthy while the underlying growth continues. Alert on restart frequency as well, and treat a rising restart rate as the leak signal it is.

Snapshots on a live instance cause the incident they were meant to diagnose. Writing a snapshot blocks the event loop for one to four seconds per gigabyte of heap. Always drain the instance first, as covered in diagnosing Node memory with heapdump and Clinic.

None of this instrumentation finds a leak on its own. What it does is turn an unreproducible production failure into a dated slope, a labelled release and a snapshot with a retainer chain — the three inputs that make the rest of this section’s diagnostic work possible at all.

FAQ

Why is RSS much higher than heapUsed?

Resident set size covers the whole process: the V8 heap, external buffers, native module allocations, thread stacks, the code segment and memory the allocator has not returned to the operating system. A gap of 100–200 MB is normal; a gap that widens steadily while heapUsed stays flat points at external buffers or native allocations rather than a JavaScript leak.

What does OOMKilled mean if the heap never filled up?

It means the kernel killed the process because the container’s total memory exceeded its limit, which V8 has no visibility of. The heap ceiling only governs the V8 old space; external buffers, native modules and thread stacks are outside it. Set --max-old-space-size to about 75% of the container limit so the runtime hits its own limit first and gives you a stack trace.

How often should memory metrics be sampled?

Every 10–15 seconds is enough for trend detection and costs well under a millisecond per sample. Sampling faster mostly captures the sawtooth between collections, which is noise for this purpose; sampling slower than a minute risks missing a fast leak entirely before the process aborts.

Should a service restart itself when memory is high?

As a bridge, yes — a graceful drain-and-restart above a threshold beats an abrupt kill mid-request. As a permanent policy it hides the slope that identifies the bug, so pair it with an alert that fires on the restart frequency itself and treat a rising restart rate as the real signal.