Fixing OOMKilled Node.js Containers in Kubernetes

An OOMKilled pod is the least informative failure a Node.js service can produce: no stack trace, no log line, no heap snapshot, just an exit code and a restart. This page belongs to production memory monitoring and container limits inside Node.js Server-Side Memory Management, and covers the configuration that converts that silence into something you can debug.

Symptom Root Cause Immediate Action
Exit code 137, reason: OOMKilled, no logs Kernel SIGKILL at the container limit Set the heap ceiling to ~75% of the limit
Restarts cluster at traffic peaks Limit sized for average, not peak concurrency Cap in-flight requests, then re-measure peak rss
Kills stop after adding workers, then return worse Each worker has its own isolate and heap Budget every isolate against one container limit
Pod killed while heapUsed is small Growth is in external or native memory Chart rss and external, not the heap alone
Same image is fine locally, killed once deployed No limit locally, so the ceiling is never reached Reproduce with the same limit under docker run -m

Root Cause: Two Limits, Only One of Which Talks to You

The kernel enforces the container’s memory limit against the whole cgroup: the V8 heap, external buffers, native allocations, thread stacks, the code segment, everything. When the total crosses the limit, the OOM killer sends SIGKILL. That signal cannot be caught, so there is no opportunity to log, flush or capture anything. The pod’s status records OOMKilled and exit code 137, and that is the entirety of the evidence.

V8 enforces its own, narrower limit on the old space, configured by --max-old-space-size. Crossing that produces a FATAL ERROR: Reached heap limit — Allocation failed message, a JavaScript stack trace, and enough time for a crash handler to run. Everything you would want during an incident comes from this path.

So the two limits should not be equal. If the heap ceiling equals the container limit, the process can never reach it — the container total includes the heap plus everything else, so the kernel always wins the race. Setting the ceiling to roughly three quarters of the limit leaves room for external memory and makes the runtime’s own abort the likely failure mode.

The remaining quarter is not arbitrary. It has to cover external buffers, native module allocations, thread stacks — including one isolate per worker thread — and the Node runtime baseline of roughly 40 MB. A service that streams large payloads may need more than a quarter; a plain JSON API may need less.

Where to put the heap ceiling In the first configuration the heap ceiling equals the container limit of one gigabyte, so the kernel kills the process before V8 reaches its own limit and no diagnostics are produced. In the second the ceiling is at seven hundred and sixty-eight megabytes, so V8 aborts first with a stack trace while two hundred and fifty-six megabytes of headroom absorbs external and native memory. Same limit, same leak, completely different incident ceiling = limit (1024 MB) heap may grow all the way to the container limit kernel wins the race → SIGKILL, exit 137, no trace, no snapshot ceiling = 768 MB (75%) V8 old space external native V8 aborts at the dashed line → FATAL ERROR with a stack trace, crash handler runs, snapshot written The 256 MB gap is not waste — it is the price of every diagnostic you will want during the incident.

Step-by-Step Fix

  1. Confirm the kill reason. Command: kubectl describe pod <name> | grep -A3 'Last State'. Expected: Reason: OOMKilled, Exit Code: 137.
  2. Read the limit from inside the container. Command: kubectl exec <pod> -- cat /sys/fs/cgroup/memory.max. Expected: the byte value the kernel enforces, which may not match what the manifest says if a LimitRange applies.
  3. Budget the non-heap portion. Add up peak external, worker isolates, native modules and ~40 MB baseline. Expected: a figure between 15% and 35% of the limit for most services.
  4. Set the ceiling explicitly. Command: --max-old-space-size=<limit MB × 0.75>, ideally computed at start-up. Verification: node -e "console.log(require('v8').getHeapStatistics().heap_size_limit)" inside the pod reports the expected value.
  5. Align requests with limits. Verification: the pod’s QoS class is Guaranteed, so the scheduler does not overcommit the node underneath it.

Command & Code Reference

The entrypoint that derives the ceiling from whatever limit the container was given, so one image is correct at any size.

#!/bin/sh
# cgroup v2 first, then v1, then unconstrained.
if [ -r /sys/fs/cgroup/memory.max ]; then
  LIMIT=$(cat /sys/fs/cgroup/memory.max)
elif [ -r /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
  LIMIT=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)
else
  LIMIT=max
fi

# An unconstrained cgroup reports "max" (v2) or a huge sentinel (v1).
case "$LIMIT" in
  max|*[!0-9]*|9223372036854771712) HEAP_MB=1024 ;;
  *) HEAP_MB=$(( LIMIT / 1048576 * 75 / 100 )) ;;
esac

echo "container limit $(( LIMIT / 1048576 ))MB → heap ceiling ${HEAP_MB}MB" >&2
exec node --max-old-space-size="${HEAP_MB}" server.js

The manifest side. Equal requests and limits give the pod a Guaranteed QoS class, which stops the node from overcommitting memory that this service is relying on.

resources:
  requests:
    memory: "1Gi"     # equal to the limit → Guaranteed QoS, no overcommit
    cpu: "500m"
  limits:
    memory: "1Gi"
# The heap ceiling is derived from this limit at start-up by the entrypoint,
# so changing the limit here is the only edit needed.

A crash handler that turns the V8 abort into an artefact, which is only reachable because the ceiling sits below the container limit.

const v8 = require('node:v8');
const fs = require('node:fs');

// Fires on the fatal heap-limit path, before the process exits.
v8.setHeapSnapshotNearHeapLimit?.(1);   // Node 18+: write one snapshot near the limit

process.on('uncaughtException', (err) => {
  if (/heap limit|Allocation failed/i.test(err.message)) {
    const path = `/tmp/oom-${process.pid}.heapsnapshot`;
    try {
      v8.writeHeapSnapshot(path);        // best effort; may fail if truly exhausted
      console.error(`heap snapshot written to ${path}`);
    } catch {}
  }
  console.error(err.stack);
  process.exit(1);
});
Three restarts, before and after the ceiling change Before the change, memory climbs to the container limit three times and each restart produces only an exit code. After the change, the climb stops at the heap ceiling, a stack trace and heap snapshot are produced, and the fourth restart is the last one because the leak is found. The same leak, before and after moving the ceiling container limit 1024 MB heap ceiling 768 MB (after) exit 137 exit 137 exit 137 FATAL ERROR + snapshot leak found Nothing about the leak changed between the two halves of this chart — only what the failure left behind.

Verification & Regression Prevention

Verify by reproducing the abort deliberately: run the image locally with docker run -m 1g and a script that allocates until it dies. You should get a FATAL ERROR and a stack trace, not a bare Killed. If you get the latter, the ceiling is still too close to the limit, or the entrypoint failed to read the cgroup file and fell back to its default.

For prevention, assert the relationship in a start-up check: read v8.getHeapStatistics().heap_size_limit and the cgroup limit, and log a warning — or refuse to start — when the ratio exceeds about 0.85. That single check catches the most common regression, which is someone raising the manifest limit without touching the ceiling, or moving to a base image whose entrypoint no longer does the calculation.

Ceiling and headroom at four container sizes At a five hundred and twelve megabyte limit the ceiling is three hundred and eighty-four with one hundred and twenty-eight of headroom, suitable for a plain API. At one gigabyte the ceiling is seven hundred and sixty-eight. At two gigabytes it is one thousand five hundred and thirty-six. At four gigabytes it is three thousand and seventy-two, with a gigabyte of headroom for stream-heavy work. The 75% rule at four common sizes Container limit Heap ceiling Headroom Comfortable for 512 MB 384 MB 128 MB JSON API, no large buffers 1 GB 768 MB 256 MB typical SSR or BFF service 2 GB 1 536 MB 512 MB worker pool, 2–4 threads 4 GB 3 072 MB 1 GB stream or media processing Shift the ratio down, never up, when the workload holds large buffers — headroom is what the kernel counts.

FAQ

Why is there no stack trace when a pod is OOMKilled?

The kernel sends SIGKILL, which cannot be caught, handled or logged. The process stops between one instruction and the next. Any diagnostics have to come from a limit the runtime enforces itself, which is why the heap ceiling belongs below the container limit rather than at it.

Does Node read the container limit automatically?

Modern Node versions size the default heap from the memory the container reports, which helps, but the default is a heuristic that does not know how much external memory your workload uses. Anything with large buffers, worker threads or native modules should set the ceiling explicitly.

Should I just raise the memory limit?

Raise it if the working set genuinely needs it — a service that legitimately holds 900 MB in a 1 GB container is under-provisioned. If the growth is unbounded, a larger limit only moves the kill later and makes each snapshot bigger and slower to capture.