RSS Growing with a Flat Heap: Native Memory and malloc
The container’s memory keeps climbing towards its limit and eventually gets OOMKilled, yet heapUsed is flat and heap snapshots show nothing growing. This guide from Production Memory Monitoring and Container Limits, part of Node.js Server-Side Memory Management, walks through the memory Node.js uses outside the V8 heap — Buffers, native addons, compression and TLS contexts, thread stacks — and the allocator behaviour that can make RSS grow without any object leaking.
| Symptom | Root Cause | Immediate Action | Measurable Impact |
|---|---|---|---|
RSS rises, heapUsed flat, arrayBuffers rising |
Buffers retained (streams, caches, pools) | Find retained Buffer owners in a snapshot | External memory bounded |
RSS rises, heap and arrayBuffers flat |
Native addon memory or allocator fragmentation | Inspect /proc/<pid>/smaps; test allocator settings |
Cause narrowed to native code or allocator |
| RSS never returns after a traffic spike | glibc malloc keeps freed memory in per-thread arenas | Set MALLOC_ARENA_MAX=2 or use jemalloc |
RSS returns closer to baseline |
| Growth correlates with image or compression work | libvips/sharp caches, zlib contexts per request | Configure library caches; reuse/limit contexts | Native memory capped |
| OOMKilled with no V8 error | Total RSS over the container limit | Budget heap vs off-heap; alert on RSS slope | Kills prevented |
Root Cause: RSS Is the Whole Process, the Heap Is One Part
process.memoryUsage() returns four numbers that answer different questions, as explained in reading RSS vs heapUsed in production Node.js. heapUsed/heapTotal cover the V8 JavaScript heap. external counts memory owned by C++ objects bound to JavaScript objects, and arrayBuffers (a subset of external) counts ArrayBuffer and Buffer backing stores. RSS is everything resident: the heap, external memory, compiled code, thread stacks, native libraries’ own allocations, and the memory the C allocator has obtained from the operating system but not yet returned.
When RSS grows and the heap does not, there are three families of causes. Retained Buffers: backing stores are external memory, so a cache of Buffers or a stream buffering chunks shows up in arrayBuffers while heapUsed barely moves; the JavaScript objects that retain them are still in heap snapshots, as small Buffer or ArrayBuffer entries with large byte lengths (see Buffer.allocUnsafe and the Node.js buffer pool). Native allocations outside V8’s view: image libraries (sharp/libvips keep operation and pixel caches), database drivers, compression (each zlib stream allocates tens to hundreds of KB of native state), TLS sessions, and native addons allocate with malloc directly; none of it appears in external unless the addon reports it.
The third family is the allocator itself. glibc’s malloc serves multi-threaded programs from multiple arenas (by default up to 8 × number of cores on 64-bit systems) to reduce lock contention. Node uses threads — libuv’s thread pool, V8’s helper threads — and allocations spread across arenas. Memory freed in an arena is kept for reuse rather than returned to the OS, and fragmentation within arenas prevents trimming. After a burst of native allocation (many concurrent compressions or image resizes), RSS can stay high indefinitely or ratchet up with each burst, even though nothing is leaked. Reducing the number of arenas (MALLOC_ARENA_MAX=2) or preloading an allocator designed to return memory more aggressively, such as jemalloc, often flattens RSS dramatically. Alpine images use musl’s allocator, which behaves differently again — sometimes better for RSS, sometimes slower.
Step-by-Step Fix
- Split the numbers over time. Log
rss,heapUsed,externalandarrayBuffersevery minute. Verification: you know whether growth is in the heap, in external/arrayBuffers, or in neither (native or allocator). - If
arrayBuffersgrows, find the owners. Take heap snapshots, filter forArrayBuffer/Buffer/Uint8Array, and sort by retained size including external bytes; follow retainers to caches or streams. Verification: you identify the structure holding the Buffers. - If neither grows, inspect process mappings. Read
/proc/<pid>/smaps_rollupand/proc/<pid>/smaps(anonymous regions), orpmap -x <pid>, and note many ~64 MB anonymous regions (typical of glibc arenas). Verification: growth is in anonymous heap/arena mappings rather than file-backed ones. - Test allocator settings. Restart one instance with
MALLOC_ARENA_MAX=2, and another with jemalloc preloaded (LD_PRELOAD=/usr/lib/.../libjemalloc.so.2). Verification: compare RSS trends under the same traffic; a large improvement points to fragmentation rather than a leak. - Bound native library caches. Configure libraries that cache natively — for example
sharp.cache({ memory: 50 })andsharp.concurrency()— and reuse or limit compression contexts. Verification: RSS during image or compression bursts is bounded. - Budget and alert on RSS. Set the container limit to cover heap limit plus measured off-heap peak and margin, and add a slope alert on RSS. Verification: RSS plateaus below the limit and no OOM kills occur.
Command and Code Reference
Use case: log the four numbers and the anonymous memory total.
// native-memory-log.js
const fs = require('node:fs');
const mb = (n) => Math.round(n / 1048576);
setInterval(() => {
const m = process.memoryUsage();
let anonKB = null;
try { // Linux only
const rollup = fs.readFileSync('/proc/self/smaps_rollup', 'utf8');
anonKB = Number(/^Anonymous:\s+(\d+) kB/m.exec(rollup)?.[1]);
} catch {}
console.log(JSON.stringify({
rssMB: mb(m.rss), heapMB: mb(m.heapUsed),
externalMB: mb(m.external), arrayBuffersMB: mb(m.arrayBuffers),
anonMB: anonKB ? Math.round(anonKB / 1024) : null,
}));
}, 60_000).unref();
Use case: allocator settings in a container image.
FROM node:22-bookworm-slim
# Option A: fewer glibc arenas (simple, often enough)
ENV MALLOC_ARENA_MAX=2
# Option B: jemalloc instead of glibc malloc (test both under real load)
RUN apt-get update && apt-get install -y --no-install-recommends libjemalloc2 && rm -rf /var/lib/apt/lists/*
ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2
COPY . /app
WORKDIR /app
CMD ["node", "server.js"]
Use case: bound a native library’s caches (sharp example).
const sharp = require('sharp');
sharp.cache({ memory: 50, files: 20, items: 100 }); // cap libvips operation cache (MB / counts)
sharp.concurrency(2); // fewer libvips threads per operation
Verification and Regression Prevention
A native-memory fix is verified when RSS under a representative workload plateaus instead of ratcheting, the gap between RSS and heap plus external memory stays stable over a day, and container OOM kills stop. Compare allocator configurations with the same traffic replay on separate instances; allocator effects are workload-specific, so measure rather than assume.
Keep the four-number log (or equivalent metrics) permanently and alert on RSS slope as well as heap slope, as in alerting on memory leaks with growth slope. Document allocator choices in the Dockerfile with the measurements that justified them, and re-test after changing base images, since glibc, musl and jemalloc versions all change behaviour.
Edge Cases and Gotchas
LD_PRELOAD paths differ by image
The jemalloc library path depends on distribution and architecture (x86_64 versus aarch64). Verify the preload works by checking /proc/<pid>/maps for the jemalloc library after startup.
MALLOC_ARENA_MAX can cost throughput
Fewer arenas mean more lock contention for heavily multi-threaded native work. Measure latency and throughput alongside RSS when changing it.
musl on Alpine
musl’s allocator has different fragmentation and performance characteristics; some services see lower RSS, others slower allocation-heavy native code. Treat the base image as part of the memory configuration.
Native leaks need native tools
If RSS grows under every allocator and nothing in JavaScript retains Buffers, the leak may be in a native addon. Tools such as heaptrack or valgrind’s massif, run in staging, attribute native allocations to C/C++ call sites.
Frequently Asked Questions
Why is RSS much higher than heapUsed in Node.js?
RSS includes everything in the process: the V8 heap, Buffers and other external memory, compiled code, thread stacks, native library allocations and memory the allocator keeps for reuse. A gap of hundreds of megabytes is common; growth of the gap is what needs attention.
Is RSS growth with a flat heap a memory leak?
Sometimes. It can be retained Buffers or a native addon leak, which are real leaks, or allocator fragmentation, where freed memory is not returned to the operating system. Splitting external/arrayBuffers from the rest and testing allocator settings tells them apart.
What does MALLOC_ARENA_MAX do?
It limits how many malloc arenas glibc creates for threads. Fewer arenas reduce fragmentation and the amount of freed-but-retained memory, often lowering RSS significantly for Node services with native workloads, at some cost in allocation concurrency.
Should I use jemalloc with Node.js?
Many teams see flatter RSS with jemalloc, especially with image processing or compression. It is not universally better; test it against the default allocator and MALLOC_ARENA_MAX=2 under your real workload before adopting it.
How do I find which native library is using memory?
Start with correlations — RSS jumps during image processing, compression or specific database operations — then configure or isolate that library. For precise attribution, profile native allocations in staging with heaptrack or massif.
Does forcing garbage collection reduce RSS?
It frees JavaScript objects and may release Buffer backing stores, but it does not make the C allocator return free memory to the OS or free native library caches. It is a diagnostic step, not a fix.
Related
- Production Memory Monitoring and Container Limits — the parent topic
- Fixing OOMKilled Node.js Containers in Kubernetes — container limits and kill behaviour
- ArrayBuffer and Blob Memory Outside the JS Heap — external memory fundamentals
- Node.js Server-Side Memory Management — the section overview