Node.js Stream Backpressure & Memory Growth

Streams are the mechanism Node.js gives you for moving data larger than the heap through a process without loading all of it at once. That promise only holds while backpressure — the flow-control signal that tells a fast producer to wait for a slow consumer — is respected. Ignore it and the stream’s internal buffer becomes an unbounded queue, resident set size (RSS) climbs in lockstep with bytes transferred, and the process drifts toward the V8 heap ceiling. This guide sits inside Node.js Server-Side Memory Management and focuses on why buffers grow, how to read the write() return value and highWaterMark, and how the pipeline() pattern keeps memory flat. For the deeper reproduction recipe see diagnosing unbounded buffering, and for the exact API comparison see pipeline vs pipe.

Conceptual Grounding #

Node.js exposes four stream base classes, and it pays to know which buffer belongs to which. A Readable produces data and holds a read buffer that fills when nobody is consuming. A Writable consumes data and holds a write buffer that fills when the underlying resource is slow. A Transform (the base for zlib.createGzip(), crypto ciphers, and your own row-mappers) is both at once — it owns a writable side facing upstream and a readable side facing downstream, so a Transform can accumulate memory on either edge. A Duplex is the general two-sided case, such as a TCP socket. Backpressure has to propagate through every one of these buffers in a chain; if any single stage ignores the signal, the whole pipeline loses flow control and memory pools at that stage.

Every Readable and Writable stream owns an internal buffer plus a threshold called highWaterMark. For byte streams the default is 16384 bytes (16 KB); for object-mode streams it is 16 objects. The threshold is not a hard cap — it is an advisory line. When you call writable.write(chunk) and the buffered length would exceed highWaterMark, the method returns false. That boolean is the entire backpressure contract: it means “I accepted this chunk, but stop sending until I emit drain”. If your producer keeps calling write() after a false, Node.js still queues every chunk — the buffer simply grows past the mark, one allocation at a time.

That is the core failure mode. Backpressure is cooperative, not enforced. A Readable connected by pipe() honours the signal automatically: it calls pause() when the destination returns false and resume() on drain. But hand-rolled loops, for await bodies that fan out writes, or a Transform whose _transform callback fires faster than the sink drains all bypass that cooperation. The buffered chunks are reachable objects held by the stream’s internal state, so the mark-and-sweep collector cannot reclaim them — they are live, not leaked, which is why heap snapshots show a growing Buffer/array retainer rather than a classic detached-object leak.

It helps to picture where those chunks physically live. Small string and buffer chunks are allocated in V8’s old space once they survive the first scavenge, and the stream keeps them referenced through a linked BufferList structure. Because they are reachable, they count against --max-old-space-size, and the collector spends escalating CPU on mark phases trying to walk an ever-larger live set. So a backpressure bug degrades throughput twice: once because the buffer holds memory, and again because major GC pauses lengthen as that buffer grows. On a server this shows up as latency creep long before the fatal OOM — p99 response times drift upward while RSS marches toward the ceiling. Recognising the pattern early, from the shape of the memory curve rather than the crash, is the single biggest time-saver when debugging stream pipelines in production.

Diagnostic Workflow #

Follow these steps to confirm backpressure — not a reference leak — is driving the growth. Each step names the exact API or flag and the output to expect. Work them in order: the early steps are cheap instrumentation you can leave in a staging build, while the heap snapshot at the end is the confirmation you reach for only once the cheaper signals point at a buffer.

1. Baseline RSS during transfer. Poll process.memoryUsage().rss on a 500 ms timer while the workload runs. Expected output: with broken backpressure, RSS rises roughly linearly with bytes moved; with healthy flow it plateaus a few MB above idle. Related heap-ceiling behaviour is covered in Node.js memory limits & OOM.

2. Log the write() return value. Wrap the sink write and print the boolean. Expected output: a stream of true values that flips to false once the buffer fills. If you see false and your code keeps writing without waiting for drain, that is the defect.

3. Read writableLength vs writableHighWaterMark. These two Writable properties expose the live buffer size and its threshold in bytes. Expected output: healthy streams show writableLength oscillating near writableHighWaterMark (16384); broken ones show writableLength in the millions and climbing.

4. Cross-check with --max-old-space-size. Run the process with a deliberately low ceiling, e.g. node --max-old-space-size=256 app.js. Expected output: a backpressure bug crashes with FATAL ERROR: Reached heap limit Allocation failed far sooner, confirming the buffer is on the old-space heap. If the same workload survives comfortably at 256 MB, the growth you saw earlier was bounded and you are chasing the wrong signal. This flag is also a useful production guard-rail: pinning it below the container limit turns a slow memory climb into a fast, restart-recoverable crash rather than a kernel OOM-kill that takes the container down uncleanly.

5. Capture a heap snapshot. In DevTools → Memory → Heap Snapshot (attach with node --inspect app.js, then open chrome://inspect), take a snapshot mid-transfer and open the Comparison view against an idle baseline. Expected output: the top retainer is an ArrayBuffer or the stream’s internal BufferList, not application objects — the signature of buffered, not leaked, memory. Sort the Comparison view by “Size Delta” to surface the buffer growth in one glance.

6. Confirm the source never paused. Instrument the Readable with a counter on its pause and resume events. Expected output: in a healthy pipeline these fire repeatedly as the buffer fills and drains; in a broken loop the pause count stays at zero for the whole transfer, which is the clearest single proof that the source ignored the sink. This distinguishes a backpressure defect from an unrelated retained-reference leak, where pause/resume cycle normally but memory still climbs.

Code Patterns & Signatures #

The broken pattern below ignores the write() return value inside a tight loop, so the sink’s buffer grows without bound.

// Anti-pattern: copy a huge file while
// ignoring backpressure entirely.
const fs = require('node:fs');
const src = fs.createReadStream('big.log');
const dst = fs.createWriteStream('out.log');

src.on('data', (chunk) => {
  // write() returns false when the buffer is
  // full, but we never check it, so every
  // chunk is queued regardless of drain.
  dst.write(chunk); // <- return value dropped
});
src.on('end', () => dst.end()); // sink flushes late

This corrected version pauses the source when the sink signals a full buffer and resumes only on drain.

// Manual backpressure: honour write()'s
// false and wait for the 'drain' event.
src.on('data', (chunk) => {
  // If write() returns false the buffer is at
  // highWaterMark; stop reading immediately.
  if (dst.write(chunk) === false) {
    src.pause(); // stop emitting 'data'
  }
});
dst.on('drain', () => {
  // Buffer has emptied below highWaterMark,
  // so it is safe to resume the source.
  src.resume(); // flow restarts, RSS stays flat
});
src.on('end', () => dst.end());

The idiomatic fix replaces all of that with stream.pipeline(), which wires up backpressure, propagates errors, and destroys every stream on failure.

// Preferred: pipeline() manages backpressure
// and cleanup for the whole chain.
const { pipeline } = require('node:stream/promises');
const fs = require('node:fs');
const zlib = require('node:zlib');

async function compress() {
  // Each stage pauses the previous one when its
  // buffer fills, so peak memory ~= sum of the
  // per-stage highWaterMarks, not the file size.
  await pipeline(
    fs.createReadStream('big.log'),   // source
    zlib.createGzip(),                // transform
    fs.createWriteStream('big.log.gz')// slow sink
  );
  // Resolves only after the last byte is flushed;
  // rejects (and destroys all streams) on error.
}
compress().catch(console.error);

For a custom Transform, keep the buffer bounded by returning from _transform only after the downstream callback fires, and set an explicit highWaterMark sized to your row payload.

// Object-mode Transform with a deliberate,
// small highWaterMark to cap retained rows.
const { Transform } = require('node:stream');

const enrich = new Transform({
  objectMode: true,       // chunks are objects
  highWaterMark: 8,       // buffer at most 8 rows
  transform(row, _enc, cb) {
    // Do async work, then signal completion so
    // the stream can apply backpressure upstream.
    lookup(row.id).then((extra) => {
      // Passing the result to cb() enqueues one
      // object; the 8-row mark throttles input.
      cb(null, { ...row, ...extra });
    }).catch(cb); // errors propagate to pipeline()
  }
});

Across all four patterns the invariant is the same: a chunk must not be produced faster than the slowest stage can accept it. The manual pause/resume version makes that explicit and is worth understanding because it is exactly what pipeline() does for you internally. In practice you should reach for the manual form only when you need custom logic between stages — throttling, sampling, or conditional routing — that a plain chain cannot express. Even then, prefer composing small Transforms inside a pipeline() over writing a bespoke event loop, because the moment you hand-roll data handling you take on responsibility for every error path and cleanup step that pipeline() would otherwise guarantee.

The peak-memory maths for a healthy pipeline() is reassuringly simple: worst-case resident buffer is roughly the sum of each stage’s highWaterMark, plus one in-flight chunk per stage. A three-stage byte pipeline at the 16 KB default therefore peaks around 48–64 KB of stream buffers regardless of whether the file is 10 MB or 10 GB. That predictability is the whole point — memory becomes a function of pipeline shape, not payload size, and capacity planning stops depending on the largest input you might ever see.

Symptom-to-Fix Reference #

Edge Cases & Gotchas #

• pipe() leaves streams half-open on error. A classic src.pipe(dst) honours backpressure but does not destroy src when dst errors, so the source keeps its buffer and file descriptor alive. Fix: use pipeline(), which calls destroy() on every stream in the chain. The exact difference is spelled out in pipeline vs pipe.

• cork() without uncork() buffers indefinitely. Calling writable.cork() to batch writes is fine only if a matching uncork() (or process.nextTick(() => writable.uncork())) follows. A forgotten uncork() holds every write in memory until the stream ends. Fix: always pair the calls, ideally in the same tick.

• Object mode measures count, not bytes. A highWaterMark of 16 objects sounds small but if each object is a 2 MB database row, the buffer can hold 32 MB. Fix: set highWaterMark explicitly based on payload size, as in the Transform example above.

• for await...of over a Readable is safe; fanning out inside it is not. The loop itself applies backpressure, but calling an unawaited async write per iteration schedules unbounded work. Fix: await each downstream write, or push the sink into a pipeline().

• data listeners silently switch to flowing mode. Attaching a data handler puts a Readable into flowing mode immediately; if you have not wired pause/resume, chunks arrive faster than you consume them. Fix: prefer pipeline() or the async iterator over manual data handling.

• HTTP responses are Writable streams too. When you stream a large payload to a client with res.write() and drop the return value, a slow client on a throttled mobile connection becomes the slow sink — the server buffers the whole response in memory per request. Under concurrency this multiplies across every in-flight request and can exhaust the heap far faster than a single file copy. Fix: pipe the source through pipeline(source, res) so a stalled socket pauses the reader.

• Readable.from() on a sync iterable can defeat backpressure expectations. Wrapping an in-memory array with Readable.from() produces a stream, but the data already lives fully in the heap, so streaming it yields no memory benefit. Fix: stream from the true origin (file, socket, database cursor) rather than materialising everything first.

Frequently Asked Questions #

Does a large highWaterMark cause a memory leak? #

No. A large highWaterMark raises the steady-state buffer ceiling but memory still plateaus at that ceiling. A genuine unbounded-growth bug comes from ignoring write() returning false, which lets the buffer grow past highWaterMark without bound. Raising the mark to 1 MB just moves the plateau up by 1 MB; it does not turn a bounded buffer into a leak.

Why does pipe() not protect me from memory growth? #

pipe() does honour backpressure for the happy path — it pauses and resumes the source correctly. The gap is error handling: on a stream error pipe() does not destroy the other streams, so a half-open pipe can keep buffers and file descriptors resident until the process exits. pipeline() guarantees every stream is destroyed on error or completion, which is why it is the memory-safe default.

How do I know backpressure is actually working? #

Log writableLength during the transfer. If it oscillates around writableHighWaterMark (16384 bytes by default) and RSS stays flat, backpressure is working. If writableLength climbs monotonically into the millions while RSS tracks the total bytes transferred, the source is not being paused and you have a broken flow-control loop. A second, orthogonal signal is the source’s pause event count: a healthy pipeline pauses and resumes the Readable many times over a large transfer, whereas a broken one never pauses it at all. Watching both together removes ambiguity — buffer size tells you the symptom, the pause count tells you the cause.

Does object-mode streaming change the buffering maths? #

Yes. In object mode highWaterMark counts objects, not bytes, and defaults to 16. A single buffered object can retain megabytes, so a 16-object buffer of large rows can hold far more memory than the byte-mode default of 16 KB implies. Size highWaterMark against your per-object payload, not against a byte figure. A practical rule for database or CSV row streams is to multiply your average row size by the mark and keep the product comfortably under a single stage’s share of the heap budget — for 500 KB rows a mark of 8 already reserves 4 MB per stage, which is usually as high as you want to go. And remember that backpressure only bounds a buffer that is meant to drain: if RSS keeps climbing after the streams close, you have a retained reference elsewhere, not a stream-buffer problem.

Related #

• Return to the Node.js Server-Side Memory Management main section for the full server-side memory map.
• Follow the reproduction recipe in the diagnosing unbounded buffering child guide.
• Compare the two wiring APIs in the pipeline vs pipe child guide.
• Understand heap-ceiling failure in the Node.js memory limits & OOM parent guide.