Choosing highWaterMark for Node.js Streams

Someone raised highWaterMark to 16 MB to “make uploads faster”, and now a file service with 500 concurrent transfers uses 8 GB of RAM — or an object-mode pipeline processing 5 MB records buffers 16 of them per stage without anyone noticing. This guide from Node.js Stream Backpressure and Memory Growth, in Node.js Server-Side Memory Management, explains what highWaterMark actually limits, how it multiplies with concurrency and pipeline depth, and how to choose values from a memory budget rather than by guessing.

Symptom Root Cause Immediate Action Measurable Impact
RSS scales with concurrent streams × large buffers Oversized highWaterMark per stream Size from a per-connection memory budget Predictable memory at peak concurrency
Object-mode pipeline uses far more memory than expected highWaterMark counts objects, not bytes Lower the object count or chunk records smaller Buffered bytes bounded
Throughput poor on fast disks with tiny buffers Very small highWaterMark causes many small reads Raise moderately (64 KB–1 MB) for bulk I/O Higher throughput, bounded memory
Memory grows despite a sensible highWaterMark Producer ignores write() return value Fix backpressure; highWaterMark is only a threshold Buffer stays near the threshold
Every stage of a long pipeline buffers Each stage has its own readable and writable buffers Count stages in the budget; reduce stage buffers Total pipeline buffer bounded

Root Cause: A Threshold, Multiplied Everywhere

Every Node.js stream keeps an internal buffer. For a Readable, highWaterMark is how much data it will read ahead from its source before it stops calling _read(); for a Writable, it is the level at which write() starts returning false to tell the producer to wait for 'drain'. It is a threshold for signalling backpressure, not a hard cap: a producer that ignores false can push the buffer far beyond it, which is the failure described in diagnosing unbounded buffering in Node.js streams. When backpressure is respected, though, a stream holds roughly up to highWaterMark of data at a time — and that is the number to budget.

For byte streams the value is in bytes; the default is 64 KiB in current Node.js versions (older versions used 16 KiB for generic streams while fs read streams used 64 KiB). For object-mode streams the value is a count of objects, 16 by default — and objects can be anything from a 50-byte row to a 5 MB parsed document. Sixteen large records per stage is easy to overlook.

The real memory cost is multiplication. A single request may involve several streams: a socket, a decompressor, a parser, a transform, a destination — each with readable and writable buffers. A service handles many requests concurrently. Total buffered memory is roughly:

concurrent flows × stages per flow × highWaterMark per stage (× average object size in object mode)

With 500 concurrent uploads, three stages each and a 16 MB highWaterMark, that is up to 24 GB of potential buffering; with 64 KB it is under 100 MB. Larger buffers improve throughput only up to the point where I/O calls are large enough to amortise their overhead — typically somewhere between 64 KB and 1 MB for disk and network — beyond which extra buffer mainly adds memory and latency. The pipeline vs pipe guide covers wiring stages so backpressure propagates end to end.

How highWaterMark multiplies One upload flow passes through a socket, a gunzip transform and a file write stream, each buffering up to highWaterMark. With 64 kilobytes per stage, one flow buffers up to about 192 kilobytes and 500 concurrent flows up to about 94 megabytes. With 16 megabytes per stage, one flow buffers up to 48 megabytes and 500 flows up to about 24 gigabytes. One flow (× concurrent flows) socket ≤ hwm buffered gunzip transform ≤ hwm in + out fs write stream ≤ hwm buffered hwm = 64 KB per flow ≈ 192 KB 500 flows ≈ 94 MB hwm = 16 MB per flow ≈ 48 MB 500 flows ≈ 24 GB

Step-by-Step Fix

  1. Inventory stream settings. Search for highWaterMark options and setDefaultHighWaterMark calls, and list object-mode streams with typical object sizes. Verification: you know every non-default value and what it counts.
  2. Compute the worst case. Multiply peak concurrent flows × stages × highWaterMark (× object size for object mode). Verification: you have a number to compare with the container’s memory budget.
  3. Set a per-flow budget. Decide how much buffering one flow may use (for example 1 MB) from the memory left after heap and baseline RSS, divided by peak concurrency. Verification: the budget × concurrency fits comfortably in the container limit.
  4. Size stages from the budget. Choose byte highWaterMarks (commonly 64 KB–1 MB for disk/network) and object counts (often 1–8 for large records) that keep each flow within budget. Verification: worst-case total stays under the limit.
  5. Benchmark throughput. Measure throughput at a few values (64 KB, 256 KB, 1 MB) with realistic concurrency. Verification: you pick the smallest value near the throughput plateau.
  6. Confirm under load. Run a peak-concurrency load test and watch RSS, arrayBuffers and per-stream writableLength/readableLength. Verification: memory tracks the computed budget and buffers stay near their thresholds.
Throughput plateaus, memory keeps rising With 200 concurrent file copies, throughput is 410 megabytes per second at 16 kilobytes, 690 at 64 kilobytes, 760 at 256 kilobytes and 770 at 1 megabyte, while peak buffered memory is 10, 38, 150 and 600 megabytes. 256 kilobytes captures nearly all the throughput at a quarter of the memory of 1 megabyte. 200 concurrent copies: throughput vs peak buffer memory 16 KB 410 MB/s · 10 MB 64 KB 690 MB/s · 38 MB 256 KB 760 MB/s · 150 MB (chosen) 1 MB 770 MB/s · 600 MB illustrative; bars show throughput, labels show peak buffered memory

Command and Code Reference

Use case: explicit, budgeted buffer sizes in a pipeline.

const { pipeline } = require('node:stream/promises');
const fs = require('node:fs');
const zlib = require('node:zlib');

const HWM = 256 * 1024;                          // chosen from benchmark + memory budget

async function storeUpload(req, path) {
  await pipeline(
    req,                                         // socket-backed readable (default hwm)
    zlib.createGunzip({ highWaterMark: HWM }),   // transform: both sides use HWM
    fs.createWriteStream(path, { highWaterMark: HWM }),
  );
}

Use case: object-mode stages sized for large records.

const { Transform } = require('node:stream');

// Each object is a parsed document of ~2–5 MB: buffer at most 2 per side
const enrich = new Transform({
  objectMode: true,
  highWaterMark: 2,                              // counts objects, not bytes
  async transform(doc, _enc, done) {
    try { done(null, await addMetadata(doc)); } catch (err) { done(err); }
  },
});

Use case: watch buffer levels under load.

// Log the largest buffers every 5 s during a load test
setInterval(() => {
  for (const s of activeStreams) {
    if (s.writableLength > 2 * s.writableHighWaterMark) {
      console.warn('buffer over threshold', s.constructor.name, s.writableLength);
    }
  }
}, 5000).unref();

Verification and Regression Prevention

The chosen values are right when a peak-concurrency load test shows buffered memory near the computed budget, throughput near the plateau of your benchmark, and per-stream writableLength staying close to writableHighWaterMark rather than growing. Re-run the benchmark when you change instance types or Node versions — the default changed between versions, and I/O characteristics differ between disks.

Record the budget calculation next to the constants in code. Add a check in code review for new streams with large highWaterMarks or object-mode stages handling big records, and export stream buffer metrics for long-lived connections. When buffering grows beyond thresholds despite reasonable values, the problem is ignored backpressure rather than sizing; fix it with async iterators and stream backpressure or pipeline().

Reading a peak-concurrency load test At peak concurrency, buffered memory near the computed budget with throughput near the benchmark plateau means the values are right. writableLength growing far past writableHighWaterMark means a writer ignores backpressure. Throughput well below the plateau with small buffers means highWaterMark is too low for this link. Peak-concurrency load test Values are right; record them buffers ≈ budget, throughput ≈ plateau A writer ignores the return value of write() writableLength ≫ highWaterMark highWaterMark too small for this link throughput below plateau

Edge Cases and Gotchas

Changing the process-wide default

stream.setDefaultHighWaterMark(objectMode, value) changes the default for every stream created afterwards, including those inside libraries. Prefer per-stream options unless you have measured the whole application.

Sockets and HTTP bodies

HTTP request and response streams use socket buffers managed partly by the kernel. Node-level highWaterMark is only one part of the memory used per connection; see WebSocket server memory per connection for the connection-level view.

Transform streams have two buffers

A Transform has a writable side and a readable side, each with its own threshold. Setting highWaterMark applies to both unless you use writableHighWaterMark and readableHighWaterMark separately.

Chunk sizes from sources

A readable’s highWaterMark influences how much it reads per call, but sources may deliver larger chunks (a network packet burst, a large write). A single chunk can exceed the threshold; budgets should allow for the largest expected chunk.

Frequently Asked Questions

What is the default highWaterMark in Node.js?

For byte streams it is 64 KiB in current Node.js versions, and 16 objects for object-mode streams. Older versions used 16 KiB for generic byte streams. fs read streams have long used 64 KiB.

Does a bigger highWaterMark make streams faster?

Up to a point. Larger buffers mean fewer, larger I/O operations, which helps throughput until overhead is amortised — typically somewhere between 64 KB and 1 MB. Beyond that, extra buffering mostly increases memory and latency.

Is highWaterMark a hard memory limit?

No. It is the threshold at which a Writable’s write() returns false and a Readable stops reading ahead. If a producer ignores backpressure, buffers grow beyond it without limit.

How does objectMode change highWaterMark?

In object mode, highWaterMark counts objects rather than bytes. Sixteen small rows are negligible; sixteen multi-megabyte documents per stage can use a lot of memory. Size it from the typical object size.

Should upload and download streams use the same value?

Not necessarily. Downloads to many slow clients multiply buffers by concurrency and tolerate smaller values well, because the client, not your disk, limits throughput. Bulk internal copies with few concurrent flows benefit more from larger buffers. Size each path from its own concurrency and throughput needs.

How do I choose a value?

Compute a per-flow memory budget from your container limit and peak concurrency, benchmark throughput at a few sizes, and pick the smallest size near the throughput plateau that fits the budget.