Alerting on Memory Leaks with Growth Slope

Your “heap above 85%” alert fires every afternoon at peak traffic and never when a slow leak is actually eating the service, which is discovered only after an OOMKilled restart. This guide from Production Memory Monitoring and Container Limits, in Node.js Server-Side Memory Management, shows how to alert on what distinguishes a leak — a rising floor — using the slope of post-GC heap troughs and a predicted time-to-limit, with queries that tolerate restarts, deploys and daily traffic patterns.

Symptom Root Cause Immediate Action Measurable Impact
Alerts fire at every traffic peak Static threshold on raw heap, which includes garbage Alert on troughs (minimum over a window), not raw values False positives drop sharply
Slow leaks found only after OOM Thresholds trigger minutes before the crash Alert on slope and predicted time-to-limit Hours of warning
Alerts reset after each deploy Restarts clear the growth history Evaluate per instance lifetime; require minimum uptime Stable alert behaviour
Leak masked by frequent autoscaling restarts Instances never live long enough to crash Track growth rate per hour of uptime Leak visible despite restarts
RSS grows while heap is flat Native or external memory growth Add slope alerts on RSS and external memory Off-heap leaks caught

Root Cause: Leaks Are About the Floor, Not the Level

Heap usage in a healthy Node.js service is a sawtooth: it rises as requests allocate and falls when garbage collection runs. Its peaks track traffic; its troughs — the heap right after collections — track what is actually retained. A static threshold on the raw value mixes the two: it fires whenever traffic is high enough to push peaks over the line, and it stays silent while a slow leak raises the troughs by a few megabytes an hour, until the heap hits its limit and the process dies, as in why a Node.js process hits the heap limit.

A leak alert should therefore watch the trend of the floor. Two signals do this well. The slope of the minimum heap over sliding windows (for example the minimum of each 10-minute window, fitted over several hours) rises steadily for leaks and stays near zero for healthy services regardless of load. The predicted time-to-limit — a linear projection of the trough trend reaching heap_size_limit (or the container memory limit for RSS) — turns the slope into something humans understand: “at this rate, this instance runs out of heap in 5 hours.”

Operational details decide whether such alerts are usable. Restarts reset memory, so growth must be evaluated per instance and only after enough uptime to see a trend. Deploys create new instances whose warm-up growth — caches filling, code compiling — looks like a leak for the first minutes, so warm-up must be excluded. Daily traffic cycles can raise troughs slightly at peak (more concurrent requests in flight), so windows should be long relative to request duration. And off-heap leaks need the same treatment for RSS and external memory, because the heap can be flat while RSS grows through native memory.

Watch the floor, not the peaks A healthy service's heap sawtooth peaks at an 85 percent static threshold during afternoon traffic but its troughs stay flat, so a threshold alert is a false positive. A leaking service's peaks stay below the threshold for hours while its troughs rise steadily; a line fitted through the troughs shows a positive slope and projects reaching the heap limit in five hours. Healthy: peaks follow traffic, floor flat threshold alert fires at peak: false positive Leaking: floor rises steadily trough slope +12 MB/h → limit in ~5 h peaks never cross the static threshold until minutes before OOM

Step-by-Step Fix

  1. Export the right gauges. Publish heap used, heap limit, RSS and external memory per instance (for example prom-client’s default Node metrics, plus a gauge for heap_size_limit). Verification: each instance has these series with its own labels.
  2. Derive the floor. Use the minimum over short windows (for example min_over_time(heap_used[10m])) as the trough series. Verification: the trough series is smooth and does not follow traffic peaks.
  3. Compute the slope. Apply deriv() over a long window (2–6 hours) to the trough series. Verification: healthy instances show near-zero slope; a staging leak test shows a clear positive slope.
  4. Project time-to-limit. Use predict_linear() on the trough series to estimate heap in N hours and compare with the limit. Verification: the prediction for the leak test crosses the limit hours before the actual crash.
  5. Exclude warm-up and short-lived instances. Require a minimum process uptime (for example 30 minutes) in the alert condition. Verification: deploys no longer trigger the alert.
  6. Alert with context. Fire when slope exceeds a budget (for example 5 MB/h) for an hour and predicted time-to-limit is under a threshold (for example 6 hours); include instance, slope and projection in the message, and link to the snapshot runbook. Verification: on-call receives actionable alerts with time to act.
Threshold versus slope alerts over a month Over a month, a static 85 percent threshold alert fired 23 times, of which 21 were traffic peaks, and gave about 8 minutes of warning before the two real OOM events. Slope and time-to-limit alerts fired 3 times, all real leaks, with between 4 and 9 hours of warning. One month of alerts Static 85% threshold 23 alerts (21 false) · ~8 min warning Trough slope + time-to-limit 3 alerts (all real) · 4–9 h warning

Command and Code Reference

Use case: Prometheus alert rules on the trough slope and time-to-limit.

# memory-leak-rules.yml — metric names follow prom-client defaults plus a heap limit gauge
groups:
  - name: node-memory-leaks
    rules:
      - record: instance:heap_trough_bytes
        expr: min_over_time(nodejs_heap_size_used_bytes[10m])

      - alert: NodeHeapLeakSlope
        # slope over 3h of the 10-minute troughs, in bytes per second
        expr: |
          deriv(instance:heap_trough_bytes[3h]) * 3600 > 5 * 1024 * 1024
          and on(instance) (time() - process_start_time_seconds) > 1800
        for: 1h
        labels: { severity: warning }
        annotations:
          summary: "Heap floor rising >5 MB/h on {{ $labels.instance }}"

      - alert: NodeHeapExhaustionPredicted
        # projected trough in 6h exceeds the V8 heap limit
        expr: |
          predict_linear(instance:heap_trough_bytes[3h], 6 * 3600)
            > on(instance) app_heap_size_limit_bytes
          and on(instance) (time() - process_start_time_seconds) > 1800
        for: 30m
        labels: { severity: critical }
        annotations:
          summary: "{{ $labels.instance }} projected to hit heap limit within 6h"

Use case: expose the heap limit so alerts can compare against it.

const v8 = require('node:v8');
const client = require('prom-client');
client.collectDefaultMetrics();                         // heap used, RSS, external, GC, uptime

new client.Gauge({
  name: 'app_heap_size_limit_bytes',
  help: 'V8 heap size limit for this process',
  collect() { this.set(v8.getHeapStatistics().heap_size_limit); },
});

Verification and Regression Prevention

Verify alerts before trusting them. In staging, run a deliberate slow leak (for example a module-level array that grows by a few kilobytes per request under steady load) and confirm that the slope alert fires within its window and the prediction alert fires hours before the process would crash — while a normal load test with traffic peaks triggers neither. Replay a month of production metrics through the rules if your monitoring system supports backtesting.

Review the thresholds quarterly: the slope budget should be below the rate that would exhaust the heap within your typical instance lifetime, and above the drift of healthy services. When an alert fires, follow the runbook to capture evidence with taking heap snapshots from a live Node.js process while the leak is still small.

Testing alerts before trusting them In staging, run a deliberate slow leak and confirm the slope alert fires within its window and the prediction alert fires hours before the process would crash, then run a normal load test with traffic peaks and confirm neither alert fires. Staging tests for each alert Slow leak → slope alert Fires within its window. Slow leak → prediction Fires hours before the projected crash. Normal peaks → silence Neither alert fires on healthy traffic.

Edge Cases and Gotchas

Caches filling look like leaks

A cache warming up after a deploy raises the floor until it reaches its bound. With bounded caches the slope flattens within an hour or two; the uptime condition and a long for duration prevent alerts during this phase. Unbounded caches will keep the slope positive — correctly, since they are leaks in practice.

Instances that restart often

If autoscaling or deploys replace instances every few hours, leaks may never cause crashes but still waste memory and CPU. Report the average slope per service as a dashboard metric even when no alert fires.

Traffic-dependent retention

Services that keep per-connection state (websockets, long polls) have floors that follow the number of open connections. Normalise the trough by connection count, or alert on memory per connection instead.

Multiple processes per container

With cluster mode, compute slopes per process, not per container, so one leaking worker is not averaged away by healthy siblings.

Frequently Asked Questions

Why not just alert when heap usage exceeds a threshold?

Raw heap includes garbage and follows traffic, so thresholds fire at busy times and stay quiet during slow leaks until the last minutes. The trough trend isolates retained memory and gives hours of warning.

What slope indicates a leak?

Any sustained positive slope of the post-GC floor after warm-up. Choose an alert threshold from your heap limit and instance lifetime — for example, a slope that would exhaust the remaining headroom within a day — and tune it against backtests.

How do I handle restarts in slope calculations?

Evaluate per instance and require a minimum uptime before alerting, so new instances’ warm-up growth is ignored. Slopes computed over windows that span a restart are unreliable; the uptime condition avoids them.

Should I alert on RSS too?

Yes. Heap-only alerts miss leaks in Buffers, native addons or allocator fragmentation. Apply the same trough-slope approach to RSS against the container memory limit.

What should the alert tell on-call engineers?

The instance, the slope in MB per hour, the projected time to the limit, and a link to the runbook for capturing heap snapshots or restarting safely. That turns the alert into a decision rather than an investigation.

Can I backtest these rules?

Many Prometheus-compatible systems can evaluate rules against historical data. Replaying past incidents and ordinary peak days shows whether the rules would have fired early for real leaks and stayed quiet otherwise.