DEV Community: Gaurav Singh

Streaming anomaly detection in Node.js with Iterflow

Gaurav Singh — Sun, 22 Feb 2026 05:38:31 +0000

If you need to flag latency spikes in a stream of request times, the usual approach is to collect everything, compute mean and std dev, then filter. That doesn't work well on continuous streams where you don't have all the data upfront.

Iterflow is a streaming statistics library for JS/TS that lets you do this incrementally. This post walks through three approaches to anomaly detection using it.

npm install @mathscapes/iterflow

The data

Some HTTP response times. Mostly 40-50ms, with a couple of spikes mixed in:

const latencies = [
  45, 42, 48, 44, 46, 43, 47, 45, 44, 43,
  46, 41, 48, 45, 200,  // spike
  44, 43, 46, 42, 45, 47, 44, 43, 45, 46,
  42, 48, 44, 250,      // spike
  45, 43, 47, 44, 46
];

The key thing: you want to flag values that are weird relative to what came before them. In a real stream you don't have the full dataset, so you can't just take the global mean.

Streaming z-score

Z-score = how many standard deviations from the mean. Iterflow's streamingZScore() computes this incrementally -- each value is scored against the mean and std dev of everything before it. The current value doesn't affect its own score.

import { iter } from '@mathscapes/iterflow';

const anomalies = iter(latencies)
  .streamingZScore()
  .enumerate()
  .filter(([_, z]) => Math.abs(z) > 3)
  .map(([i, z]) => ({ index: i, value: latencies[i], zScore: +z.toFixed(2) }))
  .toArray();

console.log(anomalies);
// [
//   { index: 14, value: 200, zScore: 8.41 },
//   { index: 28, value: 250, zScore: 9.12 }
// ]

Breaking it down:

streamingZScore() emits a z-score per element using Welford's algorithm internally. First two elements come out as NaN (need at least two prior observations for a meaningful std dev).
enumerate() pairs each z-score with its index: [0, NaN], [1, NaN], [2, 3.0], ...
filter keeps anything beyond 3 standard deviations.
map reshapes it into something readable.
toArray() kicks off the pipeline.

The z-score stage holds 3 numbers in memory (count, mean, M2). The whole pipeline is O(1). If you threw a .take(1) in there it'd stop after the first anomaly -- nothing upstream gets processed past that point.

Windowed variant

The above uses the full history, so the mean gets increasingly stable over time. If you care more about recent behavior (normal for the last hour but weird for the last 5 minutes), use a window instead:

const windowSize = 10;

const localAnomalies = iter(latencies)
  .window(windowSize)
  .map(w => {
    const mean = iter(w).mean();
    const std = iter(w).stdDev();
    return { mean, std, last: w[w.length - 1] };
  })
  .enumerate()
  .filter(([_, { mean, std, last }]) => std > 0 && Math.abs(last - mean) > 3 * std)
  .map(([i, { last, mean, std }]) => ({
    index: i + windowSize - 1,
    value: last,
    windowMean: +mean.toFixed(1),
    windowStd: +std.toFixed(1),
  }))
  .toArray();

console.log(localAnomalies);

window(10) slides a 10-element window across the stream. Memory is O(k) where k is the window size, not O(n).

EWMA approach

Exponentially weighted moving average -- weights recent values more heavily. Compare raw values against the smoothed trend:

const alpha = 0.3;
const threshold = 50;

const smoothed = iter(latencies).ewma(alpha).toArray();

const ewmaAnomalies = iter(latencies)
  .enumerate()
  .filter(([i]) => i > 0)
  .filter(([i, val]) => Math.abs(val - smoothed[i - 1]) > threshold)
  .map(([i, val]) => ({ index: i, value: val, ewma: +smoothed[i - 1].toFixed(1) }))
  .toArray();

console.log(ewmaAnomalies);
// [
//   { index: 14, value: 200, ewma: 44.5 },
//   { index: 28, value: 250, ewma: 45.1 }
// ]

One scalar of state. Higher alpha = reacts faster to changes. Lower alpha = smoother line.

With real streams

Same pattern works with generators. Doesn't have to be an array:

function* readLatencies(): Generator<number> {
  // reading from a log file, message queue, API, whatever
  while (true) {
    yield getNextLatency();
  }
}

// infinite stream, stop after 5 anomalies
const first5 = iter(readLatencies())
  .streamingZScore()
  .enumerate()
  .filter(([_, z]) => Math.abs(z) > 3)
  .take(5)
  .toArray();

Every stage is a generator, so this processes one element at a time. Constant memory no matter how long the stream runs. .take(5) propagates back through the chain -- once you have 5 anomalies, upstream generators just stop.

When this isn't worth it

Small fixed dataset that fits in memory? Just do array.filter(x => x > mean + 3 * std). Iterflow's generators add 3-5x overhead per element vs a hand-written loop. The payoff is when you're chaining multiple stages together, dealing with large/infinite streams, or need early termination.

GitHub: github.com/mathscapes/iterflow
npm: npm install @mathscapes/iterflow
Paper: doi.org/10.5281/zenodo.18610143

iterflow 1.0: What changed after someone pointed out JS already has lazy iterators

Gaurav Singh — Sun, 15 Feb 2026 02:16:35 +0000

iterflow 1.0 shipped last week. It's a different library than what I wrote about in January.

Back then, at RC2, someone rightly pointed out that JavaScript already has native Iterator Helpers - map, filter, take, drop, flatMap. Most of what iterflow offered was redundant with the language. Fair. The only things it had beyond native iterators were window(), chunk(), and a few terminal statistics with naive implementations.

Between RC2 and 1.0, the library found a different reason to exist.

Welford's algorithm changed how I thought about the library

RC2 swapped variance from two-pass sum-of-squares to Welford's online algorithm. Single pass, three scalars of state (count, running mean, running sum of squared deviations), no array materialization. You feed it values one at a time and it maintains the correct variance at every step.

I noticed this is the same contract as a JavaScript generator - values arrive one by one, state is local, output goes downstream. So I tried something: instead of variance consuming the whole iterator and returning one number, what if it yielded a running result at each step?

import { iter } from '@mathscapes/iterflow';

iter([2, 4, 4, 4, 5, 5, 7, 9]).streamingVariance().toArray();
// [0, 1, 0.889, 0.75, 0.96, 1, 1.959, 4]
// ^ population variance after each observation

That's streamingVariance() - a generator wrapping Welford's recurrence. It chains with filter, take, window like any other transform. You could write this generator yourself (Iterator Helpers are extensible), but you'd be implementing Welford's recurrence from scratch, and again for mean, and again for covariance, and again for correlation.

Once I had this working, every release after was adding another online algorithm as a pipeline stage.

What 1.0 looks like

By 1.0, iterflow has: streaming mean, streaming variance, EWMA, streaming covariance, streaming Pearson correlation (Chan et al.'s bivariate Welford extension, six scalars of state), z-score anomaly detection, and monotonic deque windowed min/max (Lemire's algorithm). All composable, all lazy.

Here's a composed pipeline vs. writing it by hand. Running Pearson correlation between price and volume, 10 observations:

import { iter } from '@mathscapes/iterflow';

const prices  = [100, 102, 104, 103, 107, 110, 108, 112, 115, 113];
const volumes = [500, 480, 520, 510, 550, 600, 570, 620, 650, 610];

iter(prices).zip(volumes).streamingCorrelation().toArray();
// [NaN, -1, 0.5, 0.543, 0.851, 0.941, 0.950, 0.968, 0.980, 0.978]

Without iterflow:

const iterP = prices[Symbol.iterator]();
const iterV = volumes[Symbol.iterator]();
let n = 0, mx = 0, my = 0, m2x = 0, m2y = 0, cxy = 0;
const results = [];
for (;;) {
  const p = iterP.next(), v = iterV.next();
  if (p.done || v.done) break;
  n++;
  const dx = p.value - mx; mx += dx / n;
  const dx2 = p.value - mx; m2x += dx * dx2;
  const dy = v.value - my; my += dy / n;
  const dy2 = v.value - my; m2y += dy * dy2;
  cxy += dx * dy2;
  const d = m2x * m2y;
  results.push(d > 0 ? cxy / Math.sqrt(d) : NaN);
}

Same output. But now try adding a filter upstream, or a threshold downstream, or swapping correlation for z-score. Each combination means rewriting the loop.

With iterflow, those combinations are just different chains. Here's anomaly detection on sensor data:

import { iter } from '@mathscapes/iterflow';

const sensorReadings = [
  { valid: true,  value: 10 },
  { valid: false, value: 999 },
  { valid: true,  value: 12 },
  { valid: true,  value: 11 },
  { valid: true,  value: 50 },  // anomaly
  { valid: true,  value: 13 },
  { valid: true,  value: 12 },
];

iter(sensorReadings)
  .filter(r => r.valid)
  .map(r => r.value)
  .streamingZScore()
  .filter(z => Math.abs(z) > 2)
  .toArray();
// [47.77]

The z-score of 47.77 looks extreme, but it's correct - the prior valid readings are [10, 12, 11] with mean 11 and stddev 0.82, so a jump to 50 is a ~48-sigma event. The z-score stage uses Welford state from prior observations only. The current value doesn't influence its own score. First two elements yield NaN (not enough data for standard deviation) and get filtered out by the threshold.

Why a standalone library and not a contribution to @stdlib?

@stdlib/stats/incr has solid streaming accumulators. But they're imperative objects - you call .update(x) in a loop and read .value. iterflow's contribution isn't the algorithms (Welford is Welford) but the composition model: online algorithms as generator stages inside JavaScript's iterator protocol. "filter, window, variance, threshold, take 100" as a single chained expression, not a loop with manual accumulator wiring.

If you only need one streaming statistic with no pipeline composition, @stdlib will be faster. Generators have per-element overhead from yield/next() dispatch. The paper benchmarks this - iterflow wins on multi-stage composition with early termination, loses on standalone single-statistic computation. I tried to be honest about both sides.

The paper

I wrote a paper formalizing the design: "iterflow: Composable Streaming Statistics for JavaScript".

It covers the algorithms (Welford, Chan et al., Lemire, Hoare's Quickselect), benchmarks against native array methods, @stdlib/stats-incr-mvariance, and hand-written loops, and documents the limitations I know about: Array.shift() on the monotonic deque is O(k) per front removal, not O(1) - fine for small windows, degrades for large k. All benchmarks on a single platform (V8/Node.js, ARM64 Linux).

Implementation choices: population variance (n denominator, not n-1) since the typical use case is complete streams. Pre-observation z-score convention. Circular buffer for window() instead of shift-based arrays.

Things I'm not planning: async iterables, streaming quantile estimation, operator fusion.

Get started

npm install @mathscapes/iterflow

Zero dependencies. 13KB uncompressed (up from 9KB at RC2 due to the streaming algorithms). TypeScript, full type inference, dual ESM/CJS.

github.com/mathscapes/iterflow | Paper (DOI)

Iterflow: Lazy Iterators for TypeScript

Gaurav Singh — Sun, 04 Jan 2026 03:50:21 +0000

JavaScript's Array methods are eager. When you chain .filter().map().reduce(), each operation creates a new array in memory before moving to the next step.

For small datasets, this is fine. For large datasets, you run out of memory.

I built Iterflow to solve this: lazy evaluation with built-in statistics and windowing for data processing in TypeScript.

The Problem with Eager Evaluation

Here's what most of us write:

const results = data
  .filter(row => row.status === 'active')
  .map(row => transformRow(row))
  .slice(0, 1000);

This looks clean, but here's what actually happens:

filter() iterates through ALL items → creates new array
map() iterates through filtered items → creates another array
slice() finally takes what we need

For 10 million rows, we're creating multiple arrays with millions of elements. That's why Node throws out-of-memory errors.

Lazy Evaluation with Iterflow

Here's the same operation with lazy evaluation:

import { iter } from '@mathscapes/iterflow';

const results = iter(data)
  .filter(row => row.status === 'active')
  .map(row => transformRow(row))
  .take(1000)
  .toArray();

The difference? Nothing executes until toArray().

Iterflow processes one item at a time through the entire pipeline. Need only 1000 items? It stops after finding 1000 matches, not after processing millions.

JavaScript has generators and the iterator protocol, but no fluent, chainable API for this pattern. So I built one.

What Makes Iterflow Different?

There are other iterator libraries for JavaScript (iter-tools, iterare, lazy.js). Iterflow focuses on:

1. Built-in Statistical Operations

const data = iter(measurements)
  .filter(m => m.valid)
  .map(m => m.value);

// Each statistical method consumes the iterator, so use separately:
const avg = iter(measurements).filter(m => m.valid).map(m => m.value).mean();
const spread = iter(measurements).filter(m => m.valid).map(m => m.value).variance();

// Or collect once if you need multiple stats:
const values = data.toArray();
const stats = {
  avg: iter(values).mean(),
  spread: iter(values).variance(),
  range: [iter(values).min(), iter(values).max()]
};

No more importing separate stats libraries. Available: sum(), mean(), median(), variance(), min(), max().

2. Windowing and Chunking

const prices = [100, 102, 101, 105, 107, 110];

// Rolling average with sliding windows
const rolling = iter(prices)
  .window(3)
  .map(w => iter(w).mean())
  .toArray();  // [101, 102.67, 104.33, 107.33]

// Process in fixed-size batches
iter(records)
  .chunk(100)
  .forEach(batch => processBatch(batch));

3. TypeScript-First Design

Full type inference throughout the chain:

const numbers = iter([1, 2, 3, 4, 5])
  .filter(n => n > 2)    // Iterflow<number>
  .map(n => n.toString()) // Iterflow<string>
  .toArray();             // string[]

4. Works with Any Iterable

Arrays, Sets, Maps, generators, your custom iterables. All work seamlessly:

// Generators for infinite sequences
function* fibonacci() {
  let [a, b] = [0, 1];
  while (true) {
    yield a;
    [a, b] = [b, a + b];
  }
}

const first20Fibs = iter(fibonacci())
  .take(20)
  .toArray();

5. Zero Dependencies, Tiny Bundle

The entire library is 2.2KB gzipped (9KB uncompressed). Zero runtime dependencies.

Example: Processing Large Datasets

Generator functions make this powerful. Here's processing millions of log entries without loading them all into memory:

import { iter } from '@mathscapes/iterflow';

// Simulate reading from a huge log file (generator = lazy)
// Note: Current version supports sync iterators; async support coming soon
function* readLogLines() {
  // In production: stream from file, database cursor, etc.
  for (let i = 1; i <= 10_000_000; i++) {
    yield `${new Date().toISOString()} [${i % 100 === 0 ? 'ERROR' : 'INFO'}] Response time: ${Math.random() * 1000}ms`;
  }
}

const avgErrorTime = iter(readLogLines())
  .filter(line => line.includes('ERROR'))
  .map(line => {
    const match = line.match(/Response time: ([\d.]+)ms/);
    return match ? parseFloat(match[1]) : 0;
  })
  .mean();

console.log(`Average error response time: ${avgErrorTime.toFixed(2)}ms`);
// Only processes ERROR lines - most lines skipped entirely

The generator yields one line at a time. Iterflow only materializes matching entries. Constant memory usage regardless of dataset size.

When Should You Use Iterflow?

Use Iterflow when:

Processing large datasets that don't fit in memory
Building ETL pipelines with filter/map/reduce chains
Need statistics on data streams (mean, median, variance)
Working with generators or infinite sequences
Want Python itertools-like patterns in TypeScript

Skip it when:

Working with small arrays (<1000 items)
You need the full array materialized anyway

Get Started

npm install @mathscapes/iterflow

import { iter } from '@mathscapes/iterflow';

// That's it. Start chaining.
const result = iter([1, 2, 3, 4, 5])
  .filter(n => n % 2 === 0)
  .map(n => n * 10)
  .sum();

Status

Released v1.0.0-rc2 on January 4, 2026. Zero dependencies, 2.2KB gzipped, 128 tests passing.

Looking for feedback from developers working with large datasets, ETL pipelines, or functional patterns in TypeScript.

GitHub: mathscapes/iterflow
Issues/feedback: GitHub Issues