DEV Community: Bill Tu

When the Patient Dies on the Table: Handling Target Process Exit During Profiling

Bill Tu — Thu, 09 Apr 2026 14:03:43 +0000

You're profiling a Node.js process that's been crashing intermittently. You attach loop-detective, start a 60-second capture, and wait. Thirty seconds in, the process crashes again.

What should happen next?

Before v1.7.0, the answer was: nothing good. loop-detective would hang waiting for a CDP response that would never come, or exit with a cryptic WebSocket error. Any lag events and slow I/O data collected during those first 30 seconds? Gone.

This is the story of how we made loop-detective handle the worst-case scenario — the target process dying mid-profiling — and why the solution required changes across every layer of the tool.

The Failure Mode

To understand the problem, you need to know how profiling works internally:

1. Connect to inspector WebSocket
2. Send Profiler.start
3. Sleep for <duration> seconds     ← target can exit here
4. Send Profiler.stop               ← this never gets a response
5. Analyze and report

Step 3 is a setTimeout wrapped in a Promise. Step 4 is a CDP command sent over the WebSocket. If the target process exits during step 3, the WebSocket closes. When step 4 tries to send Profiler.stop, it throws "Inspector not connected." The error bubbles up, and the tool exits without reporting anything useful.

But the real loss is the data. During those 30 seconds before the crash, the lag detector was running. The I/O tracker was recording slow HTTP calls. That data exists in memory — it just never gets reported because the profiling pipeline assumes it will complete normally.

The Fix: Three Layers

Layer 1: Detect the Exit Immediately

The WebSocket close event fires when the target process exits. Previously, this just emitted a disconnected event. Now it also rejects all pending CDP callbacks and emits a targetExit event:

// inspector.js
this.ws.on('close', () => {
  // Reject all pending callbacks — target is gone
  for (const { reject, timer } of this._callbacks.values()) {
    clearTimeout(timer);
    try { reject(new Error('Target process exited')); } catch {}
  }
  this._callbacks.clear();
  this.emit('disconnected');
});

This is critical. Without rejecting pending callbacks, any in-flight CDP command (like Profiler.stop) would hang until its 30-second timeout. That's 30 seconds of the user staring at a frozen terminal.

Layer 2: Cancel the Sleep

The profiling duration is implemented as a sleep:

await this._sleep(duration);

If the target exits 10 seconds into a 60-second profile, we don't want to wait the remaining 50 seconds. The sleep needs to be cancellable.

_sleep(ms) {
  return new Promise((resolve, reject) => {
    this._sleepReject = reject;
    const timer = setTimeout(() => {
      this._sleepReject = null;
      resolve();
    }, ms);
    if (timer.unref) timer.unref();
  });
}

When the disconnected event fires, the Detective cancels the sleep:

this.inspector.on('disconnected', () => {
  if (!this._stopping) {
    this._targetExited = true;
    this.emit('targetExit', { pid, message: `Target process (PID ${pid}) exited during profiling` });

    // Cancel the sleep so _captureProfile can finish
    if (this._sleepReject) {
      this._sleepReject(new Error('Target process exited'));
      this._sleepReject = null;
    }
  }
});

Layer 3: Salvage What You Can

_captureProfile now handles the case where the sleep is cancelled and the profiler can't be stopped:

async _captureProfile(duration) {
  await this.inspector.send('Profiler.enable');
  await this.inspector.send('Profiler.setSamplingInterval', { interval: 100 });
  await this.inspector.send('Profiler.start');

  try {
    await this._sleep(duration);
  } catch {
    // Sleep was cancelled (target exited) — try to stop the profiler anyway
  }

  try {
    const { profile } = await this.inspector.send('Profiler.stop');
    await this.inspector.send('Profiler.disable');
    return profile;
  } catch {
    // Target exited — no profile data available
    return null;
  }
}

The callers check for null:

const profile = await this._captureProfile(this.config.duration);
if (profile) {
  const analysis = this.analyzer.analyzeProfile(profile);
  this.emit('profile', analysis, profile);
}
// If null, targetExit event was already emitted

The key insight: even though we can't get the CPU profile (it's inside the dead process), the lag events and slow I/O events were already emitted in real-time during the profiling period. The user has already seen them scroll by in the terminal. They're not lost.

What the User Sees

Before v1.7.0:

✔ Connected to Node.js process
  Profiling for 60s with 50ms lag threshold...

⚠ Event loop lag: 245ms at 2025-03-15T14:23:45.123Z
🌐 Slow HTTP: 1820ms GET api.example.com/users → 200

<hangs for 30 seconds, then>

✖ Error: Inspector not connected

After v1.7.0:

✔ Connected to Node.js process
  Profiling for 60s with 50ms lag threshold...

⚠ Event loop lag: 245ms at 2025-03-15T14:23:45.123Z
🌐 Slow HTTP: 1820ms GET api.example.com/users → 200

  ✖ Target process (PID 12345) exited during profiling
    Any lag or I/O events collected before the exit are shown above.

✔ Disconnected cleanly

No hang. Clear message. The partial data (lag event + slow HTTP call) is visible above the exit message.

Exit Codes

We introduced a three-code system:

Code	Meaning
0	Success — profiling completed normally
1	Error — connection failed, invalid arguments, etc.
2	Target exited — process died during profiling

This matters for automation. A monitoring script can distinguish between "profiling worked" (0), "loop-detective itself had a problem" (1), and "the target crashed" (2):

loop-detective 12345 -d 60
EXIT_CODE=$?

if [ $EXIT_CODE -eq 2 ]; then
  echo "Target process crashed during profiling — check partial output"
  # Trigger incident response
elif [ $EXIT_CODE -eq 1 ]; then
  echo "loop-detective failed to connect"
elif [ $EXIT_CODE -eq 0 ]; then
  echo "Profiling completed successfully"
fi

The Irony of Profiling Crashing Processes

There's an irony here: the processes most likely to exit during profiling are the ones you most need to profile. They're crashing. They're running out of memory. They're hitting unhandled exceptions. You attached loop-detective precisely because something is wrong.

This means the "target exits during profiling" scenario isn't an edge case — it's a primary use case. If your tool can't handle it gracefully, it fails exactly when it's needed most.

The partial data is often enough. If you see three slow HTTP calls to the same endpoint right before the crash, that's a strong signal. If you see event loop lag spiking to 2 seconds, that tells you the process was CPU-bound before it died. You don't always need the full CPU profile to understand what happened.

Watch Mode Behavior

In --watch mode, a target exit stops the watch loop instead of starting a new cycle:

if (this._running && !this._targetExited) {
  setTimeout(runCycle, 1000);
}

There's no point retrying — the process is gone. The tool reports what it has and exits cleanly.

Programmatic API

For tool builders integrating loop-detective:

const { Detective } = require('node-loop-detective');

const detective = new Detective({ pid: 12345, duration: 60000 });

// Collect events as they arrive
const events = { lags: [], slowIO: [] };
detective.on('lag', (data) => events.lags.push(data));
detective.on('slowIO', (data) => events.slowIO.push(data));

// Handle target exit
detective.on('targetExit', (data) => {
  console.log(data.message);
  console.log(`Collected ${events.lags.length} lag events and ${events.slowIO.length} slow I/O events before exit`);
  // Send partial data to your monitoring system
  sendToMonitoring({ partial: true, ...events });
});

detective.on('profile', (analysis) => {
  // Full profile available — target didn't crash
  sendToMonitoring({ partial: false, analysis, ...events });
});

await detective.start();

The Broader Lesson

Diagnostic tools need to be more resilient than the systems they diagnose. If your profiler crashes when the target crashes, you've lost the most valuable debugging moment. Every diagnostic tool that attaches to external processes should answer three questions:

What happens when the target exits? Don't hang. Don't lose data. Report what you have.
How does the user know what happened? Clear messages, distinct exit codes, structured output.
What data survives? Real-time events (lag, I/O) survive because they were already emitted. Batch results (CPU profile) may be lost. Design accordingly.

The fix in v1.7.0 is about 60 lines of code across three files. But it transforms the tool from one that fails at the worst possible moment to one that handles it gracefully.

npm install -g node-loop-detective@1.7.0

Source: github.com/iwtxokhtd83/node-loop-detective

Not All Requests Are Equal: Adding Variable Cost to a Node.js Rate Limiter

Bill Tu — Thu, 09 Apr 2026 13:51:48 +0000

Rate limiters typically treat every request the same. One request, one token. But in the real world, a request that fetches a single user profile and a request that exports 10,000 records are not the same thing. Charging them the same rate limit token is like charging the same toll for a bicycle and a semi-truck.

We just shipped variable cost support in node-rate-limiter-pro v1.2.0. This post covers why it matters, how we implemented it across two algorithms and two storage backends, and the one subtle bug that almost shipped with it.

The Problem

Consider a batch API endpoint:

POST /api/users/export
Body: { "ids": ["u1", "u2", ..., "u500"] }

With a flat rate limiter set to 100 requests per minute, a client can export 50,000 users per minute by sending 100 requests of 500 IDs each. But a client making simple single-user lookups gets the same 100 requests per minute — 100 users total.

That's a 500x difference in actual resource consumption for the same rate limit.

Variable cost fixes this. Instead of consume('user:123') always deducting 1 token, you can now write:

// Each ID in the batch costs one token
const result = await limiter.consume('user:123', ids.length);

The batch export of 500 IDs now costs 500 tokens. The single lookup costs 1. The rate limiter finally reflects reality.

The API

The change is minimal — one optional parameter:

// Before (still works, backward compatible)
await limiter.consume('user:123');

// After: consume 5 tokens
await limiter.consume('user:123', 5);

For Express middleware, we added a costFn option:

app.use(limiter.middleware({
  costFn: (req) => req.body?.items?.length || 1,
}));

Implementation: Four Layers Deep

The cost parameter touches every layer of the stack: the public API, two algorithm implementations, two store backends, and two Redis Lua scripts. Here's how each one changed.

Token Bucket

The Token Bucket was the easy one. The algorithm already works with fractional tokens — we just replaced the hardcoded 1 with cost:

// Before
if (bucket.tokens >= 1) {
  bucket.tokens -= 1;
  // ...
}
const retryAfter = Math.ceil(
  ((1 - bucket.tokens) / this.maxTokens) * this.refillIntervalMs
);

// After
if (bucket.tokens >= cost) {
  bucket.tokens -= cost;
  // ...
}
const retryAfter = Math.ceil(
  ((cost - bucket.tokens) / this.maxTokens) * this.refillIntervalMs
);

The retryAfter calculation scales naturally. If you need 5 tokens and only have 2, you need to wait for 3 tokens to refill. The math is (cost - currentTokens) / refillRate.

Sliding Window — Where It Got Interesting

Our sliding window uses a weighted counter approach. Instead of storing every timestamp, it keeps two sub-window counters and calculates a weighted estimate:

estimatedCount = prevWindowCount × weight + currentWindowCount

where weight decays linearly from 1 to 0 as time progresses through the current window.

The original admission check was:

if (estimatedCount < this.maxRequests) {
  state.currCount++;
  // allowed
}

The naive variable-cost version would be:

if (estimatedCount + cost <= this.maxRequests) {
  state.currCount += cost;
  // allowed
}

This is where the bug hid. More on that in a moment. The correct implementation:

const used = Math.floor(estimatedCount);
const remaining = Math.max(0, this.maxRequests - used);

if (cost <= remaining) {
  state.currCount += cost;
  // allowed
}

We floor the estimated count to get integer remaining capacity, then check if the cost fits. The remaining field in the response now accurately reflects how many tokens are available — even when the request is denied:

// Denied response now shows actual remaining capacity
return {
  allowed: false,
  remaining,  // e.g., 2 tokens left, but you asked for 5
  // ...
};

This is a meaningful improvement over the old code, which always returned remaining: 0 on denial regardless of actual capacity.

Redis Lua Scripts

Both Redis Lua scripts needed the same treatment. The cost arrives as an additional ARGV parameter.

Token Bucket in Lua — straightforward substitution:

local cost = tonumber(ARGV[4])

if tokens >= cost then
  tokens = tokens - cost
  -- ...
  return {1, math.floor(tokens), 0}
end

local retryAfter = math.ceil(((cost - tokens) / maxTokens) * refillInterval)
return {0, math.floor(tokens), retryAfter}

Sliding Window in Lua — the ZSET approach needs to add cost members:

local cost = tonumber(ARGV[4])

if count + cost <= limit then
  for i = 1, cost do
    local seq = redis.call('INCR', key .. ':seq')
    redis.call('ZADD', key, now, now .. '-' .. seq)
  end
  return {1, limit - count - cost, 0}
end

return {0, limit - count, retryAfter}

The loop adds cost individual entries to the sorted set. Each entry gets a unique member via the atomic INCR counter (which we introduced in v1.1.0 to fix a collision bug — see our previous post). This keeps the ZSET accurate for future window calculations.

Express Middleware

The middleware layer adds a costFn option that mirrors the existing keyFn pattern:

middleware(options?: {
  keyFn?: (req: any) => string;
  costFn?: (req: any) => number;
  onLimited?: (req: any, res: any) => void;
}) {
  const costFn = options?.costFn;

  return async (req, res, next) => {
    const key = keyFn(req);
    const cost = costFn ? costFn(req) : 1;
    const result = await this.consume(key, cost);
    // ...
  };
}

The Floating Point Bug That Almost Shipped

Remember the sliding window admission check? Here's the bug we caught in testing.

Setup: limit = 3, window = 1000ms. Consume 3 tokens, wait 1100ms (just past the window boundary), then try to consume 1 more.

After 1100ms, the algorithm advances the sub-windows:

prevCount = 3, currCount = 0
elapsedInCurrent ≈ 100ms
weight = 1 - 100/1000 = 0.9
estimatedCount = 3 × 0.9 + 0 = 2.7

The old check estimatedCount < maxRequests → 2.7 < 3 → true ✅

The naive variable-cost check estimatedCount + cost <= maxRequests → 2.7 + 1 = 3.7 <= 3 → false ❌

Same scenario, different result. The request that should be allowed gets blocked.

The root cause: the weighted counter produces fractional values. The old code compared a float against an integer with <, which implicitly gave ~0.99 tokens of headroom. The new code with <= eliminated that headroom.

The fix: floor the estimate before comparing, converting the float back to integer capacity:

const used = Math.floor(estimatedCount);     // floor(2.7) = 2
const remaining = Math.max(0, limit - used); // 3 - 2 = 1
if (cost <= remaining)                       // 1 <= 1 → true ✅

This preserves the original behavior for cost = 1 while correctly handling higher costs. If estimatedCount were 2.3 and cost were 2, we'd get remaining = 1, and 2 <= 1 → false — correctly blocking a request that would exceed the limit.

Real-World Usage Patterns

Here are some patterns we've seen (or expect to see) with variable cost:

Batch APIs:

app.post('/api/batch', limiter.middleware({
  costFn: (req) => req.body?.operations?.length || 1,
}));

File uploads by size:

app.post('/upload', limiter.middleware({
  costFn: (req) => Math.ceil(Number(req.headers['content-length'] || 1) / 1_000_000),
}));

Tiered pricing (premium users get cheaper requests):

app.use(limiter.middleware({
  costFn: (req) => req.user?.plan === 'premium' ? 1 : 3,
}));

GraphQL complexity:

app.use('/graphql', limiter.middleware({
  costFn: (req) => calculateQueryComplexity(req.body.query),
}));

Design Decisions

A few choices we made along the way:

Why cost as a parameter, not a constructor option? Because cost varies per request, not per limiter. A single limiter instance might handle requests with different costs. Making it a per-call parameter is the only design that works.

Why default to 1? Backward compatibility. Every existing consume('key') call continues to work without changes. The cost parameter is purely additive.

Why floor the sliding window estimate? The weighted counter is an approximation. Flooring converts it to a conservative integer count, which is the right behavior for admission control — when in doubt, allow rather than block. This matches the original behavior and avoids surprising users with stricter-than-expected limits.

Why not validate cost > 0? We considered it, but decided against adding runtime validation on the hot path. If you pass cost = 0, you get a free peek at the current state (which is actually useful). If you pass a negative number, you're doing something wrong, but we're not going to add a branch to every consume() call to catch it.

What's Next

Variable cost was the most requested feature in our issue tracker. With it shipped, we're looking at:

get(key) — peek at rate limit state without consuming (#6)
Fixed Window Counter algorithm (#7)
IETF standard rate limit headers (#8)

The full implementation is in v1.2.0. MIT licensed, zero dependencies for in-memory mode, optional Redis for distributed deployments.

Writing 89 Tests for a Quantitative Trading Framework: Strategy and Trade-offs

Bill Tu — Thu, 09 Apr 2026 13:25:06 +0000

Adding a test suite to an existing codebase is a different exercise than writing tests alongside new code. You're reverse-engineering the implicit contracts, discovering which behaviors are intentional and which are accidental, and deciding what's worth testing versus what's noise. This article covers how we built the test suite for QuantFlow, an open-source quantitative trading framework in Python, and the decisions behind each layer of tests.

The Testing Problem in Quant Systems

Quantitative trading code has a testing problem that most software doesn't: the outputs are floating-point numbers derived from financial time series, and "correct" is often a matter of degree. An SMA of [1, 2, 3, 4, 5] with period 5 should be exactly 3.0 — that's easy. But what should the RSI of a 50-bar synthetic price series be? You can't hardcode an expected value without coupling the test to the random seed and the exact implementation.

This creates a tension between two testing philosophies:

Test against known reference values (precise but brittle)
Test invariants and contracts (robust but less specific)

We use both, choosing based on the indicator.

Test Architecture

The suite is organized into five files, each targeting a different layer:

tests/
├── conftest.py          # Shared fixtures
├── test_indicators.py   # 46 tests — all 16 indicators
├── test_risk.py         # 12 tests — risk manager rules
├── test_portfolio.py    # 16 tests — execution and position tracking
├── test_strategies.py   # 8 tests  — built-in strategy signals
└── test_engine.py       # 9 tests  — end-to-end backtest integration

The dependency flows downward: indicators have no dependencies, risk and portfolio depend on core models, strategies depend on indicators, and the engine depends on everything. Tests follow the same order — if indicator tests fail, strategy and engine tests are meaningless.

Fixtures: Deterministic Randomness

The conftest.py provides four shared fixtures that generate synthetic market data with a fixed random seed:

@pytest.fixture
def sample_prices() -> np.ndarray:
    np.random.seed(42)
    returns = np.random.normal(0.001, 0.02, 50)
    prices = 100.0 * np.cumprod(1 + returns)
    return prices

The seed 42 makes tests deterministic — same data every run. The parameters (0.1% daily drift, 2% daily volatility) produce realistic-looking price series without being tied to any real market data. This matters because tests that depend on real market data break when the data source changes, and they can't be run offline.

The sample_ohlcv fixture generates correlated OHLCV data where high > close > low holds approximately (with small random noise). This is important for indicators like ATR and Stochastic that use all four price fields — feeding them random uncorrelated data would test the math but not the real-world behavior.

Indicator Tests: Three Categories

The 46 indicator tests fall into three categories.

Warm-up Guards

Every indicator should return None when given insufficient data. This is the most important contract — it prevents strategies from acting on garbage values during the first few bars of a backtest.

def test_insufficient_data_returns_none(self):
    sma = SMA(period=20)
    assert sma.calculate(np.array([1.0, 2.0, 3.0])) is None

We test this for every indicator. It's repetitive, but it catches a common class of bugs: an indicator that returns 0.0 or NaN instead of None when it doesn't have enough data.

Known-Value Tests

For indicators with simple formulas, we test against hand-calculated values:

def test_known_value(self):
    prices = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
    sma = SMA(period=5)
    assert sma.calculate(prices) == pytest.approx(3.0)

def test_known_value(self):
    prices = np.array([100.0] * 13)
    prices[-1] = 110.0
    roc = ROC(period=12)
    assert roc.calculate(prices) == pytest.approx(10.0)

These are the strongest tests — they verify the actual computation. But they're only practical for indicators where you can compute the expected value by hand. For complex indicators like ADX or Ichimoku, hand-calculation is error-prone, so we use invariant tests instead.

Invariant Tests

For indicators where exact values are hard to predict, we test mathematical invariants that must always hold:

def test_range_0_to_100(self, sample_prices):
    rsi = RSI(period=14)
    val = rsi.calculate(sample_prices)
    assert 0.0 <= val <= 100.0

def test_all_gains_returns_100(self):
    prices = np.arange(1.0, 20.0)  # monotonically increasing
    rsi = RSI(period=14)
    assert rsi.calculate(prices) == pytest.approx(100.0)

def test_all_losses_returns_0(self):
    prices = np.arange(20.0, 1.0, -1.0)  # monotonically decreasing
    assert rsi.calculate(prices) == pytest.approx(0.0, abs=0.01)

RSI must be between 0 and 100. All-gain series must produce RSI of 100. All-loss series must produce RSI near 0. These invariants hold regardless of the implementation details.

For Bollinger Bands: upper > middle > lower (when prices have non-zero variance). For Stochastic: %K between 0 and 100. For Williams %R: between -100 and 0. These are mathematical properties of the formulas, not implementation details.

Consistency Tests

One particularly valuable test pattern verifies that calculate() and series() agree:

def test_calculate_matches_series_last(self, sample_prices):
    ema = EMA(period=12)
    calc_val = ema.calculate(sample_prices)
    series_val = ema.series(sample_prices)[-1]
    assert calc_val == pytest.approx(series_val, rel=1e-10)

After the v0.2.0 optimization where EMA.calculate() was rewritten to avoid building the full series, this test ensures the optimized path produces the same result as the reference implementation. It's a regression test disguised as a unit test.

The NaN Leakage Test

One test specifically targets a bug we fixed in v0.2.0:

def test_no_nan_leakage(self):
    prices = np.arange(1.0, 40.0)
    macd = MACD()
    m, s, h = macd.calculate(prices)
    if m is not None:
        assert not np.isnan(m)
        assert not np.isnan(s)
        assert not np.isnan(h)

The contract is: calculate() returns either valid floats or None. Never NaN. This test exists because NaN is insidious — it passes through arithmetic silently and makes comparisons return False, causing strategies to silently skip signals without any error.

The Stochastic Alignment Test

Another test targets the %D alignment bug from v0.2.0:

def test_d_alignment(self, sample_ohlcv):
    stoch = Stochastic(k_period=5, d_period=3)
    k_series, d_series = stoch.series(...)
    first_k = np.where(~np.isnan(k_series))[0][0]
    first_d = np.where(~np.isnan(d_series))[0][0]
    assert first_d == first_k + stoch.d_period - 1

%D is a moving average of %K, so it must start exactly d_period - 1 bars after %K begins. This is a structural invariant — if the alignment is wrong, the %D values are computed from the wrong %K windows.

Risk Manager Tests: State Machine Behavior

The risk manager is a stateful component — once the drawdown circuit breaker triggers, it stays triggered until explicitly reset. The tests model this as a state machine:

def test_drawdown_halts_trading(self):
    signal = Signal(SignalType.BUY, size=10)
    result = self.rm.check(signal, 100.0, 75000.0, 100000.0, 0)
    assert result.type == SignalType.HOLD

def test_drawdown_halt_is_sticky(self):
    self.rm.check(Signal(SignalType.BUY, size=10), 100.0, 75000.0, 100000.0, 0)
    # Even if equity recovers, still halted
    result = self.rm.check(Signal(SignalType.BUY, size=10), 100.0, 95000.0, 100000.0, 0)
    assert result.type == SignalType.HOLD

def test_drawdown_reset(self):
    self.rm.check(Signal(SignalType.BUY, size=10), 100.0, 75000.0, 100000.0, 0)
    self.rm.reset()
    result = self.rm.check(Signal(SignalType.BUY, size=10), 100.0, 95000.0, 100000.0, 0)
    assert result.type == SignalType.BUY

Three tests, three states: normal → halted → reset. The "sticky" test is critical — it verifies that the halt persists even when equity recovers. Without this test, a bug that resets the halt flag on every call would go undetected.

The position sizing test uses exact arithmetic:

def test_position_sizing_caps_size(self):
    # equity=100000, max_pct=10% → max $10000 → at $100/share → max 100 shares
    signal = Signal(SignalType.BUY, size=500)
    result = self.rm.check(signal, 100.0, 100000.0, 100000.0, 0)
    assert result.size <= 100

The comment documents the math. This is intentional — risk management tests should be readable by someone who isn't a developer, because the rules they encode are business logic that a portfolio manager or risk officer should be able to verify.

Portfolio Tests: Isolating Execution Mechanics

Portfolio tests use zero slippage to make assertions predictable:

def setup_method(self):
    self.portfolio = Portfolio(
        initial_capital=100000.0,
        commission_rate=0.001,
        slippage_rate=0.0,  # zero slippage for predictable tests
    )

This is a deliberate choice. Slippage adds randomness to fill prices, which makes exact assertions impossible. By setting it to zero, we can test the execution logic in isolation. Slippage behavior is tested separately at the engine level.

The most important portfolio test verifies the v0.2.0 fix for equity calculation:

def test_equity_reflects_current_price(self):
    signal = Signal(SignalType.BUY, size=100)
    self.portfolio.execute_signal(signal, "TEST", 100.0, self.ts)
    self.portfolio.update_market_prices("TEST", 120.0)
    assert self.portfolio.equity > self.portfolio.cash + 100 * 100.0

If equity used entry price (the old bug), equity would equal cash + 100 * 100. With the fix, it equals cash + 100 * 120, which is strictly greater. The test doesn't check the exact value — it checks the relationship, which is more robust.

The short margin test verifies the v0.2.0 safety fix:

def test_short_margin_check(self):
    self.portfolio.short_margin_rate = 1.0
    signal = Signal(SignalType.SELL, size=2000)  # $200k margin > $100k cash
    result = self.portfolio.execute_signal(signal, "TEST", 100.0, self.ts)
    assert result is None

This test exists because the absence of a margin check was a real bug. The test documents the expected behavior: shorts that exceed available margin are rejected.

Strategy Tests: Signal Generation

Strategy tests face a unique challenge: strategies are stateful (they track previous indicator values for crossover detection), and their outputs depend on the full price history. Testing them requires constructing price series that produce known indicator states.

def test_buy_on_oversold(self):
    strategy = RSIMeanReversion(period=14, oversold=30, size=100)
    portfolio = Portfolio()
    prices = np.linspace(200, 100, 50)  # strong downtrend
    bar = make_bar(100.0, prices)
    signal = strategy.on_bar(bar, portfolio)
    assert signal.type == SignalType.BUY

A monotonically decreasing price series guarantees RSI will be very low (near 0), which is well below the oversold threshold of 30. We don't need to know the exact RSI value — we just need to know it's below 30, which is guaranteed by the price pattern.

The SMA crossover test is more involved because it requires a crossover event:

def test_generates_buy_on_golden_cross(self):
    strategy = SMACrossover(fast_period=3, slow_period=5, size=100)
    portfolio = Portfolio()
    prices_down = np.linspace(120, 100, 20)
    prices_up = np.linspace(100, 130, 10)
    prices = np.concatenate([prices_down, prices_up])
    signals = []
    for i in range(len(prices)):
        bar = make_bar(prices[i], prices[:i + 1])
        signals.append(strategy.on_bar(bar, portfolio))
    buy_signals = [s for s in signals if s.type == SignalType.BUY]
    assert len(buy_signals) > 0

The price series goes down then sharply up. At some point during the upturn, the fast SMA (3-period) will cross above the slow SMA (5-period). We don't assert exactly when — just that it happens at least once. This makes the test resilient to minor changes in the crossover detection logic.

Engine Tests: End-to-End Integration

Engine tests verify that all components work together. They use an InMemoryDataFeed to avoid file I/O:

class InMemoryDataFeed(DataFeed):
    def __init__(self, df: pd.DataFrame):
        self._df = df

    def load(self) -> pd.DataFrame:
        return self._df

The most valuable engine test verifies that commission has a measurable effect:

def test_commission_reduces_equity(self, data_feed):
    r1 = BacktestEngine(..., commission=0.0, ...).run()
    r2 = BacktestEngine(..., commission=0.01, ...).run()
    assert r2.equity_curve[-1] <= r1.equity_curve[-1]

This is an invariant test at the system level: higher commission should never increase final equity. It doesn't test the exact commission calculation — that's covered in portfolio tests — but it verifies that commission flows through the entire pipeline correctly.

Edge case tests verify the system doesn't crash on degenerate inputs:

def test_single_bar_no_crash(self):
    df = pd.DataFrame({...}, index=pd.DatetimeIndex([datetime(2024, 1, 1)]))
    result = BacktestEngine(data_feed=InMemoryDataFeed(df), ...).run()
    assert len(result.equity_curve) == 1

def test_zero_volume_no_crash(self):
    df = pd.DataFrame({..., "volume": np.zeros(n), ...})
    result = BacktestEngine(data_feed=InMemoryDataFeed(df), ...).run()
    assert result.total_return == pytest.approx(0.0)

These tests exist because real market data contains edge cases: single-bar datasets from API errors, zero-volume bars from illiquid instruments, gaps in timestamps. The system should handle all of these without raising exceptions.

What We Chose Not to Test

A few deliberate omissions:

We don't test the YahooDataFeed because it makes HTTP requests to an external API. Testing it would require either mocking the HTTP layer (which tests the mock, not the code) or hitting the real API (which is slow, flaky, and rate-limited). The CSVDataFeed is tested indirectly through the engine tests.

We don't test the LiveEngine because it runs an infinite loop with time.sleep(). Testing it properly would require threading, timeouts, and mock brokers — significant infrastructure for a component that's architecturally identical to the backtest engine but with a different data source.

We don't test the plot() method on PerformanceReport because visual output is best verified by humans. We could assert that it doesn't raise an exception, but that's a weak test that adds maintenance burden.

Running the Suite

pip install pytest
python -m pytest tests/ -v

============================= 89 passed in 4.87s ==============================

All 89 tests run in under 5 seconds. No external dependencies, no network calls, no file I/O (except the fixtures that generate in-memory DataFrames). This matters for CI — fast tests get run; slow tests get skipped.

The test suite is part of QuantFlow v0.3.0. Contributions and additional test cases are welcome.

Who Should Own the Order ID? Moving ID Generation Inside the Matching Engine

Bill Tu — Thu, 09 Apr 2026 13:04:07 +0000

Order IDs seem trivial. Every order needs one, they need to be unique, and they show up in every trade record. In the first version of MatchEngine, the caller provided the ID as a string. It worked. Then it didn't.

This article explains why we moved ID generation inside the engine, the design choices behind the implementation, and the three new API methods that replaced the old approach.

The Problem With Caller-Provided IDs

The original API looked like this:

order := model.NewLimitOrder("my-order-1", model.Buy, price, quantity)
trades, err := e.SubmitOrder("BTC/USD", order)

The caller chose the ID. The engine accepted it. This created three problems:

1. Uniqueness is the caller's problem. The engine added duplicate detection in v0.2.0, but that just turned a silent corruption into an error. The caller still had to generate unique IDs — and every caller had to solve the same problem independently. In a system with multiple order sources (REST API, WebSocket, FIX gateway), coordinating unique IDs across all of them is a real engineering challenge.

2. No ordering guarantee. Caller-provided IDs are arbitrary strings. There is no way to determine which order came first by looking at the IDs. For debugging, auditing, and replay, monotonically increasing IDs are invaluable. They give you a total ordering of events without consulting timestamps.

3. The API is noisy. Every call site has to construct an Order struct with an ID, a side, a type, a price, a quantity, and a timestamp. The ID and timestamp are boilerplate — the caller doesn't care about them. They care about "sell 5 BTC at 50,000."

The Design: Atomic Counter

We chose the simplest possible ID generator: an atomic uint64 counter.

type Engine struct {
    nextID   uint64  // atomic counter
    idPrefix string  // optional prefix
    // ...
}

func (e *Engine) generateID() string {
    id := atomic.AddUint64(&e.nextID, 1)
    return e.idPrefix + strconv.FormatUint(id, 10)
}

atomic.AddUint64 increments the counter and returns the new value in a single atomic operation. No mutex needed. No allocation. The result is converted to a string with strconv.FormatUint, which is the fastest integer-to-string conversion in Go's standard library.

Why Not UUID?

UUIDs (google/uuid) are globally unique, which sounds appealing. But they have drawbacks for this use case:

Property	Atomic Counter	UUID v4
Uniqueness	Unique within engine instance	Globally unique
Ordering	Monotonically increasing	Random (no ordering)
Size	1-20 chars (`"1"` to `"18446744073709551615"`)	36 chars (`"550e8400-e29b-41d4-a716-446655440000"`)
Speed	~1ns (atomic add + format)	~200ns (crypto/rand + format)
Readability	`"ORD-42"`	`"550e8400-e29b-41d4-a716-446655440000"`

For a single-instance matching engine, the atomic counter is strictly better. It is faster, smaller, ordered, and human-readable. If the engine were distributed across multiple nodes, UUIDs or a distributed ID service (like Twitter's Snowflake) would be necessary. But that is a different problem for a different architecture.

Why Not Inside the Mutex?

The generateID method uses atomic.AddUint64, which is lock-free. It runs before the engine acquires its mutex in SubmitOrder. This is deliberate:

func (e *Engine) SubmitLimitOrder(symbol string, side model.Side, price, quantity decimal.Decimal) (string, []model.Trade, error) {
    id := e.generateID()  // lock-free, outside mutex
    order := model.NewLimitOrder(id, side, price, quantity)
    trades, err := e.SubmitOrder(symbol, order)  // acquires mutex internally
    return id, trades, err
}

If ID generation were inside the mutex, it would add contention for no reason. The atomic operation is already thread-safe on its own. Keeping it outside the mutex means the critical section is no longer than before.

The Prefix Option

IDs can be prefixed for readability or multi-engine disambiguation:

e := engine.New(engine.WithIDPrefix("ORD-"))
// Generates: "ORD-1", "ORD-2", "ORD-3", ...

e := engine.New(engine.WithIDPrefix("ENGINE-A-"))
// Generates: "ENGINE-A-1", "ENGINE-A-2", ...

The prefix is concatenated at generation time. No extra allocation — strconv.FormatUint returns a string, and + on two strings is a single allocation.

The New API

Three new methods replace the old pattern:

SubmitLimitOrder

func (e *Engine) SubmitLimitOrder(
    symbol string,
    side model.Side,
    price, quantity decimal.Decimal,
) (string, []model.Trade, error)

The caller provides only what they care about: symbol, side, price, quantity. The engine generates the ID and returns it.

id, trades, err := e.SubmitLimitOrder("BTC/USD", model.Sell, price, qty)
// id = "1" (or "ORD-1" with prefix)

SubmitMarketOrder

func (e *Engine) SubmitMarketOrder(
    symbol string,
    side model.Side,
    quantity decimal.Decimal,
) (string, []model.Trade, error)

Same pattern, no price parameter (market orders don't have one).

id, trades, err := e.SubmitMarketOrder("BTC/USD", model.Buy, qty)

SubmitRequest

For cases where the order type is determined at runtime (e.g., from a JSON payload), SubmitRequest accepts an OrderRequest struct:

type OrderRequest struct {
    Side     Side
    Type     OrderType
    Price    decimal.Decimal
    Quantity decimal.Decimal
}

No ID field. No timestamp. Just the intent.

req := model.NewLimitOrderRequest(model.Buy, price, qty)
id, trades, err := e.SubmitRequest("BTC/USD", req)

This is particularly useful when deserializing from external input:

// From a REST API handler
var req model.OrderRequest
json.NewDecoder(r.Body).Decode(&req)
id, trades, err := e.SubmitRequest(symbol, req)

The caller never touches an order ID. The engine owns the lifecycle.

Backward Compatibility

The original SubmitOrder method is unchanged:

func (e *Engine) SubmitOrder(symbol string, order *model.Order) ([]model.Trade, error)

It still accepts caller-provided IDs, still checks for duplicates, and still works exactly as before. The new methods are built on top of it — they generate an ID, construct an Order, and call SubmitOrder internally.

This means existing code continues to work without modification. Migration is opt-in.

How the Pieces Fit Together

The call chain for SubmitLimitOrder:

SubmitLimitOrder("BTC/USD", Sell, 50000, 1)
    │
    ├── generateID()              ← atomic, lock-free
    │   └── "ORD-7"
    │
    ├── NewLimitOrder("ORD-7", Sell, 50000, 1)
    │   └── Order{ID: "ORD-7", ...}
    │
    └── SubmitOrder("BTC/USD", order)
        ├── normalize symbol      ← "BTC/USD"
        ├── acquire mutex
        ├── validate symbol
        ├── check duplicate ID    ← "ORD-7" not in index
        ├── match against book
        ├── add to book if unfilled
        ├── record trades
        └── release mutex

The ID is generated before the mutex is acquired. If SubmitOrder returns an error (invalid price, bad symbol, etc.), the ID is "wasted" — the counter has already incremented. This is fine. Gaps in the ID sequence are harmless, and avoiding them would require either rolling back the atomic counter (unsafe) or generating the ID inside the mutex (unnecessary contention).

Testing Concurrency

The most important property of the ID generator is uniqueness under concurrent access. We test this with 10 goroutines each submitting 20 orders simultaneously:

func TestIDsUniqueAcrossGoroutines(t *testing.T) {
    e := New()

    idCh := make(chan string, 200)
    var wg sync.WaitGroup
    for g := 0; g < 10; g++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for i := 0; i < 20; i++ {
                id, _, _ := e.SubmitLimitOrder("BTC", model.Sell, d("100"), d("1"))
                idCh <- id
            }
        }()
    }
    wg.Wait()
    close(idCh)

    seen := make(map[string]bool)
    for id := range idCh {
        if seen[id] {
            t.Errorf("duplicate ID generated: %s", id)
        }
        seen[id] = true
    }
    if len(seen) != 200 {
        t.Errorf("expected 200 unique IDs, got %d", len(seen))
    }
}

200 orders, 10 goroutines, zero duplicates. The atomic counter guarantees this without any locking.

We also verify monotonic ordering:

func TestIDsAreMonotonicallyIncreasing(t *testing.T) {
    e := New()

    var ids []string
    for i := 0; i < 10; i++ {
        id, _, _ := e.SubmitLimitOrder("BTC", model.Sell, d("100"), d("1"))
        ids = append(ids, id)
    }

    for i := 1; i < len(ids); i++ {
        if ids[i] <= ids[i-1] {
            t.Errorf("IDs not monotonically increasing: %s <= %s", ids[i], ids[i-1])
        }
    }
}

Note: monotonic ordering is guaranteed only within a single goroutine. Across goroutines, the IDs are unique but the observation order depends on scheduling. Goroutine A might generate ID 5 before goroutine B generates ID 4, but B might write to the channel first. The IDs themselves are still strictly ordered by their numeric value.

Before and After

Before (caller manages IDs)

e := engine.New()

// Caller must generate unique IDs
sellID := fmt.Sprintf("sell-%d", time.Now().UnixNano())
sell := model.NewLimitOrder(sellID, model.Sell, price, qty)
trades, err := e.SubmitOrder("BTC/USD", sell)

buyID := fmt.Sprintf("buy-%d", time.Now().UnixNano())
buy := model.NewLimitOrder(buyID, model.Buy, price, qty)
trades, err = e.SubmitOrder("BTC/USD", buy)

// Hope the IDs are unique...
e.CancelOrder("BTC/USD", sellID)

After (engine manages IDs)

e := engine.New(engine.WithIDPrefix("ORD-"))

sellID, _, _ := e.SubmitLimitOrder("BTC/USD", model.Sell, price, qty)
buyID, trades, _ := e.SubmitLimitOrder("BTC/USD", model.Buy, price, qty)

// IDs are guaranteed unique
e.CancelOrder("BTC/USD", sellID)

Less code, no uniqueness bugs, and the returned IDs are useful for logging, cancellation, and client responses.

Summary

Aspect	Before	After
ID source	Caller-provided string	Engine-generated atomic counter
Uniqueness	Caller's responsibility (error on duplicate)	Guaranteed by engine
Ordering	Arbitrary	Monotonically increasing
Thread safety	N/A (caller's problem)	Lock-free atomic operation
API surface	`SubmitOrder(symbol, *Order)`	`SubmitLimitOrder`, `SubmitMarketOrder`, `SubmitRequest`
Backward compat	N/A	`SubmitOrder` still works

The change is small — one new field on the engine, one three-line method, three thin wrapper methods, and one new OrderRequest type. But it shifts the responsibility for a correctness-critical property (ID uniqueness) from every caller to a single, tested, atomic implementation inside the engine. That is the right place for it.

GitHub: https://github.com/iwtxokhtd83/MatchEngine
Release: v0.4.0
Issue: #6 — Engine should generate unique order IDs internally

Less Is More: Why We Added a Flag to Disable Our Best Feature

Bill Tu — Thu, 09 Apr 2026 06:00:31 +0000

We spent three releases building async I/O tracking into node-loop-detective. HTTP request timing. DNS lookup monitoring. TCP connection tracking. Global fetch() support. dns.promises.lookup coverage. It's the feature that fills the blind spot every other Node.js profiler has.

Then we added a flag to turn it all off.

loop-detective 12345 --no-io

This isn't a contradiction. It's a design principle: diagnostic tools that attach to production processes must give operators full control over what gets injected. Here's why.

What --no-io Does

When you pass --no-io, loop-detective skips the entire _startAsyncIOTracking() phase. No monkey-patching of http.request. No wrapping of dns.lookup. No touching net.Socket.connect. No fetch() interception. None of it.

What still works:

Event loop lag detection (the lightweight setInterval probe)
CPU profiling (V8's built-in sampling profiler via CDP)
All analysis: heavy functions, call stacks, blocking pattern detection
--save-profile for flame graph export

What's disabled:

Slow HTTP/HTTPS request tracking
Slow fetch() tracking
Slow DNS lookup tracking
Slow TCP connection tracking
The "Slow Async I/O Summary" section in the report

Why Someone Would Want This

Reason 1: The Monkey-Patching Risk

I/O tracking works by monkey-patching core Node.js modules inside the target process. We replace http.request with a wrapper that calls the original, measures the time, and records slow operations. Same for dns.lookup, net.Socket.connect, and globalThis.fetch.

This is inherently more invasive than the other diagnostic methods:

Method	How it works	Invasiveness
Event loop lag	`setInterval` with `unref()`	Minimal — one timer
CPU profiling	V8's built-in sampler via CDP	Minimal — native V8 feature
I/O tracking	Monkey-patches 6+ core functions	Moderate — modifies module behavior

For most applications, the monkey-patches are safe. We store originals, we restore them on cleanup, we don't alter the return values or error behavior. But "most" isn't "all."

Some environments have strict policies about runtime code modification. Financial systems, healthcare platforms, and security-critical services may have compliance requirements that prohibit monkey-patching production code — even temporarily, even from a diagnostic tool.

Reason 2: The CPU-Only Investigation

Sometimes you already know the problem is CPU-bound. A developer pushed a commit with an O(n²) algorithm. A regex is backtracking. A JSON.parse is choking on a 50MB payload.

In these cases, I/O tracking is noise. The report shows "No slow I/O operations detected" (because there aren't any), but the I/O patches are still active, adding a tiny overhead to every network call. With --no-io, you get a cleaner report focused on what matters:

# CPU-only investigation
loop-detective 12345 --no-io -d 30

# Output focuses entirely on:
# - Event loop lag events
# - CPU heavy functions
# - Call stacks
# - Blocking patterns (cpu-hog, json-heavy, regex, gc, sync-io, crypto)

Reason 3: Reducing Attack Surface

When you attach to a production process, every piece of injected code is a potential risk. The I/O patches interact with the application's network stack — the most security-sensitive part of most services.

A conservative operator might want to start with the least invasive option:

# Step 1: CPU profile only — zero monkey-patching
loop-detective 12345 --no-io -d 10

# Step 2: If CPU looks fine, enable I/O tracking
loop-detective 12345 -d 30 --io-threshold 500

This graduated approach lets you get initial diagnostics with minimal risk, then opt into deeper instrumentation only if needed.

Reason 4: Isolating Variables

When debugging a complex performance issue, you want to change one thing at a time. If you're seeing unexpected behavior and you're not sure whether it's caused by the I/O patches or by the application itself, --no-io lets you eliminate the patches as a variable.

The Implementation

The implementation is deliberately simple. In detective.js:

async _singleRun() {
  try {
    await this._startLagDetection();
    if (!this.config.noIO) {
      await this._startAsyncIOTracking();
    }
    const profile = await this._captureProfile(this.config.duration);
    // ...
  } finally {
    await this.stop();
  }
}

One if statement. That's it. The _startAsyncIOTracking method — with its 200+ lines of monkey-patching code — is simply never called. No patches are injected. No cleanup is needed. The target process is never touched beyond the lag detector and the CPU profiler.

The same guard exists in _watchMode():

async _watchMode() {
  await this._startLagDetection();
  if (!this.config.noIO) {
    await this._startAsyncIOTracking();
  }
  // ... profiling cycles
}

In the CLI, it's a boolean flag:

const boolMap = {
  '--no-io': 'no-io',
  // ...
};

const config = {
  noIO: values['no-io'],
  // ...
};

In the programmatic API:

const detective = new Detective({
  pid: 12345,
  duration: 10000,
  noIO: true,
});

The Design Philosophy

Every feature in a diagnostic tool should be opt-out-able. This is different from application features, where you want sensible defaults that "just work." Diagnostic tools operate in a different trust model:

You're modifying someone else's process. The target application didn't ask to be profiled. The operator is making a judgment call about acceptable risk.
Production is not staging. What's safe in development might not be acceptable in production. Giving operators granular control respects this reality.
Less can be more. A focused CPU-only report is sometimes more useful than a comprehensive report with I/O data that isn't relevant to the current investigation.

This is why node-loop-detective has --no-io but doesn't have --no-lag or --no-profile. The lag detector and CPU profiler are minimally invasive (a single unref'd timer and V8's native sampler). The I/O tracker is qualitatively different — it modifies the behavior of core modules. That difference warrants an opt-out.

When to Use Each Mode

Scenario	Command
General diagnosis (default)	`loop-detective <pid>`
Known CPU issue	`loop-detective <pid> --no-io`
Sensitive production system	`loop-detective <pid> --no-io` (start here, add I/O later if needed)
Suspected slow I/O	`loop-detective <pid> --io-threshold 200`
Full investigation with export	`loop-detective <pid> -d 60 --save-profile ./profile.cpuprofile`
Minimal risk, maximum info	`loop-detective <pid> --no-io --save-profile ./profile.cpuprofile`

Try It

npm install -g node-loop-detective@1.6.0

# Full diagnostics (default)
loop-detective <pid>

# CPU and event loop only — no monkey-patching
loop-detective <pid> --no-io

Source: github.com/iwtxokhtd83/node-loop-detective

The best diagnostic tool is one you trust enough to run in production. Sometimes that means giving you the option to use less of it.

The Invisible Network Calls: Tracking fetch() and dns.promises in Node.js

Bill Tu — Wed, 08 Apr 2026 13:24:57 +0000

You've instrumented your Node.js app with I/O tracking. You're catching slow HTTP requests, slow database connections, slow DNS lookups. Your monitoring dashboard looks comprehensive.

Then a developer refactors a service client from axios to the built-in fetch(). Or switches from dns.lookup callbacks to dns.promises.lookup. The code is cleaner. The tests pass. The deployment goes out.

And your I/O tracking goes blind.

This is the story of two API surfaces that silently bypass traditional Node.js instrumentation — and how we patched them in node-loop-detective v1.5.0.

The Problem: Two Parallel Worlds

Node.js has evolved its networking APIs over the years, creating parallel paths that don't share the same internal plumbing.

fetch() vs http.request

Before Node.js 18, if you wanted to make an HTTP request, you used http.request (or a library built on top of it like axios, got, or node-fetch). Every HTTP client in the npm ecosystem ultimately called http.request or https.request under the hood.

Then Node.js 18 introduced a global fetch() function. It looks like the browser Fetch API. It's convenient. It's modern. And it's backed by undici — a completely separate HTTP client that does NOT use http.request internally.

This means if you monkey-patch http.request to track slow HTTP calls, fetch() calls are completely invisible. They go through undici's own connection pool, its own socket management, its own DNS resolution. Your instrumentation sees nothing.

// This is tracked (goes through http.request)
const axios = require('axios');
await axios.get('https://api.example.com/users');

// This is NOT tracked (goes through undici, bypasses http.request)
const res = await fetch('https://api.example.com/users');

As more codebases adopt fetch() — it's the standard API, it's built-in, it doesn't need a dependency — this blind spot grows.

dns.promises.lookup vs dns.lookup

A similar split exists in the DNS module. The original dns.lookup uses callbacks:

const dns = require('dns');
dns.lookup('example.com', (err, address) => {
  // ...
});

Node.js 10.6 introduced dns.promises.lookup, the promise-based equivalent:

const dns = require('dns');
const { address } = await dns.promises.lookup('example.com');

These are separate functions with separate code paths. Patching dns.lookup doesn't intercept dns.promises.lookup. Modern async/await code increasingly uses the promise API, making callback-only instrumentation incomplete.

Why This Matters in Practice

Consider a typical Node.js microservice in 2025:

// auth.js — uses fetch (Node.js 18+)
async function verifyToken(token) {
  const res = await fetch(`${AUTH_SERVICE_URL}/verify`, {
    method: 'POST',
    headers: { 'Authorization': `Bearer ${token}` },
    body: JSON.stringify({ token }),
  });
  return res.json();
}

// dns-cache.js — uses dns.promises
const dns = require('dns');
async function resolveService(hostname) {
  const { address } = await dns.promises.lookup(hostname);
  return address;
}

If the auth service starts responding slowly (800ms instead of 20ms), or DNS resolution starts hanging (2 seconds in a misconfigured container), your I/O tracker reports nothing. The event loop isn't blocked. The CPU profile looks healthy. But every request is paying a multi-second tax.

This is exactly the scenario users reported: "The report says healthy, but my app is slow."

The Fix: Patching Both Worlds

In v1.5.0, node-loop-detective now patches both the old and new API surfaces.

Patching fetch()

The global fetch is a regular function on globalThis. We wrap it the same way we wrap http.request — call the original, measure the time, record if it's slow:

(function patchFetch() {
  if (typeof globalThis.fetch !== 'function') return;
  const origFetch = globalThis.fetch;

  globalThis.fetch = function patchedFetch(input, init) {
    const startTime = Date.now();
    const callerStack = captureCallerStack();

    // Extract URL and method
    let target = 'unknown';
    let method = 'GET';
    if (typeof input === 'string') {
      target = input;
    } else if (input && typeof input === 'object') {
      target = input.url || input.href || String(input);
      method = (input.method || 'GET').toUpperCase();
    }
    if (init && init.method) {
      method = init.method.toUpperCase();
    }

    return origFetch.call(this, input, init).then(
      (res) => {
        const duration = Date.now() - startTime;
        if (duration >= threshold) {
          recordSlowOp({
            type: 'fetch', method, target,
            statusCode: res.status, duration,
            timestamp: Date.now(), stack: callerStack,
          });
        }
        return res;
      },
      (err) => {
        const duration = Date.now() - startTime;
        if (duration >= threshold) {
          recordSlowOp({
            type: 'fetch', method, target,
            error: err.message, duration,
            timestamp: Date.now(), stack: callerStack,
          });
        }
        throw err;
      }
    );
  };
})();

Key details:

Graceful detection: if (typeof globalThis.fetch !== 'function') return — on Node.js 16/17 where fetch doesn't exist, the patch is silently skipped.
Input parsing: fetch() accepts a string URL, a URL object, or a Request object. We handle all three.
Promise wrapping: fetch() returns a Promise, so we chain .then() to measure timing without altering the response.
URL shortening: We parse the URL to show host + pathname instead of the full URL with query parameters, keeping the output readable.
Cleanup: The original fetch is stored and restored when loop-detective disconnects.

Patching dns.promises.lookup

The promise-based DNS API wraps similarly:

if (dns.promises && dns.promises.lookup) {
  const origPromiseLookup = dns.promises.lookup;

  dns.promises.lookup = function patchedPromiseLookup(hostname, options) {
    const startTime = Date.now();
    const callerStack = captureCallerStack();

    return origPromiseLookup.call(dns.promises, hostname, options).then(
      (result) => {
        const duration = Date.now() - startTime;
        if (duration >= threshold) {
          recordSlowOp({
            type: 'dns', target: hostname, duration,
            timestamp: Date.now(), stack: callerStack,
          });
        }
        return result;
      },
      (err) => {
        const duration = Date.now() - startTime;
        if (duration >= threshold) {
          recordSlowOp({
            type: 'dns', target: hostname, duration,
            error: err.message, timestamp: Date.now(), stack: callerStack,
          });
        }
        throw err;
      }
    );
  };
}

This sits alongside the existing dns.lookup callback patch. Both are active simultaneously, covering code that uses either API style.

What the Output Looks Like

With both patches active, the report now catches everything:

🌐 Slow FETCH: 1820ms GET api.example.com/users → 200
    at loadUsers (/app/services/user.js:23:18)
    at handleRequest (/app/routes/api.js:45:5)

🌐 Slow HTTP: 950ms POST payment-gateway.com/charge → 200
    at processPayment (/app/services/payment.js:67:12)

🔍 Slow DNS: 2100ms lookup internal-service.local
    at Agent.createConnection (node:_http_agent:321:5)

🔌 Slow TCP: 1520ms db-server:5432
    at createConnection (/app/db.js:12:5)

  ⚠ Slow Async I/O Summary
    Total slow ops: 8

    🌐 FETCH — 3 slow ops, avg 1400ms, max 1820ms
      GET api.example.com/users
        3 calls, total 4200ms, avg 1400ms, max 1820ms

    🌐 HTTP — 2 slow ops, avg 900ms, max 950ms
      POST payment-gateway.com/charge
        2 calls, total 1800ms, avg 900ms, max 950ms

    🔍 DNS — 2 slow ops, avg 1800ms, max 2100ms
      internal-service.local
        2 calls, total 3600ms, avg 1800ms, max 2100ms

    🔌 TCP — 1 slow ops, avg 1520ms, max 1520ms
      db-server:5432
        1 calls, total 1520ms, max 1520ms

Notice that FETCH and HTTP are reported as separate types. This is intentional — it tells you which API surface the code is using, which matters when you're deciding how to add timeouts or caching.

The Full Coverage Map

Here's what node-loop-detective now tracks across Node.js versions:

I/O Type	What's Patched	Node.js Version	Covers
HTTP/HTTPS	`http.request`, `http.get`, `https.*`	All	axios, got, node-fetch, superagent, needle, etc.
Fetch	`globalThis.fetch`	18+	Built-in fetch, any code using the Fetch API
DNS (callback)	`dns.lookup`	All	Internal hostname resolution for http/https
DNS (promise)	`dns.promises.lookup`	10.6+	Modern async/await DNS code
TCP	`net.Socket.connect`	All	MySQL, PostgreSQL, Redis, MongoDB, AMQP, etc.

On Node.js 16-17, the fetch patch is silently skipped (it doesn't exist). On Node.js < 10.6, the dns.promises patch is skipped. Everything degrades gracefully.

A Note on undici Internals

You might wonder: if fetch() uses undici internally, and undici uses TCP sockets, wouldn't the net.Socket.connect patch catch undici's connections?

It depends. undici maintains a connection pool. If a connection is already established and pooled, a fetch() call reuses it without creating a new socket. The TCP patch only fires on new connections. So:

First fetch() to a new host: TCP connect is tracked + fetch timing is tracked
Subsequent fetch() calls to the same host: only fetch timing is tracked (connection is reused)

This is actually the correct behavior. You want to know "this fetch call took 1.8 seconds" regardless of whether it created a new connection or reused one. The fetch-level timing captures the full request lifecycle.

Upgrading

npm install -g node-loop-detective@1.5.0

# Track everything including fetch() and dns.promises
loop-detective <pid> --io-threshold 200

If your Node.js app uses fetch() or dns.promises, the slow calls that were previously invisible will now appear in the report. No configuration needed — the patches are applied automatically based on what's available in the target process.

Source: github.com/iwtxokhtd83/node-loop-detective

From O(n log n) to O(log p): Rebuilding the Order Book for Performance

Bill Tu — Wed, 08 Apr 2026 13:02:24 +0000

When we first built MatchEngine, the order book used a sorted slice. Every time an order was inserted, the entire slice was re-sorted. It was simple, correct, and easy to understand. It was also O(n log n) per insertion, which means it got slower with every order added to the book.

This article walks through the redesign: replacing the sorted slice with a price-level map backed by binary search insertion. The result is O(log p) insertion where p is the number of distinct price levels — typically 50-200 even when the book holds tens of thousands of orders.

The Original Design

The order book stored all orders in a flat slice, sorted by price-time priority:

type OrderBook struct {
    Bids []*model.Order  // highest price first
    Asks []*model.Order  // lowest price first
}

Insertion appended the order and re-sorted:

func (ob *OrderBook) AddOrder(order *model.Order) {
    ob.Asks = append(ob.Asks, order)
    sort.SliceStable(ob.Asks, func(i, j int) bool {
        if ob.Asks[i].Price.Equal(ob.Asks[j].Price) {
            return ob.Asks[i].Timestamp.Before(ob.Asks[j].Timestamp)
        }
        return ob.Asks[i].Price.LessThan(ob.Asks[j].Price)
    })
}

sort.SliceStable is O(n log n). For a book with 10,000 orders, every single insertion triggers a comparison-sort across all 10,000 elements. The 10,001st order is just as expensive to insert as rebuilding the entire book from scratch.

Cancellation was O(n) too — a linear scan to find the order by ID:

func removeFromSlice(orders []*model.Order, id string) []*model.Order {
    for i, o := range orders {
        if o.ID == id {
            return append(orders[:i], orders[i+1:]...)
        }
    }
    return orders
}

This was fine for a prototype. But the issue was clear: the data structure didn't match the access pattern.

The Access Pattern

A matching engine's order book has a very specific access pattern:

Orders cluster at price levels. A book with 10,000 orders might have only 100 distinct prices.
The most frequent operations are insert and best-price lookup.
Orders at the same price are processed in FIFO order.
Removal by ID needs to be fast (for cancellation).

The sorted slice treats every order as an independent element. It doesn't exploit the fact that orders group by price. It re-sorts everything even when the new order's price already exists in the book.

The New Design: Price-Level Map

The new architecture has three layers:

OrderBook
├── bids: orderSide (highest price first)
├── asks: orderSide (lowest price first)
└── orders: map[string]*Order (ID index)

orderSide
├── prices: []decimal.Decimal (sorted, binary search)
├── levels: map[string]*priceLevel (price -> queue)
└── count: int

priceLevel
└── orders: []*Order (FIFO queue)

Each side of the book (orderSide) maintains:

A sorted slice of distinct price keys
A map from price string to a priceLevel
A total order count

Each priceLevel is a simple FIFO queue — a slice of orders at that exact price, in arrival order.

Why This Structure?

It matches the access pattern exactly:

Insert at an existing price: O(1) map lookup + O(1) append to the queue. No sorting.
Insert at a new price: O(log p) binary search to find the insertion point in the price slice + O(p) shift to make room. This happens rarely compared to inserts at existing prices.
Best price: O(1) — it's prices[0].
Cancel by ID: O(1) map lookup to find the order, then O(k) scan within its price level (where k is typically small).

The Implementation

priceLevel: The FIFO Queue

The simplest component. Each price level holds orders in arrival order:

type priceLevel struct {
    orders []*model.Order
}

func newPriceLevel() *priceLevel {
    return &priceLevel{orders: make([]*model.Order, 0, 4)}
}

func (pl *priceLevel) append(order *model.Order) {
    pl.orders = append(pl.orders, order)
}

func (pl *priceLevel) front() *model.Order {
    if len(pl.orders) == 0 {
        return nil
    }
    return pl.orders[0]
}

The initial capacity of 4 is a small optimization — most price levels have a handful of orders, and this avoids the first few slice growths.

The removeFilled method uses an in-place compaction pattern to avoid allocating a new slice:

func (pl *priceLevel) removeFilled() int {
    n := 0
    for _, o := range pl.orders {
        if !o.IsFilled() {
            pl.orders[n] = o
            n++
        }
    }
    pl.orders = pl.orders[:n]
    return n
}

This overwrites filled orders with unfilled ones from left to right, then truncates. Zero allocations, single pass.

orderSide: Sorted Prices + Level Map

This is where the interesting logic lives. Each side of the book is an orderSide:

type orderSide struct {
    prices []decimal.Decimal
    levels map[string]*priceLevel
    less   func(a, b decimal.Decimal) bool
    count  int
}

The less function determines sort order — GreaterThan for bids (highest first), LessThan for asks (lowest first). This lets both sides share the same code with different comparators.

Insertion: Binary Search

The core of the optimization. When an order arrives:

func (s *orderSide) insert(order *model.Order) {
    key := order.Price.String()
    pl, exists := s.levels[key]
    if !exists {
        pl = newPriceLevel()
        s.levels[key] = pl
        idx := sort.Search(len(s.prices), func(i int) bool {
            return s.less(order.Price, s.prices[i]) || order.Price.Equal(s.prices[i])
        })
        // Insert price at idx
        s.prices = append(s.prices, decimal.Zero)
        copy(s.prices[idx+1:], s.prices[idx:])
        s.prices[idx] = order.Price
    }
    pl.append(order)
    s.count++
}

Two paths:

Path 1: Price level exists (common case). Map lookup is O(1). Append to the queue is O(1) amortized. Total: O(1).

Path 2: New price level (rare case). sort.Search does a binary search over the price slice — O(log p). Then we insert the price at the found index using the standard Go slice insertion pattern (append + copy). The copy is O(p) in the worst case, but p is the number of distinct prices, not the number of orders.

In a real order book, most orders arrive at prices that already exist. The common path is O(1). The rare path is O(log p + p). Compared to the old O(n log n) on every insert, this is a massive improvement.

Best Price: O(1)

func (s *orderSide) best() *model.Order {
    if len(s.prices) == 0 {
        return nil
    }
    key := s.prices[0].String()
    return s.levels[key].front()
}

The best price is always prices[0]. The best order at that price is the front of the FIFO queue. Two lookups, both O(1).

Removal

When an order is cancelled, the engine already knows the order (via the order ID index). The orderbook looks up the price level and removes the order from the queue:

func (s *orderSide) remove(order *model.Order) {
    key := order.Price.String()
    pl, exists := s.levels[key]
    if !exists {
        return
    }
    if pl.remove(order.ID) {
        s.count--
        if pl.len() == 0 {
            s.removePrice(key, order.Price)
        }
    }
}

If the price level becomes empty after removal, it is cleaned up — deleted from the map and removed from the price slice. This keeps the data structure compact.

Cleanup: removeFilled

After each matching cycle, filled orders need to be removed. The new implementation scans each price level, compacts it in-place, and collects empty levels for removal:

func (s *orderSide) removeFilled() []*model.Order {
    var filled []*model.Order
    pricesToRemove := make([]int, 0)

    for i, price := range s.prices {
        key := price.String()
        pl := s.levels[key]
        for _, o := range pl.allOrders() {
            if o.IsFilled() {
                filled = append(filled, o)
                s.count--
            }
        }
        remaining := pl.removeFilled()
        if remaining == 0 {
            delete(s.levels, key)
            pricesToRemove = append(pricesToRemove, i)
        }
    }

    // Remove empty price levels in reverse order to preserve indices
    for i := len(pricesToRemove) - 1; i >= 0; i-- {
        idx := pricesToRemove[i]
        s.prices = append(s.prices[:idx], s.prices[idx+1:]...)
    }

    return filled
}

The reverse-order removal of empty price indices is a subtle but important detail. Removing index 3 before index 7 would shift index 7 to 6, invalidating the stored index. Removing in reverse (7 first, then 3) avoids this.

The OrderBook: Putting It Together

The public OrderBook struct wires the two sides together:

type OrderBook struct {
    bids   *orderSide
    asks   *orderSide
    orders map[string]*model.Order
}

func New() *OrderBook {
    return &OrderBook{
        bids: newOrderSide(func(a, b decimal.Decimal) bool {
            return a.GreaterThan(b)  // highest first
        }),
        asks: newOrderSide(func(a, b decimal.Decimal) bool {
            return a.LessThan(b)     // lowest first
        }),
        orders: make(map[string]*model.Order),
    }
}

The orders map provides O(1) lookup by ID, used by RemoveOrder (cancellation) and HasOrder (duplicate detection).

One notable change: Bids and Asks are no longer public fields. They are now methods that return flattened slices:

func (ob *OrderBook) Bids() []*model.Order {
    return ob.bids.flatten()
}

This is a deliberate API change. The internal representation is no longer a flat slice, so exposing it as one would either leak the abstraction or require materializing the slice on every access. Making it a method makes the cost explicit.

Complexity Comparison

Operation	Sorted Slice (before)	Price-Level Map (after)
AddOrder (existing price)	O(n log n)	O(1)
AddOrder (new price)	O(n log n)	O(log p + p)
RemoveOrder	O(n)	O(1) + O(k)
BestBid / BestAsk	O(1)	O(1)
RemoveFilled	O(n)	O(n)
Snapshot	O(n)	O(n)

Variables: n = total orders, p = distinct price levels, k = orders at a given price.

The key insight is that p is much smaller than n. A liquid market might have 10,000 resting orders spread across 100 price levels. The old design paid O(10000 × log 10000) ≈ 130,000 comparisons per insert. The new design pays O(1) for the common case (existing price) or O(log 100 + 100) ≈ 107 operations for a new price level.

What Didn't Change

The matching algorithm is completely untouched. It still calls BestAsk(), checks the price, calls executeTrade(), and loops. The engine code changed only in one place — cleanFilled now calls book.Bids() and book.Asks() methods instead of accessing public fields.

The test suite passes without modification to test logic (only field access syntax changed to method calls). This is the benefit of a clean abstraction boundary — the order book's public API (AddOrder, RemoveOrder, BestBid, BestAsk, Depth, Spread) stayed the same.

Trade-offs

Nothing is free:

Memory: The price-level map uses more memory than a flat slice. Each price level has its own slice header, and the map has per-bucket overhead. For a book with 100 price levels and 10,000 orders, the overhead is roughly 100 × (slice header + map entry) ≈ a few KB. Negligible.

Flatten cost: Bids() and Asks() now allocate a new slice and copy all orders into it. The old design returned the slice directly (O(1)). This matters for Snapshot(), which calls flatten() twice. For a 10,000-order book, that is two allocations of 10,000 pointers. Acceptable for a snapshot operation that is already O(n).

Complexity: The code is more complex. Three types (orderSide, priceLevel, OrderBook) instead of one. The binary search insertion with slice shifting is trickier than sort.SliceStable. But the complexity is localized — the matching engine doesn't know or care about price levels.

Benchmarks

We added three benchmarks to measure the impact:

func BenchmarkInsertOnly(b *testing.B) {
    e := New(WithMaxTradeLog(0))
    price := decimal.NewFromInt(100)
    qty := decimal.NewFromInt(1)
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        id := fmt.Sprintf("s%d", i)
        e.SubmitOrder("BTC", model.NewLimitOrder(id, model.Sell, price, qty))
    }
}

BenchmarkInsertOnly: All orders at the same price. Tests the O(1) common path.
BenchmarkInsertAndMatch: Alternating buy/sell at the same price. Tests insert + match + cleanup.
BenchmarkInsertMultiplePriceLevels: Orders spread across 100 price levels. Tests the O(log p) path.

Run with:

go test -bench=. -benchmem ./pkg/engine/

The single-price insert benchmark is the most dramatic improvement. With the old sorted slice, each insert re-sorted the growing slice. With the new design, each insert is a map lookup + slice append — constant time regardless of book depth.

Conclusion

The optimization boils down to one idea: exploit the structure of the data. Orders in a matching engine cluster by price. A flat sorted list ignores this structure and pays O(n log n) to maintain global order. A price-level map respects the structure and pays O(1) for the common case.

The change touched two files (orderbook.go rewritten, pricelevel.go added) and required only syntax changes in consumers (book.Bids → book.Bids()). The matching algorithm, the engine, and all existing tests remained functionally identical.

For most open-source matching engines, this is the right level of optimization. It eliminates the obvious bottleneck without introducing the complexity of a red-black tree or skip list. If profiling later shows that the O(p) slice shift during new-price insertion is a problem, the prices slice can be replaced with a balanced tree — but that is a localized change within orderSide, invisible to the rest of the system.

GitHub: https://github.com/iwtxokhtd83/MatchEngine
Release: v0.3.0
Issue: #4 — Order book uses O(n log n) sort on every insert

From Terminal to Flame Graph: Exporting CPU Profiles From Production

Bill Tu — Wed, 08 Apr 2026 09:52:49 +0000

loop-detective gives you a text-based diagnostic report in your terminal. It tells you which functions are heavy, what patterns are blocking the event loop, and where the slow I/O is. For most debugging sessions, that's enough.

But sometimes you need more. You need to see the full call tree. You need to zoom into a 200ms window and understand every function that executed. You need a flame graph.

With v1.4.0, loop-detective can now export the raw V8 CPU profile to a .cpuprofile file — the same format Chrome DevTools uses. One flag:

loop-detective 12345 --save-profile ./profile.cpuprofile

This article explains what's in that file, how to use it, and why combining terminal diagnostics with visual profiling gives you the best of both worlds.

What's Inside a .cpuprofile File

When loop-detective profiles a Node.js process, it uses the V8 Profiler via the Chrome DevTools Protocol. The profiler works by statistical sampling: every ~100 microseconds, it records which JavaScript function is currently executing. Over thousands of samples, this builds a statistical picture of where CPU time is spent.

The raw output is a JSON object with three key fields:

{
  "nodes": [
    {
      "id": 1,
      "callFrame": {
        "functionName": "processRequest",
        "url": "/app/server.js",
        "lineNumber": 41,
        "columnNumber": 2
      },
      "hitCount": 342,
      "children": [2, 3, 7]
    }
  ],
  "startTime": 1705312345000000,
  "endTime": 1705312355000000,
  "samples": [1, 1, 2, 3, 3, 3, 1, 7, ...],
  "timeDeltas": [120, 98, 105, 112, ...]
}

nodes: A call tree. Each node is a function with its file path, line number, and child nodes. This represents every unique call path the profiler observed.
samples: A time series of node IDs. Each entry is "which function was on top of the stack at this sample tick."
timeDeltas: The time gap (in microseconds) between consecutive samples.

This is the same format that Chrome DevTools produces when you record a performance profile in the browser. The .cpuprofile extension is a convention — the file is just JSON.

What loop-detective Already Tells You

The built-in analysis processes this raw data and gives you a focused report:

  Top CPU-Heavy Functions
────────────────────────────────────────────────────────────
   1. heavyComputation
      ██████████████░░░░░░ 6245ms (62.3%)
      /app/server.js:42:1
   2. JSON.parse
      ███░░░░░░░░░░░░░░░░░ 823ms (8.2%)
      (native):1:1

  Diagnosis
────────────────────────────────────────────────────────────
   HIGH  cpu-hog
         Function "heavyComputation" consumed 62.3% of CPU time
         → Consider breaking this into smaller async chunks

This is great for quick diagnosis. You know the top offenders, you know the patterns, you know where to look. But the analysis is lossy — it shows you the top 20 functions and the top 5 call stacks. The raw profile has thousands of nodes and tens of thousands of samples.

When You Need the Full Picture

There are cases where the summary isn't enough:

"The top function is only 15% of CPU, but the app is still slow." The problem might be spread across many functions. A flame graph shows the full distribution at a glance — wide bars are slow, narrow bars are fast.

"I need to see the exact call path." The built-in call stacks show the path to the top 5 heavy functions. But what if the problem is a function that's called from 20 different places? A flame graph shows every call path and how much time each one contributes.

"I need to share this with my team." A .cpuprofile file can be opened by anyone with Chrome. No special tools needed. Drop it in a Slack thread, attach it to a Jira ticket, include it in a post-mortem.

"I want to compare before and after." Save a profile before your fix, save another after. Open both in speedscope and compare side by side.

How to Use the Saved Profile

Chrome DevTools

The most accessible option — everyone has Chrome.

Open Chrome (or any Chromium browser)
Open DevTools (F12)
Go to the Performance tab
Click the ↑ upload button (or the "Load profile" option)
Select your .cpuprofile file

You get:

Flame chart: A timeline view showing function execution over time. The x-axis is time, the y-axis is call depth. Wide bars = slow functions.
Bottom-up view: Functions sorted by self time. Shows where CPU time was actually spent (not including children).
Call tree: Top-down view from the entry point. Shows how time flows through your call hierarchy.
Search: Ctrl+F to find specific functions by name.

speedscope.app

A dedicated profile viewer that's faster and more interactive than Chrome DevTools for large profiles.

Visit speedscope.app
Drag and drop the .cpuprofile file
Choose your view:
- Time Order: flame chart showing execution over time
- Left Heavy: aggregated flame graph, merged by call stack
- Sandwich: shows callers and callees for any selected function

speedscope handles large profiles (100MB+) smoothly and supports keyboard navigation for zooming and panning.

VS Code

Several extensions can open .cpuprofile files directly in VS Code:

vscode-js-profile-flame: Shows an inline flame graph
Flame Chart Visualizer: Interactive flame chart in a VS Code tab

This is convenient when you're already in your editor and want to jump directly from the profile to the source code.

The Combined Workflow

Here's how terminal diagnostics and visual profiling complement each other:

# Step 1: Quick diagnosis — what's the problem?
loop-detective 12345 -d 10

# Output tells you: cpu-hog on processPayload, 54% CPU
# But you want to understand the full call tree

# Step 2: Capture a longer profile with export
loop-detective 12345 -d 60 --save-profile ./investigation.cpuprofile

# Step 3: Open in Chrome DevTools or speedscope
# - Flame graph shows processPayload is called from 3 different routes
# - One route accounts for 80% of the calls
# - Inside processPayload, JSON.parse on line 67 is the real hotspot

# Step 4: Fix the issue, then verify
loop-detective 12345 -d 60 --save-profile ./after-fix.cpuprofile
# Compare the two profiles in speedscope

Step 1 takes 10 seconds and gives you the direction. Steps 2-4 give you the depth. You don't need the flame graph for every debugging session, but when you do, it's there.

Practical Tips

Profile duration matters. A 5-second profile might miss intermittent issues. For flame graph analysis, 30-60 seconds gives you a representative sample. The file size is typically 1-5MB for a 60-second profile.

Name your files. Include context in the filename:

loop-detective 12345 -d 60 --save-profile ./profiles/api-server-$(date +%Y%m%d-%H%M%S).cpuprofile

Watch mode overwrites. In --watch mode, each cycle overwrites the same file. If you need to keep every cycle's profile, use --json output and extract the raw profile data programmatically.

Combine with I/O tracking. The CPU profile shows where compute time goes. The slow I/O summary shows where wait time goes. Together they explain the full request lifecycle:

loop-detective 12345 -d 30 --io-threshold 200 --save-profile ./profile.cpuprofile

The profile includes everything. The saved file contains the complete V8 profile — all nodes, all samples, all time deltas. loop-detective's built-in analysis only shows the top 20 functions. The flame graph shows everything, including functions that individually account for <1% of CPU but collectively matter.

Under the Hood

The implementation is minimal. The Detective class already captures the raw V8 profile via CDP:

async _captureProfile(duration) {
  await this.inspector.send('Profiler.enable');
  await this.inspector.send('Profiler.start');
  await this._sleep(duration);
  const { profile } = await this.inspector.send('Profiler.stop');
  return profile;
}

The profile object is the raw V8 CPU profile. Previously, it was only passed to the Analyzer. Now it's also emitted alongside the analysis:

this.emit('profile', analysis, rawProfile);

The CLI catches it and writes to disk:

detective.on('profile', (analysis, rawProfile) => {
  reporter.onProfile(analysis);
  if (config.saveProfile && rawProfile) {
    fs.writeFileSync(path.resolve(config.saveProfile), JSON.stringify(rawProfile));
  }
});

No transformation, no filtering. The file is the exact JSON that V8 produced. This means any tool that reads .cpuprofile files will work correctly — Chrome DevTools, speedscope, VS Code extensions, or your own analysis scripts.

Programmatic API

If you're building tooling on top of loop-detective, you can access the raw profile via the event:

const { Detective } = require('node-loop-detective');

const detective = new Detective({ pid: 12345, duration: 30000 });

detective.on('profile', (analysis, rawProfile) => {
  // analysis — the processed report (heavy functions, patterns, etc.)
  // rawProfile — the raw V8 CPU profile object

  // Save it
  fs.writeFileSync('profile.cpuprofile', JSON.stringify(rawProfile));

  // Or analyze it yourself
  console.log(`${rawProfile.nodes.length} nodes, ${rawProfile.samples.length} samples`);
});

await detective.start();

You can also use the Analyzer class standalone if you already have a .cpuprofile file:

const { Analyzer } = require('node-loop-detective');
const profile = JSON.parse(fs.readFileSync('profile.cpuprofile', 'utf8'));
const analyzer = new Analyzer({ threshold: 50 });
const result = analyzer.analyzeProfile(profile);

Try It

npm install -g node-loop-detective@1.4.0

# Profile and export
loop-detective <pid> --save-profile ./profile.cpuprofile

# Then open in your browser
# Chrome DevTools → Performance → Load profile
# Or drag into https://www.speedscope.app

The terminal report tells you what's wrong. The flame graph shows you why.

Source: github.com/iwtxokhtd83/node-loop-detective

Lessons from Fixing 8 Bugs in a Quantitative Trading Framework

Bill Tu — Wed, 08 Apr 2026 07:36:26 +0000

QuantFlow is an open-source quantitative trading system written in Python. After the initial release, a thorough code review uncovered 8 bugs — ranging from silent correctness errors to quadratic performance traps. This article walks through each bug, explains why it matters in the context of financial software, and shows the fix.

The bugs fall into three categories: correctness errors that produce wrong numbers, safety gaps that allow dangerous operations, and performance issues that make the system unusable at scale.

Bug 1: Python Properties That Silently Ignore Arguments

The PerformanceReport class defined sharpe_ratio and sortino_ratio as Python @property decorators, but the method signatures included parameters:

@property
def sharpe_ratio(self, risk_free_rate: float = 0.0, periods: int = 252) -> float:
    ...

This is a subtle Python gotcha. The @property decorator transforms a method into a descriptor that takes no arguments beyond self. The risk_free_rate and periods parameters exist in the signature but can never be passed by the caller. result.sharpe_ratio always uses the defaults. Attempting result.sharpe_ratio(risk_free_rate=0.02) raises TypeError: 'float' object is not callable.

In quantitative finance, the risk-free rate matters. A Sharpe ratio calculated with a 0% risk-free rate versus a 4% rate can tell very different stories about a strategy's risk-adjusted performance. Silently hardcoding this to zero is a correctness problem.

The fix removes @property and makes them regular methods:

def sharpe_ratio(self, risk_free_rate: float = 0.0, periods: int = 252) -> float:
    if len(self.equity_curve) < 2:
        return 0.0
    returns = np.diff(self.equity_curve) / self.equity_curve[:-1]
    excess = returns - risk_free_rate / periods
    if np.std(excess) == 0:
        return 0.0
    return float(np.mean(excess) / np.std(excess) * np.sqrt(periods))

This is a breaking change — callers must update from result.sharpe_ratio to result.sharpe_ratio() — but it's the right trade-off. A method that pretends to accept parameters but ignores them is worse than a clean API change.

Bug 2: Drawdown Calculated from Cash, Not Equity

The risk manager's drawdown circuit breaker compared cash against equity_peak:

# Before: uses cash only
signal = self.risk_manager.check(
    signal=signal,
    current_price=bar.close,
    capital=portfolio.cash,        # BUG: just cash, not total equity
    equity_peak=portfolio.equity_peak,
    ...
)

The equity_peak tracks the highest total equity (cash + positions). But the drawdown check only looked at cash. When a position is open, cash drops by the position cost — even if the position is profitable. This creates false drawdown triggers.

Example: you start with $100,000 and buy $80,000 worth of stock. Cash is now $20,000. The stock goes up 10%, so your equity is $108,000. But the drawdown check sees $20,000 vs $100,000 peak — an 80% "drawdown" — and halts trading on a profitable portfolio.

The fix passes total equity instead:

signal = self.risk_manager.check(
    signal=signal,
    current_price=bar.close,
    equity=portfolio.equity,       # FIXED: total equity
    equity_peak=portfolio.equity_peak,
    ...
)

The risk manager's parameter was renamed from capital to equity to make the semantics explicit.

Bug 3: Position Market Value Frozen at Entry Price

The Position.market_value property was defined as:

@property
def market_value(self) -> float:
    return self.quantity * self.entry_price

This returns the cost basis, not the current market value. Since Portfolio.equity sums market_value across all positions, the equity curve never reflects unrealized gains or losses. A position that doubled in value still reports its original cost. A position that lost 50% still reports full value.

This means the equity curve, the Sharpe ratio, the max drawdown — every metric derived from equity — is wrong whenever positions are open.

The fix adds a current_price field to Position that gets updated on every bar:

@dataclass
class Position:
    symbol: str
    side: OrderSide
    quantity: float
    entry_price: float
    entry_time: datetime
    current_price: float = 0.0    # updated each bar
    ...

    @property
    def market_value(self) -> float:
        return self.quantity * self.current_price

    @property
    def cost_basis(self) -> float:
        return self.quantity * self.entry_price

The engine calls portfolio.update_market_prices(symbol, bar.close) on every bar before computing equity. The old entry_price-based value is preserved as cost_basis for PnL calculations.

Bugs 2 and 3 are related — fixing the equity calculation (Bug 3) makes the drawdown fix (Bug 2) actually meaningful. Without accurate equity, even the correct drawdown formula produces wrong results.

Bug 4: Unlimited Short Selling Without Margin

The portfolio's execute_signal method checked cash sufficiency for buy orders but had no check for sell (short) orders:

# Only checked for buys
if side == OrderSide.BUY and cost > self.cash:
    return None
# Shorts? No check at all.

In real markets, short selling requires margin — you need collateral to borrow shares. Without any check, the system could open unlimited short positions regardless of available capital. A strategy bug that generates continuous sell signals would create unbounded short exposure.

The fix adds a configurable margin requirement:

if side == OrderSide.SELL:
    margin_required = fill_price * quantity * self.short_margin_rate + commission
    if margin_required > self.cash:
        logger.info("Insufficient margin for short: need %.2f, have %.2f",
                     margin_required, self.cash)
        return None

The short_margin_rate defaults to 1.0 (100% margin), meaning you need the full position value in cash to short. This is conservative but safe. Users can adjust it for their broker's actual margin requirements.

Bug 5: O(n²) Memory Allocation in Data Feed

The DataFeed.bars() method built the close price history by appending to a Python list and creating a new numpy array on every bar:

closes = []
for _, row in df.iterrows():
    closes.append(row["close"])
    yield Bar(..., close_prices=np.array(closes, dtype=np.float64))

For a backtest with n bars, this creates n numpy arrays of sizes 1, 2, 3, ..., n. Total memory allocated: O(n²). Total copy operations: O(n²). For 10,000 bars, that's ~50 million float copies just for the price history.

The fix pre-allocates the full array once and passes slices:

all_closes = df["close"].values.astype(np.float64)
for i, (_, row) in enumerate(df.iterrows()):
    yield Bar(..., close_prices=all_closes[:i + 1])

all_closes[:i + 1] is a numpy view, not a copy. Zero additional memory allocation. The total memory for close prices drops from O(n²) to O(n), and the allocation count drops from n to 1.

On a 10,000-bar backtest, this eliminates roughly 400MB of transient memory allocation.

Bug 6: EMA Recomputes Full Series on Every Bar

The EMA.calculate() method called self.series(prices)[-1]:

def calculate(self, prices: np.ndarray) -> Optional[float]:
    if len(prices) < self.period:
        return None
    return float(self.series(prices)[-1])

series() builds the complete EMA array for all prices. Since strategies call calculate() on every bar with a growing price array, the total work over a backtest is O(n²) — computing the full series of length 1, then 2, then 3, ..., then n.

The fix computes only the final value:

def calculate(self, prices: np.ndarray) -> Optional[float]:
    if len(prices) < self.period:
        return None
    alpha = 2.0 / (self.period + 1)
    ema = float(np.mean(prices[:self.period]))
    for i in range(self.period, len(prices)):
        ema = alpha * prices[i] + (1 - alpha) * ema
    return ema

This is still O(n) per call (it walks the full price array), but it avoids the numpy array allocation overhead. A further optimization would cache the previous EMA value and compute incrementally in O(1), but that requires the indicator to be stateful — a design change for a future version.

Bug 7: Stochastic %D Misaligned with %K

The Stochastic oscillator's %D line (a moving average of %K) was computed by:

Extracting non-NaN values from %K into a compact array
Computing SMA on the compact array
Mapping results back to original indices

d = SMA(self.d_period).series(k[~np.isnan(k)])
d_full = np.full(n, np.nan)
valid_idx = np.where(~np.isnan(k))[0]
d_full[valid_idx[:len(d)]] = d

The problem: SMA.series() returns an array with leading NaN values (the warm-up period). When mapped back via valid_idx[:len(d)], these NaN values land on the wrong indices. The %D line ends up shifted relative to %K.

The fix computes %D directly with proper alignment:

d = np.full(n, np.nan)
for i in range(self.k_period - 1 + self.d_period - 1, n):
    window = k[i - self.d_period + 1: i + 1]
    if not np.any(np.isnan(window)):
        d[i] = np.mean(window)

This is straightforward: for each position, take the last d_period values of %K, check they're all valid, and average them. The index arithmetic ensures %D starts exactly d_period - 1 bars after %K begins.

Bug 8: MACD Returns NaN Instead of None

The MACD.calculate() method checked for insufficient data and returned (None, None, None). But when there was enough data for the MACD line but not enough for the signal line's EMA to fully warm up, series() would produce NaN values. The calculate() method then returned float(NaN):

# Before: NaN leaks through
return float(macd_line[-1]), float(signal_line[-1]), float(histogram[-1])

Downstream strategies check if hist is None but don't check for NaN. A NaN histogram passes the None check, enters the crossover logic, and produces garbage signals — NaN comparisons in Python always return False, so prev_hist <= 0 < hist silently fails.

The fix adds a NaN guard:

m, s, h = macd_line[-1], signal_line[-1], histogram[-1]
if np.isnan(m) or np.isnan(s) or np.isnan(h):
    return None, None, None
return float(m), float(s), float(h)

This is a general principle for indicator APIs: the contract should be "returns a number or None", never NaN. NaN propagates silently through arithmetic and comparisons, making bugs extremely hard to trace. None forces explicit handling.

Patterns Across the Bugs

Looking at all 8 bugs together, a few patterns emerge:

Financial software has a low tolerance for "close enough." Bugs 2, 3, and 8 all produce numbers that look plausible but are wrong. A Sharpe ratio of 1.8 vs 1.7 might not raise eyebrows, but if the difference comes from using cost basis instead of market value, every decision based on that number is compromised. In most software, off-by-a-little is acceptable. In quantitative finance, it's the difference between a profitable strategy and a losing one.

Python's flexibility is a double-edged sword. Bug 1 (@property with parameters) is a language-level trap. Python doesn't warn you. The code runs fine. The parameters just don't do anything. Static type checkers like mypy would catch this, which is a strong argument for running them in CI.

Performance bugs compound. Bugs 5 and 6 are both O(n²) issues that individually might be tolerable on small datasets. But they multiply: the data feed creates O(n²) arrays, and then the EMA indicator does O(n²) work on each one. On a 10,000-bar backtest, the combined effect is severe.

Safety checks are not optional. Bug 4 (unlimited short selling) is the kind of bug that doesn't matter until it does — and then it matters a lot. A strategy bug that generates spurious sell signals would create unbounded short exposure. In a live trading system, this could be catastrophic.

Verification

After applying all fixes, the multi-strategy backtest produces more accurate results:

================================================================================
  MULTI-STRATEGY COMPARISON
================================================================================
  Strategy                      Return   Trades   Win Rate   Sharpe     Max DD
--------------------------------------------------------------------------------
  SMACrossover                 228.29%       17     17.65%     1.80     -1.24%
  RSIMeanReversion              71.01%        4     25.00%     1.02    -22.57%
  MACDTrend                    366.14%       28     42.86%     2.18     -1.11%
  BollingerBreakout            857.60%       55     30.91%     2.84    -11.26%
================================================================================

The max drawdown numbers changed (e.g., RSIMeanReversion went from -16.86% to -22.57%) because they now reflect true equity-based drawdown instead of the cash-only approximation. The previous numbers were optimistic — they understated risk.

All fixes are included in QuantFlow v0.2.0.

QuantFlow is available at github.com/iwtxokhtd83/QuantFlow. Issues and contributions are welcome.

Four Bugs We Found in Our Node.js Rate Limiter (And How We Fixed Them)

Bill Tu — Wed, 08 Apr 2026 07:09:26 +0000

We recently shipped node-rate-limiter-pro, a high-performance rate limiter for Node.js with Token Bucket, Sliding Window, and Redis support. It benchmarks at ~2M ops/sec in memory and ~10x faster than express-rate-limit.

Then we did a proper code review.

We found four bugs — none of them obvious at first glance, all of them the kind that only show up in production under real load. This post walks through each one: what went wrong, why it matters, and the fix.

Bug #1: The Middleware That Swallowed Errors

The problem: Our Express middleware called await this.consume(key) without a try/catch.

// Before (broken)
return async (req, res, next) => {
  const key = keyFn(req);
  const result = await this.consume(key); // 💥 if Redis is down, this throws
  res.setHeader('X-RateLimit-Limit', result.limit);
  // ...
  next();
};

If you're using the in-memory store, this will never bite you — the memory store can't fail. But the moment you plug in Redis and the connection drops, consume() throws, and that error has nowhere to go. No catch, no next(err). Just an unhandled promise rejection that crashes your process.

This is particularly insidious because it works perfectly in development and testing (where Redis is always up), and only fails in production (where Redis occasionally hiccups).

The fix:

// After (fixed)
return async (req, res, next) => {
  try {
    const key = keyFn(req);
    const result = await this.consume(key);
    res.setHeader('X-RateLimit-Limit', result.limit);
    // ...
    next();
  } catch (err) {
    next(err);
  }
};

Now errors flow through Express's error handling pipeline, and users can implement their own strategy:

app.use(limiter.middleware());

app.use((err, req, res, next) => {
  if (err.message.includes('Redis')) {
    // Fail open: let the request through
    return next();
  }
  res.status(500).json({ error: 'Internal Server Error' });
});

Lesson: Every await in middleware is a potential throw. If you're writing Express middleware with async operations, wrap the entire body in try/catch and forward errors to next(). This is Express Middleware 101, but it's easy to forget when the happy path works so cleanly.

Bug #2: The Birthday Problem in Redis

The problem: Our Redis sliding window implementation uses a sorted set (ZSET) to track request timestamps. Each request adds a member with the current timestamp as the score. But sorted set members must be unique — if two members have the same value, the second one overwrites the first instead of adding a new entry.

To make members unique, we appended a random number:

-- Before (broken)
redis.call('ZADD', key, now, now .. '-' .. math.random(1000000))

See the problem? math.random(1000000) generates a number between 1 and 1,000,000. At high throughput — say 1,000 requests per second for a single key — multiple requests land in the same millisecond. The birthday problem tells us that with ~1,000 values drawn from a pool of 1,000,000, the probability of at least one collision is roughly:

P ≈ 1 - e^(-n² / 2m) = 1 - e^(-1000² / 2000000) ≈ 39%

A 39% chance of collision per second. When a collision happens, ZADD overwrites instead of adding, the count is off by one, and the rate limiter allows one extra request. Under sustained load, this adds up.

The fix: Replace math.random() with an atomic Redis counter:

-- After (fixed)
local seq = redis.call('INCR', key .. ':seq')
redis.call('ZADD', key, now, now .. '-' .. seq)
redis.call('PEXPIRE', key, window)
redis.call('PEXPIRE', key .. ':seq', window)

INCR is atomic and monotonically increasing — no collisions, ever. The sequence counter key gets the same TTL as the main key, so it cleans itself up.

Lesson: When you need uniqueness in a distributed system, don't use random numbers unless the space is astronomically large. Monotonic counters are simpler, faster, and provably collision-free. And always do the birthday problem math — your intuition about "how likely is a collision" is almost certainly wrong.

Bug #3: The Slow Memory Leak

The problem: Both the Token Bucket and Sliding Window algorithms store per-key state in a JavaScript Map:

private windows = new Map<string, WindowState>();  // sliding window
private buckets = new Map<string, BucketState>();   // token bucket

Keys are added on first consume() call. Keys are never removed (except via explicit reset()).

If you're rate limiting by IP address and your API serves 100,000 unique IPs per day, after 30 days you have 3 million entries in the Map. Each entry is small (~24-40 bytes of data, plus the Map overhead and the key string), but it adds up. More importantly, it never stops growing.

This is the kind of bug that doesn't show up in benchmarks, doesn't show up in tests, and doesn't show up in the first week of production. It shows up when your ops team notices the Node.js process is using 2GB of RAM and climbing.

The fix: Lazy eviction. Every 1,000 consume() calls, we scan up to 500 keys and remove the ones that have expired:

const EVICTION_INTERVAL = 1000;
const EVICTION_BATCH = 500;

export class SlidingWindow {
  private callCount = 0;

  consume(key: string): RateLimitResult {
    const now = Date.now();

    if (++this.callCount >= EVICTION_INTERVAL) {
      this.callCount = 0;
      this.evict(now);
    }
    // ... rest of consume logic
  }

  private evict(now: number): void {
    const expiry = this.windowMs * 2;
    let scanned = 0;
    for (const [key, state] of this.windows) {
      if (scanned++ >= EVICTION_BATCH) break;
      if (now - state.currStart >= expiry) {
        this.windows.delete(key);
      }
    }
  }
}

The same pattern applies to Token Bucket — evict keys that have been idle long enough to have fully refilled (meaning they're back to their initial state and can be safely recreated on next access).

Why lazy eviction instead of a setInterval timer?

No timer overhead — no background work when the limiter is idle
Amortized cost — the eviction work is spread across consume calls
Bounded per-call cost — scanning at most 500 keys per pass keeps the hot path fast
No concurrency issues — everything runs synchronously in the consume path

The tradeoff is that memory isn't freed immediately when keys expire. But with a 1:1000 ratio (one eviction pass per 1,000 calls), stale keys don't accumulate meaningfully.

Lesson: If you're storing per-key state in memory, you need an eviction strategy. This applies to caches, rate limiters, session stores — anything with unbounded key cardinality. The simplest approach is often the best: piggyback on existing operations and do a little cleanup each time.

Bug #4: The `any` That Defeated TypeScript

The problem: The Redis client was typed as any:

export interface RedisStoreOptions extends RateLimiterOptions {
  client: any;  // 🤷
}

This means you could pass literally anything — a string, a number, null, a random object — and TypeScript would happily compile it. The error would only surface at runtime when the code tries to call client.eval().

We originally used any because ioredis is an optional peer dependency. We didn't want to import its types and create a hard dependency. But any is never the answer.

The fix: Define a minimal structural interface that describes what we actually need from the client:

export interface RedisLike {
  eval(...args: any[]): Promise<any>;
  del(...keys: string[]): Promise<number>;
}

export interface RedisStoreOptions extends RateLimiterOptions {
  client: RedisLike;
}

This is TypeScript's structural typing at its best. Any object with eval() and del() methods will satisfy the interface — ioredis, the redis package, a mock object in tests, or a custom wrapper. No import needed, no hard dependency, but you get compile-time checking that catches the obvious mistakes.

Lesson: any is a code smell. When you're tempted to use it because you don't want a hard dependency on a specific library, define a structural interface instead. Describe the shape you need, not the specific implementation. This is one of TypeScript's superpowers — use it.

Wrapping Up

Four bugs, four different categories:

#	Bug	Category	When It Bites
1	Missing try/catch in middleware	Error handling	Redis goes down in production
2	Random collision in ZADD	Distributed systems math	High throughput (>1K req/sec/key)
3	No key eviction	Memory management	After days/weeks of uptime
4	`any` typed Redis client	Type safety	Passing wrong object at integration time

None of these showed up in our test suite. The tests all passed. The benchmarks looked great. The code was clean and well-structured.

That's the thing about these kinds of bugs — they live in the gaps between what you test and what production actually does. They require thinking about failure modes (what if Redis dies?), mathematical edge cases (what's the collision probability?), long-running behavior (what happens after a month?), and developer experience (what happens when someone passes the wrong thing?).

If you're building a library, especially one that sits in the critical path of every HTTP request, these are the questions worth asking before your users ask them for you.

All four fixes shipped in v1.1.0. The project is MIT licensed and available on GitHub and npm.

Five Bugs That Would Have Killed TitanMQ in Production

Bill Tu — Wed, 08 Apr 2026 06:12:19 +0000

We recently ran a deep audit of TitanMQ's codebase and found five bugs that ranged from "data loss on restart" to "unbounded disk growth." None of them showed up in unit tests. All of them would have been catastrophic in production.

This post walks through each bug, why it existed, and how we fixed it.

Bug 1: The Broker That Forgot Everything on Restart

Issue: #5 — CommitLog has no recovery on restart

The original CommitLog constructor looked like this:

public CommitLog(Path directory, long maxSegmentSize) throws IOException {
    this.directory = directory;
    this.maxSegmentSize = maxSegmentSize;
    this.directory.toFile().mkdirs();
    rollNewSegment();  // Always starts fresh at offset 0
}

Every time the broker restarted, it created a new segment starting at offset 0. The old segment files were still on disk, but the broker had no idea they existed. Consumers would see an empty topic. Producers would get offset 0 for their next message. Months of data, invisible.

The Fix

On startup, scan the data directory for existing .log files, sort them by base offset (the filename is the zero-padded offset), and recover state from each one:

private void recover() throws IOException {
    List<Path> segmentFiles = new ArrayList<>();
    if (Files.exists(directory)) {
        try (DirectoryStream<Path> stream = Files.newDirectoryStream(directory, "*.log")) {
            for (Path path : stream) {
                segmentFiles.add(path);
            }
        }
    }

    if (segmentFiles.isEmpty()) {
        rollNewSegment();
        return;
    }

    segmentFiles.sort(Comparator.comparing(p -> p.getFileName().toString()));

    for (Path segFile : segmentFiles) {
        String fileName = segFile.getFileName().toString();
        long baseOffset = Long.parseLong(fileName.replace(".log", ""));
        LogSegment segment = new LogSegment(directory, baseOffset, maxSegmentSize, flushPolicy);
        segment.recover();
        segments.add(segment);
    }

    LogSegment lastSegment = segments.getLast();
    if (lastSegment.endOffset() >= 0) {
        nextOffset.set(lastSegment.endOffset() + 1);
    } else {
        nextOffset.set(lastSegment.baseOffset());
    }
    activeSegment = lastSegment;
}

Each LogSegment also needed a recover() method that scans the file entry by entry, rebuilds the in-memory index, and handles the edge case where the broker crashed mid-write (a partial entry at the end of the file):

void recover() throws IOException {
    long fileSize = channel.size();
    long pos = 0;
    index.clear();

    while (pos + ENTRY_HEADER_SIZE <= fileSize) {
        ByteBuffer header = ByteBuffer.allocate(ENTRY_HEADER_SIZE);
        int bytesRead = channel.read(header, pos);
        if (bytesRead < ENTRY_HEADER_SIZE) break;
        header.flip();

        long offset = header.getLong();
        int messageSize = header.getInt();

        if (messageSize <= 0 || pos + ENTRY_HEADER_SIZE + messageSize > fileSize) {
            // Partial write from crash — truncate here
            channel.truncate(pos);
            break;
        }

        index.add(new OffsetIndex(offset, pos));
        endOffset = offset;
        pos += ENTRY_HEADER_SIZE + messageSize;
    }
    currentPosition = pos;
}

The truncation of partial entries is important. If the broker crashed between writing the header and the payload, we'd have a valid-looking header pointing to garbage data. By checking pos + ENTRY_HEADER_SIZE + messageSize > fileSize, we detect this and truncate the file to the last complete entry.

Bug 2: Messages That Survived the JVM but Not the OS

Issue: #6 — No fsync, messages can be lost on OS crash

The original LogSegment.append() called channel.write() and moved on. Java's FileChannel.write() writes to the OS page cache, not to disk. If the JVM crashes, the OS will eventually flush the pages. But if the OS crashes (kernel panic, power loss), those pages are gone.

This is the classic fsync trade-off that every storage system has to make. Kafka defaults to relying on the OS page cache (no explicit fsync) and compensates with replication. But for a single-node deployment or when you need strong durability guarantees, you need fsync.

The Fix

We introduced a FlushPolicy enum with three options:

public enum FlushPolicy {
    EVERY_MESSAGE,  // fsync after each append — safest, slowest
    PERIODIC,       // fsync every N milliseconds — good balance
    NONE            // rely on OS page cache — fastest, least safe
}

In LogSegment.append():

public synchronized void append(long offset, ByteBuffer data) throws IOException {
    // ... write header and data ...

    if (flushPolicy == FlushPolicy.EVERY_MESSAGE) {
        channel.force(false);
    }
}

For PERIODIC mode, CommitLog runs a background thread:

if (flushPolicy == FlushPolicy.PERIODIC && flushIntervalMs > 0) {
    scheduler.scheduleAtFixedRate(this::periodicFlush,
            flushIntervalMs, flushIntervalMs, TimeUnit.MILLISECONDS);
}

private void periodicFlush() {
    try {
        if (activeSegment != null) {
            activeSegment.flush();
        }
    } catch (IOException e) {
        log.error("Periodic flush failed", e);
    }
}

The channel.force(false) call is fdatasync — it flushes data but not file metadata (size, timestamps). This is faster than force(true) (fsync) and sufficient for our use case since we track file position in memory.

The performance impact is significant. On a typical SSD, EVERY_MESSAGE reduces throughput by roughly 10x compared to NONE. PERIODIC with a 1-second interval is a good middle ground: you lose at most 1 second of data on an OS crash, but throughput stays close to NONE.

Bug 3: The Raft Cluster That Couldn't Remember Its Own Term

Issue: #7 — Raft log is in-memory only

This was the scariest bug. The entire Raft state — currentTerm, votedFor, and all log entries — was stored in an ArrayList in memory. When a broker restarted, it came back as a blank slate at term 0 with an empty log.

The Raft paper (§5.2) is explicit: currentTerm and votedFor must be persisted to stable storage before responding to any RPC. Without this:

A node could vote for candidate A, restart, then vote for candidate B in the same term — violating election safety
Committed entries would vanish, violating state machine safety
A restarted node would start an election at term 0, potentially disrupting a healthy cluster

The Fix

We created RaftPersistence with two files:

{dataDir}/raft-meta    — currentTerm (8B) + votedFor
{dataDir}/raft-log     — sequential log entries

The meta file uses atomic write-then-rename to prevent corruption:

public void saveMeta(long currentTerm, String votedFor) throws IOException {
    byte[] votedForBytes = votedFor != null ? votedFor.getBytes() : new byte[0];
    ByteBuffer buf = ByteBuffer.allocate(8 + 4 + votedForBytes.length);
    buf.putLong(currentTerm);
    buf.putInt(votedForBytes.length);
    if (votedForBytes.length > 0) buf.put(votedForBytes);
    buf.flip();

    Path tmpFile = metaFile.resolveSibling("raft-meta.tmp");
    try (FileChannel ch = FileChannel.open(tmpFile,
            StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.TRUNCATE_EXISTING)) {
        ch.write(buf);
        ch.force(true);  // fsync before rename
    }
    Files.move(tmpFile, metaFile,
            StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
}

The write-to-temp-then-rename pattern is critical. If we wrote directly to raft-meta and crashed mid-write, we'd have a corrupted file. With the rename approach, the file is either the old version or the new version — never a partial write.

Every place in RaftNode where currentTerm or votedFor changes now calls persistMeta() before proceeding:

startElection() — increments term, votes for self
stepDown() — updates term, clears votedFor
handleVoteRequest() — may update term and grant vote

On startup, RaftNode.start() loads the persisted state before joining the cluster:

public void start() {
    if (persistence != null) {
        RaftPersistence.MetaState meta = persistence.loadMeta();
        if (meta != null) {
            currentTerm.set(meta.currentTerm());
            votedFor = meta.votedFor();
        }
        List<RaftLog.LogEntry> entries = persistence.loadLogEntries();
        for (RaftLog.LogEntry entry : entries) {
            raftLog.append(entry.term(), entry.command());
        }
    }
    // ... start election timer ...
}

Bug 4: The Lock-Free Counter That Wasn't

Issue: #8 — BackPressureController race condition

The original tryAcquire():

public boolean tryAcquire() {
    long current = inFlightCount.incrementAndGet();  // Already incremented!
    if (current > highWaterMark) {
        inFlightCount.decrementAndGet();  // Undo
        return false;
    }
    return true;
}

The problem: incrementAndGet() is atomic, but the increment happens before the check. If 10 threads call tryAcquire() simultaneously when the count is at highWaterMark - 1, all 10 will increment the counter (pushing it to highWaterMark + 9), then all 10 will see current > highWaterMark and decrement. The counter ends up correct, but for a brief moment it exceeded the watermark — and during that moment, memory pressure could spike.

More importantly, the pattern is just wrong. The intent is "only increment if below the limit," but the code increments unconditionally and then checks.

The Fix

A proper CAS (Compare-And-Swap) loop:

public boolean tryAcquire() {
    while (true) {
        long current = inFlightCount.get();
        if (current >= highWaterMark) {
            if (!throttled) {
                throttled = true;
                log.warn("Back-pressure activated: in-flight={}, highWaterMark={}",
                         current, highWaterMark);
            }
            return false;
        }
        if (inFlightCount.compareAndSet(current, current + 1)) {
            return true;
        }
        // CAS failed — another thread modified the count, retry
    }
}

This guarantees the counter never exceeds highWaterMark. The compareAndSet only succeeds if no other thread has modified the value since we read it. If it fails, we re-read and try again. Under low contention, this completes in one iteration. Under high contention, it may take a few retries, but correctness is guaranteed.

The performance difference is negligible — both approaches are lock-free and operate on a single AtomicLong. But the CAS version is correct.

Bug 5: The Disk That Never Stopped Growing

Issue: #9 — No segment cleanup

The original CommitLog created new segments when old ones filled up, but never deleted anything. On a broker processing 100K messages/second with 1KB messages, that's roughly 100MB per second, 8.6TB per day. Without cleanup, the broker would fill any disk in hours.

The Fix

Two retention policies, configurable independently:

// Delete segments older than 7 days
commitLog.retentionTime(Duration.ofDays(7));

// Or: keep total size under 100GB
commitLog.retentionBytes(100L * 1024 * 1024 * 1024);

The cleanup runs on a background thread every 60 seconds:

synchronized void cleanupExpiredSegments() {
    if (segments.size() <= 1) return;  // Never delete the active segment

    List<LogSegment> toDelete = new ArrayList<>();

    // Time-based
    if (retentionMs > 0) {
        for (int i = 0; i < segments.size() - 1; i++) {
            LogSegment seg = segments.get(i);
            long lastModified = Files.getLastModifiedTime(seg.filePath()).toMillis();
            if (System.currentTimeMillis() - lastModified > retentionMs) {
                toDelete.add(seg);
            }
        }
    }

    // Size-based
    if (retentionBytes > 0) {
        long totalSize = segments.stream().mapToLong(LogSegment::size).sum();
        for (int i = 0; i < segments.size() - 1 && totalSize > retentionBytes; i++) {
            LogSegment seg = segments.get(i);
            if (!toDelete.contains(seg)) toDelete.add(seg);
            totalSize -= seg.size();
        }
    }

    for (LogSegment seg : toDelete) {
        seg.delete();
        segments.remove(seg);
    }
}

The key safety invariant: we never delete the last segment (the active one). The loop runs i < segments.size() - 1 to enforce this. Even if both retention policies say "delete everything," the active segment survives.

LogSegment.delete() closes the file handles before deleting:

public void delete() throws IOException {
    close();
    if (!filePath.toFile().delete()) {
        log.warn("Failed to delete segment file {}", filePath);
    }
}

What We Learned

These five bugs share a common theme: they all involve state that lives beyond a single JVM process lifetime. In-memory data structures are easy to reason about, but the moment you need durability, recovery, or bounded resource usage, you need explicit code to handle it.

The fixes added roughly 500 lines of code across 7 files. None of them changed the public API. All of them were invisible to users — until the broker crashed, restarted, or ran for more than a few hours.

If you're building a storage system, here's the checklist we wish we'd had from day one:

What happens on restart? If your constructor creates fresh state, you've lost everything.
What happens on OS crash? If you're not calling fsync, your data is in the page cache, not on disk.
What happens after a week? If you're not deleting old data, you're filling the disk.
What happens under contention? If your "atomic" operation has a check-then-act gap, it's not atomic.
What state must survive a restart? If it's in a volatile field, it won't.

All five fixes are in commit ab0578d. Issues #5, #6, #7, #8, #9 are closed.

GitHub: https://github.com/iwtxokhtd83/TitanMQ

Four Bugs That Would Break a Matching Engine in Production

Bill Tu — Wed, 08 Apr 2026 05:37:40 +0000

MatchEngine is an open-source order matching engine written in Go. After the initial release, we opened issues for every defect we could find through code review. This article covers the four bugs we fixed in v0.2.0, why each one matters, and the exact code changes that resolved them.

These are not exotic edge cases. They are the kind of bugs that survive code review, pass unit tests, and then blow up under real traffic.

Bug #1: Duplicate Order IDs Corrupt the Book

The Problem

The engine accepted any order ID provided by the caller. There was no uniqueness check. Submitting two orders with the same ID caused both to exist in the book:

e.SubmitOrder("BTC", model.NewLimitOrder("dup", model.Sell, d("100"), d("5")))
e.SubmitOrder("BTC", model.NewLimitOrder("dup", model.Sell, d("200"), d("10")))
// Both orders now live in the book with ID "dup"

When CancelOrder("BTC", "dup") was called, the old RemoveOrder did a linear scan and stopped at the first match. The second order became an orphan — still in the book, still matchable, but invisible to cancellation.

Worse, if a downstream system used order IDs as keys (for position tracking, risk management, or reconciliation), the duplicate would silently corrupt its state.

The Fix

We added an order index to the engine — a map[string]string mapping order ID to symbol:

type Engine struct {
    mu         sync.Mutex
    books      map[string]*orderbook.OrderBook
    orderIndex map[string]string  // order ID -> symbol
    // ...
}

On every SubmitOrder, the engine checks the index before proceeding:

if _, exists := e.orderIndex[order.ID]; exists {
    return nil, fmt.Errorf("duplicate order ID: %s", order.ID)
}

When an order enters the book, it is registered in the index. When it is filled or cancelled, it is removed:

// On add to book
e.orderIndex[order.ID] = symbol

// On cancel
if book.RemoveOrder(orderID) {
    delete(e.orderIndex, orderID)
    return true
}

// On fill (during cleanup)
if o.IsFilled() {
    delete(e.orderIndex, o.ID)
}

This means order IDs can be reused after an order is fully filled or cancelled — a deliberate design choice. The ID lifecycle is: available → active (in book) → available (after fill or cancel).

We also upgraded the order book itself to maintain an internal orders map[string]*model.Order index, which gives RemoveOrder O(1) lookup instead of the previous O(n) linear scan:

type OrderBook struct {
    Bids   []*model.Order
    Asks   []*model.Order
    orders map[string]*model.Order  // O(1) lookup by ID
}

func (ob *OrderBook) RemoveOrder(orderID string) bool {
    order, ok := ob.orders[orderID]
    if !ok {
        return false
    }
    delete(ob.orders, orderID)
    if order.Side == model.Buy {
        ob.Bids = removeFromSlice(ob.Bids, orderID)
    } else {
        ob.Asks = removeFromSlice(ob.Asks, orderID)
    }
    return true
}

Because we know which side the order is on from the map lookup, we only scan one slice instead of both. The map also eliminates the ambiguity of the old code — there is exactly one entry per ID, always.

The Tests

func TestDuplicateOrderID(t *testing.T) {
    e := New()
    e.SubmitOrder("BTC", model.NewLimitOrder("dup", model.Sell, d("100"), d("5")))

    _, err := e.SubmitOrder("BTC", model.NewLimitOrder("dup", model.Sell, d("200"), d("10")))
    if err == nil {
        t.Error("expected error for duplicate order ID")
    }
}

func TestDuplicateOrderIDAfterFill(t *testing.T) {
    e := New()
    e.SubmitOrder("BTC", model.NewLimitOrder("reuse", model.Sell, d("100"), d("5")))
    e.SubmitOrder("BTC", model.NewLimitOrder("buyer", model.Buy, d("100"), d("5")))

    // Filled ID should be reusable
    _, err := e.SubmitOrder("BTC", model.NewLimitOrder("reuse", model.Sell, d("100"), d("3")))
    if err != nil {
        t.Errorf("expected reuse to succeed, got: %v", err)
    }
}

Bug #2: GetOrderBook Leaks Internal State

The Problem

GetOrderBook returned a direct pointer to the engine's internal order book:

func (e *Engine) GetOrderBook(symbol string) *orderbook.OrderBook {
    e.mu.Lock()
    defer e.mu.Unlock()
    return e.books[symbol]  // returns the real thing
}

The mutex is released as soon as the method returns. The caller now holds a raw pointer to mutable shared state with no lock protection. Any read from another goroutine races with writes from SubmitOrder:

// Goroutine 1: reading the book
book := e.GetOrderBook("BTC")
for _, ask := range book.Asks {  // iterating over a slice that may be
    fmt.Println(ask.Price)        // modified concurrently
}

// Goroutine 2: submitting an order (modifies book.Asks)
e.SubmitOrder("BTC", someOrder)

This is a textbook data race. Under Go's race detector (go test -race), this would be flagged immediately. In production without the race detector, it manifests as corrupted reads, panics from slice index out of range, or silently wrong data.

The Fix

GetOrderBook now returns a deep-copy snapshot:

func (e *Engine) GetOrderBook(symbol string) *orderbook.OrderBook {
    symbol = normalizeSymbol(symbol)
    e.mu.Lock()
    defer e.mu.Unlock()

    book, ok := e.books[symbol]
    if !ok {
        return nil
    }
    return book.Snapshot()
}

The Snapshot method on OrderBook creates a completely independent copy:

func (ob *OrderBook) Snapshot() *OrderBook {
    snap := &OrderBook{
        Bids:   make([]*model.Order, len(ob.Bids)),
        Asks:   make([]*model.Order, len(ob.Asks)),
        orders: make(map[string]*model.Order),
    }
    for i, o := range ob.Bids {
        cp := *o          // copy the Order struct by value
        snap.Bids[i] = &cp
        snap.orders[cp.ID] = &cp
    }
    for i, o := range ob.Asks {
        cp := *o
        snap.Asks[i] = &cp
        snap.orders[cp.ID] = &cp
    }
    return snap
}

Key detail: we copy each *model.Order by dereferencing the pointer (cp := *o), which creates a new Order value on the stack, then take its address. This ensures the snapshot's orders are completely independent from the engine's orders. Mutating snap.Asks[0].Remaining does not touch the engine's internal state.

The Tests

func TestGetOrderBookReturnsSnapshot(t *testing.T) {
    e := New()
    e.SubmitOrder("SNAP", model.NewLimitOrder("s1", model.Sell, d("100"), d("10")))

    snap := e.GetOrderBook("SNAP")
    snap.Asks[0].Remaining = d("999")  // mutate the snapshot

    snap2 := e.GetOrderBook("SNAP")
    if snap2.Asks[0].Remaining.Equal(d("999")) {
        t.Error("snapshot mutation affected internal book")
    }
}

func TestGetOrderBookConcurrentAccess(t *testing.T) {
    e := New()
    e.SubmitOrder("CONC", model.NewLimitOrder("s1", model.Sell, d("100"), d("10")))

    var wg sync.WaitGroup
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            book := e.GetOrderBook("CONC")
            _ = book.Depth()
            _ = book.Spread()
        }()
    }
    wg.Wait()
}

The concurrent test spawns 100 goroutines all reading the order book simultaneously. With the old code, this would trigger the race detector. With snapshots, each goroutine gets its own independent copy.

Bug #3: Unbounded Trade Log Leaks Memory

The Problem

Every executed trade was appended to e.trades and never removed:

e.trades = append(e.trades, trades...)

GetTrades() copied the entire slice on every call:

func (e *Engine) GetTrades() []model.Trade {
    e.mu.Lock()
    defer e.mu.Unlock()
    result := make([]model.Trade, len(e.trades))
    copy(result, e.trades)
    return result
}

For a long-running engine processing thousands of trades per second, this is a memory leak. After a million trades, the slice holds a million Trade structs in memory permanently. GetTrades() allocates another million-element slice on every call. Eventually, the process runs out of memory.

The Fix

We introduced two mechanisms: a bounded in-memory log with eviction, and a callback for real-time trade processing.

The engine now accepts functional options:

e := engine.New(
    engine.WithMaxTradeLog(5000),
    engine.WithTradeHandler(func(symbol string, trade model.Trade) {
        // persist to database, publish to Kafka, etc.
    }),
)

The trade recording logic:

for _, t := range trades {
    if e.onTrade != nil {
        e.onTrade(symbol, t)
    }
    if e.maxTrades > 0 {
        if len(e.trades) >= e.maxTrades {
            evict := e.maxTrades / 10
            if evict < 1 {
                evict = 1
            }
            e.trades = e.trades[evict:]
        }
        e.trades = append(e.trades, t)
    }
}

Three modes of operation:

Configuration	Behavior
Default (`maxTrades=10000`)	Keeps last ~10,000 trades, evicts oldest 10% when full
`WithMaxTradeLog(0)`	No in-memory log at all. Use `WithTradeHandler` for processing.
`WithTradeHandler(fn)`	Callback invoked for every trade, regardless of log setting

The eviction strategy removes 10% at a time rather than one-by-one. This amortizes the cost of slice shifting — removing one element from the front of a 10,000-element slice requires shifting 9,999 elements. Removing 1,000 at once means we only pay that cost every 1,000 trades instead of every trade.

The callback runs inside the mutex, which is intentional. It guarantees that the handler sees trades in order and that no concurrent SubmitOrder can interleave. If the handler needs to do slow I/O, it should buffer internally and process asynchronously.

The Tests

func TestTradeLogBounded(t *testing.T) {
    e := New(WithMaxTradeLog(10))

    for i := 0; i < 20; i++ {
        sid := fmt.Sprintf("s%d", i)
        bid := fmt.Sprintf("b%d", i)
        e.SubmitOrder("BND", model.NewLimitOrder(sid, model.Sell, d("100"), d("1")))
        e.SubmitOrder("BND", model.NewLimitOrder(bid, model.Buy, d("100"), d("1")))
    }

    trades := e.GetTrades()
    if len(trades) > 10 {
        t.Errorf("expected at most 10 trades, got %d", len(trades))
    }
}

func TestTradeLogDisabled(t *testing.T) {
    e := New(WithMaxTradeLog(0))
    e.SubmitOrder("DIS", model.NewLimitOrder("s1", model.Sell, d("100"), d("1")))
    e.SubmitOrder("DIS", model.NewLimitOrder("b1", model.Buy, d("100"), d("1")))

    if len(e.GetTrades()) != 0 {
        t.Error("expected empty trade log when disabled")
    }
}

func TestTradeHandler(t *testing.T) {
    var received []model.Trade
    e := New(WithTradeHandler(func(symbol string, trade model.Trade) {
        received = append(received, trade)
    }))

    e.SubmitOrder("CB", model.NewLimitOrder("s1", model.Sell, d("100"), d("5")))
    e.SubmitOrder("CB", model.NewLimitOrder("b1", model.Buy, d("100"), d("3")))

    if len(received) != 1 || !received[0].Quantity.Equal(d("3")) {
        t.Error("trade handler did not receive expected trade")
    }
}

Bug #4: No Symbol Validation

The Problem

The engine accepted any string as a symbol, including empty strings, whitespace, and mixed-case variants:

e.SubmitOrder("", order)          // creates a book keyed by ""
e.SubmitOrder("   ", order)       // creates a book keyed by "   "
e.SubmitOrder("btc/usd", order)   // different book from "BTC/USD"
e.SubmitOrder("BTC/USD", order)   // different book from "btc/usd"

A user typing "btc/usd" and "BTC/USD" would unknowingly place orders in two separate books. They would never match against each other. No error, no warning — just silent failure.

The Fix

Three layers of defense:

1. Normalization: Every symbol is uppercased and trimmed before use.

func normalizeSymbol(s string) string {
    return strings.ToUpper(strings.TrimSpace(s))
}

This is applied in SubmitOrder, CancelOrder, and GetOrderBook. The caller can pass "btc/usd", " BTC/USD ", or "BTC/USD" — they all resolve to the same book.

2. Empty rejection: After normalization, empty strings are rejected.

func (e *Engine) validateSymbol(symbol string) error {
    if symbol == "" {
        return fmt.Errorf("symbol cannot be empty")
    }
    if e.symbols != nil && !e.symbols[symbol] {
        return fmt.Errorf("symbol %q is not registered", symbol)
    }
    return nil
}

3. Optional registration: For strict environments, symbols can be pre-registered.

e := engine.New()
e.RegisterSymbol("BTC/USD")
e.RegisterSymbol("ETH/USD")

// This works
e.SubmitOrder("BTC/USD", order)

// This returns an error: symbol "DOGE/USD" is not registered
e.SubmitOrder("DOGE/USD", order)

Registration is opt-in. If no symbols are registered, the engine accepts any non-empty symbol (the default behavior for development and testing). Once the first symbol is registered, strict mode activates.

The Tests

func TestEmptySymbolRejected(t *testing.T) {
    e := New()
    _, err := e.SubmitOrder("", model.NewLimitOrder("s1", model.Sell, d("100"), d("5")))
    if err == nil {
        t.Error("expected error for empty symbol")
    }
    _, err = e.SubmitOrder("   ", model.NewLimitOrder("s2", model.Sell, d("100"), d("5")))
    if err == nil {
        t.Error("expected error for whitespace-only symbol")
    }
}

func TestSymbolNormalization(t *testing.T) {
    e := New()
    e.SubmitOrder("btc/usd", model.NewLimitOrder("s1", model.Sell, d("100"), d("5")))
    trades, _ := e.SubmitOrder("BTC/USD", model.NewLimitOrder("b1", model.Buy, d("100"), d("3")))
    if len(trades) != 1 {
        t.Fatal("expected match across normalized symbols")
    }
}

func TestRegisteredSymbolsOnly(t *testing.T) {
    e := New()
    e.RegisterSymbol("BTC/USD")

    _, err := e.SubmitOrder("DOGE/USD", model.NewLimitOrder("s1", model.Sell, d("1"), d("100")))
    if err == nil {
        t.Error("expected error for unregistered symbol")
    }
}

The Pattern

Looking at these four bugs together, a pattern emerges. Each one is a boundary problem — a place where the engine's internal state meets the outside world:

Bug	Boundary
Duplicate IDs	Caller-provided identifiers entering the engine
Leaked pointer	Internal state leaving the engine
Unbounded log	Time (state accumulating without limit)
No validation	Caller-provided symbols entering the engine

The fixes follow a consistent strategy: validate at the boundary, copy across the boundary, and bound what accumulates inside.

These are not matching-engine-specific lessons. They apply to any stateful system that accepts external input and exposes internal state. The matching algorithm itself — the price-time priority loop — was never the problem. The problems were all in the plumbing around it.

Summary

Issue	Problem	Fix	Tests Added
#2	Duplicate order IDs	Order index map + rejection	3
#3	GetOrderBook leaks pointer	Deep-copy snapshot	2
#5	Unbounded trade log	Bounded log + callback	3
#10	No symbol validation	Normalize + validate + optional registration	3

Total: 11 new tests, 0 changes to the matching algorithm.

GitHub: https://github.com/iwtxokhtd83/MatchEngine
Release: v0.2.0