System Design: Stock Surveillance System

A capstone system design walkthrough — designing a market/stock surveillance system end to end — covering real-time ingestion of order and trade data, the alert pipeline that detects manipulative and anomalous trading patterns, the case management workflow that turns alerts into investigations, replay and backtesting against historical data, and the specific correctness, latency, and regulatory demands that make surveillance a uniquely unforgiving system design problem.

---

Table of Contents

1. Introduction
2. Why Stock Surveillance Is a Different Kind of Hard
3. The Core Domain Model
4. The Market Data Log: Ordered, Immutable History as the Source of Truth
5. Ingestion: Normalizing and Time-Ordering Multi-Venue Data
6. Detecting Patterns: The Alert Engine
7. The Alert State Machine and Case Management
8. Entity Resolution: Linking Orders to Real Actors
9. Streaming vs. Batch: Coordinating Real-Time and Historical Detection
10. Backtesting and Replay
11. Tuning Detection: Precision, Recall, and Analyst Trust
12. Data Security and Compliance
13. Consistency, Availability, and the CAP Trade-off for Surveillance
14. Scaling the System
15. Observability for a Surveillance System
16. Common Pitfalls
17. Quick Reference Table
18. Conclusion

---

Introduction

A stock surveillance system takes the general system design vocabulary covered in this series' System Design guide — streaming ingestion, event logs, rules engines, case management — and applies it to a domain where the ordinary consequences of a missed detection or a false one are dramatically higher than most systems tolerate: a missed instance of spoofing or insider trading is a genuine regulatory failure with legal exposure, and a flood of false positives buries analysts and erodes trust in the system entirely. This guide walks through designing such a system end to end, drawing directly on this series' Event-Driven Architecture, DDD, Stream Processing, and Data Retention guides, each of which turns out to be load-bearing infrastructure for getting surveillance right rather than optional architectural polish.

Exchange Feeds → Ingestion/Normalization → Market Data Log (source of truth) → Alert Engine (streaming + batch)
                                                                                        ↓
                                                                              Alert Queue → Case Management → Analyst

---

1. Why Stock Surveillance Is a Different Kind of Hard

The cost of a miss and the cost of a false alarm are both genuinely high

Most systems covered in this series can tolerate an occasional wrong decision with a bounded, recoverable cost — a mis-ranked search result, a slightly stale recommendation. A surveillance system's failure modes sit on two sides of a much sharper trade-off: missing a real instance of manipulation (spoofing, layering, wash trading, insider trading ahead of an announcement) is a regulatory and reputational failure that can trigger fines or sanctions, while flagging too much noise means human analysts — a fundamentally limited, expensive resource — drown in alerts and stop trusting, and eventually stop carefully reviewing, the system's output. This is why detection tuning (Section 10) and case management (Section 6) get as much design attention in this guide as raw ingestion throughput does.

You must reconstruct market state as it actually was, not as it is now

An order book snapshot queried "live" reflects the CURRENT state.
An investigation into an event three weeks ago needs the book EXACTLY as it stood
  at that millisecond — reconstructed from history, not approximated from what's cached today.

Unlike most systems in this series where "current state" is what matters, surveillance is fundamentally retrospective and evidentiary — every alert, and every investigation that follows it, must be reconstructable from immutable historical data with the same precision available at the time, which is precisely why the append-only market data log (Section 3) is this system's true foundation, more so than any live dashboard sitting on top of it.

You are almost never the only source of truth for what actually happened

A critical, freeing realization for the design that follows: a surveillance system, in the overwhelming majority of real-world designs, does not generate the trading activity it watches — it ingests order, trade, and quote data from exchanges and internal order management systems, and its job is to detect patterns in that data, correlate it with reference data (accounts, traders, related entities), and hand well-supported cases to humans — not to adjudicate whether misconduct occurred, which remains a human, and often legal, determination. This mirrors the "don't reimplement what the specialist system already provides" guidance echoed in this series' API Integration and Data Pipeline guides, applied here to exchange connectivity and market data.

---

2. The Core Domain Model

Modeled with DDD, per this series' companion guide

public record OrderId(string Value);
public record InstrumentId(string Symbol, string Venue);
public record TraderId(Guid Value);

public enum AlertStatus { Open, UnderReview, Escalated, Closed, FalsePositive }

public class SurveillanceAlert // the AGGREGATE ROOT, per this series' DDD guide
{
    public Guid AlertId { get; }
    public string PatternType { get; }        // e.g. "Spoofing", "Layering", "WashTrade"
    public IReadOnlyList<OrderId> RelatedOrders { get; }
    public TraderId SubjectTrader { get; }
    public AlertStatus Status { get; private set; }
    private readonly List<AlertEvent> _domainEvents = new();

    public void AssignForReview(string analystId)
    {
        if (Status != AlertStatus.Open)
            throw new InvalidOperationException($"Cannot assign an alert in status {Status}");
        Status = AlertStatus.UnderReview;
        _domainEvents.Add(new AlertAssignedEvent(AlertId, analystId));
    }

    public void Close(string resolution, string analystId)
    {
        if (Status != AlertStatus.UnderReview)
            throw new InvalidOperationException($"Cannot close an alert in status {Status}");
        Status = resolution == "false_positive" ? AlertStatus.FalsePositive : AlertStatus.Closed;
        _domainEvents.Add(new AlertClosedEvent(AlertId, resolution, analystId));
    }
}

This directly applies this series' DDD guide's aggregate pattern — SurveillanceAlert is the aggregate root, enforcing its own state transitions (an alert cannot be closed before it's under review) rather than trusting every caller to check status before mutating it, and raising domain events at exactly the points those transitions genuinely occur.

The order/trade event as a distinct, immutable value object

public record OrderEvent(
    OrderId OrderId, InstrumentId Instrument, TraderId Trader,
    OrderEventType Type, // New, Modify, Cancel, Fill
    decimal Price, long Quantity, DateTimeOffset ExchangeTimestamp, DateTimeOffset IngestTimestamp);

As covered in this series' DDD guide's value object discussion, modeling each order lifecycle event as an immutable value — never mutated once ingested — is what makes the entire downstream system (Section 3's log, Section 5's alert engine) trustworthy: the raw evidentiary record is never touched again after ingestion, only interpreted.

---

3. The Market Data Log: Ordered, Immutable History as the Source of Truth

Why a "current order book" view alone is insufficient

-- ❌ Only ever knowing the CURRENT book state has no path back to "what did the book look like at 09:31:04.223"
UPDATE order_book SET quantity = quantity - 100 WHERE order_id = 'X';

A surveillance system needs more than "what does the book look like now" — it needs an immutable, precisely time-ordered record of every order, modification, cancellation, and fill that ever occurred, and the ability to reconstruct book state (and detect patterns) as of any historical instant. A mutable "current state" table, updated in place, destroys exactly the history an investigation depends on.

The log as the append-only, time-ordered backbone

CREATE TABLE market_data_log (
    sequence_id BIGINT PRIMARY KEY,       -- strictly increasing, per-instrument-partition
    instrument_id VARCHAR NOT NULL,       -- partition key: keeps one instrument's events strictly ordered
    exchange_timestamp TIMESTAMPTZ NOT NULL,  -- the timestamp that matters for reconstruction and evidence
    ingest_timestamp TIMESTAMPTZ NOT NULL,    -- when WE received it — used to detect ingestion delay, not for reconstruction
    event_type VARCHAR NOT NULL,
    payload JSONB NOT NULL
);

In practice this table's role is usually filled by a distributed log (Kafka/Pulsar) rather than a relational table directly — every ingested market event is first durably appended to the log, partitioned by instrument, before any alert logic runs against it. This gives a durable, replayable record (rebuild any historical order book from scratch, exactly, for an investigation or a backtest), natural per-instrument ordering (single partition per instrument = strict order for reconstructing that instrument's book), and a backbone for downstream consumers via the outbox/CDC pattern, directly echoing this series' Event-Driven Architecture guide's discussion of avoiding dual-write inconsistency between "update derived state" and "publish the event."

Exchange timestamp vs. ingest timestamp — a distinction that matters more here than almost anywhere else

Exchange timestamp: when the event ACTUALLY happened, per the venue's own clock — this is what
  ordering, reconstruction, and evidence are built on.
Ingest timestamp: when OUR system received it — used only to monitor our own ingestion latency.

Conflating these two timestamps is a subtle, serious bug class specific to this domain: detection logic and legal evidence must be built on the exchange's own sequencing (per this series' Event Sourcing discussion of event time vs. processing time), while ingestion health monitoring (Section 14) is a separate concern that should never leak into the reconstructed record itself.

---

4. Ingestion: Normalizing and Time-Ordering Multi-Venue Data

Why raw exchange feeds can't be consumed as-is

Exchange A: FIX protocol, timestamps to the microsecond, prices in decimal
Exchange B: proprietary binary protocol, timestamps to the millisecond, prices in fixed-point ticks
Exchange C: a different symbology entirely for the SAME underlying instrument

A surveillance system watching multiple venues (or multiple asset classes) faces a normalization problem before any detection logic can run at all — as covered in this series' Data Pipeline and ETL guides, ingestion adapters translate each venue's native protocol and symbology into one canonical OrderEvent schema (Section 2), resolving cross-venue instrument identity so that a pattern spanning two venues can actually be detected as one pattern rather than two unrelated ones.

Handling out-of-order and late-arriving events

Network jitter, venue-side batching, and multi-path delivery mean events don't always arrive
  in exchange-timestamp order — the ingestion layer must buffer briefly and re-sort by
  exchange timestamp before events reach the alert engine, per this series' Stream Processing
  guide's watermarking and out-of-order handling patterns.

As covered in this series' Kafka Streams / Stream Processing guide, a small, bounded buffering window with watermarks lets the system tolerate realistic out-of-order arrival without either blocking ingestion indefinitely or emitting events for detection in an order that would produce spurious pattern matches (a cancel appearing to precede the order it cancels, for instance).

Sequence gap detection

if (incoming.SequenceId != lastSeenSequenceId + 1)
{
    await _alerting.RaiseAsync("Sequence gap detected on feed", instrumentId, lastSeenSequenceId, incoming.SequenceId);
    // per this series' Health Checks guide: a gap here means the log's completeness itself is now in question
}

Unlike most streaming systems where a dropped message degrades a downstream metric slightly, a gap in the market data log means the surveillance system's fundamental evidentiary record is incomplete for that window — sequence gap detection is a first-class alerting concern here, not a minor data-quality nicety, since a missed order due to a gap could be the exact order that made a manipulative pattern detectable.

---

5. Detecting Patterns: The Alert Engine

Rules-based detection for well-understood manipulative patterns

// Simplified spoofing heuristic: large order placed and cancelled quickly, on the opposite side
// of a smaller order that then executes — a classic layering/spoofing signature
public bool DetectsSpoofing(OrderBookWindow window)
{
    var largeCancelledOrder = window.Orders
        .Where(o => o.Quantity > window.AverageOrderSize * 10 && o.WasCancelledWithinMillis(500))
        .FirstOrDefault();
    return largeCancelledOrder is not null &&
        window.HasOppositeSideExecutionShortlyAfter(largeCancelledOrder);
}

Well-characterized manipulative patterns — spoofing, layering, wash trading, marking the close — have known structural signatures that a deterministic rules engine can detect reliably and, critically, explainably, which matters enormously here: per this series' guidance on explainability in rules-driven systems, an analyst (and eventually a regulator) needs to understand exactly why an alert fired, not just that a model scored it highly.

Statistical and ML-based detection for less-defined anomalies

Rules engine: catches KNOWN pattern shapes reliably, explainably.
Statistical/ML layer: flags STATISTICAL outliers (unusual volume, unusual price movement
  ahead of a corporate announcement) that don't match a predefined rule shape but warrant review.

As covered in this series' Machine Learning Systems guide, a purely rules-based engine only catches patterns someone has already thought to encode — a complementary statistical layer (volume/price anomaly detection, unusual correlation with insider-adjacent trading ahead of news) catches genuinely novel or evolving manipulation techniques, at the cost of being harder to explain and requiring the tuning discipline covered in Section 10.

Windowed, stateful computation over the event stream

// Per this series' Stream Processing guide's windowing patterns
var window = _streamProcessor
    .KeyBy(e => e.InstrumentId)
    .TumblingWindow(TimeSpan.FromSeconds(5))
    .Aggregate(new OrderBookWindowAggregator());

Most detection patterns require state accumulated over a window of time (an order book's recent history, a trader's recent order/cancel ratio) rather than a single event in isolation — this is a direct application of this series' Stream Processing guide's windowing and stateful aggregation patterns, keyed by instrument or by trader depending on which patterns a given rule is designed to catch.

---

6. The Alert State Machine and Case Management

An explicit, enumerable set of states and legal transitions

Open → UnderReview → (Closed | Escalated | FalsePositive)

As covered in Section 2's SurveillanceAlert aggregate, an alert's lifecycle is a small, explicit state machine — and the aggregate's own methods (AssignForReview(), Close()) are what enforce that only legal transitions are ever possible, throwing rather than silently succeeding if called out of order (closing an alert that was never assigned for review, for instance).

Why an explicit, auditable state machine matters more here than for most domain objects

Given this guide's emphasis on regulatory exposure (Section 1), having every legal and illegal state transition explicitly enumerated, enforced by the aggregate itself, and logged with the acting analyst's identity is precisely the kind of rigor this series' DDD guide argues pays for itself most clearly in domains with genuine legal and audit stakes, and few domains fit that description more clearly than surveillance case management.

Case management as the human-in-the-loop workflow

Alert generated → routed to a queue by pattern type / instrument / desk →
  analyst reviews evidence (Section 3's reconstructed data) → disposition recorded →
  disposition FEEDS BACK into detection tuning (Section 10)

Case management is where the system hands off from automated detection to human judgment — per this series' Workflow Engine guide's routing and assignment patterns, alerts are queued and routed to the right analyst or desk, and every disposition an analyst records is itself a genuinely valuable signal that should feed back into the detection layer, not just close out the individual case.

---

7. Entity Resolution: Linking Orders to Real Actors

Why "trader ID" alone is often not enough

Manipulation is frequently attempted across MULTIPLE accounts, sometimes at MULTIPLE firms,
  coordinated by the same underlying actor — detecting it requires linking orders to the
  real-world entity behind them, not just the account that submitted each individual order.

A pattern that looks innocuous from any single account's perspective can be a clear violation once orders from related accounts (the same beneficial owner, a household, a known associated-party network) are considered together — this is why entity resolution, linking accounts and trader IDs to a broader real-world identity graph, is treated as its own explicit subsystem rather than an incidental join.

Building and maintaining the identity graph

public class EntityResolutionService
{
    public async Task<IEnumerable<TraderId>> GetRelatedTradersAsync(TraderId trader)
    {
        // per this series' Graph Database guide: traverse known relationships
        // (shared address, shared beneficial owner, historically correlated trading) up to N hops
        return await _graphStore.TraverseRelatedEntitiesAsync(trader, maxHops: 2);
    }
}

As covered in this series' Graph Database guide, modeling accounts and traders as nodes with explicit relationship edges (shared ownership, shared address, historically correlated trading behavior) lets detection logic (Section 5) query "who is plausibly the same actor as this trader" as a graph traversal, rather than every rule needing to independently reimplement relationship inference.

---

8. Streaming vs. Batch: Coordinating Real-Time and Historical Detection

Why some patterns need to be caught in near-real-time, and others don't

Spoofing/layering: needs near-real-time detection — the manipulative intent is time-sensitive,
  and same-day intervention may be required.
Insider trading ahead of an announcement: often only detectable in hindsight, once the
  announcement has happened and trading ahead of it can be meaningfully assessed.

Not every manipulative pattern has the same latency requirement, and treating them all identically wastes either latency budget or analytical depth — per this series' Lambda/Kappa Architecture discussion, a streaming layer handles patterns genuinely time-sensitive enough to need near-real-time detection, while a batch layer re-runs richer, more expensive detection logic over the full historical log at end-of-day or on a schedule, catching patterns whose evidence only becomes clear with hindsight or additional context (a corporate announcement, a related filing).

Keeping streaming and batch detection consistent

Both layers detect against the SAME market data log (Section 3) and the SAME rule definitions —
  the difference is WHEN each runs and how much lookback/context each has access to,
  not a separate, divergent codebase per layer.

As covered in this series' Kappa Architecture guide's critique of maintaining two divergent codebases for streaming and batch, this design deliberately shares rule logic between the two layers wherever possible — the streaming layer runs a fast, bounded-context version of a rule, and the batch layer reruns the same underlying logic with a fuller window, rather than maintaining two separately-evolving implementations that can silently drift apart.

---

9. Backtesting and Replay

Why every new or modified rule needs to run against history before going live

A new detection rule, deployed directly to production: unknown false-positive rate,
  unknown coverage of historical known-bad cases, discovered only after analysts are already
  drowning in alerts (or after a real case was missed).

As covered in this series' A/B Testing and Data Pipeline guides' general principle of validating a change against real data before it affects real users, a new or modified detection rule here must first be run — via replay of the market data log (Section 3) — against a substantial historical window, measuring both its alert volume against known-clean periods and its recall against previously confirmed cases, before it is ever enabled against live traffic.

Replay as a first-class capability of the log, not a special-case tool

public async Task<IEnumerable<Alert>> BacktestAsync(DateRange range, RuleDefinition rule)
{
    var events = _marketDataLog.ReadRange(range); // the SAME log the live system reads from
    return await _alertEngine.EvaluateAsync(events, rule);
}

Because the market data log is the append-only, replayable source of truth (Section 3), backtesting a rule is structurally the same operation as running it live — feed the rule the same event stream, just from history instead of the live tail — which is precisely why treating the log as genuinely immutable and complete matters so much: a log with gaps or silent mutations makes backtesting results untrustworthy in exactly the way this guide's Section 1 stakes can't tolerate.

---

10. Tuning Detection: Precision, Recall, and Analyst Trust

The false-positive problem is a genuine, ongoing engineering concern, not a one-time calibration

An overly sensitive rule: hundreds of alerts a day, nearly all benign →
  analysts start triaging superficially, or de-prioritizing the queue → a REAL case
  gets buried in noise and missed. The false-positive rate is itself a risk factor.

Every fraud-and-risk-adjacent system covered elsewhere in this series faces some version of the precision/recall trade-off, but here it compounds: too many false positives doesn't just waste analyst time, it measurably degrades detection of real cases by eroding the attention and trust a human reviewer brings to each alert — this is why alert volume and analyst disposition rates are tracked as core product metrics (Section 14), not just detection accuracy in isolation.

Feedback loops from analyst dispositions back into rule tuning

Analyst marks alert as FalsePositive with a reason code → aggregated over time →
  surfaces which rule/threshold combinations are producing disproportionate noise →
  feeds a deliberate, reviewed tuning process (Section 9's backtest gate applies to every change)

As covered in this series' Feedback Loop and MLOps discussions, analyst dispositions are themselves valuable signal that should be aggregated and fed back into detection tuning — but per Section 9, every tuning change still goes through backtesting before deployment, since an untested threshold change can just as easily suppress real cases as reduce noise.

---

11. Data Security and Compliance

Access control commensurate with genuinely sensitive data

Surveillance data is inherently sensitive — it contains trading activity, account relationships, and open investigations that could themselves be market-moving or reputationally damaging if leaked. The practical strategy mirrors this series' Secret Management and Identity guides' least-privilege principle: analysts see only the alerts and cases assigned to their desk or mandate, access to raw market data and entity-resolution graphs is separately scoped and audited, and access to open investigations is restricted well beyond ordinary application-level roles.

Chain-of-custody logging for evidentiary integrity

logger.LogInformation("Alert {AlertId} evidence viewed by {AnalystId} at {Timestamp}", alertId, analystId, DateTimeOffset.UtcNow);

As covered in this series' Structured Logging and OWASP Top 10 guides, every access to an alert's underlying evidence needs to be logged with enough context (who, what, when) to support both regulatory audit requirements and, in escalated cases, formal chain-of-custody requirements — this is a stricter, more comprehensive logging bar than most systems require, precisely because of Section 1's legal stakes.

Retention requirements that outlast typical system design defaults

Regulatory retention requirements for surveillance data commonly run into MULTIPLE YEARS —
  far longer than most systems' default "keep hot data for 90 days, archive/delete after" policy.

Per this series' Data Retention guide, the market data log and case records here need a retention policy driven by regulatory requirement rather than storage-cost convenience, with cold storage tiers (Section 13) explicitly designed to keep years of historical data genuinely queryable for backtesting (Section 9) and investigation, not just archived and effectively inaccessible.

---

12. Consistency, Availability, and the CAP Trade-off for Surveillance

Why the log favors durability and completeness over raw ingestion availability

As covered in this series' System Design guide's CAP theorem discussion, most systems in this series lean toward availability where possible — surveillance is one of the clearer exceptions on the ingestion side: it is generally preferable for ingestion to apply backpressure or briefly buffer under load than to silently drop market events, since a dropped event isn't just a missing data point, it's a potential gap in the evidentiary record that Section 4's sequence-gap detection exists specifically to catch.

Where eventual consistency is still acceptable, deliberately scoped

The MARKET DATA LOG (Section 3) → durability and completeness required, no compromise
The real-time analyst DASHBOARD showing "alerts today" → eventual consistency, a few seconds, is fine
Cross-venue entity-resolution graph updates → eventually consistent is acceptable and expected

Not every part of a surveillance system needs the same bar — the log's completeness absolutely does, but downstream, read-only projections (dashboards, alert-volume reporting) can and should tolerate the eventual consistency this series' Event-Driven Architecture and CQRS discussions describe generally, since those are convenience views, not the evidentiary record itself.

---

13. Scaling the System

Applying this series' System Design guide's building blocks, with surveillance-specific emphasis

Partitioning the log (per this series' Kafka guide): by instrument, so per-instrument order-book
  reconstruction and detection stay embarrassingly parallel across instruments
Tiered storage (per this series' Data Retention guide): hot log for recent data driving real-time
  detection, cold/archival storage for the multi-year regulatory retention window (Section 11),
  queryable for backtesting and investigation without needing to stay in the hot path
Stream processing scale-out (per this series' Kafka Streams guide): the alert engine scales
  horizontally by instrument partition, matching the log's own partitioning scheme

Every technique from this series' System Design guide applies here, with the caveat that each one needs to be evaluated against this guide's completeness and retention requirements (Sections 3 and 11) before being applied — the general principle "identify the bottleneck, then apply the specific technique" holds, but surveillance narrows which trade-offs (dropping data, aggressive TTLs) are actually acceptable.

Isolating the batch/backtest workload from the live detection path

Backtesting and end-of-day batch detection (Section 8, Section 9) read from the SAME log,
  but run on separate compute — a heavy historical replay must never compete for resources
  with the real-time detection path a live spoofing pattern depends on.

Per this series' Resource Isolation and Bulkhead pattern discussion, keeping batch/backtest workloads on separate compute (even though they share the same underlying log) prevents a large historical replay job from degrading the latency-sensitive real-time detection path — the two workloads have fundamentally different latency requirements (Section 8) and shouldn't contend for the same resources.

---

14. Observability for a Surveillance System

Every guide in this series' observability trio, applied with surveillance-specific stakes

Structured logs (per this series' Structured Logging guide): every alert generated, every
  disposition recorded, every evidence access — with enough context for both debugging and audit
Distributed tracing (per this series' Distributed Tracing guide): tracing a single event's journey
  from ingestion through the alert engine to (possibly) an alert — essential for diagnosing why
  a pattern that should have fired an alert didn't, or why one fired unexpectedly
Metrics (per this series' Prometheus/Grafana guide): ingestion lag per feed, sequence-gap count
  (Section 4), alert volume per rule, analyst disposition rates (Section 10) — the aggregate
  health signals a surveillance operations team watches continuously

Every technique from this series' observability guides applies directly, with one surveillance-specific addition worth stating explicitly: ingestion lag itself is a compliance-relevant metric here, not just an operational one — a feed falling meaningfully behind means real-time detection is running on stale data, which is exactly the kind of gap Section 1's stakes make unacceptable to discover only after the fact.

Alerting on system-health symptoms, distinct from surveillance alerts themselves

# Per this series' Prometheus/Grafana guide's symptom-based alerting principle
increase(market_data_sequence_gap_total{instrument="$instrument"}[5m]) > 0

A sequence gap (Section 4) or a feed falling behind its expected latency budget is exactly the kind of system-health symptom this series' Prometheus/Grafana guide argues alerts should be built around — and it's worth keeping this category of alert (about the surveillance system's own health) clearly distinct from the surveillance alerts it produces about trading activity, since conflating the two in dashboards or paging rotations creates genuine confusion for on-call responders.

---

15. Common Pitfalls

Pitfall	Why it hurts	Better approach
Mutable "current order book" as the only stored state	No path to reconstruct historical book state for an investigation	Append-only, time-ordered market data log; book state as a derived, replayable projection
Conflating exchange timestamp with ingest timestamp	Corrupts event ordering used for detection and evidence	Store both explicitly; use exchange timestamp for reconstruction, ingest timestamp only for latency monitoring
Deploying new/modified detection rules directly to production	Unknown false-positive rate and unknown historical recall, discovered only after analysts are overwhelmed or a case is missed	Backtest every rule change against historical data before enabling it live
Treating all manipulative patterns as needing the same detection latency	Wastes latency budget on patterns that are only detectable in hindsight, or under-resources genuinely time-sensitive ones	Split streaming (real-time-sensitive patterns) from batch (hindsight-dependent patterns), sharing rule logic
Ignoring the false-positive rate as a "someday" tuning concern	Alert fatigue measurably degrades detection of real cases, not just analyst efficiency	Track disposition rates as a core metric; feed them into a reviewed, backtested tuning process
No entity resolution across related accounts	Coordinated manipulation across multiple accounts goes undetected even though each account's activity is monitored	Maintain an identity/relationship graph; detection queries consider related entities, not just a single account
Applying a short, storage-cost-driven retention policy	Violates regulatory retention requirements and makes historical backtesting impossible	Retention policy driven by regulatory requirement, with tiered storage keeping years of data genuinely queryable
Silently dropping market events under ingestion load	A gap in the evidentiary record, discovered (if at all) only during an investigation that needs the missing data	Backpressure and buffering over silent drops; explicit sequence-gap detection and alerting

---

Quick Reference Table

Concept	Purpose
SurveillanceAlert aggregate + state machine	Enforces only legal alert state transitions, with an auditable trail, per this series' DDD guide
Append-only, time-ordered market data log	The provably complete, replayable evidentiary record all detection and investigation rests on
Exchange timestamp vs. ingest timestamp	Keeps event ordering/evidence separate from ingestion-latency monitoring
Rules engine + statistical/ML layer	Explainable detection of known patterns, complemented by anomaly detection for novel ones
Entity resolution graph	Detects coordinated manipulation across related accounts, not just a single account in isolation
Streaming + batch detection sharing rule logic	Matches detection latency to each pattern's actual time-sensitivity without maintaining two divergent codebases
Backtesting/replay against the log	Validates every detection rule change against real historical data before it goes live
Analyst disposition feedback loop	Continuously improves precision without bypassing the backtest gate on any tuning change
Tiered, regulation-driven retention	Keeps years of historical data genuinely queryable for backtesting and investigation

---

Conclusion

A stock surveillance system takes every general system design technique covered throughout this series and applies it under a correctness-and-trust bar strict enough that both misses and false alarms carry real cost — because the system exists specifically to catch behavior someone is actively trying to hide, while not burying the humans who review its output in noise. The design that actually holds up under that bar rests on a small number of non-negotiable foundations: an append-only, precisely time-ordered market data log as the provable evidentiary source of truth; detection logic that's explainable where it can be and continuously validated by backtesting wherever it changes; entity resolution that looks past any single account to the real actor behind a pattern; and a disciplined feedback loop between analyst judgment and detection tuning that never bypasses that same backtest gate.

Nearly every architectural pattern covered elsewhere in this series shows up here in service of that bar — DDD's aggregates enforcing an auditable case lifecycle, Stream Processing's windowing and watermarking for out-of-order data, Kappa Architecture's shared logic across streaming and batch, and the full observability trio watching over both the system's own health and the quality of what it produces. Stock surveillance is, in that sense, less a distinct discipline from everything else in this series than the place where its cumulative lessons about immutable history, explainability, and honest reconciliation between automated detection and human judgment matter more visibly, and more unforgivingly, than almost anywhere else.

---

Found this useful? Feel free to star the repo, open an issue with corrections, or share the sequence-gap incident that turned out to matter far more than a missing data point ever should.