System Design: Investment Monitoring / Alert System
A capstone system design walkthrough — designing a system that continuously monitors portfolios, positions, and market conditions to generate timely, trustworthy alerts — covering the ingestion of market and position data, the rules and threshold engine that evaluates conditions, per-user alert subscriptions and notification delivery, deduplication and alert fatigue management, and the specific freshness, correctness, and delivery-guarantee demands that make investment monitoring a uniquely time-sensitive system design problem.
Table of Contents
- Introduction
- Why Investment Monitoring Is a Different Kind of Hard
- The Core Domain Model
- The Market and Position Data Log: A Consistent, Ordered View of "What's True Now"
- Ingestion: Market Data, Position Data, and Corporate Actions
- The Rule Engine: Evaluating Conditions at Scale
- The Alert Lifecycle State Machine
- Deduplication and Alert Fatigue Management
- Notification Delivery: Multi-Channel, At-Least-Once, User-Controlled
- Idempotency and Exactly-Once Alert Semantics
- Backtesting and Simulating New Rules
- Data Security and Compliance
- Consistency, Availability, and the CAP Trade-off for Alerts
- Scaling the System
- Observability for an Alerting System
- Common Pitfalls
- Quick Reference Table
- Conclusion
Introduction
An investment monitoring and alert system takes the general system design vocabulary covered in this series' System Design guide — streaming ingestion, rules engines, notification delivery, subscription management — and applies it to a domain where staleness and missed delivery both carry a real, sometimes irreversible cost: an alert that a stop-loss threshold was crossed, delivered ten minutes late or not at all, can mean a user loses meaningfully more money than the system existed to protect them from. This guide walks through designing such a system end to end, drawing directly on this series' Event-Driven Architecture, Stream Processing, Notification Systems, and Rate Limiting guides, each of which turns out to be load-bearing infrastructure for getting monitoring and alerting right rather than optional architectural polish.
Market Data Feeds + Position/Account Updates → Ingestion → Market/Position Data Log (source of truth)
↓
Rule Engine (per-user conditions, streaming)
↓
Alert Generated → Dedup/Fatigue Filter → Notification Delivery
1. Why Investment Monitoring Is a Different Kind of Hard
Freshness is not a nice-to-have — it's the entire point of the system
Most systems covered in this series can tolerate a few seconds, or even minutes, of staleness with a bounded, recoverable cost — a slightly outdated recommendation, a delayed notification. An investment alert system's core value proposition is timeliness: a price-threshold alert or a margin-call warning that arrives after the market has already moved further isn't just degraded, it's close to useless for the decision it was meant to support. This is why the freshness of the underlying data (Section 3) and the latency of the rule engine (Section 5) dominate this guide's concerns more than almost any other design axis.
A missed alert and a duplicate alert are both genuinely costly, in different ways
A missed alert: a user takes no action when they needed to — potentially real financial loss.
A duplicate or repeated alert for the same condition: users start ignoring or
muting the channel entirely — the NEXT alert, possibly a critical one, goes unseen too.
Unlike a typical notification system where an occasional duplicate is a minor annoyance, here a pattern of duplicate or excessive alerts erodes the exact trust the system depends on to be useful at the one moment it matters — this is why deduplication and fatigue management (Section 7) get as much design attention in this guide as delivery reliability itself.
The system must evaluate a very large number of conditions against constantly-changing data, continuously
A critical, freeing realization for the design that follows: an investment monitoring system, in the overwhelming majority of real-world designs, does not evaluate every user's every rule against every incoming price tick from scratch — it indexes rules by the instruments and conditions they actually depend on, and evaluates each incoming update only against the (much smaller) set of rules it could plausibly affect. This mirrors the "know which subset of state actually changed" discipline covered in this series' Caching and Change Data Capture guides, applied here to rule evaluation at scale.
2. The Core Domain Model
Modeled with DDD, per this series' companion guide
public record UserId(Guid Value);
public record InstrumentId(string Symbol, string Exchange);
public record AlertRuleId(Guid Value);
public enum AlertRuleStatus { Active, Paused, Triggered, Expired }
public enum ConditionType { PriceAbove, PriceBelow, PercentChange, VolumeSpike, MarginCallRisk, CorporateAction }
public class AlertRule // the AGGREGATE ROOT, per this series' DDD guide
{
public AlertRuleId Id { get; }
public UserId Owner { get; }
public InstrumentId Instrument { get; }
public ConditionType Condition { get; }
public decimal Threshold { get; }
public AlertRuleStatus Status { get; private set; }
private readonly List<AlertRuleEvent> _domainEvents = new();
public void Trigger(MarketSnapshot snapshot)
{
if (Status != AlertRuleStatus.Active)
throw new InvalidOperationException($"Cannot trigger a rule in status {Status}");
Status = AlertRuleStatus.Triggered;
_domainEvents.Add(new AlertRuleTriggeredEvent(Id, Owner, snapshot));
}
}
This directly applies this series' DDD guide's aggregate pattern — AlertRule is the aggregate root, enforcing its own state transitions (a paused rule cannot trigger) rather than trusting every caller to check status before mutating it, and raising domain events at exactly the points those transitions genuinely occur.
Separating the rule (what the user wants to know about) from the alert (a specific firing of it)
public record Alert(Guid AlertId, AlertRuleId RuleId, UserId Owner, MarketSnapshot TriggerContext, DateTimeOffset FiredAt, AlertDeliveryStatus DeliveryStatus);
As covered in this series' DDD guide's aggregate-sizing discussion, keeping AlertRule (the durable, user-configured condition) separate from Alert (an immutable record of one specific occurrence of that condition firing) means a single rule can fire many times over its life, each producing its own independently-delivered, independently-tracked Alert, without the rule itself needing to carry delivery-tracking state that has nothing to do with the condition it represents.
3. The Market and Position Data Log: A Consistent, Ordered View of "What's True Now"
Why a "latest price" cache alone is insufficient
❌ A single mutable "current price" field, overwritten on every tick, has no way to detect
that an update was skipped, arrived out of order, or that the feed itself went stale.
✅ An ordered, timestamped log of every price/position update — "current" is always a
query against that log, not a separately-maintained, unverifiable number.
An investment monitoring system needs more than "what is the latest known price" — it needs an ordered, timestamped record of updates it can reason about staleness against, detect gaps in, and (for compliance and dispute resolution, Section 11) reconstruct historically. A mutable "latest value" field, updated in place with no history, provides no way to distinguish "genuinely just updated" from "hasn't updated in twenty minutes because the feed died."
The log as the append-only backbone for both market and position data
CREATE TABLE market_position_log (
sequence_id BIGINT PRIMARY KEY,
entity_id VARCHAR NOT NULL, -- instrument_id OR account_id — partition key
entity_type VARCHAR NOT NULL, -- 'instrument' or 'position'
source_timestamp TIMESTAMPTZ NOT NULL, -- when the update actually occurred, per its source
ingest_timestamp TIMESTAMPTZ NOT NULL, -- used only for staleness/latency monitoring
payload JSONB NOT NULL
);
In practice this table's role is usually filled by a distributed log (Kafka/Pulsar) rather than a relational table directly — every incoming market tick and position/account update is first durably appended to the log, partitioned by instrument or account, before the rule engine evaluates it. This gives a durable, replayable record (essential for Section 10's backtesting and Section 11's audit requirements), natural per-entity ordering (single partition per instrument/account = strict order), and a backbone for downstream consumers via the outbox/CDC pattern, directly echoing this series' Event-Driven Architecture guide's discussion of avoiding dual-write inconsistency between "update derived state" and "publish the event."
Source timestamp vs. ingest timestamp — staleness must be measurable, not assumed
Source timestamp: when the price/position update actually occurred at its origin.
Ingest timestamp: when OUR system received it — the gap between the two IS the
system's actual freshness, and must be monitored explicitly (Section 14), not assumed to be near-zero.
Conflating these two timestamps hides exactly the failure mode Section 1 identifies as most costly — a feed that's silently fallen behind still looks "current" if only ingest timestamps are tracked; per this series' Event Sourcing discussion of event time vs. processing time, both must be stored and the gap between them treated as a first-class, alertable health signal in its own right.
4. Ingestion: Market Data, Position Data, and Corporate Actions
Three genuinely different input streams, each with its own reliability characteristics
Market data (prices, volume): high-frequency, vendor-fed, generally reliable but can gap or lag.
Position/account data: lower-frequency, sourced from internal systems (brokerage, custodian) —
a stale position feed means alerts evaluate against a portfolio that no longer reflects reality.
Corporate actions (splits, dividends, delistings): low-frequency but HIGH-IMPACT if missed —
a stock split not accounted for can make a price-threshold rule fire on a phantom move.
As covered in this series' Data Pipeline guide, each of these input streams needs its own ingestion adapter, its own staleness monitoring (Section 14), and — critically for corporate actions — its own explicit handling logic, since a missed or late-applied corporate action doesn't just delay one alert, it can make an otherwise-correct rule evaluate against fundamentally wrong data (a 2-for-1 split misread as a 50% price crash).
Normalizing across venues and vendors into one canonical schema
Different market data vendors report timestamps, symbology, and even trading halts
differently — normalization into one canonical instrument/price schema happens
BEFORE the rule engine ever sees the data, per this series' ETL guide's adapter pattern.
Per this series' Data Pipeline and ETL guides, ingestion adapters translate each vendor's native feed format and symbology into one canonical schema, resolving cross-vendor instrument identity, so that rules defined against a symbol behave consistently regardless of which upstream feed happened to deliver the update that triggered evaluation.
Handling feed gaps and trading halts explicitly, not silently
if (timeSinceLastUpdate > instrument.ExpectedUpdateInterval * StalenessMultiplier)
{
await _alerting.RaiseAsync("Feed gap detected", instrumentId, timeSinceLastUpdate);
// per this series' Health Checks guide: a gap here means every rule depending on
// this instrument is now evaluating against data of unknown freshness
}
Unlike most streaming systems where a gap degrades a downstream metric slightly, a gap in market data here means every alert rule depending on that instrument is now silently evaluating stale, possibly misleading data — gap detection is a first-class alerting concern (distinct from the user-facing alerts the system produces, per Section 14), not a minor data-quality nicety.
5. The Rule Engine: Evaluating Conditions at Scale
Indexing rules by what they depend on, not evaluating every rule against every update
// Rules are indexed by instrument, so an incoming price tick only triggers evaluation
// of the (small) set of rules that actually reference that instrument
public class RuleIndex
{
private readonly Dictionary<InstrumentId, List<AlertRuleId>> _rulesByInstrument = new();
public IEnumerable<AlertRuleId> GetRulesForUpdate(InstrumentId instrument)
=> _rulesByInstrument.GetValueOrDefault(instrument, new List<AlertRuleId>());
}
As covered in this series' Rules Engine and Complex Event Processing guides, evaluating every active rule against every incoming update simply doesn't scale once the number of active rules and the update rate both grow — indexing rules by the instrument(s) or account(s) they depend on turns each incoming update into a bounded, targeted evaluation against only the rules it could plausibly affect, directly mirroring Section 1's "know which subset of state actually changed" principle.
Simple threshold conditions vs. stateful, windowed conditions
"Alert if price crosses $150": stateless — evaluate the incoming tick against the threshold directly.
"Alert if price drops more than 5% in a 10-minute window": STATEFUL — requires tracking
a rolling window of recent prices per instrument, per this series' Stream Processing guide's
windowing patterns.
"Alert if margin usage exceeds 80% of account limit": requires combining a POSITION
update with a MARKET price update — a join across two different input streams.
Not every condition type has the same evaluation complexity — simple threshold rules are cheap, stateless comparisons, but percent-change-over-time and margin-risk conditions require windowed, stateful computation (per this series' Stream Processing guide) or joining across the market and position streams (Section 4) — the rule engine's architecture needs to support both cheaply, since the cheap majority of rules shouldn't pay the overhead the stateful minority requires.
Evaluation latency as a design constraint, not an afterthought
Per this series' Stream Processing guide's latency budget discussion: the gap between
"market data ingested" and "rule evaluated" is itself a component of the system's
overall freshness (Section 1) — a correct rule engine that's slow is functionally
the same failure as a stale feed.
Given Section 1's emphasis on freshness being the system's core value, rule evaluation latency deserves the same design attention as ingestion latency — an engine that evaluates correctly but with a multi-second lag under load has effectively reintroduced the staleness problem the ingestion pipeline was designed to avoid.
6. The Alert Lifecycle State Machine
An explicit, enumerable set of states and legal transitions
Rule: Active → Triggered → (Active again, if recurring) | Expired
Alert (one firing): Generated → Deduplicated/Suppressed | Queued → Delivered | Failed
As covered in Section 2's AlertRule aggregate, both the rule's lifecycle and each individual alert firing's delivery lifecycle are small, explicit state machines — the aggregate's own methods (Trigger()) are what enforce that only legal transitions are ever possible, throwing rather than silently succeeding if called out of order (triggering a rule that's already paused, for instance).
Why an explicit state machine matters more here than for most notification systems
Given this guide's emphasis on both missed and duplicate alerts being genuinely costly (Section 1), having every legal and illegal state transition explicitly enumerated and enforced — including whether a rule is a one-shot alert (fires once, then expires) or a recurring condition (re-arms after a cooldown, Section 7) — is precisely the kind of rigor this series' DDD guide argues pays for itself most clearly in domains where a state-machine bug directly produces either a missed or a duplicated user-facing alert.
Re-arming recurring rules without immediately re-triggering
public void ReArm(TimeSpan cooldown)
{
if (Status != AlertRuleStatus.Triggered) throw new InvalidOperationException();
if (DateTimeOffset.UtcNow - LastTriggeredAt < cooldown) return; // still cooling down
Status = AlertRuleStatus.Active;
}
A recurring rule (e.g., "alert every time this crosses $150, in either direction") needs an explicit cooldown before re-arming, or a single volatile period around the threshold would fire dozens of alerts for what a user experiences as one event — this connects directly to Section 7's deduplication logic, but starts here, in the state machine itself, as a structural guard rather than a filter applied after the fact.
7. Deduplication and Alert Fatigue Management
Why naive re-evaluation produces alert storms around a threshold
A price oscillating just above and below $150.00 for several minutes, evaluated
tick-by-tick against "alert if price crosses $150," fires dozens of times for
what a user experiences as ONE noteworthy event.
This is the investment-monitoring equivalent of the false-positive problem covered in this series' Fraud Detection and Surveillance system design guides — an alert engine that's technically correct on every individual evaluation can still produce an unusable, trust-eroding stream of near-duplicate alerts if it has no concept of "this is the same underlying event as the one I just fired."
Hysteresis and cooldown windows as the primary defense
public bool ShouldSuppress(AlertRule rule, DateTimeOffset now)
{
if (rule.LastTriggeredAt is null) return false;
return now - rule.LastTriggeredAt < rule.CooldownWindow; // per Section 6's re-arm logic
}
Per this series' Rate Limiting guide's cooldown/hysteresis patterns, requiring a rule to "cool down" for a configurable window after firing — and, for threshold rules specifically, requiring the price to move meaningfully past the threshold again (not just oscillate at the boundary) before re-arming — is the primary, structural defense against alert storms, applied at the rule-evaluation layer rather than as a downstream filter trying to guess which alerts are "really" duplicates after the fact.
User-facing controls over sensitivity, not just system-side defaults
Per this series' Notification Preferences discussion: users should be able to tune
cooldown windows, aggregate multiple related alerts into a single digest, or
set quiet hours — because the "right" level of alert frequency is genuinely
user- and context-dependent, not a single global constant the system can guess correctly.
As covered in this series' Notification Systems guide, giving users direct control over sensitivity and delivery cadence (rather than the system unilaterally deciding what counts as "too many" alerts) respects that different users have genuinely different risk tolerances and attention budgets — a day trader and a long-term retirement-account holder have very different definitions of a useful alert frequency for the same underlying rule type.
8. Notification Delivery: Multi-Channel, At-Least-Once, User-Controlled
Why delivery itself needs the same reliability discipline as detection
A correctly-detected, correctly-deduplicated alert that fails to actually reach the
user is functionally identical, from the user's perspective, to a missed alert
(Section 1) — delivery reliability is not a lesser concern than detection accuracy.
As covered in this series' Notification Systems guide, delivery across push notification, email, and SMS channels each has its own failure modes (a push token expiring, an email bouncing, an SMS carrier delay) — the system needs to treat delivery confirmation, not just alert generation, as the actual success criterion, and retry or fall back across channels when a preferred channel fails.
Multi-channel fallback for high-priority alerts
public async Task DeliverAsync(Alert alert)
{
foreach (var channel in alert.Rule.PreferredChannelsInOrder)
{
var result = await _channelSenders[channel].SendAsync(alert);
if (result.Confirmed) return; // per this series' Resilience guide's fallback chain pattern
}
await EscalateToDeadLetterAsync(alert); // every channel failed — this needs human/ops attention
}
Per this series' Resilience guide's fallback-chain pattern, a high-priority alert (a margin call, a stop-loss trigger) attempts delivery across a user's configured channels in priority order, falling back to the next channel on failure or non-confirmation — rather than accepting silent failure on the first channel attempted, which would reintroduce exactly the "technically generated but never seen" failure mode Section 1 warns against.
At-least-once delivery semantics, with idempotent client-side handling
The delivery layer itself provides AT-LEAST-ONCE guarantees (per this series' Kafka/
messaging guides) — a retried delivery attempt after an ambiguous failure (timeout,
unclear ack) is safer than risking a silent drop, and the CLIENT (mobile app, email
client) naturally de-duplicates by alert ID if a duplicate notification does arrive.
Given Section 1's asymmetric cost (a missed alert is worse than an occasional duplicate delivery attempt, unlike the alert-generation layer where duplicates actively erode trust), the delivery layer deliberately favors at-least-once over exactly-once — Section 9 covers the idempotency needed to make that safe.
9. Idempotency and Exactly-Once Alert Semantics
Why this matters at both the generation and delivery layers, for different reasons
As covered throughout this series' RabbitMQ, Kafka, and Event-Driven Architecture guides, every messaging layer provides at-least-once delivery, and a worker crash mid-evaluation or mid-delivery is a routine, expected occurrence at scale — an un-idempotent rule engine means a retried evaluation could generate a duplicate Alert record for the same underlying trigger event, and an un-idempotent delivery layer means a retried send could notify a user twice for the same alert, both of which directly undermine Section 7's fatigue management.
Deduplication keys tying an alert firing to its specific trigger event
var alertKey = $"{ruleId}:{triggerEvent.SequenceId}"; // ties the alert to the EXACT log entry that caused it
var existing = await _alertStore.FindByKeyAsync(alertKey);
if (existing is not null) return existing; // this exact trigger was already processed
Per this series' Idempotency Key pattern, keying a generated alert to the specific (rule_id, triggering_log_sequence_id) pair — rather than just a timestamp or a loosely-defined "this rule fired around now" — means a retried evaluation of the same log entry against the same rule always produces the same, single Alert, regardless of how many times the evaluation is retried.
Idempotency at the delivery layer, keyed by alert ID and channel
A delivery attempt is keyed by (alert_id, channel) — a retried send for the same
alert on the same channel is recognized and suppressed if a confirmed delivery
already exists, per this series' Idempotency guide's delivery-layer application.
Delivery-layer idempotency, keyed separately from generation-layer idempotency, ensures that a retried notification send (after an ambiguous timeout, say) doesn't produce a second, redundant push notification for an alert that already delivered successfully — closing the loop Section 8's at-least-once delivery semantics deliberately leaves open.
10. Backtesting and Simulating New Rules
Why a new rule type or threshold change needs validation against history before going live
A newly-added rule TYPE (e.g., "volatility spike detection"), deployed directly
to production: unknown alert volume, unknown false-positive rate, discovered
only after users are already receiving noisy or unhelpful alerts.
As covered in this series' A/B Testing and Data Pipeline guides' general principle of validating a change against real data before it affects real users, a new rule type or a change to default thresholds should first be run — via replay of the market/position log (Section 3) — against a substantial historical window, measuring both alert volume and how often it would have fired around genuinely noteworthy events versus noise, before being offered to users or enabled by default.
Replay as a first-class capability of the log, not a special-case tool
public async Task<IEnumerable<Alert>> SimulateAsync(DateRange range, RuleDefinition ruleTemplate)
{
var events = _marketPositionLog.ReadRange(range); // the SAME log the live system reads from
return await _ruleEngine.EvaluateAsync(events, ruleTemplate);
}
Because the market/position log is the append-only, replayable source of truth (Section 3), simulating a new rule is structurally the same operation as running it live — feed the rule engine the same event stream, just from history instead of the live tail — which is precisely why treating the log as genuinely complete and gap-free (Section 4) matters so much: a log with silent gaps makes simulation results untrustworthy in exactly the way this guide's Section 1 stakes can't tolerate.
11. Data Security and Compliance
Position and account data is inherently sensitive financial information
A user's holdings, account balances, and trading activity are sensitive by nature — the practical strategy mirrors this series' Secret Management and Data Privacy guides' least-privilege principle: access to position data is scoped strictly to the owning user and to services that genuinely need it to evaluate rules, and any aggregate or cross-user analysis (e.g., "how many users have an alert on this symbol") is handled through anonymized or access-controlled views, never raw position joins.
Audit logging of rule changes and alert delivery, for dispute resolution
logger.LogInformation("AlertRule {RuleId} modified by {UserId}: {Change}", ruleId, userId, change);
logger.LogInformation("Alert {AlertId} delivery attempted via {Channel}, result: {Result}", alertId, channel, result);
As covered in this series' Structured Logging and OWASP Top 10 guides, both rule configuration changes and delivery attempts need to be logged with enough context to resolve a genuinely common and consequential dispute type here: "I should have been alerted and wasn't" — having an auditable record of exactly what the rule was, when it was evaluated, and what delivery was attempted (and whether it was confirmed) is essential for investigating that claim credibly.
Regulatory considerations around investment-adviser-like functionality
Depending on jurisdiction and how alerts are framed, a system that goes beyond
"notify on a user-configured condition" into implying trading recommendations
can cross into regulated investment-advice territory — this is a genuine legal
and compliance question for the product, not just a system design detail, and
should be reviewed accordingly rather than assumed away.
Worth flagging explicitly, distinct from the purely technical design: the line between "monitoring and alerting on user-defined conditions" and "providing investment advice" is a real regulatory boundary in most jurisdictions, and how alert copy, rule templates, and any suggested thresholds are framed has compliance implications well outside this guide's system design scope — genuinely worth involving legal/compliance review on, not something to design around unilaterally.
12. Consistency, Availability, and the CAP Trade-off for Alerts
Why the ingestion and rule-evaluation path favors availability with monitored staleness, while alert generation favors correctness
As covered in this series' System Design guide's CAP theorem discussion, the ingestion pipeline generally favors staying available and accepting data even under partial degradation (better a slightly delayed price update than none at all, with staleness explicitly monitored per Section 4) — but alert generation itself needs enough consistency within a rule's evaluation that a duplicate or missed firing isn't produced by a race between concurrent evaluations of the same rule against overlapping data.
Where eventual consistency is deliberately, explicitly scoped in
The MARKET/POSITION LOG (Section 3) → durability and gap-free completeness required
Alert GENERATION (Section 5, 9) → strong consistency required per rule evaluation
A user-facing "alert history" DASHBOARD → eventual consistency, a few seconds, is fine
Not every part of the system needs the same bar — the log's completeness and alert generation's per-rule consistency both do, but downstream, read-only projections (a user's alert history view, aggregate usage analytics) can and should tolerate the eventual consistency this series' Event-Driven Architecture and CQRS discussions describe generally, since those are convenience views, not the trigger-and-delivery path itself.
13. Scaling the System
Applying this series' System Design guide's building blocks, with monitoring-specific emphasis
Rule indexing (Section 5) is the primary scaling lever for evaluation — without it,
evaluation cost grows with (rules × updates) rather than staying bounded per update
Partitioning the log and rule index (per this series' Kafka guide): by instrument,
so evaluation stays embarrassingly parallel across instruments
Stream processing scale-out (per this series' Kafka Streams guide): the rule engine
scales horizontally by instrument/account partition, matching the log's partitioning
Notification delivery (per this series' Notification Systems guide): scales
independently of evaluation, since delivery throughput and evaluation throughput
have different bottlenecks (external channel rate limits vs. internal compute)
Every technique from this series' System Design guide applies here, with the caveat that each one needs to be evaluated against this guide's freshness requirements (Section 1) before being applied — batching or queueing that would be a reasonable throughput optimization elsewhere can directly undermine the timeliness this system exists to provide, so the trade-off needs to be made deliberately, not by default.
Isolating the delivery layer from evaluation to prevent cross-contamination of backlogs
A slow or degraded notification channel (Section 8) must never block rule evaluation
from proceeding for other rules — per this series' Bulkhead pattern discussion,
evaluation and delivery run as separately-scaled, queue-decoupled stages.
Per this series' Resource Isolation and Bulkhead pattern discussion, decoupling evaluation from delivery via a queue means a degraded email provider or SMS carrier backs up only the delivery stage, not the latency-critical evaluation path — an alert can be generated promptly and queued for delivery even if the delivery layer itself is temporarily struggling to keep up.
14. Observability for an Alerting System
Every guide in this series' observability trio, applied with freshness-specific stakes
Structured logs (per this series' Structured Logging guide): every rule evaluation
decision, every alert generated, every delivery attempt and its outcome — with
rule and alert IDs for correlation
Distributed tracing (per this series' Distributed Tracing guide): tracing a single
update's journey from ingestion through rule evaluation to (possibly) an alert
and its delivery — essential for diagnosing why a specific alert was late or missing
Metrics (per this series' Prometheus/Grafana guide): ingest-to-evaluation latency
(the freshness gap, Section 3), feed gap count (Section 4), alert generation
rate per rule type, delivery confirmation rate per channel — the aggregate
health signals an on-call engineer watches continuously
Every technique from this series' observability guides applies directly, with one freshness-specific addition worth stating explicitly: end-to-end latency, from source timestamp to confirmed delivery, is the single most important metric this system produces about itself — per Section 1, a system that's otherwise correct but consistently slow has failed at its actual purpose just as thoroughly as one that's fast but wrong.
Alerting on the alerting system's own health, kept distinct from user-facing alerts
# Per this series' Prometheus/Grafana guide's symptom-based alerting principle
histogram_quantile(0.99, rate(source_to_delivery_latency_seconds_bucket[5m])) > 30
A p99 end-to-end latency exceeding an acceptable threshold, a feed gap (Section 4), or a delivery-channel failure spike are exactly the kind of symptoms this series' Prometheus/Grafana guide argues alerts should be built around — and it's worth keeping this category of alert (about the monitoring system's own health) clearly distinct from the investment alerts it produces for users, since conflating the two in dashboards or paging rotations creates genuine confusion for on-call responders.
15. Common Pitfalls
| Pitfall | Why it hurts | Better approach |
|---|---|---|
| Conflating source timestamp with ingest timestamp | Hides real feed staleness; a stale feed still "looks current" | Store both explicitly; treat the gap between them as a first-class, monitored metric |
| Evaluating every active rule against every incoming update | Evaluation cost scales as (rules × updates), which doesn't hold up at real volume | Index rules by the instrument/account they depend on; evaluate only the affected subset per update |
| No cooldown/hysteresis around threshold rules | A price oscillating near a threshold fires dozens of near-duplicate alerts, eroding trust | Cooldown windows and re-arm logic at the rule-evaluation layer, not a downstream filter |
| Treating delivery as "fire and forget" once an alert is generated | A generated-but-undelivered alert is functionally a missed alert from the user's perspective | Track delivery confirmation as the actual success criterion; fall back across channels on failure |
| No idempotency at the generation or delivery layer | A retried evaluation or send duplicates an alert or a notification for the same event | Idempotency keys tying alerts to their exact triggering log entry, and deliveries to (alert_id, channel) |
| Missing or late corporate action handling | A stock split or similar action makes an otherwise-correct rule fire on a phantom price move | Explicit, high-priority ingestion and handling path for corporate actions, separate from routine price ticks |
| Deploying new rule types or threshold defaults directly to production | Unknown alert volume and false-positive rate, discovered only after users are already annoyed or unhelped | Backtest/simulate new rules against historical data before enabling them live or by default |
| Batching or queueing on the evaluation path as a default throughput optimization | Directly undermines the timeliness the whole system exists to provide | Apply batching only on stages where it doesn't compromise freshness (e.g., delivery, not evaluation) |
Quick Reference Table
| Concept | Purpose |
|---|---|
AlertRule aggregate + state machine |
Enforces only legal rule/alert state transitions, per this series' DDD guide |
| Append-only market/position log with dual timestamps | The provable, gap-detectable source of truth freshness monitoring depends on |
| Rule indexing by dependency | Bounds evaluation cost per incoming update instead of scaling with total rule count |
| Cooldown / hysteresis / re-arm logic | Prevents alert storms around a threshold, protecting user trust in the channel |
| Multi-channel delivery with fallback | Treats delivery confirmation, not generation, as the true success criterion |
| Idempotency keys at generation and delivery | Prevents duplicate alerts and duplicate notifications from routine retries |
| Backtesting/simulation against the log | Validates new rule types or threshold defaults before they affect real users |
| End-to-end latency as the primary health metric | Makes freshness — the system's core value proposition — directly observable |
Conclusion
An investment monitoring and alert system takes every general system design technique covered throughout this series and applies it under a freshness bar strict enough that latency itself becomes a correctness concern, not just a performance one — because the entire value of the system collapses if a correct alert arrives too late to act on. The design that actually holds up under that bar rests on a small number of non-negotiable foundations: an append-only, dual-timestamped data log that makes staleness measurable rather than assumed; a rule engine that indexes by dependency so evaluation cost stays bounded as rules and volume both grow; deduplication and cooldown logic built into the evaluation layer itself, not bolted on after; delivery treated as genuinely complete only once confirmed, with fallback across channels; and idempotency enforced at both the generation and delivery layers so that routine retries never become duplicate, trust-eroding alerts.
Nearly every architectural pattern covered elsewhere in this series shows up here in service of that bar — DDD's aggregates enforcing a clean separation between a durable rule and its individual firings, Stream Processing's windowing for stateful conditions, Resilience's fallback chains for delivery, and the full observability trio watching over end-to-end latency as the system's single most important self-reported metric. Investment monitoring is, in that sense, less a distinct discipline from everything else in this series than the place where its cumulative lessons about freshness, idempotency, and honest signal-versus-noise management matter more visibly, and more unforgivingly, than almost anywhere else.
Found this useful? Feel free to star the repo, open an issue with corrections, or share the alert-storm-around-a-threshold incident that turned out to matter far more than a single missed tick ever should.
Top comments (0)