Rizwan Saleem

Posted on Jun 2

Designing a scalable event-sourced analytics pipeline from scratch

#frontend #webdev

Designing a scalable event-sourced analytics pipeline from scratch

Event-sourced analytics systems capture every user interaction as an immutable event, enabling rich time-travel queries, accurate replays, and robust fault tolerance. This guide walks you through designing a practical, production-ready event-sourced analytics pipeline suitable for apps with millions of events per day. It covers data modeling, storage, stream processing, compaction, query patterns, observability, and operational considerations. You’ll get concrete choices, tradeoffs, and sample code to get started.

Overview and design goals

Capture every user interaction as an immutable event with a concise schema.
Durable, scalable storage that supports append-only writes and efficient reads.
Real-time and batch processing paths for dashboards, alerts, and downstream systems.
Proven techniques for event deduplication, idempotence, and schema evolution.
Observability: metrics, tracing, and replayable tests to ensure data correctness.

Key decisions you’ll make:

Event schema and identifiers
Storage layer (log streams + read models)
Messaging/streaming architecture
Processing models (real-time vs. micro-batch)
Data retention, compaction, and aging
Query interface and APIs ### 1) Event schema and identity

Principles:

Immutable, append-only events
Each event has: type, aggregate_id (entity), timestamp, version, payload, metadata
Use a global sequence or partitioned key for ordering by event time

Example event schema (JSON-like):

event_id: string (UUID)
type: string (e.g., "page_view", "button_click", "purchase")
aggregate_id: string (e.g., user_id, session_id)
timestamp: int64 (epoch milliseconds)
version: int32 (per-aggregate version)
payload: object (event-specific data)
metadata: object (source, device, etc.)
context: object (trace_id, correlation_id)

Guidelines:

Avoid storing large payloads; reference external blobs if needed.
Normalize common fields to enable efficient filtering (e.g., user_id, city, app_version).
Version your event types to handle schema evolution without breaking readers.

Event identifiers and deduplication:

Use a deterministic event_id (e.g., UUID v4) and idempotent writes at the producer side.
Include a correlation_id for tracing and cross-service deduplication.
Consider a deduplication window on the consumer side for exactly-once semantics in downstream systems. ### 2) Storage and data layout

Architecture pattern:

Append-only log segments as the primary source of truth (an event log or message queue backed by durable storage).
Read models (materialized views) built from the event stream, optimized for queries.

Recommended layers:

Event log: log-based storage with partitioning by aggregate_id or by time range. Examples: Apache Kafka, Apache Pulsar, or a cloud-managed service like Kinesis/Event Hubs.
Durable sink: a write-ahead storage (e.g., object storage) for long-term archival and replay.
Read models store: a query-optimized database or data lake (e.g., columnar store like ClickHouse, Apache Druid, Elasticsearch, or a scalable relational store depending on queries).

Data lifecycle:

Hot store: in-memory or fast storage for recent data (for real-time dashboards).
Warm store: columnar store for longer retention.
Cold store: compressed archives for compliance and long-term access.

Compaction and schema evolution:

Use versioned events to handle schema changes without rewriting history.
Implement a schema registry to track event formats and evolution rules.
For some event types, consider topic compaction if events carry only latest state (not typical for analytics; prefer immutable events).

Observe data locality:

Partition by time (e.g., daily or hourly) to enable efficient queries over a window.
If you need user-centric queries, consider partitioning by user_id where feasible. ### 3) Streaming architecture and processing models

Two core paths:

Real-time path: streaming processing for near-real-time dashboards and alerts.
Batch path: periodic re-aggregation and complex analytics.

Recommended setup:

Ingest: producers publish events to a log (Kafka topics or equivalent).
Stream processor: microservices or serverless functions consuming from the log, updating read models, and emitting derived events if needed.
Orchestration: a lightweight orchestrator to schedule batch jobs.

Processing patterns:

Event-to-read-model: compute user sessions, funnels, and aggregates by consuming events and updating derived tables.
Windowed analytics: sliding windows (e.g., 1m, 5m, 1h) for metrics like active users, conversions, and retention.
Exactly-once vs at-least-once:
- At-least-once is common in streaming; implement idempotent upserts to read models.
- For critical aggregates, use idempotent processors with per-event idempotency keys.

Sample real-time pipeline idea:

Ingest: events into Kafka topics (e.g., events.user, events.purchases).
Processor: Apache Flink or ksqlDB reads streams, performs windowed aggregations, writes results to a read-model store (e.g., ClickHouse).
Downstream: dashboards query the read models; anomalies trigger alerts via a separate alerting topic.

Code sketch (conceptual, language-agnostic):

Consumer reads events from a topic.
Maintains state per aggregate/window.
Writes derived metrics to a read-model store and emits derived-events for downstream systems.

Key considerations:

Backpressure handling and at-least-once semantics.
Exactly-once ingestion where feasible, using transactional producers/consumers.
Time synchronization across producers and processors to maintain consistent windowing. ### 4) Read models and query patterns

Design read models tailored to common analytics questions:

User-level funnels: sequences of actions within a session.
Global metrics: daily active users, sessions, events per second.
Cohort analyses: retention by cohorts defined at first interaction.
Purchase funnels: conversions across stages of checkout.

Storage choices:

Real-time dashboards: in-memory stores or columnar engines with fast aggregations (e.g., ClickHouse, Apache Druid).
Ad-hoc analysis: data lake with Parquet files, queryable via a SQL engine (e.g., DuckDB for local, Presto/Trino for distributed).

Example read model schema for daily metrics (in ClickHouse-style):

date: Date
app_version: String
country: String
active_users: UInt64
sessions: UInt64
events: UInt64
revenue_usd: Float64
avg_session_duration_sec: Float64

Materialization approach:

Incremental updates from the streaming path.
Periodic batch recomputation for complex metrics that are expensive to compute in real time.
Rollups by time granularity (hourly/daily) to optimize drill-down.

Query patterns:

Time-bounded ranges: SELECT date, active_users FROM daily_metrics WHERE date BETWEEN '2026-01-01' AND '2026-01-31'
Cohort analysis: JOIN user cohorts with events to compute retention.
Segmentation: filter by country, device, or app_version. ### 5) Observability, testing, and reliability

Telemetry:

Metrics: event ingestion rate, lag, error rate, read-model update latency, query latency.
Tracing: propagate trace context through producers and processors for end-to-end visibility.
Logs: structured logs with event IDs for traceability.

Testing strategy:

End-to-end replay tests: replay a known set of events and verify read models reproduce expected results.
Backfill tests: simulate historical data to ensure processing logic handles old schemas.
schema-evolution tests: verify readers can cope with versioned events and that new readers ignore unknown fields gracefully.

Operational reliability:

Idempotent upserts in read models to tolerate retries.
Dead-letter queues for failed event processing with automatic retry policies.
Schema registry to coordinate changes across producers and consumers.

Backups and retention:

Regular backups of read-model stores and the event log.
Data retention policies by event type and privacy requirements.
Data deletion or anonymization for GDPR/CCPA compliance, implemented in a governed way (not by soft deletes in the log).

6) Security, privacy, and compliance
Pseudonymize or hash personal identifiers where possible in the event payload.
Encrypt data at rest and in transit; use access-controlled streams and least-privilege intents.
Data governance: maintain an inventory of data sources, event types, and retention rules.
Audit trails: log who accessed which data and when, especially for sensitive metrics.

7) Example end-to-end implementation plan

Phase 1: MVP with a single product

Choose tech stack: Kafka for the event log, Flink for streaming processing, ClickHouse for read models.
Define essential events: page_view, add_to_cart, purchase.
Implement producers: lightweight SDK to emit events with required fields.
Build a Flink job: aggregate daily active users and revenue; write to ClickHouse.
Expose dashboards with SQL clients or BI tools connected to ClickHouse.

Phase 2: Real-time enhancements

Add sliding-window computations (e.g., 15-minute MAU) and alerting on anomalies.
Introduce a real-time user funnel read model with low-latency updates.

Phase 3: Governance and scalability

Implement schema registry and versioning.
Add backfill tooling and automated tests for new event types.
Consider alternative storage paths (e.g., managed services) as needs scale.

Phase 4: Privacy and compliance refinements

Implement data masking, access controls, and retention automation.
Prepare for data deletion workflows and consent management. ### 8) A concise code sketch: event emission and a simple processor

Note: this is a high-level illustration to help you start. Adapt to your language and stack.

Event emission (pseudocode):
def emit_event(event_type, aggregate_id, payload, metadata=None, context=None):
event = {
"event_id": generate_uuid(),
"type": event_type,
"aggregate_id": aggregate_id,
"timestamp": current_time_millis(),
"version": fetch_and_increment_version(aggregate_id, event_type),
"payload": payload,
"metadata": metadata or {},
"context": context or {},
}

kafka_producer.send(topic="events", value=serialize(event))

Simple processor (pseudocode):
def process_event(event):
```
if event.type == "page_view":
```

    update_read_model("daily_metrics", event.date, event.country, increment="events")

```
elif event.type == "purchase":
```

    update_read_model("revenue_by_day", event.date, event.country, increment=event.payload.amount)

# idempotent upsert via event_id or dedup key

ensure idempotence by upserting with a unique key (event_id) and using upsert semantics in the read-model store.

9) Practical pitfalls and how to avoid them
Pitfall: Schema drift causing read-model failures.
- Solution: enforce a schema registry, log strictly the event_version, and maintain compatibility rules.
Pitfall: Late-arriving events breaking window calculations.
- Solution: design windows to tolerate late data; implement watermarking strategies and grace periods.
Pitfall: Read-model stores becoming bottlenecks.
- Solution: partitioned stores, parallelized writes, and tiered storage with hot/creeze paths.
Pitfall: Overcomplicated pipelines that are hard to operate.
- Solution: start simple, with clear SLAs; iteratively add components as needs grow. ### 10) Next steps
Map your specific analytics questions to a minimal viable event schema.
Prototyping plan: build a small in-memory or local test environment to validate end-to-end flow.
Start with a clear data governance policy and a schema registry to manage evolution.

If you’d like, I can tailor this design to your stack (language, cloud provider, or preferred data stores) and provide a concrete codebase scaffold to bootstrap your project. Which tech choices are you considering for ingestion, processing, and read-model storage?

Rizwan Saleem | https://rizwansaleem.co