You're Optimizing Real-Time Analytics for the Wrong Thing

Speed is seductive. When teams pitch real-time analytics internally, the word that does the most work in the room is fast. Sub-second queries. Instant dashboards. Live data.

But speed is a red herring. The actual property that makes real-time analytics valuable is recency — knowing that the data you're looking at reflects the world as it exists right now, not six hours ago when the batch job last ran. Those are different problems, and conflating them sends teams down expensive architectural rabbit holes.

Here's where it goes wrong, and how to think through it before you're six months into a painful rework.

The Architecture Decision Nobody Takes Seriously Enough

Before you pick a database or write a single line of ingestion code, you need to decide whether you're building a Lambda or Kappa architecture. At low event volumes this feels academic. At 10 billion events per day, it is the decision everything else inherits from.

Lambda keeps a batch layer and a streaming layer running in parallel. Your batch layer produces accurate, reprocessable historical results. Your streaming layer produces approximate, low-latency recent results. You merge them at query time. The appeal is resilience — if your stream processor falls over, the batch layer still works. The cost is that you're maintaining two separate codebases that need to produce semantically identical results, which they will eventually fail to do.

Kappa eliminates the batch layer entirely. Streaming is the system of record. When you need to reprocess, you replay from a durable log. This is simpler to reason about and operate, but it places enormous demands on your streaming infrastructure to be both reliable and replayable.

The teams that skip this decision and bolt on streaming as an afterthought to an existing batch pipeline always end up with accidental Lambda — without the intentional design choices that make Lambda workable. That's the worst of both worlds.

Tools like Turboline's streaming engine sit precisely at this ingestion layer, which makes the Lambda vs. Kappa question concrete very quickly: it forces you to decide whether your stream is authoritative or supplemental before data ever reaches your analytical store.

Your Storage Engine Is Not Interchangeable

Once you've committed to an architecture, the next failure mode is treating ClickHouse, Apache Druid, and Apache Pinot as roughly equivalent OLAP options. They're not, and picking the wrong one for your query pattern will cost you performance you can never fully claw back with tuning.

ClickHouse is exceptionally good at large analytical queries over wide tables — the kind of ad hoc exploration where a data analyst is writing SQL and expecting results in under a second. It's column-oriented, compresses aggressively, and vectorizes query execution well. If your access pattern is "give me everything that happened last Tuesday filtered by these four dimensions," ClickHouse is hard to beat.

Apache Druid was designed specifically for high-concurrency, low-latency queries over time-series event data with many simultaneous users. It pre-aggregates at ingestion time, which means query latency is predictable but your data model needs to be defined before the data arrives. It handles 100 concurrent dashboard queries far better than ClickHouse will.

Apache Pinot sits closest to Druid in design philosophy but adds stronger support for upserts and real-time mutations — scenarios where event records get corrected after the fact. If you're in financial services or anywhere that late-arriving corrections to events are normal, Pinot is worth the operational overhead.

The practical rule: match your storage engine to your dominant query shape, not to what the largest company with a public engineering blog happened to choose.

The Hidden Computation Behind "Instant" Dashboards

Here's the part nobody talks about in real-time analytics demos: that live dashboard that refreshes every five seconds and feels instant is almost certainly not querying raw events.

It's querying pre-computed aggregations.

Materialized views, continuous aggregations, rollup tables — whatever you call them in your stack, the "real-time" experience users perceive is built on a foundation of work that happened before the query. Your streaming pipeline is continuously computing SELECT event_type, COUNT(*), SUM(value) GROUP BY window(5 minutes) and writing the result somewhere cheap to read. The dashboard queries that result, not the underlying event stream.

This is not a cheat. It's the correct architecture. But it has a real implication: if you don't know in advance what your users will want to see, you cannot pre-compute it, and your "real-time" dashboard will either be slow or built on stale batch rollups.

The upfront design work in a real-time analytics system is almost entirely about predicting query patterns and building the right aggregations before the data arrives. The streaming infrastructure is the easy part.

A rough pattern for a continuous aggregation in ClickHouse looks like this:

CREATE MATERIALIZED VIEW events_per_minute
ENGINE = SummingMergeTree
ORDER BY (event_type, window_start)
AS SELECT
    event_type,
    toStartOfMinute(occurred_at) AS window_start,
    count() AS event_count,
    sum(value) AS total_value
FROM raw_events
GROUP BY event_type, window_start;

Your dashboard queries events_per_minute. It never touches raw_events directly.

The Concrete Takeaway

Real-time analytics projects fail or become expensive to maintain because teams optimize for the wrong property (speed instead of recency), defer the wrong decision (Lambda vs. Kappa until it's too late to change), and pick storage engines based on popularity rather than query shape.

Get the architecture decision right before you write ingestion code. Define your query patterns before you choose your storage engine. Build your aggregations before your dashboards demand them. The infrastructure will look boring and deliberate, which means it's working.