Most streaming diagrams stop at the stream processor. Data flows in, something processes it, and the diagram ends with an arrow pointing at a box labeled "analytics."
That last box is where the real design work lives. A stream is not something an analyst can query. A table is. The pipeline below is the shape I keep coming back to: sources, Kafka, Flink, Iceberg via TableFlow, Athena, QuickSight. Events in, answers out.
The pipeline at a glance
Six stages, each doing one job:
- Sources — applications and services emitting events
- Kafka — the topics that hold those events
- Apache Flink — real-time processing
- Iceberg tables via TableFlow — stored on S3, cataloged in Glue
- Athena — plain SQL queries
- QuickSight — dashboards
No stage reaches around another. Each one consumes the output of the previous stage and produces the input for the next. That discipline is most of the design.
Stages 1 and 2: Sources and Kafka
Everything starts with applications and services emitting events — an order placed, a payment attempted, a reading reported. Those events land on Kafka topics.
Kafka earns its place here for two reasons. It decouples producers from everything downstream, so a source system never waits on analytics and never knows analytics exists. And it retains the events, so the processing layer can re-read history when logic changes or a job restarts from a failure.
The topic becomes the contract: a named, ordered, replayable record of what happened. Everything downstream trusts the topic instead of trusting the source systems.
Stage 3: Flink does the work in real time
Flink reads from the topics and processes events as they arrive. This is where the raw stream gets shaped: noise filtered out, streams joined, events enriched, values aggregated over time windows.
The important property is timing. The work happens once, at event time, while the data is fresh. The alternative is a batch job that re-reads an entire day of raw events every night and redoes the same shaping work from scratch. By the time records leave Flink, they are already in the shape the table needs.
State lives in Flink for exactly this reason. A join or a window does not care that the data arrived as a stream. It cares that the right records meet each other, and Flink holds them until they do.
Stage 4: TableFlow lands the stream in Iceberg
This is the stage that changed the architecture for me. TableFlow materializes the processed stream as an Apache Iceberg table, continuously. The data files live in Amazon S3. The table is registered in the Glue Catalog, and that catalog entry is what makes it visible to the rest of the analytics stack.
Iceberg matters because it gives files on object storage real table semantics: a defined schema that can evolve, snapshots of the table at points in time, and safe reads while new data keeps arriving. The stream stops being a stream at exactly this point. It becomes a table, and it stays current because the materialization never stops.
The practical win is what disappears. There is no separate ETL job copying Kafka data into a warehouse on a schedule. There is no second version of the pipeline maintained for the analytics side, drifting out of sync with the streaming version. The streaming path and the table are the same path.
Stages 5 and 6: Athena and QuickSight
With the table in the Glue Catalog, Athena queries it with plain SQL. There is no cluster to size and no data to load. The query runs where the data already sits, on S3.
QuickSight sits on top of that for dashboards. The people consuming the data never touch Kafka or Flink. They open a dashboard, or run a SQL query against a table that was a stream moments earlier.
That separation is the point of the whole design. The streaming stack does streaming work. The analytics stack does analytics work. Iceberg and TableFlow are the seam between them, and the seam is a table format and a catalog — boring, standard, inspectable pieces.
Takeaways
- A streaming pipeline is only finished when the data lands somewhere queryable. The last box on the diagram deserves the most design attention.
- Kafka works best as the durable contract between producers and processing: ordered, replayable, and decoupled from both sides.
- Processing in Flink at event time removes the nightly batch re-processing step entirely.
- Materializing a stream into Iceberg with TableFlow makes one pipeline serve as both the streaming path and the analytics path.
- Analysts should query tables with SQL. They should never need to know a stream exists underneath.
Top comments (0)