DEV Community

Cover image for Is Amazon Data Firehose serverless? Kinesis vs Firehose, Flink and MSK on AWS
will peixoto for AWS Community Builders

Posted on Originally published at willpeixoto.dev

Is Amazon Data Firehose serverless? Kinesis vs Firehose, Flink and MSK on AWS

Two questions bring most people to this post, so here are the short answers. Yes, Amazon Data Firehose (formerly Kinesis Data Firehose) is serverless in practice. And the difference between Kinesis Data Streams and Firehose comes down to who reads the data: Data Streams keeps the stream for your own consumers to read and replay, while Firehose delivers it straight to a destination. Below I compare the four AWS streaming services (Kinesis Data Streams, Amazon Data Firehose, Managed Service for Apache Flink, and Amazon MSK, short for Managed Streaming for Apache Kafka) and show when to use each.

Some data can't wait. The suspicious transaction you need to block now, not tomorrow in a report. The user click becomes a sharp recommendation if you read it on the spot, and a missed chance if you read it later. For a long time we treated everything the same way: collect a pile of data, store it in a database, and process it in a batch later. But some things live in the now.

The picture that helps me explain this is a river. Notice that the water doesn't stop to be used. It flows by, and whoever is on the bank takes advantage of it along the way: one turns the mill wheel, another irrigates the field, another generates power. Nobody dams it all up first just to use it afterward. Now swap the water for data and you have data streaming: information arrives in a continuous flow and you react the instant it passes, instead of stacking it all up to process down the line.

That difference is what separates reacting from only finding out later. A bank that blocks fraud the second the transaction happens, an e-commerce that recommends on the click, a factory that adjusts the machine before it breaks: in all of them, processing later is almost not processing, because by the time the midnight batch runs, the moment is gone.

The trouble is that "streaming on AWS" gets confusing, because there are four services with similar names and even good people pick the wrong one. Come with me and I'll sort them out.

Updated November 2026. Streaming services move fast: mode, quota and price. If something doesn't match the screen in front of you, check the official docs for Kinesis Data Streams and Amazon Data Firehose (links at the end) and tell me in the comments so I can fix it.

Before the service, the concept: stream or batch

Batch is collecting a pile of data and processing it every so often, like the report that runs overnight. Streaming is processing event by event, as it arrives. They don't compete: each solves a different kind of problem. Month-end accounting close is batch and it's just fine. A fraud alert is streaming, because one minute of delay is money lost. The mistake is using batch where the business needs to react on the spot.

Which AWS services support continuous data streaming?

Four managed services cover it: Amazon Kinesis Data Streams captures and holds the stream, Amazon Data Firehose delivers it to a destination, Amazon Managed Service for Apache Flink processes it with state, and Amazon MSK runs Apache Kafka for you (with MSK Serverless if you don't want to size a cluster). They're the same four AWS lists under data streaming in its guide to modern data streaming architectures. AWS Lambda shows up next to them all the time, in the role of the consumer that reads the stream, and I show that one in the hands-on below.

The map: the four services (and what each one actually does)

Before diving into each one, two warnings that save you confusion. First, AWS renamed two services: Kinesis Data Firehose became Amazon Data Firehose (Feb 2024) and Kinesis Data Analytics became Amazon Managed Service for Apache Flink (Aug 2023). If you find a tutorial with the old name, it's the same service, just a new sign on the door. And Data Streams got a new mode, On-demand Advantage, which I'll explain in a moment.

Second, for those coming from Kafka: what's a topic there is a stream here in Kinesis; what's a partition there is a shard here. The map helps, but it isn't identical: Kinesis's PartitionKey runs through a hash that picks the shard, so several different keys can share the same shard. From here on I use each service's real name, but know that the two worlds mirror each other.

Data Streaming pipeline on AWS: producers send events to Kinesis Data Streams, which feeds a Lambda consumer in real time, Amazon Data Firehose delivering to S3, and Managed Service for Apache Flink processing to a dashboard; Amazon MSK appears as an alternative entry point.

Kinesis Data Streams: the river you can re-read

It's the heart of the story. The stream (the topic, remember?) is durable: producers write, consumers read, and the data is retained for a while (up to 365 days), so you can reprocess it. It's the "log" of the river. Use it when you need a durable stream, with replay, and you'll build the consumer (a Lambda, or an app with the Kinesis Client Library, the KCL, which handles shard distribution and checkpointing for you). Capacity comes split into shards (the partitions), each with a write and read ceiling, and you pick between three capacity modes:

  • Provisioned: you set the number of shards and pay for them, whether they're busy or not.

  • On-demand: AWS manages the shards for you and you pay for the throughput you use.

  • On-demand Advantage (the newest): brings warm throughput, with instant capacity for spikes, up to 10 GiB/s. One detail the sales page doesn't shout: it's an ACCOUNT billing mode, with a minimum usage commitment (25 MiB/s for at least 24h, charged even if you use less). In exchange, throughput comes out 60%+ cheaper and the per-stream-hour charge goes away. It makes sense from real volume up; for a small stream, stick with the first two modes.

Amazon Data Firehose: just deliver it to a destination

This one was renamed from Kinesis Data Firehose in 2024, same thing, new name. Here you don't write a consumer, it's the role Kafka Connect plays in the Kafka world (the sink connectors). You point at a source and a destination (S3, OpenSearch, Redshift, and others) and Firehose delivers, with buffering, optional transformation, and compression along the way. No replay, it's near-real-time delivery. Use it when the goal is to take the stream and drop it somewhere, with no consumption logic of your own.

Managed Service for Apache Flink: STATEFUL processing

This one was Kinesis Data Analytics, renamed in 2023. It runs managed Apache Flink for stateful processing, what Kafka Streams or ksqlDB do in the Kafka world: windows (summing per minute), joins between streams, and exactly-once on state recovery. When the question is "what's the moving average of the last 5 minutes per user," the answer lives here. The state, checkpoint, and recovery infrastructure is the service's responsibility; end-to-end semantics (what your sources and sinks guarantee) and state compatibility across versions remain your decisions.

Amazon MSK (and MSK Serverless): managed Kafka

If your world is already Kafka (ecosystem, Kafka API, portability across clouds, a team that knows it), MSK is AWS's managed Kafka. MSK Serverless provisions and scales capacity on its own and manages the topic's partitions, without you sizing a cluster. Use it when you need real Kafka, not a native equivalent.

When to use each

You want... AWS service Kafka equivalent
a durable, replayable stream with your own consumer Kinesis Data Streams Apache Kafka (Amazon MSK)
to just deliver the stream to a destination, no code Amazon Data Firehose Kafka Connect (sink)
stateful processing (window, join, aggregation) Managed Service for Apache Flink Kafka Streams / ksqlDB

In real life, a lot of architectures combine these services in three layers: ingest, delivery, and processing. Data Streams at the entrance, Firehose delivering a raw copy to S3 for history, and Flink processing in real time for a dashboard. And the entrance doesn't have to be Data Streams: you can have MSK up front and Flink after, because Managed Flink reads from both Kinesis and MSK.

Is Amazon Data Firehose serverless?

Yes, in practice. Amazon Data Firehose, called Kinesis Data Firehose until AWS renamed it in February 2024, has no servers or shards for you to provision, scales on its own with your throughput, and charges per GB ingested with no minimum fees or upfront commitments. AWS describes it as "fully managed" and doesn't use the word serverless, but there's nothing for you to size or patch, and you pay for what you send.

The docs say Firehose "automatically scales to match the throughput of your data and requires no ongoing administration." The rename only swapped the sign on the door: APIs, service endpoints and access policies stayed the same, so code written for Kinesis Data Firehose keeps working.

The bill follows the data. On the pricing page, you pay for the volume you ingest, and with Direct PUT or a Kinesis data stream as the source, each record is rounded up in 5 KB increments: a 3 KB record is billed as 5 KB. If your events are tiny, that rounding changes the math, so run it with your real record size. Format conversion (to Apache Parquet, for example), dynamic partitioning, and delivery into a VPC (Virtual Private Cloud) are optional and billed separately.

Delivery is near real time and depends on the buffer you configure. The buffering interval goes from zero seconds up to 15 minutes (900 seconds). Zero buffering arrived in December 2023, and with it AWS says most streams with no additional processing are delivered within five seconds.

Serverless doesn't mean unlimited, though. By default, each Firehose stream with Direct PUT takes 5 MiB/s, 2,000 requests/s and 500,000 records/s in US East (N. Virginia), US West (Oregon) and Europe (Ireland), and 1 MiB/s, 1,000 requests/s and 100,000 records/s in the other Regions, São Paulo included. When throttling starts, Firehose raises the limit on its own until the throttling is contained, and you can request an increase ahead of a spike you already know is coming.

What Firehose doesn't give you is replay. With Direct PUT, it holds records for up to 24 hours only while the destination is unavailable. That's a delivery safety net: once the data lands, Firehose has nothing left to reread. If you need to reprocess, put Kinesis Data Streams in front.

Kinesis Data Streams vs Firehose: what's the difference?

Kinesis Data Streams keeps the stream so your own consumers can read it and replay it, while Firehose reads the stream for you and delivers it to a destination. Back to the river: Data Streams is the river itself, where anyone on the bank can draw water, and Firehose is the pipe that carries the water to a reservoir.

Decision flowchart: if you need replay, several consumers, or your own logic or ordering per key, use Kinesis Data Streams; if the goal is just landing data in S3, Redshift, OpenSearch, Splunk, Iceberg or an HTTP endpoint, use Amazon Data Firehose

Kinesis Data Streams Amazon Data Firehose
Who reads the data your consumers (Lambda, a KCL app, Managed Flink), several at once Firehose itself, writing to the destination
Replay yes, 24 hours by default, up to 365 days no (Direct PUT holds data up to 24 hours only while the destination is down)
Latency as fast as your consumer reads buffered, from zero seconds to 900 seconds
Capacity Provisioned shards, On-demand, or On-demand Advantage automatic
Billing per shard-hour plus PUT payload units (Provisioned), or per GB written and read (On-demand, with a per-stream hourly charge in On-demand Standard) per GB ingested, plus optional features
Code you write producer and consumer producer only, or none if the source is a data stream, an MSK cluster, or a log service
Pick it when you need custom logic, multiple consumers, ordering per key, or replay the goal is landing data in S3, Redshift, OpenSearch, Splunk, Apache Iceberg tables, or an HTTP endpoint

And you don't have to pick just one. Firehose accepts a Kinesis data stream (or an MSK cluster) as its source, which is exactly the three-layer design from a few paragraphs up: Data Streams receives, your consumers react, and Firehose drops a raw copy in S3.

Hands-on: producer and consumer on Kinesis Data Streams

The producer writes events to the stream. Look at the PartitionKey: it runs through a hash that decides which shard the record lands in. Same key, same shard, and the shard is the territory where order can exist.

import { KinesisClient, PutRecordsCommand } from "@aws-sdk/client-kinesis";

const kinesis = new KinesisClient({});

await kinesis.send(new PutRecordsCommand({
  StreamName: "customer-events",
  Records: [
    {
      Data: Buffer.from(JSON.stringify({ userId: "u-42", event: "click", ts: Date.now() })),
      PartitionKey: "u-42", // same key = same shard (the territory of order)
    },
  ],
}));
Enter fullscreen mode Exit fullscreen mode

Now the gotcha I learned the hard way: PutRecords, the batch one, does not guarantee order. Not even with the same key. It processes each record individually, accepts partial success (half the batch goes in, half fails and you resend), and the docs are explicit: if you need to read in the order you wrote, the path is PutRecord in the singular, serial, chaining each write's SequenceNumberForOrdering into the next one:

import { KinesisClient, PutRecordCommand } from "@aws-sdk/client-kinesis";

const kinesis = new KinesisClient({});

const putEvent = async (payload, previousSequence) =>
  kinesis.send(new PutRecordCommand({
    StreamName: "customer-events",
    PartitionKey: payload.userId, // same entity, same shard
    Data: Buffer.from(JSON.stringify(payload)),
    // chain the previous write: this is what guarantees the sequence
    ...(previousSequence ? { SequenceNumberForOrdering: previousSequence } : {}),
  }));

const login = await putEvent({ userId: "u-42", event: "login" });
await putEvent({ userId: "u-42", event: "purchase" }, login.SequenceNumber);
Enter fullscreen mode Exit fullscreen mode

Order lives inside the shard: the PartitionKey runs through the hash and picks the shard; events with the same key stay in sequence in the same shard; across shards there is no global order. Batch PutRecords does not guarantee order, strict order needs serial chained PutRecord.

With that, the consumer always sees "login" before "purchase." The price is honest: you traded batch throughput for the sequence guarantee. If it's thousands of events per second and strict order isn't a requirement, PutRecords is the right call. The guarantee you ask for changes the bill you pay; that's what architecture is.

On the other side, the serverless consumer is a Lambda with an event source mapping on the stream. And here's the second truth that separates blog examples from production code: delivery is at-least-once, the same event can arrive twice. So the handler needs to be idempotent and know how to fail per item, without taking down the whole batch:

export const handler = async (event) => {
  const batchItemFailures = [];

  for (const record of event.Records) {
    try {
      const payload = JSON.parse(Buffer.from(record.kinesis.data, "base64").toString("utf8"));

      // In production: check and write an eventId to an idempotent store
      // BEFORE the side effect. A duplicate processed twice = a customer charged twice.
      console.log(payload.userId, payload.event);
    } catch (error) {
      // fail ONLY this item; the rest of the batch continues
      batchItemFailures.push({ itemIdentifier: record.kinesis.sequenceNumber });
    }
  }

  return { batchItemFailures };
};
Enter fullscreen mode Exit fullscreen mode

Returning partial failures only takes effect with ReportBatchItemFailures enabled on the event source mapping. And for real production I'd still add persistent idempotency, Logger and Metrics from AWS Lambda Powertools, a retry limit, a maximum record age, and an on-failure destination. A poison record can't stall the shard forever. I go through more of these resilience patterns, with code, in Serverless Isn't Failure-Proof.

One detail to close: order is per shard, not across the whole stream. If order matters for an entity (a user, an order), the key is that entity's. Otherwise, events spread across shards and global order becomes an illusion.

The honest trade-offs

  • Ordering and scale: order is per shard, not global, and strict order needs serial PutRecord (the batch one doesn't guarantee it). The partition key choice isn't a detail, it's a design decision that defines how your application behaves AND how it scales. A skewed key throws too much traffic at a single shard (the hot shard) and you lose throughput even paying for several; and each partition key handles at most 1 MiB/s, no matter the warm throughput you configured. This alone is worth a whole post, and it will get one.

  • Duplicates and observability: delivery is at-least-once, so replay and retry require an idempotent consumer; reprocessing an event that charges the customer twice just trades one incident for another. And consumer lag needs an alarm: streaming without observability turns into accidental batch, the event arrives now and the consumer processes it half an hour later. The Well-Architected Framework names these bills: Reliability asks for idempotency and recovery, Operational Excellence asks for metrics and alarms, Cost Optimization asks for a capacity mode that follows real traffic, not the team's hopes.

  • Retention and replay: Data Streams retains and lets you reprocess; Firehose doesn't, it delivers and moves on.

  • Cost: provisioned pays for shards even when idle; on-demand pays a per-stream-hour rate plus usage. A stream on 24/7 with predictable traffic is sometimes cheaper provisioned. Cost is an architecture decision, as Werner Vogels hammers in the Frugal Architect, and it's the same talk as availability has a price.

  • Latency: streaming is low, but not zero. Firehose still has buffering (seconds to minutes), so don't count on it for instant reaction.

MSK vs Kinesis: which one should you pick?

If you have no commitment to Kafka, pick Kinesis: it's simpler, native, and pairs better with Lambda and the rest of serverless. If you already live on Kafka (connectors, tooling, multi-cloud, a trained team), pick MSK and get Kafka without operating the cluster by hand. And if what scares you about Kafka is sizing brokers, MSK Serverless provisions and scales capacity for you and charges by throughput. The choice is about context and ecosystem; "which is best" is the wrong question.

What you take away

  • Data streaming is processing in the flow, in real time, to react in the moment (fraud, recommendation, IoT, the Internet of Things). Batch is for when the delay doesn't hurt.

  • Kinesis Data Streams is the durable, replayable stream; Amazon Data Firehose delivers to a destination; Managed Service for Apache Flink does stateful processing; MSK is managed Kafka.

  • Amazon Data Firehose (the old Kinesis Data Firehose) is serverless in practice: no shards, automatic scaling, pay per GB ingested. The trade is buffering and no replay.

  • Order is per shard, so choose the partition key with intent. And STRICT order needs serial PutRecord with SequenceNumberForOrdering: batch PutRecords doesn't guarantee it.

  • Lambda with Kinesis is at-least-once: idempotency and per-item failure (batchItemFailures) belong in the design from day one.

  • Provisioned or on-demand is a cost decision, and real architecture combines the services instead of picking just one.

Your turn

Do you use streaming in a project? Tell me which of these four made it into your design, and whether picking the wrong one has ever burned you. Drop that like, share it with someone who still processes everything in the midnight batch, and let's talk. Thanks a lot!

BUILD. SCALE. REPEAT. =D

Sources


Originally published on willpeixoto.dev.

Top comments (0)