RIP Amazon Data Firehose Change Data Capture

#dataengineering #aws #cloud #firehose

When a couple of weeks before re:Invent 2024, AWS announced in a blog post titled Replicate changes from databases to Apache Iceberg tables using Amazon Data Firehose (in preview) that we could now get a live copy of any relational database, as an Iceberg table stored in S3, I was quite excited, as it would make data lakes accessible to smaller companies. And for nearly a year, it really did fulfil its promise. Alas, I kinda of missed the key word here: preview. A few weeks ago, AWS ~~announced~~ let us know they would discontinue this feature, leaving us in trouble.

An amazing feature (despite a few shortages)

Quite soon after reading Sebastien's blog post, I started implementing this new capability. At first, setting this up was a bit cumbersome: you had to create a VPC Endpoint Service (why not! after all, using it meant enabling some AWS-owned machines connect to our database), configuration options were limited (destination database named was hard-wired after the source database).

Yet, it did an amazing job: for me it was truly a game changer for small companies to start a data lake / lakehouse

with ever-fresh data, making OLAP workloads accessible right after data was saved in the OLTP workload
without the heavy lifting of ETL jobs, managing upsert/merge, deletion events and so on.
with an announced pricing on a $/Gb basis, not a $$$/h (as Cloud customers we always like this serverless type of pricing, which makes it easy to have multiple environments without going broke).

Its only major needs for improvement were

the lack of configurable on-the-fly data partitioning. Data was always partitioned with the database primary key.
the absence of "reset" feature (full-copy in case the CDC job irremediably misses some transactions, for example due to the database reachability and short binlog depth).

I actually enjoyed this feature so much I submitted a Pull Request so it could be supported by the Terraform AWS Provider. By then, I was totally oblivious of it being a Preview feature.

With some customers, we've been using it since February 2025 in staging environment, March for production (though not yet with customer-facing applications) and we were about to have some customer-facing stuff relying on this next month.

When things go south..

Suddenly, all hell broke loose.

On Sept. 17th, in a casual support ticket, I learn with great horror that the Change Data Capture feature won't go GA, and that Preview was being discontinued.

However, I just want to inform you again that, as previously announced by us here at AWS, our Firehose CDC solution, which was initially offered as a preview feature, has been deprecated. Support for this feature officially ended on September 15, 2025.

Support engineer referred to a communication "that was distributed to all customers on August 24th" (maybe it did, but we couldn't trace it).
On Sept. 24th, the feature disappears in the Console. Which was critical for us, as we needed to recreate stream in case of failure (no buil-in redrive / full reload feature).
On Sept. 29th, support tells us

"The internal team has informed us that CDC Firehose streams will no longer be operational after September 30th, 2025."

When AWS leaves us with only bad options

Don't get me wrong: we should have been more careful. The first rule of Preview stuff is "don't use it in a production workload".
(in our defence, the fact it was in preview was visible neither in the Console UI nor in the documentation)

Yet, such a short notice is quite a hit. Especially when all options left are bad.

Support suggested we replaced it with MSK (Managed Service for Kafka) with Debezium, but running an MSK Cluster is at least $0.476/h = $4k per year, and MSK Connect $0.128/h = $1.1k per year, so it's a $5k at best to replace a feature that was so "serverlessly perfect".
DMS was my second idea, but for a serverless (mono AZ) option it already costs $900/y and it doesn't natively write to Iceberg table, so I should either keep writing to Kinesis Data Stream (another $450 at best) or use a Glue Streaming job ($3.7k). Another $5k
But.. didn't AWS announce Zero-ETL from RDS to Sagemaker Lakehouse in July??? Brilliant !!! Well.. it's not yet available in eu-west-3, where our workloads sit. And the new shiny SageMaker Unified Studio + Lakehouse stuff feels like another layer on already complicated, multi-layered service (so much that, as of today, one year after it was released and in spite of AWS intense marketing, the community hasn't added it to the AWS Terraform Provider).

The frustration of not knowing why.

It's ok for AWS to launch new features in preview. My bad for being too much of an early adopter here.

But getting no feedback on why this feature was killed is a little bit frustrating.

Was the feature technically fragile? Were the technical challenges of aforementioned improvements too complicated to solve?
Was it too costly to run to offer a competitive pricing? or to offer a $/Gb pricing model (I would have ended up using it on many small DBs with little activity)
Was it to make space for the Zero-ETL feature and Sagemaker Lakehouse? Then they could at least wait until they're able to give customers a nice migration path.
Was it for some internal politics? I hope not.

If I manage to get some insights from the service team, I'll let you know.

Back to step one.

TL,DR; don't use Preview features for production. Unfortunately that means unless you're a big company, you'd better wait to use them and not be an early adopter.

So now, we're back to ETL.

RIP Amazon Data Firehose Change Data Capture.
We'll miss you.
Dearly.