DEV Community: Robin Moffatt

Interesting links - June 2026

Robin Moffatt — Mon, 29 Jun 2026 13:39:39 +0000

June has been a busy month—113 links below for your enjoyment and delectation.

I’m going to share one extra link up here with you though, but it’s not my fault if it wrecks your productivity! My friend Kris Jenkins has written this devishly simple but addictive browser-based game: Escape the Moon.

AI

I warned you previously…this AI stuff is here to stay, and it’d be short-sighted to think otherwise. As I read and learn more about it, I’m going to share interesting links (the clue is in the blog post title) that I find—whilst trying to avoid the breathless hype and slop.

AI impact, big picture, and cultural impact on software engineering

I’ll stop featuring posts from Charity Majors when she stops writing such on-point and well-argued content. For now though, enjoy yet two more great posts:
- 🔥 Make AI Boring Again.
- AI enthusiasts are in a race against time, AI skeptics are in a race against entropy.
🔥 Armin Ronacher - Communities of Not.
DHH - Let the agents democratize open source.
Gergely Orosz - Why is Meta destroying its engineering organization?
Dian Fay - what llms are doing to conference programs.
What do you do if you maintain a testing library and feel that anyone using AI coding agents should be punished? What about having that testing library use prompt injection to delete your user’s code? That’s what happened with the jqwik project: jqwik: intent of printMessageForCodingAgents() — visible to agents, invisible to humans.

AI in action

🔥 DataDog - How we migrated a live routing system using AI-assisted refactoring.
🔥 PostHog - Andrej Karpathy’s AI Autoresearch found a 3-year-old bug in PostHog’s query engine.
Mark Rittman - Making Agentic Analytics More Accurate using Anthropic’s Agentic Data Stack and the Wire Framework.
DoorDash - Building a unified consumer memory for personalization at scale.
DoorDash - Building DoorDash Assistant: An engineering overview.
StateFarm - Using LLM as a Judge to Monitor AI Agents in Production.

Building with AI

🔥 Armin Ronacher - The Coming Loop.
Petrica Leuca uses Python and llama.cpp to explore the question: What’s an AI agent exactly?
John Kutay - Building an AI Database for Agentic GTM Operations.
Izzy Miller - How Hex built a lab to evaluate data agents.
Nicolas Fränkel - AI gateways: why and how.
Vicki Boykis - Running local models is good now.
Databricks have open-sourced Omnigent "a meta-harness for building and running AI agents". The Multi-AI agents caught my eye in particular as something that looks interesting to try.
ktx is an open-source project that describes itself as "a self-improving context layer that teaches agents how to query your warehouse accurately". You can see it in action from a couple of articles published by Madison Schott and Erfan Hesami.

Kafka and Event Streaming

🔥 Mina Tafreshi - Kafka Rebalances: What’s Actually Happening Under the Hood.
🔥 Michał Matłoka has been busy this month, first creating a tool that converts Kafka ACLs to Confluent RBAC, and then writing about a simulator for understanding how Kafka works. You can try the Kafka Simulator for yourself in your web browser. Related to this, he’s also written an article addressing the question Are Kafka Replicas Synchronous or Asynchronous?
OpenData’s Jason Gustafson and Almog Gavra (both Confluent-alumni) reckon there’s a better way to implement one of the two primary use cases for Kafka that they call 'routing', and they’re releasing OpenData Log to do it.
A trio of posts from Jack Vanlightly looking at Kafka Share Groups and beyond:
On the subject of share groups in Kafka, Sage Pierce has an alternative proposal, in the form of Atleon.
Another visualiser for Kafka, this one from Sandon Jacobs is specifically for KIP-932 Share Groups.
Trivago’s ZhongLi Shen details How We Cut Kafka Consumer Deployment Costs by 83%.
Ângelo Galvão shows how to Create a simple Mutual TLS (mTLS) authentication for Strimzi.
Sam Barker puts Kroxylicious through its paces, and shares a benchmark harness and results.
Elad Eldor - The Cheapest Kafka Consumer Is One That Doesn’t Read From Kafka.
Aswin A writes up their experiments with Multi-Cluster Kafka on Kubernetes with Strimzi.

Stream Processing

🔥 Apache Flink 2.3.0 has been released, including a Native S3 FileSystem that is detailed in this post.
- Apache Flink Agents 0.3.0 has also been released.
Francisco Morillo takes a look at the performance of Spark 4.1 Real-time Mode (RTM) compared to Flink.
🔥 Excellent deep-dive from Gorgias' Matthieu Bonneviot about their experience Building Billing on Apache Flink and the impact of event time on the system implementation.
Zander Matheson - Introducing the dbt Adapter for Confluent Cloud Flink SQL.
Katya Gorshkova - Calling LLMs from Flink.
Salva Alcántara has a nice write-up about Multi-Way Joins in Flink’s DataStream API.
Jevin Maltais was a guest on Data Engineering Podcast and talked about many Kafka-related topics including his project TypeStream for declaring Kafka Streams pipelines as config.
Yaroslav Tkachenko has launched Streamling, describing it as "a performant and extensible data streaming runtime built with the RAD stack (Rust, Arrow, DataFusion)".
Gilles Philippart has published a set of Data Streaming Cheatsheets under the banner of StreamSheets, including this one for Flink.
🔥 A deep-dive from Junaid Effendi looking at How Feldera Works: A True Incremental View Maintenance Engine.
On the subject of IVM, Databricks published a paper about Enzyme: Incremental View Maintenance for Data Engineering.

Analytics

🔥 Kyle Cheung - DuckDB Internals: Why is DuckDB Fast? (Part 1).
🔥 Building ClickCannon - a tool for benchmarking ClickHouse.
It’s ten years since ClickHouse was created, and their blog has a series of interesting posts reflecting on both the tool and the community around it:
- Alexey Milovidov - Ten years of ClickHouse in open source.
- Tyler Hannan - Ten years of open source: a stained-glass view.
- Al Brown - What a difference 10 years of open source makes.
- Al Brown - The open ecosystem around ClickHouse.

Data Platforms, Architectures, and Modelling

🔥 Jack Vanlightly asks: Can We Agree on a Storage/Workload Architecture Taxonomy?
Animesh Kumar et al assert that Most enterprise AI-for-data agents failed in 2025 because they lacked context, and Data Products, built above the engine, are the fix.
A recording and transcript of a talk that OpenAI’s Bonnie Xu did at InfoQ discussing their use of AI Agents to Make Sense of Data at OpenAI.
Pavlina Mitsou and Jonathan Warburton write about The Context Layer Behind Spotify’s Data Assistant.
Chen Chang and colleagues at Anthropic discuss How Anthropic enables self-service data analytics with Claude.
🔥 Fresha continue their run of excellent blog posts, with this one from Daniel Wiszowaty: Everything Everywhere As Of Once: Rebuilding Postgres Inside Snowflake.
Yordan Ivanov - How I Made My Data Platform’s Failures Public and Earned My Stakeholders' Trust.
Goutham Budati - Why Technically Excellent Data Teams Still Fail.
Booking.com’s Tiago Ferreira writes about scaling a shared DWH across multiple teams without turning governance into a delivery bottleneck.
Amer Hesson and colleagues at Netflix discuss Managing Data Assets at Netflix Scale.
Uber’s Daniel Musgrave and colleagues write about Simplifying Data and Product Integrations with a Data Abstraction Layer.
Details of GuideWire’s Federated Query Platform for ML at Scale.
Patrick Lam - Airbnb evolved its data architecture.
Rohit Channe and Simran Mirchandani - How Lyft Governs and Scales Key Data Definitions.
René Luijk discusses the use of the C4 model in the context of data platforms.

Data Engineering and Pipelines

🔥 Joe Reis - The Turf Wars Are Over. Time to Cross-Train.
🔥 Ben Rogojan - In 2026 The Data Fundamentals Matter More Than Ever.
Madison Mae - 4 Analytic Engineering Fundamentals That Haven’t Changed.
A reminder of Roche’s Maxim of Data Transformation:

Data should be transformed as far upstream as possible, and as far downstream as necessary.
Bruno Masciarelli - How to Build a Simple, Bulletproof Data Pipeline.
Couple of good posts from Joshua Kim about building a pipeline from scratch with dbt part 1 / part 2.
Charly Clairmont - Why dbt-state Matters More Than You Think.
Joachim Hodana - 5 dbt mistakes I see in every startup.
dbtLabs published their 2026 State of Analytics Engineering Report, with some interesting data around use of AI (ofc).
Details of dbt Core v2 from Joel Labes and Grace Goheen.
Vikas Rai describes the Data Profiling Framework at Halodoc.
Artem Golubin - Using local ClickHouse for data processing.

CDC

🔥 George Zefko - Building a CDC pipeline, part 3: From Kafka events to an analytical event log (previously: part 1, part 2).
Andreas Andreakis - Why DBLog Is Snapshot-Equivalent (see also the original DBLog paper and the more recent A Theoretical Study of DBLog: Certified Virtual Cuts - both also by Andreas).
DoorDash’s Vinay Chella and Akshat Goel spoke at InfoQ about Write-Ahead Intent Log: A Foundation for Efficient CDC at Scale.

Open Table Formats (OTF), Catalogs, Lakehouses etc.

🔥 Gunnar Morling has announced the 1.0 release of Hardwood, which is, in his words, a fast, lightweight Apache Parquet reader for the JVM. He’s published some impressive benchmark figures too, showing just how much of a performance improvement can be had when reading Parquet files by using multi-threading.
vortex-java is a project that builds on Hardwood, implementing the Vortex columnar format in pure Java.
If you want to see first-hand how open-source projects navigate issues that are not straightforward and have strong arguments on either side, this issue on the Iceberg project is fascinating: #14797 Implement Iceberg Kafka Connect with Delta Writer Support in DV Mode.
The Apache Hudi project has been busy, releasing 1.2.0 with support for Stateless Global Upserts for Flink, along with publishing details of how Hudi is used at Southwest Airlines and Penn Entertainment.
Sivabalan Narayanan discusses on the Hudi blog Why Metadata Has to Be Mutation-Friendly.
Iceberg Doctor is a tool from Sarthak Singh for diagnostics on your Iceberg metadata.
A three part series (1, 2, 3) from Giannis Polyzos about Apache Fluss and lakehouse tiering, along with a separate introductory overview to the concepts.
A useful post from Alex Merced about partition evolution in Iceberg.
Two posts from Soumil Shah about shredding VARIANT fields in Iceberg.

RDBMS and General Data Stuff

🔥 Kelsey Hightower - Thoughts on Open Source (2024).
🔥 Datadog’s Shree Sampath writes about how they implement HA for Postgres on Kubernetes.
Warner Music Group’s Yask Srivastava describes why we shrank our TimescaleDB chunks from 30 days to 7.
Tom Pang - The only scalable delete in Postgres is DROP TABLE.
pg_durable brings durable execution to Postgres.
🔥 This is less "data" blog post and more just "cool tech blog post" :) Conor Gallagher from Zalando details how they do Client-Side Load Balancing at a Million Requests Per Second.
Eric Sun - The Join-Aware Materialized View Query Rewrite Gap.
A good write-up from dbt’s Tristan Handy about recent tech events and trends: Hunting for Tokens. Snowflake Summit. Agent Use Cases.

And finally…

Nothing to do with data, but stuff that I’ve found interesting or has made me smile.

Work

You’ll Get Re-Orged Again This Year. Here’s How to Be Ready.
🔥 A great write-up from Apurva Mehta about his company, Responsive, and what went wrong: Our first customers were the exception.
Simonida Jovanovic - How do you sell a forever-free product? PostHog’s answer is 10 emails and a pet hedgehog.

Life

🔥 Casey Neistat - yeah, 730 days no exceptions.
Murat Demirbas - 5 Lessons at 50.

Fun

TikTok is my guilty-pleasure, and this account always brings a smile to my face for how completely and utterly weird-yet-engaging it is: Le Triangle.
I definitely am adding this to my todo-list: Writerdeck. Not sure if this one is Work or Fun. I’ll probably convince myself it’s the former whilst doing the latter ;)

Misc

🔥 I love this: using ML to read burnt papyrus scrolls from Vesuvius. You can find details of the full project here.
The Cypherpunk Library.
The Absurdly Optimized Pancake.
I started 'scrobbling' my music well over 20 years ago, so this one was a blast from the past for me: Last.fm is now independent.

Interesting links - May 2026

Robin Moffatt — Fri, 29 May 2026 10:00:19 +0000

Welcome to May’s Interesting Links! This month saw the Current conference in London with the usual 5k run, lots of familiar faces and friendly conversations—and plenty of excellent breakout sessions too. It seems live-tweeting conferences isn’t a thing any more, with only myself and Thomas Cooper seeming to post anything, but if you want you can go review the hashtag feed on BlueSky for some highlights of the conference.

I got my first Hacker News front page hit with AI Slop is Killing Online Communities (51k views and climbing!), and a nice little halo boost for another rant from earlier this year, AI will fsck you up if you’re not on board.

Oh, and I got involved in some thought leadering over on LinkedIn (which a non-zero number of people thought was serious) with my shitposting about fried breakfasts.

Kafka and Event Streaming

🔥 Apache Kafka 4.3.0 has been released. Check out the release announcement, as well as a video from Sandon Jacobs covering the new features.
🔥 After a few quiet months on his blog, Jack Vanlightly is back with a bang! He’s written a new tool, Dimster, a performance benchmarking tool for Apache Kafka, and has written several more blog posts off the back of it:
- Benchmarking Apache Kafka Consumer Groups vs Share Groups (overhead test).
- Kafka Share Groups and Parallelizing Consumption Part 1: Tuning max.poll.records, Part 2: Producer Batches and share.acquire.mode.
🔥 I had the absolute pleasure to watch Victor Rentea present at Devoxx UK earlier this month. This guy redefines what it means to be an entertaining, energetic, enthusiastic—and educational presenter. Whilst his specific talk, "Event-Driven Architecture Pitfalls" isn’t online yet, you can find the slides here, and a recording from Devoxx last year of a similar talk.
The Parallel Consumer library from Confluent has been marked as no longer maintained, prompting a discussion of alternatives (and the concept itself) on LinkedIn, as well as a fork from one of the original authors, Tony Stubbs.
Mariano Gonzalez - Benchmarking KPipe against the parallel-Kafka libraries you would actually pick.
Michel Tricot - Event-Driven vs. Polling Architectures for Agent Triggers.
An interesting idea from Florent Ramiere and colleagues: what if you specify a set of interesting additions to Kafka’s functionality, with strict rules around the implementation, and then have LLMs take their best shot at it? You can see the ideas and results in the branches of this repository.
Viquar Khan - Architecting Cloud-Native Kafka: From Tiered Storage Towards a Diskless Future.
Elad Eldor - Kafka’s Real Compression Problem Is Batch Depth, and Kafka Compute Is Cheap. Network Is Not.
Kroxylicious version 0.21.0 has been released, and Sam Barker from the Kroxylicious project has been running some benchmarks to look at the impact that the proxy has, both pass-through and when encrypting records.
Aiven’s Juha Mynttinen explores why they think Apache Kafka Deserves Topic Types.
Details of a Coinbase outage involving their Kafka provider, which based on blogs from 2022 and 2023 is MSK.
Andy Muir - Kafka Schema Registry doesn’t guarantee compatibility (and what actually does).
Bruno Cadonna - OpenData Buffer: HA pipelines without Kafka.
Jeffrey J. Jennings - Kafka’s quiet observability superpower - Kafka Interceptors.
Grzegorz Kocur - Do Kafka metrics have to be so difficult?

Stream Processing

Flink’s Stateful Functions (StateFun) is not maintained by the project any more, so kzmlabs' Oleksandr Kazimirov forked it to continue developing it.
Olena Vodzianova - How Chandy-Lamport Inspired Apache Flink Checkpointing.
🔥 Two good posts from the team at Grab:
- Enhancing Flink deployment with shadow testing.
- Details of how they built their one-click data ingestion platform with Apache Flink.
Details of how Smartsheet use Flink for optimising both costs and performance by filtering messages.
flink-state-explorer is, as the name suggests, a tool for exploring Apache Flink 1.20 canonical savepoints interactively.
A hands-on github repo from Patrick Neff showing off Stream processing pipeline using dbt and Flink on Confluent Cloud.
Shuva Jyoti Kar - Designing stateful serverless Agentic Loop with Kafka and Flink.
A couple of security issues for Flink to be aware of if you’re running it:
- CVE-2026-35194 (SQL injection).
- CVE-2026-40564 (K8s operator).

Analytics

🔥 Tristan Handy - BI’s Second Unbundling.
A good writeup from Cloudflare’s James Morrison and Christian Endres about tracing performance issues in ClickHouse.
Several posts from StarRocks covering new features in 4.1:
Two BigQuery optimisation/cost saving articles, from Christophe Oudar and Azeem Jalageri.
Daniel Beach - Spark is Dead. Long Live DuckDB.
Alibaba added DuckDB into their fork of MySQL, AliSQL, providing storage and query for OLAP workloads.
Simon Aubury - I don’t need an untrusted LLM to tell me I’m spending too much on coffee.
The DuckDB team announced Quack: The DuckDB Client-Server Protocol.
Ben Fleis explores DuckDB’s support for Delta and Unity Catalog.
🔥 I’ve been a fan of Mark Litwintschik’s no-nonsense blog posts showing current technologies and exploring interesting data sets for many years. In this one he uses DuckDB to analyse details of 10K+ Satellites in Space.

Data Platforms, Architectures, and Modelling

🔥 Nikola Ilic - Data Modeling for Analytics Engineers: The Complete Primer.
AirTable’s Matthew Jin details how they optimised their costs by moving PBs of cold data from MySQL to S3, and wrote a query engine using Data Fusion to serve it.
Brian Brunner and his colleagues at Cloudflare published details of how they built Cloudflare’s data platform and an AI agent on top of it.

Data Engineering, Pipelines, and CDC

Caesario Kisty - A Practical Implementation of Medallion Architecture Using ClickHouse.
Xinran Waibel - Data Engineering Open Forum 2026 Recap.
🔥 After doing a bit of fairly naïve experimentation with Claude and dbt earlier this year, I was very interested to read Jason Ganz’s article What data agent benchmarks do and don’t tell us, and hope to try out the referenced ADE-bench ("a framework for evaluating AI agents on data analyst tasks") soon.
Whilst Thijs Nieuwdorp’s article about Handling Schema Issues in Polars is specific to Polars, it’s a useful reference for the kind of schema changes one will want to make in a data pipeline, and the challenges it can cause depending on how or if your implementation technology of choice supports it.
🔥 Pedram Navid - We need to talk about dbt.
A summary/re-write by Alex Yu (a.k.a. ByteByteGo) looking at How Figma Upgraded Data Pipeline from Multi-Day Latency to Real-Time (based on the original blog post by Yichao Zhao from last year).
Netflix - The Evolution of Cassandra Data Movement at Netflix.
Alexey Makhotkin has a two parter on Data Quality part 1 / part 2.
Chris Hillman - Don’t Go Dark: Visibility Is a Data Engineering Skill.
After the excellent survey and results that Joe Reis published about data engineering earlier this year, he’s now following up with a survey on The Organizational State of Data Engineering (open for submissions until Sunday, June 21).
Mahendran Vasagam - From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines.
Dana Rabba - Building Self-Healing Data Pipelines at Halodoc.
rocky is a dbt alternative that looks quite interesting. It describes itself as "the trust plane for your warehouse", and targeting Databricks users primarily, with Snowflake and BigQuery to follow. There’s a built-in playground feature that’s worth poking around to get a feel for it.

CDC

Marc Bowes describes how Aurora DSQL’s CDC feature works. If you want more, there’s further details of its use from Vijay Karumajji.
🔥 George Zefko - Building a CDC pipeline, part 2: Debezium Internals (I featured part 1 last month, if you missed it it’s here).
A couple of good posts from the Debezium team:
- Chris Cranford discusses What Nobody Explains About Debezium in 2026 (But Should).
- Jiri Pechanec explores Change Data Capture with Debezium and Jupyter.

Open Table Formats (OTF), Catalogs, Lakehouses etc.

Apache Iceberg 1.11 has been released (I even got some small contributions merged 🎉). There are more details of the release in these blog posts:
- Alex Stephen & Talat Uyarer - Announcing Apache Iceberg 1.11.0.
- Ajantha Bhat - Apache Iceberg 1.11.0 Adds registerView: Closing a Catalog Migration Gap.
- Alex Merced - An In-Depth Overview of the Apache Iceberg 1.11.0 Release.
🔥 The talks from Iceberg Summit 2026 are now online.
Alex Merced begins an epic 15-part series about Apache Iceberg by looking at What Are Table Formats and Why Were They Needed?
Yelp’s Nick Del Nano looks at How Partition Access Visualizations Reduced Data Lake S3 Cost by 33%.
Honest words from Fresha’s Samuel Valente as he looks at the use of Iceberg with Snowflake in practice: Snowflake with Iceberg: Lakekeeper, dbt, and some Sparks Flying.

RDBMS

Daniel Guzman-Burgos describes bintrail which provides time-travel SQL for MySQL. Renato Losio has a summary on InfoQ.
Teiva Harsanyi - How Linux 7.0 Broke PostgreSQL: The Preemption Regression Explained.
Radim Marek covers the ORDER BY jungle, as well as PostgreSQL’s TOAST.
Markus Winand also looks at ORDER BY and the evolution of support in different RDBMS.
🔥 James Blackwood-Sewell writes up details of the benchmarking platform they built, whilst Ben Dicken muses on benchmarking at PlanetScale too.

General Data Stuff

An opinionated, and fairly concise, set of recommendations for the use of different Open Standards for Modern Data Architecture.
LinkedIn’s Pratikmohan Srivastav writes about a performance troubleshooting experience - The 58-Million-Key Freeze: What a HashMap Resize Taught Us About Memory Allocation at Scale.
Sem Sinchenko - Same buffers, same instructions, same hardware. Where Is the JVM Tax?
Gergely Orosz shares some excerpts from Martin Kleppmann’s second edition of Designing Data Intensive Applications.

AI

🔥 Ben Evans - AI Eats the World.
🔥 TikTok is my guilty pleasure, but instead of dogs misbehaving in comical ways, here’s an excellent piece to camera from Scott Hanselman reflecting on the impact of AI in our lives as software developers.

Pro-tip:
yt-dlp works great with TikTok, so you don’t have to actually open the page if you still wanna view the video.

Nate Berkopec - Thoughts on LLMs in 2026.
Julien Hurault - Time for AI Coding to Turn Boring?
Kate Holterhoff - AI Slop & the Vulnerability Treadmill.
Paulo Arruda - What I Learned Building Multi-Agent Systems from Scratch at Shopify.
🔥 Loris Cro - Contributor Poker and Zig’s AI Ban.
Lucia Cerchie - Why You Need More Than a SKILL.md.

And finally…

Nothing to do with data, but stuff that I’ve found interesting or has made me smile.

Work and Career

🔥 An oldie (2008!) but a goodie: Jeff Atwood - Don’t Go Dark.
🔥 Lara Hogan - Be a thermostat, not a thermometer.
As an IC, I endorse this pitch from Elena Verna ;-) IC work is the new career flex.
🔥 Ana Rodrigues - It’s 2026 and women are still asked to teach others to think a little bit and not be a prick.
Leyla Kazim - I did no work for a year and no one noticed.

Community and Blogging

🔥 Kevin Powell wrote this article which resonated hard for me. I think it’s a boiling-frog situation; if I think about my motivation to write today, vs a year ago, vs 5, it’s definitely very different. AI noise drowns things out, kinda like SEO marketing 'content factories' did but on a bigger and more destructive scale, so as an author is it even worth writing original material? Is anyone even gonna read it?
An excellent writeup from Vicki Boykis about Tagging my blog posts with BERTopic and LLMs - definitely need to try this.
Mike McQuaid - Open Source Resistance: Keep OSS alive on company time.
Very cool idea for conference badges from Shy Ruparel, with an excellent writeup to boot.

Read, Watch, Listen

I couldn’t think of a good subheading for these :)

Some fun nostalgia, with screenshots of various old OSes at typewritten.org and The Virtual OS Museum (the latter even has, IIUC, runnable VMs for download!)
Dan Carlin is probably my favourite podcaster, and as well as his well-known Hardcore Histories he has occasional thoughts on more current affairs, including this one: The Water in Which We Swim.
The Middle Class Museum (A memorial to affordable living).
📷 The Classic American Diner.
Is my blue your blue?
Fast16: The Cyberweapon That Predates Stuxnet by Five Years.

Unresolved directive in <stdin> - include::../../asciidoc-includes/il-footer.adoc[]

AI Slop is Killing Online Communities

Robin Moffatt — Thu, 07 May 2026 09:25:55 +0000

Like a young child coming home from kindergarten with their latest crayon scrawls, the internet is currently awash with people sharing their AI-generated work. And just like the young child's drawings, much of that work should be proudly put up on the walls within the artist's house---and no further.

Prologue: I ❤️ AI

I am not an AI-hater. In fact, I think AI-haters are on the wrong side of history.

It's just that I know when to keep my crayon-drawings to myself ;) And I am getting increasingly sad and frustrated seeing communities that I value slowly wilting under the onslaught of shit. Often that shit is perhaps naïvely shared with no deleterious intent, but shit nonetheless it is.

Congratulations, you entered a prompt and pressed return.

"I rewrote Kafka in COBOL"

Great, enter it at your next science fair. Meanwhile stop begging for stars on your brand new GitHub repo that no-one's touching with a bargepole.

"I wrote a blog post about Kafka"

Did you though? We can tell that Claude wrote it, and it's a piece of garbage.

"I made this video about Kafka"

Cool story bro. Except AI made it, and it's only of interest as a curiosity, not a useful learning artefact.

"I'm self-publishing an ebook that I wrote about Kafka"

What you mean is, you got Claude to scrape the internet and crap out a "book" that you should be ashamed to give away for free.

Any fool can feed coins into a fruit machine and pull the arm.

Step 0: Profit

The pattern I see over and over seems to be:

Step 1: Discover agentic coding. Mind blown.
Step 2: Chuck a project up onto GitHub (if it's actually up </snark>).
Step 3: Have AI write a breathless blog post about your vibe-coded project. Share blog post and repo to any subreddit and Slack group that you can find. Not sure which is suitable? Post to all of them---people will love to see it! /s

Let me tell you now: pause after step 2. Take a really long breath. Think really hard about what you've created, and why you want to share it. If it's "because it's cool" then I've got news for you: agentic coding is no longer a novelty. It's just how shit gets done now.

If you can think of the prompt, AI can write it. Big deal. That's so early-2026. Move on.

Still want to share it far and wide? Is it actually useful? Are you using it? Has it got really good documentation? Is it usable? Have you actually come back to the code again and again and put it through its paces? Or was it a one-night stand with Claude and the next morning neither of you thinks it was such a good idea?

Still want to share it? If it's software, are you prepared to stand behind it as something people will raise issues against, maybe submit PRs for? If it's written, is it something you'd want to read? Is it actually adding to the cumulative understanding of the community, or is it just an LLM auto-completing its way through text that you can't be arsed to write and I can't be arsed to read?

Who cares?

No one forces me to read this stuff. Why am I so bothered by it?

Because like bindweed, it's slowly strangling the organic life out of communities. When I open up Reddit now, it's increasingly overrun with vibe-coded AI stuff. Whilst much of it is well-intentioned I'm sure, it does nothing to contribute to the community.

AI slop is driving up the noise, and making the signal more and more difficult to discern in communities. This risks becoming a downward spiral; as communities become more polluted by this stuff, members will get frustrated from wading through AI slop and draw back, thus diminishing the life of the organic community even further.

Carrying on like this, online communities will either wither and die, or converge on something like the dystopian-but-banal MoltBook in which AI agents "talk" to each other with no humans present.

There's good 'slop' and bad slop

You may have noticed that AI Slop has become the mot du jour.

The broad use of the term that I'm generally familiar with is as a negative description for low-effort material created by AI and foisted upon those to whom it is of no benefit. However I learnt recently that there are those---probably correlating strongly with the AI-hating crowd---who brand anything written about AI as "AI Slop", even if not written by AI.

Material created with the assistance of AI is not bad in itself. It's the purpose to which it's put.

A good use of AI is when it enables people to do something they couldn't do before, to contribute to a community when they couldn't before. Done with the care and good intent of a human behind it, this is a nett positive.

Bad AI slop, on the other hand, is monkeys throwing crap over the fence for a purpose other than furthering the community. This includes spam, engagement farming, and simply thoughtless noise in a space which is not for that purpose.

OK, but who made you gatekeeper of the internet?

The standards of common decency and taste, that's who.

Let's take a step back. Sharing content online is wonderful. It's pretty much what made the internet what it is today.

The knack is to understand what you're sharing, to whom, and why.

If you were born before around 1980 you'll know that there was the Geocities era. Every high-school nerd had a homepage (mine was in Vienna since you're asking).

Just because I built a homepage on Geocities, complete with 'Under Construction' anigifs, a web counter and a web ring banner, does not mean that I should be sharing it to anyone who'll listen. Amongst my friends, sure. My parents, of course---they'll be proud of anything I build. But to the general internet? Who cares.

And now with AI-generated content, whether a vibe-coded app or a blog post, the same applies. The internet went through a collective convulsion in early 2026 as everyone discovered the power of Claude Opus 4.5 (and don't get me wrong, it is damn cool). And what does anyone do when they discover anything neat? They want to share it with their friends!

Combine that with the deafening AI-hype machine of grifters already in overdrive---and suddenly subreddits and Slacks are overrun with AI-generated material.

So what should I share?

Built with AI, not by AI

This excellent section heading is taken from my friend and colleague Gunnar Morling's recent article. As I outlined above, AI is a powerful tool, and I will argue with anyone for the case that it's pretty much a dereliction of one's job to not be including it in one's toolbox. Gunnar nails the nuance though:

Build with AI.

AI is just a tool.

You need to do the thinking, the instructing, the checking.

Gunnar has built a fantastic new project (Hardwood; a new parser for Apache Parquet), using AI. Does that mean it falls foul of my wrath and judgement? No, of course not. It's a project that's taken four months so far, with a solid roadmap, a burgeoning community, and a thoughtful and careful design behind it.

Contribution

Does your offering contribute anything to the community?

If you boil it down to its essentials, is what you're sharing anything other than the manifestation of a prompt fed into an agentic coding tool? If I took your prompt and ran it, would I end up with something similar? Prompt engineering is fun and an interesting study, but it's tangential to the subject itself. Consider a community of ornate furniture enthusiasts (I'm sure such a thing exists); it's the equivalent of bombarding them with Ikea-esque pieces simply because you've got a really interesting set of chisels that you want to show off.

Just like I'm not posting my kid's drawings off to the National Gallery just yet, I'm also not sharing every cool app that I can build with Claude. Not that "software is art" (though some of the best actually is), but there's nothing much interesting in the puerile output of a process. Anyone with a few tokens can prompt their way to a bit of software. Throwaway tools are just fine. They're great, in fact---the internet is built on weird little scripts that people have built and shared. But chuck them on gist/GitHub---they don't need a launch blog post as if you're the incarnation of Steve Jobs.

Respect the community

This is a tale as old as time. Well, the internet anyway, once it got beyond ARPANET and BBSes.

Whether Usenet, Reddit, lobste.rs, or any other online platform, the netiquette is always to "lurk". Hang around, read what gets written, get a feel for "the vibe".

I'm not the arbiter of what's acceptable in a given community. The community members are. Vibed an amazing new implementation of the Kafka protocol, but not sure if people want to see it? Read the room, and get a feel for whether they'll welcome with open arms your AI slop---or not. If in doubt, ask!

As well as lurking, another way of showing respect to the community is to be very open and clear about if, how, and where you're using AI in your contribution.

The Asymmetry of Bullshit

What's the impact on others of your contribution?

The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.

--- Alberto Brandolini

If you splurge out a gobbledegook article, you're putting that workload onto your reader to realise that it's not worth wading through. If you dump a complex PR into a project without due care, you're obligating the reviewers to go through the code and for them to explain to you why it can't be merged. In both of these scenarios the community would be better off without your contribution.

Unfortunately the sheer volume of AI slop (and I mean that in the bad way) has meant that communities and projects are struggling to deal with the impact of contributions, in some cases even slamming their doors shut on anything that's touched AI. Projects like Vouch have sprung up to try and address this, but the Rubicon has been crossed and it's unclear where things will end up.

Pre-AI, the effort required for contributions was sufficient proof of work to either deter people or demonstrate an actual commitment. Communities could deal with sub-par contributions. Those well-intentioned and willing to learn could be mentored and often would develop into important members of the community. Those less well-intentioned and not doing much more than spamming could be dealt with because the volume was so low.

With great power...

or perhaps that should be "With a great number of tokens"

Communities are powerful yet fragile things. Don't be the bindweed that suffocates the life out of them.

Explore with great joy the power that LLMs and agentic coding tools bring. Enjoy the frisson of jfc **that* is cool* that it invariably brings.

But respect the community, and only share what is truly relevant. Save the crayon pictures for your kitchen fridge.

Bindweed photo by Joshua Ralph on Unsplash.
All other pictures by my kids. Which is ironic, given my exhorting for people to literally keep their childish drawings to themselves ;)

🏃🚶 The unofficial Current London 2026 Run/Walk 🏃🚶

Robin Moffatt — Tue, 05 May 2026 10:51:06 +0000

Another year, another Current---another 5k run/walk for anyone who'd like to join!

Did I mostly copy-and-paste this from last year's post? You bet I did!

This is a tradition that started back at Kafka Summit San Francisco in 2019 over the Golden Gate Bridge and has continued since then.

London's Docklands offer up what would charitably be described as not the most...picturesque of routes---but with a group of similarly minded people, what could be a better way to start day 1 of Current 2026 London than a 5k run or walk :) Plus, it's flat! 🙌

Who? What? Where? When?

Who: Anyone who would like to is very welcome :)
What: A 5k (3.1 miles) run or walk
Where: Starting from the Aloft Hotel (next to the ExCeL centre), heading anti-clockwise around the quayside at ExCeL.
When: Tuesday 19th May at 07:30 BST.

We'll meet at 07:30 in the carpark in front of the hotel. That'll give us time to congregate and take some #streamingselfies if you want them, before setting off at 07:35 sharp to make our way around the 5k route.

Go at your own pace; this isn't a race. You can run, or walk. Or skip, if you prefer. You can download the route to follow at your leisure. In fact, it's a good idea to do so because this is not an organised event, so it's your responsibility not to get lost :-D

If you Strava, feel free to join the Strava Club.

Cool, where do I register?

This is an entirely unofficial, barely-organised, thing. Turn up and run, if you want. Don't, if you don't. Fully intend to but sleep in from the jetlag and miss it, that's completely fine 😁

It's the Smell

Robin Moffatt — Tue, 05 May 2026 10:40:51 +0000

It's now a joy to simply read any blog post that's not AI-generated.

If you actually have a good thing to write about, for the love of your readers and the good of the internet, please just write about it. Claude might shit out a completed article but it'll stink. Just like boomer art, readers are recognising this crap at a hundred yards and running away from it. It's horrible to read, it makes me feel gross just opening it, and it's going straight in /dev/null.

Use AI to refine what you want to say, but I (and I'm sure many others) would rather read something imperfect-but-real than a pristine-but-anodyne article that's in the same style as every other bloody post these days.

Interesting links - April 2026

Robin Moffatt — Thu, 30 Apr 2026 14:21:40 +0000

A bit of a streamlined edition, this month. Lots of interesting links still, but less commentary. You can put that down to me prevaricating on getting my previous blog about Materialized Tables in Apache Flink finished, and leaving myself little time to work on this one :) Not including the detailed narration actually knocks a bunch of time off the preparation---I'd be interested in your feedback as to how much the absence of narration impacts (if at all) your enjoyment of reading it. Let me know in the comments below!

Something that I'm slowly changing is how I categorise links to do with AI. A few months back anything "AI" got its own section. It wasn't much more than a novelty really; certainly not something worth distracting the regular link sections with. But now AI is just part-and-parcel of many people's workflows, a regular component in their toolbox. So where an article is about credibly using AI as part of an existing topic (such as data engineering), I'll file it in that section. (And if this news makes you cross because you abhor anything AI, well, I've got news for you).

Current London 2026 - wanna free ticket? 🎟️

If you're in the UK and interested in Kafka, Flink, Iceberg, etc etc (which, since you're reading this blog post, I assume you at least have a passing interest in) then you might be interested in Current London in May---and I have a free ticket code for you to use! Register with code L-CMP-LDNKafka and it's all yours :)

Unresolved directive in <stdin> - include::../../asciidoc-includes/il-header.adoc[]

Analytics

Ben Sykes - Interval-Aware Caching for Druid at Netflix Scale
Dorothée Clerc - How BlaBlaCar PMs use AI to self-serve data
DuckDB 1.5.2 has been released, with support for DuckLake 1.0, even better Iceberg support, and fixes as a result of initial Jepsen testing.
Randy Au - Dashboard rot as org attention grave markers
Ahmed Youssef - Nobody Is Making Decisions With Your Dashboards
🔥 Torsten Grust has published a course about the Design and Implementation of DuckDB Internals
Hamel Husain - The Revenge of the Data Scientist

Data Platforms, Architectures, and Modelling

Antonia Badarau and team at Monzo - A "meshy" approach to Data: Enabling 100+ teams to build Data Models
Justina Bartulevičienė & Benediktas Kazanavičius (Vinted) - Serving Personalised Search Autocomplete
Rishabh Kumar (Airbnb) - Building a fault-tolerant metrics storage system at Airbnb
Matt Lawhon and team at Pinterest - Scaling Recommendation Systems with Request-Level Deduplication
Facundo Agriel (Dropbox) - Improving storage efficiency in Magic Pocket, our immutable blob store
🔥 A couple of interesting posts from the teams at Notion: Enabling Multi-Region Data Systems, and Two years of vector search: 10x scale, 1/10th cost
Nikola Ilic - Data Modeling for Analytics Engineers: The Complete Primer
Chris Gambill - The Medallion Masterclass: Why Knowing the Colors Isn't Enough
Joe Reis - Why Time Matters in Data Modeling

Data Engineering, Pipelines, and CDC

Alexander Goida - Three Kafka S3 Sink Settings for Easier File Processing
Chris Gambill - AI Agents are Failing Your Data Engineers
Sugat Mahanti (Zapier) - Lessons from using the outbox pattern at scale
Couple of good posts from Chris Hillman - Your Data Platform Costs More Than It Should, and Why Your Pipeline Finishes Later Every Month
Jin-won Park (Karrot) - In the AI era where everyone handles data, how has the data team changed over the past year? (original Korean)
Tristan Handy (dbt Labs) - Five things I believe about the future of analytics
Igor Shurmin (Riskified Tech) - Data Exploration for Software Engineers: Evaluating and Integrating External Datasets
Aleksandr Klein (Just Eat Takeaway) - Daedalus and the Data Labyrinth
🔥 An excellent deep-dive from George Zefkilis, looking at PostgreSQL WAL Internals in the context of building a CDC pipeline.
Debezium 3.6.0.Alpha1 and Debezium 3.5.0.Final have been released.
Yaroslav Tkachenko analysed the performance of different technologies for getting data from Postgres into Iceberg.
Leonard Xu looks at good practices when building Large-Scale Lake Ingestion with Flink CDC and Paimon
Real-world details from Nathan Smit of how they've been using Debezium with Oracle for four years, and how they addressed issues with Oracle CDC Replication Lag.
🔥 Yanquan Lv published the announcement of the release of Apache Flink CDC 3.6.0 as well as an excellent Deep Dive into Apache Flink CDC 3.6.0
Jason Ganz & Benoit Perigaud (dbt Labs) - Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update

Kafka and Event Streaming

Zapier - Reducing Kafka connections by 10x with a sidecar pattern
Yunhong Zheng - How Apache Fluss Achieves True Pruning in Streaming Storage
Bibek Maharjan - AI-Driven Autonomous Optimization of Apache Kafka on AWS MSK for High-Volume Financial Systems
Piotr Minkowski - Deep Dive into Kafka Offset Commit with Spring Boot
StrimziCon 2026 is on 3rd June, and the schedule has been published.

Flink

🔥 Robin Moffatt (that's me!) - Materialized Tables in Apache Flink
Yaroslav Tkachenko - Apache Flink: Reading and Modifying Kafka Consumer Offsets Using the State Processor API
Lee Seung-min / Choi Won-yong - Extending Real-time Ad Frequency Capping Aggregation to One Week with Apache Flink + RocksDB Tuning (original)
Katya Gorshkova - Hands-On with Flink --- Part 5: Managing State (previously: 1, 2, 3, 4)
Viktor Gamov digs out the open source toolbox to use Kafka, Flink, Iceberg, Superset and more to build Building a Streaming Lakehouse.

Open Table Formats (OTF), Catalogs, Lakehouses etc.

Gunnar Morling's Hardwood project has had its second beta release, which includes a very cool TUI for working with Parquet files.
Laurent Saint-Félix has written aq - "query and transform Parquet, Arrow IPC, CSV, and NDJSON files using jq-style expressions."
Yusuf Gözübüyük (TOM Tech) - The Performance Improvement Journey in Apache Iceberg Tables
Ved Prakash - Deep Dive into Apache Iceberg Architecture
🔥 CMU-DB tech talk - Kurt Westerfeld & Mark Cusack - Floe: A SQL Compute Service for the Data Lakehouse
Apache Iceberg has moved to "adopt" on the latest Technology Radar from Thoughtworks
Qiegang Long - Preliminary Notes on Open-Source Variant Performance
Steve Loughran - Benchmarking Parquet Variants through Iceberg
Anahita Singla (Picnic) - Leveraging contextual data in real-time analytics with Apache Iceberg
DuckLake version 1.0 has been released, and thus is now deemed production-ready. AFAIK it's only got real support within DuckDB, but do let me know if you see it supported elsewhere. Thoughtworks have marked it as "assess" on their Tech Radar.
A nice hands-on guide for setting up a local playground with Iceberg using Minio and Gravitino
Pedro Holanda describes how DuckLake deals with the small-files problem (often encountered when one starts streaming data to these types of table format). Using Data Inlining in DuckLake, they saw vast performance improvements over the same kind of processing done with Iceberg.

RDBMS

🔥 Ohad Ravid - The Best (Query) Plans of Mice and Men
Radim Marek - PostgreSQL MVCC, Byte by Byte
Simeon Griggs - Keeping a Postgres queue healthy
Thomas Kejser - Joins are NOT Expensive!
Mike Freedman - Introducing TigerFS - a filesystem backed by PostgreSQL, and a filesystem interface to PostgreSQL (Renato Losio wrote an InfoQ article about it)
Nikita Volkov - My 14-Year Journey Away from ORMs
Chess in Pure SQL

General Data Stuff

Almog Gavra - The Broken Economics of Databases
Kirill Bobrov - The Power of Data Sketches: A Comprehensive Guide
🔥 Gergely Orosz (a.k.a. The Pragmatic Engineer) interviews Martin Kleppmann about the second edition of Designing Data-intensive Applications.
Animesh Kumar - AI-Ready Data vs. Analytics-Ready Data
I'm slightly fascinated by the idea of ggsql, which brings SQL to the world of ggplot2 and the Grammar of Graphics.
Akshat Vig & Andrew Davidson (MongoDB) - Open Source, Community, and Consequence: The Story of MongoDB (InfoQ London 2026)

AI

I warned you previously...this AI stuff is here to stay, and it'd be short-sighted to think otherwise. As I read and learn more about it, I'm going to share interesting links (the clue is in the blog post title) that I find---whilst trying to avoid the breathless hype and slop.

🔥 Joe Reis - Why Electricity (Not Dot-Com) Is the Right AI Analogy. I like this idea from Joe. It also makes me think of the lift-and-shift that folk did with on-premises workloads to VMs in the Cloud, instead of re-architecting properly.
Jason Ganz - A Dispatch from the Jagged Frontier of Analytics Engineering (referencing Ethan Mollick's jagged frontier article from 2023).
Industry legends Mark Russinovich and Scott Hanselman wrote this opinion piece for ACM: Redefining the Software Engineering Profession for AI.

Without the hiring of early-in-career developers, the profession's talent pipeline will collapse, and organizations will face a future without the next generation of experienced engineers.
🔥 Elena Verna - Confessions of a Millennial in Tech
🎥 Vik Gamov - If Memento was about AI Agents. I watched Memento in preparation for this...I still have no idea what was going on in either 😆
Addy Osmani - Agent Harness Engineering
🔥 Hamel Husain - LLM Evals: Everything You Need to Know
Robin Moffatt - Kicking the Tyres on Harbor for Agent Evals
🔥 Bryan Cantrill - The peril of laziness lost
Adam Jacob - Laziness, Impatience, and Hubris
Alex Woods - Don't Let AI Write For You. (Reminder: I disclose my use of AI, and it's NEVER for writing!)

And finally...

Nothing to do with data, but stuff that I've found interesting or has made me smile.

Mitchell Hashimoto - Ghostty Is Leaving GitHub

Tool

I love the agility with which one can collaboratively work in GDocs, but I also prefer working with plain text and Markdown (or even better, Asciidoc). mist brings the concept of GDocs collaboration to Markdown files. It's pretty neat, and it's now open source.
A useful reminder from Christian Hofstede-Kuhn of Shell Tricks That Actually Make Life Easier (And Save Your Sanity)

Watch/Listen

🔥 A very cool example from the demo-scene: Razor1911
The Internet Archive isn't just about finding webpages that have gone offline---it also hosts tons of media, like this recording of Nirvana Live at Dreamerz 1989-07-08
I love this idea: TrainJazz: Every train, a note.

Nerd

😸 Not all specification drafts published are serious. Meow.
The ways in which one can play Doom continue to increase, with DOOM, played over cURL, and Can it Resolve DOOM? Game Engine in 2,000 DNS Records
HackerNews members share their memories of What was it like in the era of BBS before the internet?

My own memories are around Acorn-based BBSes. My favourite was Arcade BBS. Ah, memories. Fidonet, filebases...good times :)
What's more important than the code that you're writing Claude's writing for you? Getting it in the right font of course! Shave many a yak and waste plenty of time at Codingfont picking just the right font...

Unresolved directive in <stdin> - include::../../asciidoc-includes/il-footer.adoc[]

Materialized Tables in Apache Flink

Robin Moffatt — Tue, 28 Apr 2026 16:53:48 +0000

Flink added support for what it calls Materialized Tables in 1.20, released in 2024. You can read about the design and motivations in FLIP-435. In a nutshell, Materialized Tables provide a way to include the SQL to populate and refresh a table as part of its definition.

Let’s take a look!

Note:
In this blog post I’m using Flink 2.2 for the examples; some things have been added since the initial 1.20 release.

Materialized Table basics

Tip:
Materialized Tables are a new type of object in Flink and require:

A Catalog that supports Materialized Tables

A scheduler (if you want automated refreshes)
- The Flink SQL Gateway provides a basic one for test purposes
test-filesystem:
In this blog post I’m using the test-filesystem catalog that is, as the name implies, for testing only. It was added to Flink to support testing of Materialized Tables, and is currently the only option other than Apache Paimon for demonstrating all the capabilities.

The test-filesystem catalog also includes support for storing tables. Both catalog metadata and table data are written to disk.

Since I’m using a Docker-based stack, I’m going to launch the SQL Client like this:

docker compose exec -it jobmanager bash -c \
    "./bin/sql-gateway.sh start && \ 
    sleep 2 && \
    ./bin/sql-client.sh gateway --endpoint http://localhost:8083"

Start up the SQL Gateway
Use the SQL Gateway from the SQL Client

First up we’ll create the catalog and set it as active:

SET 'execution.checkpointing.savepoint-dir' = 'file:///shared/savepoints';

CREATE CATALOG mt_cat WITH (
    'type' = 'test-filesystem',
    'path' = '/shared/catalog',
    'default-database' = 'mydb'
);

USE CATALOG mt_cat;

Our source table for the Materialized Table is going to be a table reading data from Kafka:

CREATE TABLE simple_orders (
  order_id INT,
  total_gbp INT
) WITH (
  'connector' = 'kafka', 'topic' = 'orders', 'properties.bootstrap.servers' = 'broker:9092', 'properties.group.id' = 'flink-orders', 'scan.startup.mode' = 'earliest-offset', 'format' = 'json'
);

with some data in it:

INSERT INTO simple_orders
    VALUES  (2, 5),
            (3, 10),
            (4, 5),
            (1, 5),
            (5, 10);

Flink SQL> SELECT * FROM simple_orders;

    order_id   total_gbp
           2           5
           3          10
           4           5
           1           5
           5          10

And what about the total of these orders? This is where it starts to get funky:

Flink SQL> SELECT SUM(total_gbp) FROM simple_orders;
+----+-------------+
| op |      EXPR$0 |
+----+-------------+
| +I |           5 |
| -U |           5 |
| +U |          15 |
| -U |          15 |
| +U |          20 |
| -U |          20 |
| +U |          25 |
| -U |          25 |
| +U |          35 |
█

Notice the cursor, and the absence of a Flink SQL> prompt?

This query is doing two things that you might not recognise from the RDBMS world:

We see the changelog; as each record is read, the SUM() is restated (+U), with the previous value being replaced (-U). The final record (35) is the value we’d expect.
The query doesn’t complete; it keeps on running. That’s because the source is unbounded, and the values shown are just as of the data that has been read so far.

Now let’s create a Materialized Table:

CREATE MATERIALIZED TABLE orders_total_mt (
    PRIMARY KEY (total_gbp) NOT ENFORCED 
)
WITH (
  'connector' = 'upsert-kafka', 'topic' = 'orders-total-mt', 'properties.bootstrap.servers' = 'broker:9092', 'key.format' = 'json', 'value.format' = 'json'
)
AS SELECT COALESCE(SUM(total_gbp), 0) AS total_gbp FROM simple_orders;

I’m specifying a primary key because of the connector that I’m using ('upsert-kafka' tables require to define a PRIMARY KEY constraint. The PRIMARY KEY specifies which columns should be read from or write to the Kafka message key. The PRIMARY KEY also defines records in the 'upsert-kafka' table should update or delete on which keys.)
Configuration for the upsert-kafka connector here to persist the table’s data to Kafka.
Because of the primary key (see <1>), this needs to be non-nullable (Could not create a PRIMARY KEY with nullable column 'total_gbp'. A PRIMARY KEY column must be declared on non-nullable physical columns.)

And did it work? Have we got data in it?

Flink SQL> SELECT * FROM orders_total_mt;
+----+-------------+
| op |   total_gbp |
+----+-------------+
| +I |           5 |
| -D |           5 |
| +I |          15 |
| -D |          15 |
| +I |          25 |
| -D |          25 |
| +I |          35 | 
| +I |          20 | 
| -D |          20 | 
█

This is the latest value
These two cancel out

Well…yes. Again with that changelog business, but still giving us the correct answer.

So far, this Materialized Table is nothing extraordinary. In fact, it’s doing pretty much the same as the CTAS we’ll see below and with which you may already be familiar; creating a table that stores its data in Kafka, and setting a query running to populate it.

You can see the query running as a job, just as you would with a regular INSERT or CTAS:

Before we get much more into Materialized Tables, let’s do a quick run through of the alternatives in Flink SQL for defining a table and populating it.

Without Materialized Tables: It’s kinda DDL, and sorta DML

Materialized Tables move in the direction of solving the sometimes-awkward situation in Flink SQL that a table is part definition (data types, columns, etc) and part job (how to populate it), with the job needing managing and maintaining like any other code.

Let’s create the orders_total table like we saw in batch above, using the first pattern (CREATE, INSERT):

As before, the syntax is a little bit more fiddly because of the underlying persistence

CREATE TABLE orders_total (
    total_gbp INT,
    PRIMARY KEY (total_gbp) NOT ENFORCED
  ) WITH (
    'connector' = 'upsert-kafka', 'topic' = 'orders-total', 'properties.bootstrap.servers' = 'broker:9092', 'key.format' = 'json', 'value.format' = 'json'
  );

This is just the CREATE statement; we’ve not populated it yet.

Now to get data into the table:

Flink SQL> INSERT INTO orders_total (SELECT SUM(total_gbp) FROM simple_orders);
[INFO] Submitting SQL update statement to the cluster...
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: 8f666e58599217ad00feab629f8a5a9c

Check it out ma, we’ve got a Job ID!

This doesn’t look like your regular RDBMS INSERT, which will return (sometimes after a while) once it’s done. This returned straight away, but with chat about submitting statements to a cluster. But what’s going on with this "job"? We can see it directly in Flink SQL:

Flink SQL> DESCRIBE JOB '8f666e58599217ad00feab629f8a5a9c';
+----------------------------------+-----------------------------------+---------+-------------------------+
|                           job id |                          job name |  status |              start time |
+----------------------------------+-----------------------------------+---------+-------------------------+
| 8f666e58599217ad00feab629f8a5a9c | insert-into_cat.mydb.orders_total | RUNNING | 2026-04-21T10:30:48.054 |
+----------------------------------+-----------------------------------+---------+-------------------------+
1 row in set

Flink SQL>

as well as in the Flink Web UI:

And with the job running, there’s data loaded to the table:

Flink SQL> SELECT * FROM orders_total;
+----+-------------+
| op |   total_gbp |
+----+-------------+
| +I |           5 |
| -D |           5 |
| +I |          15 |
| -D |          15 |
| +I |          25 |
| -D |          25 |
| +I |          35 |
| +I |          20 |
| -D |          20 |

Other than CREATE TABLE/INSERT, the other non-Materialized Table route is to use CREATE TABLE…AS SELECT (CTAS). Using the CTAS approach, we get a similar behaviour; a populated table, and a job running on the Flink cluster:

CREATE TABLE orders_total_ctas (
    total_gbp INT,
    PRIMARY KEY (total_gbp) NOT ENFORCED
) WITH (
    'connector' = 'upsert-kafka', 'topic' = 'orders-total', 'properties.bootstrap.servers' = 'broker:9092', 'key.format' = 'json', 'value.format' = 'json'
) AS SELECT COALESCE(SUM(total_gbp),0) AS total_gbp FROM simple_orders;

[INFO] Submitting SQL update statement to the cluster...
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: dfebb27cd20653d93616945966244c66

Flink SQL> DESCRIBE JOB 'dfebb27cd20653d93616945966244c66';
+----------------------------------+--------------------------------------------+---------+-------------------------+
|                           job id |                                   job name |  status |              start time |
+----------------------------------+--------------------------------------------+---------+-------------------------+
| dfebb27cd20653d93616945966244c66 | insert-into_mt_cat.mydb.orders_total_ctas  | RUNNING | 2026-04-24T09:58:53.895 |
+----------------------------------+--------------------------------------------+---------+-------------------------+
1 row in set

Now we’ve got a total of three jobs, all doing the same thing:

When I insert a record to the source table simple_orders…

INSERT INTO simple_orders VALUES  (6, 7);

…the three running jobs update their respective tables, and I see the expected result on them:

Flink SQL> SET 'sql-client.execution.result-mode' = 'table'; 

Flink SQL> SELECT * FROM orders_total_mt;

   total_gbp
          42

Flink SQL> SELECT * FROM orders_total;

   total_gbp
          42

Flink SQL> SELECT * FROM orders_total_ctas;

   total_gbp
          42

This means I just see the latest calculated value, not the changelog as shown above

So that’s the three options: CREATE MATERIALIZED TABLE, CREATE TABLE/INSERT, CREATE TABLE…AS SELECT. Let’s now look at what makes Materialized Tables of interest to us.

Lifecycle Management

With CREATE TABLE plus INSERT (hereafter noted as CT/I), the query to populate the table is a separate entity from the definition. Even with the combined CREATE TABLE … AS SELECT (CTAS), it’s just a marriage of convenience; in the background a separate INSERT is fired off, never to be associated with its CTAS parent again.

Since we’re using the test-filesystem catalog we can easily inspect what’s in the catalog just by looking at what’s on disk. Here’s the catalog entry for both the orders_total (CT/I) and orders_total_ctas (CTAS)—it’s identical for both:

{
  "tableKind" : "TABLE",  "catalogTableInfo" : {
    "properties.bootstrap.servers" : "broker:9092",
    "connector" : "upsert-kafka",
    "schema.0.data-type" : "INT NOT NULL",
    "value.format" : "json",
    "schema.primary-key.name" : "PK_total_gbp",
    "schema.primary-key.columns" : "total_gbp",
    "key.format" : "json",
    "topic" : "orders-total",
    "schema.0.name" : "total_gbp"
  }
}

Contrast this to the Materialized Table, in which the DDL and DML are intertwined and the DML query literally persisted as part of the table definition, along with the query’s execution details:

{
  "tableKind" : "MATERIALIZED_TABLE",
  "catalogTableInfo" : {
    "schema.0.data-type" : "INT NOT NULL",
    "definition-query" : 
        "SELECT COALESCE(SUM(`simple_orders`.`total_gbp`), 0) AS `total_gbp`\nFROM `mt_cat`.`mydb`.`simple_orders` AS `simple_orders`",
    "refresh-status" : "ACTIVATED",
    "refresh-handler-bytes" : "rO0ABXNyADdvcmcuYXBhY2hlLmZsaW5rLnRhYmxlLnJlZnJlc2guQ29udGludW91c1JlZnJlc2hIYW5kbGVyAAAAAAAAAAECAARMAAljbHVzdGVySWR0ABJMamF2YS9sYW5nL1N0cmluZztMAA9leGVjdXRpb25UYXJnZXRxAH4AAUwABWpvYklkcQB+AAFMAAtyZXN0b3JlUGF0aHEAfgABeHB0ABNTdGFuZGFsb25lQ2x1c3RlcklkdAAGcmVtb3RldAAgNmFlNjBjZGU5N2VlMmMxZGU2MGVjNDhjMWZmZWY1NjFw",
    "freshness-interval" : "3", 
    "refresh-mode" : "CONTINUOUS", 
    "properties.bootstrap.servers" : "broker:9092",
    "connector" : "upsert-kafka",
    "value.format" : "json",
    "freshness-unit" : "MINUTE",
    "refresh-handler-desc" : "{\n executionTarget=remote,\n clusterId=StandaloneClusterId,\n jobId=6ae60cde97ee2c1de60ec48c1ffef561\n}",
    "schema.primary-key.name" : "PK_total_gbp",
    "schema.primary-key.columns" : "total_gbp",
    "key.format" : "json",
    "topic" : "orders-total-mt",
    "logical-refresh-mode" : "AUTOMATIC",
    "schema.0.name" : "total_gbp"
  }
}

These fields all hold information about the refresh query itself, not just the standard table DDL

As a simple example of the benefits, consider lifecycle management of the query.

Here are all three table population statements running:

What happens if we restart the task managers?

Ooops. Only one left, for the Materialized Table. The two INSERT statements (one directly from INSERT, the other a product of the CTAS) got killed, and not restarted.

Why would they get restarted? They’re just INSERT statements, with no lifecycle guarantees attached to them. Contrast that to the Materialized Table, which includes this persistence of the definition of how it is populated, and thus Flink can resurrect the query.

Schema evolution

What if we want to add a new column to the table being populated? In this example, that could be a count of the total orders (to go with the sum of the orders which is what we’ve already got).

With both CT/I and CTAS it gets messy. I’ve dropped and recreated both (since they got killed in the restart above), so their INSERT statements are running again now (as, presumably, they would be by default when we want to evolve the schema). Let’s see how it works.

Flink SQL> ALTER TABLE orders_total ADD order_ct INT;
[INFO] Execute statement succeeded.

Flink SQL> SELECT * FROM orders_total;
   total_gbp    order_ct
          42      <NULL>

No value for order_ct. Remember; the INSERT from before is still running:

INSERT INTO orders_total (SELECT SUM(total_gbp) FROM simple_orders);

And this only populates the existing total_gbp field.

Let’s kill the existing INSERT:

Flink SQL> SHOW JOBS;
+----------------------------------+--------------------------------------+----------+-------------------------+
|                           job id |                             job name |   status |              start time |
+----------------------------------+--------------------------------------+----------+-------------------------+
| 12af3aa38e4815ca3faa04be0fe51a51 | insert-into_mt_cat.mydb.orders_total |  RUNNING | 2026-04-24T10:54:24.990 |
[…]
Flink SQL> STOP JOB '12af3aa38e4815ca3faa04be0fe51a51';

Now to run a new one, including the additional field:

Flink SQL> INSERT INTO orders_total (SELECT SUM(total_gbp),COUNT(*) FROM simple_orders);
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.api.ValidationException: Column types of query result and sink for 'mt_cat.mydb.orders_total' do not match.
Cause: Incompatible types for sink column 'order_ct' at position 1.

Query schema: [EXPR$0: INT, EXPR$1: BIGINT NOT NULL]
Sink schema:  [total_gbp: INT, order_ct: INT]

Uh oh, schema mismatch. This is fiddly, right? Try again:

Flink SQL> ALTER TABLE orders_total DROP order_ct;
[INFO] Execute statement succeeded.

Flink SQL> ALTER TABLE orders_total ADD order_ct BIGINT NOT NULL;
[INFO] Execute statement succeeded.

Flink SQL> INSERT INTO orders_total (SELECT SUM(total_gbp),COUNT(*) FROM simple_orders);
[INFO] Submitting SQL update statement to the cluster...
[INFO] SQL update statement has been successfully submitted to the cluster:
Job ID: 8c197789f2b4966e97ed8d4002feb775

The new INSERT runs, but it turns out that the table’s still not happy when it comes to querying it:

Flink SQL> SELECT * FROM orders_total;
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.runtime.operators.sink.constraint.EnforcerException: Column 'order_ct' is NOT NULL, however, a null value is being written into it.
You can set job configuration 'table.exec.sink.not-null-enforcer'='DROP' to suppress this exception and drop such records silently.

Tip:
The reason is almost by-the-by here, but for completeness: the table is backed by a Kafka topic, and when we read from the table the first records fetched will be those populated by the initial INSERT with just one column in. (When we DROP the table it doesn’t touch the topic from which the table reads/writes). Hence, when we try to read those back, as the error message says, it can’t cope with null values for a NOT NULL field.

Yes there’s a workaround, but this is not the happy path that it could be.

There are similar troubles with CTAS—the process (and problems) are exactly the same as I just demonstrated for CT/I above. As a bonus we get just a little bit more ambiguity because of the coupled-but-not issue with the INSERT query that it spawns (as in, CTAS includes the definition, but subsequently it’s just an INSERT running on its own, for us to care for and water).

Compare all this to life with Materialized Tables:

Flink SQL> ALTER MATERIALIZED TABLE orders_total_mt
                AS SELECT COALESCE(SUM(total_gbp), 0) AS total_gbp,
                          COUNT(*) AS order_ct
                    FROM simple_orders;
[INFO] Execute statement succeeded.

Flink SQL> SELECT * FROM orders_total_mt;

   total_gbp             order_ct
          42                    6

No errors, no drama.

In the background, Flink stopped the first job, and started up a new one to include the new column:

Flink SQL> SHOW JOBS;
+----------------------------------+----------+-----------------------------------------------------------------------------+-------------------------+
|                           job id |   status |                                                                    job name |              start time |
+----------------------------------+----------+-----------------------------------------------------------------------------+-------------------------+
| 6ae60cde97ee2c1de60ec48c1ffef561 | FINISHED | Materialized_table_`mt_cat`.`mydb`.`orders_total_mt`_continuous_refresh_job | 2026-04-24T10:05:30.647 |
| a7a77aa3f909b6debb25c25d5d3c5de0 |  RUNNING | Materialized_table_`mt_cat`.`mydb`.`orders_total_mt`_continuous_refresh_job | 2026-04-24T11:09:07.887 |
+----------------------------------+----------+-----------------------------------------------------------------------------+-------------------------+

See the documentation for full details of what’s supported in schema evolution, as well as caveats. Significantly, the docs do note:

The new refresh job starts from the beginning and does not restore from the previous state.

The new refresh job will not restore from the state of the original refresh job.

Pause and Resume execution

With our Materialized Table we can suspend its execution:

ALTER MATERIALIZED TABLE orders_total_mt SUSPEND;

Then, if we add some data to the source table:

INSERT INTO simple_orders VALUES (7,10);

We’ll see that these new records aren’t reflected in the Materialized Table:

Flink SQL> SELECT SUM(total_gbp) AS total_gbp, COUNT(*) AS order_ct FROM simple_orders;
   total_gbp             order_ct
          52                    7

Flink SQL> SELECT * FROM orders_total_mt;
   total_gbp             order_ct
          42                    6

We can then RESUME the Materialized Table:

ALTER MATERIALIZED TABLE orders_total_mt RESUME;

and the Materialized Table shows the updated data:

Flink SQL> SELECT * FROM orders_total_mt;
   total_gbp             order_ct
          52                    7

The key thing here is that when we SUSPEND the Materialized Table Flink takes a savepoint. This is much easier than stopping a job (as we’d do with CT/I / CTAS) and having to start a new one from the start, since Flink will use the savepoint to start its processing from where it got to before it was suspended.

This very clean, SQL-driven, way of managing a table and its job could come in handy in several ways:

In the event of upstream data quality issues, rather than continuing to propagate suspect data—but without pulling the parachute cord of killing the job and losing its state—you can just suspend it whilst you investigate the problem.
Server maintenance, Flink upgrades, etc.
If there are availability problems with the table’s source (remember, Flink tables can use connectors against numerous systems, such as RDBMS, flat files, etc etc) you can suspend reading from it rather than dealing with a flood of errors.

Batch, Streaming, or Both?

One of the pitches for Materialized Tables in Flink is that it 'unifies' batch and streaming, and dispenses with the Lambda/Kappa concept. By having one SQL concept (Materialized Tables) with different configuration options you can define your data manipulation logic once and execute it as you want in both streaming or batch. This is defined by a parameter called REFRESH_MODE.

In the example above the Materialized Table was using an effective REFRESH_MODE of CONTINUOUS, i.e. a streaming query, per the Flink job that we saw running.

To explore the other REFRESH_MODE, which is FULL, we need a source connector that supports bounded reads. You can find a list of connectors within the Flink project here which includes annotation as to what kind of source mode they support. Here we’ll stick with test-filesystem just for convenience:

CREATE TABLE orders_file (
    order_id INT,
    total_gbp INT
) WITH (
    'format' = 'json'
);

INSERT INTO orders_file VALUES (1, 5), (2, 10), (3, 15);

SELECT * FROM orders_file;

    order_id   total_gbp
           1           5
           2          10
           3          15

So now for our Materialized Table:

CREATE MATERIALIZED TABLE orders_total_mt_file_full
WITH ( 'format' = 'json' ) 
FRESHNESS = INTERVAL '5' MINUTE 
REFRESH_MODE = FULL 
AS SELECT COALESCE(SUM(total_gbp), 0) AS total_gbp FROM orders_file;

We’re using the test-filesystem catalog to store the data for this table.
We’ll talk about FRESHNESS in the next section
FULL refresh mode

The SELECT works…but there’s no data in the table 😿

Flink SQL> SET 'sql-client.execution.result-mode' = 'tableau';
[INFO] Execute statement succeeded.

Flink SQL> SELECT * FROM orders_total_mt_file_full;
+----+-------------+
| op |   total_gbp |
+----+-------------+
Received a total of 0 row (0.18 seconds)

Flink SQL>

Unlike CONTINUOUS refresh, which runs as a job on the task manager, FULL refresh is managed on a scheduler that’s provided by the SQL Gateway. When we create the Materialized Table we see it added an entry in the scheduler on SQL Gateway:

2026-04-27 10:33:06,828 INFO  org.apache.flink.table.gateway.workflow.scheduler.EmbeddedQuartzScheduler []
Create quartz schedule job for materialized table `mt_cat`.`mydb`.`orders_total_mt_file_full` successfully,
job info: default_group.quartz_job_`mt_cat`.`mydb`.`orders_total_mt_file_full`,
cron expression: 0 0/5 * * * ? *.                                                                                                                              │

Check out the cron expression—that’s driven by the FRESHNESS = INTERVAL '5' MINUTE configuration.

We can sit tight and wait, or we can manually kick a refresh off ourselves:

Flink SQL> ALTER MATERIALIZED TABLE orders_total_mt_file_full REFRESH;
+----------------------------------+---------------------------+
|                           job id |              cluster info |
+----------------------------------+---------------------------+
| 2215db412d0ef0551ae0704bebb978bd | {execution.target=remote} |
+----------------------------------+---------------------------+
1 row in set

Flink SQL> SHOW JOBS;
+----------------------------------+----------+-------------------------------------------------------------------------------------+-------------------------+
|                           job id |   status |                                                                            job name |              start time |
+----------------------------------+----------+-------------------------------------------------------------------------------------+-------------------------+
| 2215db412d0ef0551ae0704bebb978bd | FINISHED | Materialized_table_`mt_cat`.`mydb`.`orders_total_mt_file_full`_one_time_refresh_job | 2026-04-27T11:27:32.863 |

And now we have data:

Flink SQL> SELECT * FROM orders_total_mt_file_full;
+----+-------------+
| op |   total_gbp |
+----+-------------+
| +I |          30 | 
+----+-------------+
Received a total of 1 row (0.19 seconds)

Note how it’s a single value, rather than a changelog series of +I, -U, +U rows like we saw with the Kafka sink above

Over in the Flink Web UI we can see the periodic refreshes of the table successfully executing:

One thing that’s important to note about the refresh jobs is the SQL that gets run:

INFO  org.apache.flink.table.gateway.service.materializedtable.MaterializedTableManager [] -
Begin to refreshing the materialized table
`mt_cat`.`mydb`.`orders_total_mt_file_full`, statement:

INSERT OVERWRITE `mt_cat`.`mydb`.`orders_total_mt_file_full` 
    SELECT * FROM (SELECT COALESCE(SUM(`orders_file`.`total_gbp`), 0) AS `total_gbp`
                    FROM `mt_cat`.`mydb`.`orders_file` AS `orders_file`)

INSERT OVERWRITE replaces the contents of the target table (or partition; see below).

Batch to Kafka? Nope.

In the earlier section when we created a Materialized Table reading and writing from Kafka topics, we didn’t set a REFRESH_MODE. What happened was that Flink set an implicit value for it, which we can see if we examine the DDL that Flink reads back for the table:

Flink SQL> SHOW CREATE MATERIALIZED TABLE orders_total_mt;

CREATE MATERIALIZED TABLE `mt_cat`.`mydb`.`orders_total_mt` (
  CONSTRAINT `PK_total_gbp` PRIMARY KEY (`total_gbp`) NOT ENFORCED
)
WITH (
  'connector' = 'upsert-kafka',
  'key.format' = 'json',
  'properties.bootstrap.servers' = 'broker:9092',
  'topic' = 'orders-total-mt',
  'value.format' = 'json'
)
FRESHNESS = INTERVAL '3' MINUTE 
REFRESH_MODE = CONTINUOUS 
AS SELECT COALESCE(SUM(`simple_orders`.`total_gbp`), 0) AS `total_gbp`, COUNT(*) AS `order_ct`
FROM `mt_cat`.`mydb`.`simple_orders` AS `simple_orders`

These weren’t in the submitted SQL; Flink added these.

This is where it gets a bit funky though, and also exposes some of the slightly rough edges on the current implementation of Materialized Tables. What if we decide we’d like to set the Materialized Table to refresh on a batch schedule instead?

Here’s what happens:

Flink SQL> CREATE MATERIALIZED TABLE orders_total_mt__full (
                PRIMARY KEY (total_gbp) NOT ENFORCED
            )
            WITH (
                'connector' = 'upsert-kafka', 'topic' = 'orders_total_mt__full', 'properties.bootstrap.servers' = 'broker:9092', 'key.format' = 'json', 'value.format' = 'json'
            )
            FRESHNESS = INTERVAL '5' MINUTE
            REFRESH_MODE = FULL 
            AS SELECT COALESCE(SUM(total_gbp), 0) AS total_gbp FROM simple_orders;
[INFO] Execute statement succeeded.

Use batch refresh for the Materialized Table reading and writing from Kafka.

Flink tells me it’s succeeded. But has it actually?

The table’s there, sure:

Flink SQL> SHOW TABLES;
+-----------------------+
|            table name |
+-----------------------+
| orders_total_mt__full |
[…]

But…

Flink SQL> SELECT * FROM orders_total_mt__full;

[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.UnknownTopicOrPartitionException:
This server does not host this topic-partition.

The target Kafka topic hasn’t been populated (or even created). Something went wrong!

So whilst the Materialized Table created successfully, it looks like it’s not got any data in it. Let’s take a look at the SQL Gateway log. As if on cue, at the next 5-minute interval since the epoch, the job kicks off:

org.apache.flink.table.gateway.service.utils.SqlExecutionException: Refreshing the materialized table `mt_cat`.`mydb`.`orders_total_mt__full` occur exception.
Caused by: org.apache.flink.table.api.ValidationException:
Querying an unbounded table 'mt_cat.mydb.simple_orders' 
in batch mode is not allowed. The table source is unbounded.

It fails per the reason on the final line there—the source for the table is a Flink table reading from a Kafka topic, and since Kafka topics are unbounded, Flink won’t do a batch read from it.

Logically, this makes sense; but some kind of "canary test" of the Materialized Table’s refresh query might be a useful thing to add to the CREATE process, instead of the silent failure and log-diving necessary to surface it.

Refresh frequency

I’ve already trailed some of the details about the frequency with which Materialized Tables can be set to refresh above. It’s configured by the FRESHNESS setting, which per the documentation defines:

Data freshness defines the maximum amount of time that the materialized table’s content should lag behind updates to the base tables.

For tables using FULL refresh mode, the refresh query is run on a schedule determined by the FRESHNESS value that you set, translated by Flink into a cron schedule. By default, the schedule is implemented on the built-in scheduler on the SQL Gateway. It’s worth noting that if it takes a long time to execute, the actual freshness may be noticeably longer than that specified.
For CONTINUOUS refresh, the FRESHNESS defines how often the table’s refresh job will checkpoint the data. This is important for connectors (e.g. Iceberg—although perhaps a bad example as Iceberg doesn’t yet support Materialized Tables) that rely on a checkpoint to flush data to cause it to be readable.

One interesting sidenote for FULL refresh mode is that since the schedule uses a cron expression, there are restrictions on what the value for FRESHNESS can be. The docs say it must be:

a common divisor of the respective time interval unit.

So if you’re using an HOUR interval, the value must be divisible into 24, for MINUTE into 60, etc. If you get it wrong, Flink will tell you:

org.apache.flink.table.api.ValidationException:
In full refresh mode, only freshness that are factors of 60 are currently
supported when the time unit is MINUTE.

Defaulting on the defaults

When I introduced the first Materialized Table query at the top of this article, I didn’t specify REFRESH_MODE or FRESHNESS. I’m all for minimum-required defaults enabling users to get up and running with software, but here there is potential for some serious ambiguity. I’ve gone through the docs and code to try and distil down how Flink will handle the non-provision of both or either of these settings.

If you do specify REFRESH_MODE but not FRESHNESS then for FRESHNESS Flink uses the values of system configuration option relating to the refresh mode you’ve chosen:

If you do specify FRESHNESS but not REFRESH_MODE then Flink determines REFRESH_MODE by comparing FRESHNESS to the configuration materialized-table.refresh-mode.freshness-threshold (which has a default of 30 minutes).

If your value of FRESHNESS is less than the freshness-threshold Flink will use CONTINUOUS, otherwise it’ll use FULL.

Tip:
If you want to use a different REFRESH_MODE than this logic sets, then just configure REFRESH_MODE directly.

And finally, what happens if you don’t specify a REFRESH_MODE or FRESHNESS at all? This is where it gets less clear :)

If you don’t specify a REFRESH_MODE or FRESHNESS then Flink will default to an internal value of AUTOMATIC for the REFRESH_MODE. You can actually see that if you look closely at the catalog entry that I quoted above, which includes "logical-refresh-mode" : "AUTOMATIC".

Flink then defaults the FRESHNESS to the value of materialized-table.default-freshness.continuous (default: 3 minutes):

DefaultMaterializedTableEnricher.java#L89-L94

// User omitted freshness, choose default based on logical mode
if (table.getLogicalRefreshMode() == LogicalRefreshMode.FULL) {
    finalFreshness = defaultFullFreshness;
} else {
    // For AUTOMATIC or CONTINUOUS modes, use the continuous default
    finalFreshness = defaultContinuousFreshness;
}

From there, now that it has a value for FRESHNESS, it follows the logic above using materialized-table.refresh-mode.freshness-threshold.

My tl;dr from this is that you don’t want to leave these values to the defaults for some poor sod who ends up maintaining your DDL having to try and pick through. Be deliberate and explicit in how you configure your Materialized Table REFRESH_MODE, that way the person operating the pipeline will have a clear handle at least on the FRESHNESS that will be used—and indeed, you may not want to hardcode freshness for each Materialized Table and manage it at a system level.

Partitions

Let’s now look at the idea of partition-specific updates and refreshes.

Note:
At the moment there’s limited support for Materialized Tables in the ecosystem beyond Flink. For example, whilst it would make tons of sense in Iceberg (particularly with the partitioning stuff I’m about to show you), it’s not been implemented yet.

Currently (April 2026) only Paimon supports partitioned Materialized Tables (along with a test-filesystem catalog/connector).

In the examples above I took a set of transactions and aggregated them up to one single total. What is much more common in the real world is aggregating by one or more dimensions. These dimensions could be things like date, customer, product, business unit, etc. Sometimes the volume of data can dictate that physically partitioning this data by a dimension at the storage layer will help performance and manageability.

For this example I’m going to use a base transaction table that includes customer and date dimensions, and then aggregate it up by date which I’ll also partition by.

Here’s the source table:

CREATE TABLE user_orders (
  order_id    BIGINT,
  order_date  DATE NOT NULL,
  user_id     BIGINT,
  user_name   STRING,
  amount      BIGINT
) WITH ( 'format' = 'json' );

with some data:

INSERT INTO user_orders (order_id, order_date, user_id, user_name, amount)
VALUES (1001, DATE '2026-04-18', 1, 'alice', 1500),
        (1002, DATE '2026-04-18', 2, 'bob', 2300),
        (1003, DATE '2026-04-18', 3, 'carol', 800),
        (1004, DATE '2026-04-19', 1, 'alice', 1200),
        (1005, DATE '2026-04-19', 2, 'bob', 3100),
        (1006, DATE '2026-04-19', 3, 'carol', 1900),
        (1007, DATE '2026-04-20', 1, 'alice', 500),
        (1008, DATE '2026-04-20', 4, 'dave', 4200);

Now to create the Materialized Table (using the test-filesystem connector)

CREATE MATERIALIZED TABLE daily_order_summary
    PARTITIONED BY (order_date) 
WITH ('format'='json')
REFRESH_MODE = FULL
AS SELECT   order_date,
            SUM(amount)  AS total_amount,
            COUNT(*)     AS order_count
    FROM    user_orders
    GROUP BY order_date;

Ooooh partitions!

Check the data:

Flink SQL> SELECT * FROM daily_order_summary;
+----+------------+----------------------+----------------------+
| op | order_date |         total_amount |          order_count |
+----+------------+----------------------+----------------------+
Received a total of 0 row (0.23 seconds)

Oh noes! But wait…what did we learn above, both about REFRESH_MODE = FULL and the absence of FRESHNESS? With FULL the loading of data is done on a schedule, and without a value for FRESHNESS Flink will default to materialized-table.default-freshness.full, which is 1 hour.

Given that we created the table at 16:56 (as we can see from its schedule being created on the SQL Gateway)…

16:56:16,282 INFO  org.apache.flink.table.gateway.workflow.scheduler.EmbeddedQuartzScheduler []
Create quartz schedule job for materialized table `mt_cat`.`mydb`.`daily_order_summary` successfully, job info:
default_group.quartz_job_`mt_cat`.`mydb`.`daily_order_summary`, cron expression: 0 0 0/1 * * ? *.

…it means we should see the job run in the next few minutes…

and there it goes!

17:00:00,016 INFO  org.apache.flink.table.gateway.workflow.scheduler.EmbeddedQuartzScheduler [] -
                    Execute refresh operation for workflow: WorkflowInfo{materializedTableIdentifier='`mt_cat`.`mydb`.`daily_order_summary`', dynamicOptions={}, initConfig={execution.checkpointing.savepoint-dir=file:///shared/savepoints}, executionConfig={}, restEndpointUrl='http://0.0.0.0:8083'}.
17:00:00,039 INFO  org.apache.flink.table.gateway.service.materializedtable.MaterializedTableManager [] -
                    Begin to refreshing the materialized table `mt_cat`.`mydb`.`daily_order_summary`,
                    statement: INSERT OVERWRITE `mt_cat`.`mydb`.`daily_order_summary`

Flink SQL> SHOW JOBS;
+----------------------------------+----------+-------------------------------------------------------------------------------+-------------------------+
|                           job id |   status |                                                                      job name |              start time |
+----------------------------------+----------+-------------------------------------------------------------------------------+-------------------------+
| 261181049fca90b15786b09e383eae18 | FINISHED | Materialized_table_`mt_cat`.`mydb`.`daily_order_summary`_periodic_refresh_job | 2026-04-27T17:00:00.370 |
[…]

But, alas!

Flink SQL> SELECT * FROM daily_order_summary;
+----+------------+----------------------+----------------------+
| op | order_date |         total_amount |          order_count |
+----+------------+----------------------+----------------------+
[ERROR] Could not execute SQL statement. Reason:
java.time.format.DateTimeParseException:
Text '20561' could not be parsed at index 0

Under the covers Flink is writing the date as an epoch day (20561 being the epoch day for 2026-04-18), whilst reading it in a different way and failing. Let’s try again, and be explicit in how to store it:

DROP MATERIALIZED TABLE daily_order_summary;

CREATE MATERIALIZED TABLE daily_order_summary
    PARTITIONED BY (order_date)
WITH ('format'='json',
      'partition.fields.order_date.date-formatter' = 'yyyy-MM-dd') 
REFRESH_MODE = FULL
AS SELECT   CAST(order_date AS STRING) AS order_date, 
            SUM(amount)  AS total_amount,
            COUNT(*)     AS order_count
    FROM    user_orders
    GROUP BY order_date;

Specify the format for the date when reading and writing
Needs to be written as a STRING

Note:
Specifying a date as a string gives me strong ick vibes. A date is a date, not a string. Date-based partitioning is cool, but ending up with a STRING in the target Materialized Table feels off to me (now your end users have to figure out how to handle it, what format it is, etc).

So now our Materialized Table is recreated, let’s force a load of it instead of waiting for the next :00/:30 to roll around on the clock:

Flink SQL> ALTER MATERIALIZED TABLE daily_order_summary REFRESH;
+----------------------------------+---------------------------+
|                           job id |              cluster info |
+----------------------------------+---------------------------+
| 0490ff33c9a8a530592b8451f7da43df | {execution.target=remote} |
+----------------------------------+---------------------------+

Flink SQL> SHOW JOBS;
+----------------------------------+----------+-------------------------------------------------------------------------------+-------------------------+
|                           job id |   status |                                                                      job name |              start time |
+----------------------------------+----------+-------------------------------------------------------------------------------+-------------------------+
| 0490ff33c9a8a530592b8451f7da43df | FINISHED | Materialized_table_`mt_cat`.`mydb`.`daily_order_summary`_one_time_refresh_job | 2026-04-27T17:26:08.799 |

Flink SQL> SELECT * FROM daily_order_summary;
+----+------------+--------------+-------------+
| op | order_date | total_amount | order_count |
+----+------------+--------------+-------------+
| +I | 2026-04-20 |         4700 |           2 |
| +I | 2026-04-18 |         4600 |           3 |
| +I | 2026-04-19 |         6200 |           3 |
+----+------------+--------------+-------------+
Received a total of 3 rows (0.22 seconds)

Woo, we have data!

We can see it on disk too, laid out in partitions:

$ tree flink-shared/catalog/mydb/
flink-shared/catalog/mydb/
├── daily_order_summary
│   ├── data
│   │   ├── order_date=2026-04-18
│   │   │   └── part-b86894d8-b96f-4add-857e-9a29121e45f9-task-0-file-0
│   │   ├── order_date=2026-04-19
│   │   │   └── part-b86894d8-b96f-4add-857e-9a29121e45f9-task-0-file-1
│   │   └── order_date=2026-04-20
│   │       └── part-b86894d8-b96f-4add-857e-9a29121e45f9-task-0-file-2
│   └── schema
│       └── daily_order_summary_schema.json

And since it’s in partitions, we can take advantage of the functionality in Materialized Tables to only refresh a specific partition:

Flink SQL> ALTER MATERIALIZED TABLE daily_order_summary REFRESH PARTITION (order_date='2026-04-20');
+----------------------------------+---------------------------+
|                           job id |              cluster info |
+----------------------------------+---------------------------+
| f7b979f004cbf88d574bb07509ed57e8 | {execution.target=remote} |
+----------------------------------+---------------------------+
1 row in set

Flink SQL> SHOW JOBS;
+----------------------------------+----------+--------------------------------------------------------------------------------+-------------------------+
|                           job id |   status |                                                                       job name |              start time |
+----------------------------------+----------+--------------------------------------------------------------------------------+-------------------------+
| f7b979f004cbf88d574bb07509ed57e8 | FINISHED |  Materialized_table_`mt_cat`.`mydb`.`daily_order_summary`_one_time_refresh_job | 2026-04-28T09:13:55.198 |
[…]

How do we know it only refreshed the data for a single partition (2026-04-20)? In the SQL Gateway logs we can see the SQL used:

INSERT OVERWRITE `mt_cat`.`mydb`.`daily_order_summary`
  SELECT * FROM (SELECT CAST(`user_orders`.`order_date` AS STRING) AS `order_date`, SUM(`user_orders`.`amount`) AS `total_amount`, COUNT(*) AS `order_count`
FROM `mt_cat`.`mydb`.`user_orders` AS `user_orders`
GROUP BY `user_orders`.`order_date`)
  WHERE order_date = '2026-04-20'

Predicate on order_date as expected, matching the […] REFRESH PARTITION (order_date='2026-04-20') statement that we ran

So we can do partition-specific refreshes—pretty useful for things like recomputing once missing data has arrived for a date.

The other thing that partitioned Materialized Tables gives us though is this: when a regular scheduled refresh of the Materialized Table runs, it only refreshes the current partition. So given that today is 28 Apr 2026, if we inspect the SQL for the most recent scheduled refresh of the partitioned Materialized Table we see this:

INSERT OVERWRITE `mt_cat`.`mydb`.`daily_order_summary`
  SELECT * FROM (SELECT CAST(`user_orders`.`order_date` AS STRING) AS `order_date`, SUM(`user_orders`.`amount`) AS `total_amount`, COUNT(*) AS `order_count`
FROM `mt_cat`.`mydb`.`user_orders` AS `user_orders`
GROUP BY `user_orders`.`order_date`)
  WHERE order_date = '2026-04-28'

That’s today’s date!

This is a much more efficient way of loading data if you’re going to use a batch process. Instead of pulling all the data from the source and recomputing it all, you just pick the data for the latest partition. Now, there are a few considerations to bear in mind:

You need to have set your partition date carefully, based on the business logic of your data. In our simplistic example we’re just capturing the date an order was placed. What if the source table includes things like order status, and our aggregate table also needs to show the total value of fulfilled orders? If an order is placed yesterday but fulfilled today, we still need to pick that change up—which we won’t if we only refresh the partition of the date on which the order was placed. In this situation you’d perhaps end up adding a new field such as date_updated and partitioning on that.
It’s still a batch-based process. In the example of orders changing and you wanting to reflect the latest state in your aggregate table, you’d probably be using CONTINUOUS mode and this batch-based partitioning refresh would be irrelevant.

It’s not explicitly stated in the docs, but so far as I can tell since you (1) must specify date-formatter (as we saw above) and (2) the docs state that date-formatter only works in FULL refresh mode, partitioned Materialized Tables won’t work for streaming updates.

It also means, since this is not a configurable thing, that your partitioned Materialized Table will never pick up data earlier than today unless you manually force a full refresh (ALTER MATERIALIZED TABLE…REFRESH) or refresh of the specific partition (as seen above).

That means you could end up with holes in your data unwittingly. Consider the following scenario. We saw above that we’ve currently got data in the Materialized Table for 18-20th April:

Flink SQL> SELECT * FROM daily_order_summary;
+----+--------------------------------+----------------------+----------------------+
| op |                     order_date |         total_amount |          order_count |
+----+--------------------------------+----------------------+----------------------+
| +I |                     2026-04-18 |                 4600 |                    3 |
| +I |                     2026-04-19 |                 6200 |                    3 |
| +I |                     2026-04-20 |                 4700 |                    2 |
+----+--------------------------------+----------------------+----------------------+
Received a total of 3 rows (0.21 seconds)

Some more data gets added to the source table for both 21st April (let’s say it came in late) and for today (28th April):

INSERT INTO user_orders (order_id, order_date, user_id, user_name, amount) VALUES
(1011, DATE '2026-04-21', 1, 'alice', 1500),
(1012, DATE '2026-04-21', 2, 'bob', 2300),
(1021, DATE '2026-04-28', 1, 'alice', 1500),
(1022, DATE '2026-04-28', 2, 'bob', 2300);

Imagine being on-call and someone complains that the data in the aggregate table (the Materialized Table) is incomplete. Bleary-eyed (it’s 03:00 and you were up late studying the source code for Apache Flink, naturally) you check the table and gasp in horror as you (and your angry end-user) only see four rows of data, not five:

Flink SQL> SELECT * FROM daily_order_summary;

order_date         total_amount          order_count
2026-04-18                 4600                    3
2026-04-19                 6200                    3
2026-04-20                 4700                    2
2026-04-28                 3800                    2

Note, no row for 2026-04-21

But when you query the base table you see the full set of data:

Flink SQL> SELECT order_date, SUM(amount) AS `total_amount`, COUNT(*) AS `order_count`
            FROM user_orders
            GROUP BY order_date;

 order_date         total_amount          order_count
 2026-04-18                 4600                    3
 2026-04-19                 6200                    3
 2026-04-20                 4700                    2
 2026-04-21                 3800                    2 
 2026-04-28                 3800                    2

The missing date!

You hurriedly check and confirm that the scheduled refresh of the Materialized Table has run and completed successfully:

Flink SQL> SHOW JOBS;
+----------------------------------+----------+-------------------------------------------------------------------------------+-------------------------+
|                           job id |   status |                                                                      job name |              start time |
+----------------------------------+----------+-------------------------------------------------------------------------------+-------------------------+
| d17cc4458346ed8cb4675cac8170b059 | FINISHED | Materialized_table_`mt_cat`.`mydb`.`daily_order_summary`_periodic_refresh_job | 2026-04-28T11:00:00.157 |

But then you remember: when the scheduled refresh of the Materialized Table ran, it only refreshed the current partition (because it’s a partitioned Materialized Table).

In order to see the full set of data, we need to force a full refresh:

Flink SQL> ALTER MATERIALIZED TABLE daily_order_summary REFRESH;
+----------------------------------+---------------------------+
|                           job id |              cluster info |
+----------------------------------+---------------------------+
| 9f0d3e3c03fe821b66cfe642c589f649 | {execution.target=remote} |
+----------------------------------+---------------------------+
1 row in set

Flink SQL> SELECT * FROM daily_order_summary;

order_date         total_amount          order_count
2026-04-18                 4600                    3
2026-04-19                 6200                    3
2026-04-20                 4700                    2
2026-04-21                 3800                    2 
2026-04-28                 3800                    2

The partition’s data that was missed in the scheduled refresh because it’s not the current date

A full refresh is no big deal if we’re aggregating a handful of rows of source data, but if this were a huge table this could be quite a consideration. And then you’re into targeting the refresh of specific partitions, and hoping you’ve got them all.

The scheduler that didn’t

The last bit to cover here is to highlight a bit of a gap, IMHO, in the state of Materialized Tables in Flink 2.2 as they currently stand.

I’ve already addressed the limited catalog support for Materialized Tables (necessitating the use of test-filesystem for the simple examples above). The other "fine for a blog post but only just" component is the scheduler.

The scheduler is a crucial part of batch-based Materialized Tables in Flink 2.2. Per the FLIP’s design document, it’s designed to be pluggable, so I bring no criticism per se about the state of it in SQL Gateway, just a "buyer beware" note. In fact, the docs themselves even say so:

This embedded scheduler is mainly used for testing scenarios and is not suitable for production environment.

Here’s an example of why, which had me scratching my head somewhat when I was writing this. If you create a Materialized Table, and then restart the SQL Gateway, its refresh schedule is not restored. That’s because the SQL Gateway is using an embedded memory-based scheduler which doesn’t persist schedules to disk.

When you create a Materialized Table you’ll see the schedule get added:

2026-04-28 11:16:44,111 INFO  org.apache.flink.table.gateway.workflow.scheduler.EmbeddedQuartzScheduler []
Create quartz schedule job for materialized table `mt_cat`.`mydb`.`order_summary_1min` successfully,
job info: default_group.quartz_job_`mt_cat`.`mydb`.`order_summary_1min`, cron expression: 0 0/1 * * * ? *.

But that’s the only time it gets added. And if there’s no schedule (such as after a restart), there’s no refresh of your Materialized Table.

If you think, aha! I can just RESUME the job, you’re gonna have a bad time:

Flink SQL> ALTER MATERIALIZED TABLE order_summary_1min RESUME;
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.gateway.service.utils.SqlExecutionException:
Materialized table `mt_cat`.`mydb`.`order_summary_1min` refresh workflow has been resumed.

Hang on…an [ERROR] but refresh workflow has been resumed? Sounds bad, but promising?

It’s actually just a misleading error message; what it’s actually saying is indeed an ERROR: Flink thinks that the Materialized Table is already active, and therefore will not resume it.

Okay, so what about if we SUSPEND it then, so that it’s then in a state from which to resume it?

Flink SQL> ALTER MATERIALIZED TABLE order_summary_1min SUSPEND;
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.gateway.workflow.scheduler.SchedulerException:
Failed to suspend a non-existent quartz schedule job: default_group.quartz_job_`mt_cat`.`mydb`.`order_summary_1min`.

You can’t suspend it because there’s no schedule!

The quick option is just to drop & recreate the Materialized Table. The fiddly option is to manually recreate the schedule using the SQL Gateway’s REST API to hit the scheduler directly:

curl -s -X POST http://localhost:8083/v3/workflow/embedded-scheduler/create \
  -H "Content-Type: application/json" \
  -d '{
    "materializedTableIdentifier": "`mt_cat`.`mydb`.`order_summary_1min`",
    "cronExpression": "0 0/1 * * * ? *",
    "initConfig": {"execution.checkpointing.savepoint-dir": "file:///shared/savepoints"},
    "executionConfig": {},
    "restEndpointUrl": "http://0.0.0.0:8083"  }'

This then brings the refresh job back to life, and we see it start running again:

Flink SQL> SHOW JOBS;
+----------------------------------+------------------------------------------------------------------------------+----------+-------------------------+
|                           job id |                                                                     job name |   status |              start time |
+----------------------------------+------------------------------------------------------------------------------+----------+-------------------------+
| 8969c414cd83a569b91f3127be509cf2 | Materialized_table_`mt_cat`.`mydb`.`order_summary_1min`_periodic_refresh_job | FINISHED | 2026-04-28T11:32:00.931 |
+----------------------------------+------------------------------------------------------------------------------+----------+-------------------------+

So it works…but I wouldn’t be going anywhere near a production environment with this. Which, to be fair, is literally what the docs say too :)

FLIP-448 does define a pluggable scheduler, but I’m only aware of Apache DolphinScheduler having added support for it (and even then, I can’t find much info on how to put it into practice).

Wrapping Up

Apache Flink’s Materialized Tables are a solid concept. However, whilst they give me warm fuzzy memories of my time in the trenches with Materialized Views in Oracle 11g, I think streaming complicates things, as do the issues around scheduling and partitioning—so I’m not convinced yet that the implementation is complete enough across the bits of the ecosystem that I’m working most with for me to wholeheartedly adopt them. That said, the schema evolution and query lifecycle management is a really good development.

Definitely worth playing with and making up your own mind!

If you want to try this out for yourself, you can find the Docker Compose stack on my examples repo on GitHub.

Kicking the Tyres on Harbor for Agent Evals

Robin Moffatt — Fri, 10 Apr 2026 14:26:41 +0000

After cobbling together my own eval for Claude, I was interested to discover harbor. It’s described as:

A framework for evaluating and optimizing agents and models in container environments.

Which sounds kinda cool, right?

It ships with a bunch of pre-created tests and benchmarks, such as the mandatory hello-world to more complex and multi-task examples such as terminal-bench.

Harbor’s unit of execution is a task, which is basically a prompt for a coding agent (such as Claude Code). Harbor works with multiple coding agents, and multiple models. Which is basically what it says on the tin above, right?

Here’s an example task:

Create a file called hello.txt with "Hello, world!" as the content.

Trying it out

Let’s try out hello-world:

harbor run  --model anthropic/claude-sonnet-4-6 \ 
            --agent claude-code \                 
            --dataset hello-world \

Use Sonnet 4.6 model
Run the test using Claude Code
Run the pre-packaged "Hello, World" test

After a short time this completes:

┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Metric              ┃ Value                           ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Agent               │ claude-code (claude-sonnet-4-6) │
│ Dataset             │ hello-world                     │
│ Trials              │ 1                               │
│ Errors              │ 0                               │
│                     │                                 │
│ Mean                │ 1.000                           │
│                     │                                 │
│ Reward Distribution │                                 │
│   reward = 1.0      │ 1                               │
└─────────────────────┴─────────────────────────────────┘

Harbor ships (see what I did there? 😉) with a nice dashboard for exploring test runs. Spin it up by pointing it at the output folder (jobs, in this case):

harbor view jobs

Status and timing breakdown:

Under the covers, Harbor spins up a Docker container, within which Claude runs with --dangerously-skip-permissions so that it can go about its business without any of that pesky permission seeking. It takes the defined task or dataset prompt, and runs it, as we can see here:

Scoring

A task’s performance is scored using a verifier that’s part of the task definition. For the above "hello world", all we need to do is check if the agent (a) created the file with the correct name and (b) with the correct content. Which is exactly what this Python script does:

def test_hello_file_exists():
    hello_path = Path("/app/hello.txt")

    assert hello_path.exists(), f"File {hello_path} does not exist"


def test_hello_file_contents():
    hello_path = Path("/app/hello.txt")

    content = hello_path.read_text().strip()
    expected_content = "Hello, world!"

    assert content == expected_content, (
        f"File content is '{content}', expected '{expected_content}'"
    )

Its wrapper script gives it a pass/fail score—either it worked, or it didn’t:

if [ $? -eq 0 ]; then
  echo 1 > /logs/verifier/reward.txt
else
  echo 0 > /logs/verifier/reward.txt
fi

If you run the test multiple times, you’ll get scores (rewards); unsurprisingly "hello-world" doesn’t pose any challenges or show variability:

So that’s Hello World…what about the Real World?

Using it with dbt

The driver to looking at Harbor was my curiosity as to whether I could have used Harbor in place of my hacky homebrew bespoke and artisanal scripts, and if so what it would look like.

Note:
There is an imbalance here in that I now know more about evals, deterministic testing and LLM-as-judge than I did before creating my custom harness. If I were to write it again, it’d be a lot cleaner I’m sure. So almost by definition, something like Harbor is probably going to be better.

As you’d expect by now, I didn’t write the dbt task myself; I told Claude about my previous work, and had it build a Harbor-compliant task. Its key components are this:

+-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Component | Description |
+=========================================+========================================================================================================================================================================================================+
| Dockerfile | Installs dbt-duckdb and the dbt-agent-skills in the same container that Claude Code will run. |
+-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| instruction.md | The same prompt as before |
| | |
| | > […] Build a dbt project using DuckDB for this data using idiomatic patterns and good practices. […] Run dbt build to verify your work. If it fails, fix the errors and re-run until it passes. |
+-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tests/test.sh | Verifier script, which does several things: |
| | |
| | 1. Deterministic checks (similar to the ones I used before - is the dbt project there, does it have models defined, etc) |
| | |
| | 2. Non-deterministic checks, via LLM-as-judge |
| | |
| | 3. If the LLM-as-judge succeeds, uses its score for the Harbor reward; if it fails, fall back on the deterministic checks score |
+-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| tests/llm_judge.py, tests/rubric.md | Script to call out to LLM to judge the work, using the rubric provided (similar to the one used before) |
+-----------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

There are plenty of gaps in this, such as only using the non-deterministic score. The final Harbor reward value should probably be a weighted version of the deterministic and non-deterministic verification. However, this was more about understanding the scope of Harbor than building the perfect test.

With this in place I could then run my test:

harbor run \
    --agent claude-code \
    --model "claude-sonnet-4-6" \
    --path "tasks/f-rich-with-plugin" \ 
    --artifact /app \                   
    --n-attempts 3

Custom task definition
Capture the output of the agent as an artifact (i.e. don’t throw it away once the test finishes)
Run the same task multiple times

and see how multiple iterations of the same prompt and model scored, and the variance between them:

I still Harbor some doubt…

(…and not just about the scope for so many awful puns)

I think Harbor would be extremely useful for tightly defined tasks against which one wanted to evaluate new models or agentic coding tools (for example, alternatives to Claude Code). In fact, it’s perfect for that, and it’s literally what it’s designed for.

Where I’m not going to be rushing to use it though is for evaluating the effectiveness of different prompts or skills, particularly as I flail around trying all the things and randomly changing lots of stuff.

A lot of that is down to my inexperience in this area; Harbor adds a layer of complexity, and I’m almost certain my random jiggling to unbreak things would often break my Harbor test rig (or invalidate the integrity of the test results). For example; to vary the prompt (the task’s instruction.md) means building another task (if I understand Harbor correctly), which then means duplicating the verifier. Once that’s duplicated, it has the scope to get out of sync across tasks. Maybe there’s a way to align it (symlinks, perhaps - but I’d need to test it, and that’s more tooling work), but maybe Harbor’s not really designed to be used in this way.

Anyway - an interesting tool, and definitely one to keep in mind as I continue to explore this bamboozling world of agents and AI :)

Look Ma, I made a JAR! (Building a connector for Kafka Connect with Claude Code)

Robin Moffatt — Fri, 27 Mar 2026 14:40:58 +0000

As a non-Java coder, for the last ten years I've stumbled my way through the JVM-centric world of "big data" (as it was called then), relying on my wits with SQL and config files to just about muddle through. One of the things that drew me to Kafka Connect was that I could build integrations between Kafka and other systems without needing to write Java, and the same again for ksqlDB and Flink SQL---now stream processing was available to mere RDBMS mortals and not just the Java adonises.

One thing defeated me though; if a connector didn't exist for Kafka Connect, then I was stuck. I'd resort to cobbled-together pipelines leaning heavily on kafkacat kcat, such as I did in this blog post. I built some cool analytics on top of maritime AIS data about ships' locations, but the foundations were shaky at best:

nc ais.host.com 5631 | gpsdecode | kafkacat -b broker:9092 -P

No failure logic, no schema handling, no bueno.

What I really needed was a connector for Kafka Connect. However for that, you need Java.

I don't write Java.

But Claude can write Java.

I've already waxed lyrical about the power of LLMs for coding, and written about the risks of not keeping up with this revolution in how we build things, and the limitations of what they can currently do. So far all this work with coding agents was generally within the realm of something I could eyeball (HTML/CSS), or actually debug and validate (dbt/SQL).

But what about getting a coding agent to build a connector for me?

Skillin Up

There were two things I wanted to get out of this experiment - a working connector for Kafka Connect, and a skill that I could use to have Claude build more in the future. Simply prompting it to build a connector throws away a lot of useful context and knowledge that it builds up in the process.

Since I don't know anything about building connectors, I figured I'd just get Claude to figure it out:

inspect the Kafka Connect API and prepare a Claude skill. Use source available repos on GitHub from Confluent to inform good design practice, as well as their docs: https://docs.confluent.io/platform/current/connect/devguide.html https://www.confluent.io/blog/how-build-custom-kafka-connector/ Make no mistakes ;)

Claude went away and busily set fire to a bunch of tokens. It came back like a proud six year old, having completely missed the point.

# Plan: Kafka Connect REST API Skill

Robin needs a Claude skill that provides deep knowledge of the Kafka Connect REST API and common connector configuration patterns. [...]

Wait, what? provides deep knowledge of the Kafka Connect REST API and common connector configuration patterns?

[rejected tool --- user feedback:] wow, you completely screwed up the brief. I want a skill for building new connectors for Kafka Connect.

You're right, I misread the brief completely. The docs you linked are about developing custom Kafka Connect connectors (Java implementation of SourceConnector/SinkConnector), not using the REST API. Let me restart the research properly.

Once back on course, it came up with a plan. I may not code Java, but I know enough to nudge Claude to really do a thorough job learning as much as it can, as well as reminding it about security. As a side note, this is one of the fundamental beauties of source-available code: people, and now agents, can learn from it.

does this plan include security (not logging passwords, etc)? Can it learn anything from other source-available connectors of note such as Debezium?

After that, it was full-stream ahead, and Claude duly wrote the skill, nbd ¯\_(ツ)_/¯

# Kafka Connect — Connector Development

Build production-grade Kafka Connect source and sink connector plugins in Java.

## Workflow

1. **Gather requirements** — source or sink? external system? schema? auth? offset model?
2. **Scaffold** — Maven project, directory layout, POM dependencies
3. **Implement Connector** — config validation, task partitioning
4. **Implement Task** — data movement (poll or put), offset tracking
5. **Define configuration** — ConfigDef with types, validators, groups
6. **Handle schemas** — SchemaBuilder, Struct, logical types
7. **Security review** — credentials, logging, SSL, error sanitization
8. **Test** — unit, integration (Testcontainers), failure scenarios
9. **Package & deploy** — plugin directory or uber-JAR, install to plugin.path
[…]

You can see the complete skill here.

This is a proof of concept. Literally that. Proving a concept. Don't use it unless you know what gaps might exist with it, both in the skill and the code that it creates.

Claude, build me a connector

As I mentioned above, a few years ago I wrote a blog post about stream processing using AIS data as the source. This data is emitted from ships, with information about their heading etc. Unless you have an AIS receiver, you need to find a service that emits this relayed AIS data, which fortunately the Norwegian government provides for free, including a map viewer:

There's also a TCP endpoint, which I figured Claude could use to test and validate a connector.

build a source connector for AIS data. there is a live endpoint at 153.44.253.27 5631 that you can test with

The user wants to build a Kafka Connect source connector for AIS (Automatic Identification System) data. AIS is a maritime tracking system used by ships. The endpoint at 153.44.253.27:5631 appears to be a live AIS data feed.

Let me first understand what AIS data looks like and explore the codebase to see if there's any existing work. Let me also test the endpoint to understand the data format.

I should: 1. Explore what AIS data format looks like (NMEA sentences typically) 2. Test the endpoint to see what data comes through 3. Check if there's any existing connector or project structure

Let me start with Phase 1 - understanding.

Let me explore the AIS data format and test the live endpoint to understand what we're working with.

If you're interested in how Claude works, using Simon Willison's claude-code-timeline tool is a great way to see quite what it gets up to. Poking around through the JSONL file that Claude writes to, you get an insight into its "thought" process:

Now I have real AIS data from the endpoint. Let me analyze what I see:

[...]

Now I'm ready to design the connector architecture. I need to build a source connector that can handle this tag block format, parse the Norwegian station identifiers, and manage multi-sentence message reassembly.

The key architectural decisions are whether to decode AIS messages internally using a Java library or pass raw NMEA downstream, how to organize the output topic structure, and how to handle the offset model for a live stream with no meaningful replay capability.

Actually, let me think about this more carefully. The user said "a source connector for AIS data" - they want a proper Kafka Connect connector. Given their existing experience with the nc+gpsdecode+kcat pipeline, they probably want something more robust and native.

How did it know about my existing nc+gpsdecode+kcat pipeline? If you look at the prompt above, I just told it the AIS endpoint. Turns out it had a good rummage around my existing projects!

Check if there are any existing AIS or Kafka Connect connector projects in /Users/rmoff/git/. Look for:

Any directories with "ais" or "connect" or "connector" in the name
Any pom.xml or build.gradle files related to Kafka Connect connectors
Any NMEA or maritime-related code

Just do a quick search - this is a brief reconnaissance task.

From that it found its way to my local clone of demo-scene which includes the code it references. Clever stuff.

Its first iteration of the plan is pretty good:

The plan is written. Summary:

What: A Kafka Connect source connector that connects to AIS TCP endpoints, decodes NMEA/AIS messages using AisLib (Apache 2.0), and produces structured Kafka records keyed by MMSI (vessel ID)
Where: /Users/rmoff/git/kafka-connect-ais
Output: Single topic with a flat schema --- common fields (mmsi, msg_type, timestamp) plus nullable type-specific fields (lat/lon/speed for position reports, ship_name/callsign for static data)
Replaces: The nc | gpsdecode | kafkacat pipeline from your existing demos

8 implementation steps: Maven skeleton → config → TCP manager → NMEA parser → schema/converter → connector/task → tests → build & verify.

Now comes the bit that Claude (or a real developer, without domain knowledge) can't do: does it actually make sense what's being proposed?

I picked up on its plan to write one single topic. AIS data has different record types, with different fields across them. I also wanted it to "make no mistakes" a.k.a. sense-check its architecture that it wasn't going to bug-out on a flaky TCP connection.

check the conceptual architecture is right. usually KC would be reading from a finite source such as RDBMS. does anything change for sucking on a TCP pipe?

also topics; what about 1:N design to split by type? is that idiomatic or not? config option maybe?

Other than the ass-kissing ("Great questions!") its reply is useful:

Great questions. Let me address both.

TCP pipe architecture: Yes, this changes things meaningfully. Most source connectors follow a request/response pattern (query DB → get rows → return records). A live TCP stream is fundamentally different:

No replay: If the connector is down, messages are lost. The offset is for observability only, not resume.
Long-lived connection: The TCP socket persists across poll() calls (opened in start(), read in poll(), closed in stop()). This is similar to MQTT or syslog source connectors.
No backoff needed: Data flows continuously --- poll() just reads whatever's buffered on the socket. No "sleep when no data" pattern (though SO_TIMEOUT handles the case where the stream pauses).
No snapshot phase: No existing data to catch up on.
Reconnection is critical: TCP connections drop. This is the main failure mode to handle well.

The architecture is sound --- it's just a different flavor of source connector. The Connect framework handles this fine.

Topic strategy: You're right, 1:N is worth considering. With separate topics, each gets a tighter schema (no nullable fields from other types). Let me think through the options:

Single topic (ais): simpler, matches your existing ksqlDB pipeline
Split by category (ais.position, ais.static, ais.base_station, ais.other): cleaner schemas, easier for consumers that only care about one type
Configurable: best of both worlds

At this point, I am full of sheer joy. As an end user of Kafka Connect and its connectors for many years I've often thought wouldn't it be nice if the connector had config option <x>. And now, they can!

Claude enables you to write tools that work as you want them, not make do and adapt your workflow to fit them. For example: one topic would be fine if that's all I could get. But I'd probably need some post-processing (e.g. Flink SQL) to split out records of interest. Or a Single Message Transform to filter and route particular message types to a sink.

also Kafka message headers - anything clever to do there? And talk to me more about the reconnection of TCP, how are we gonna handle this well and not stall?

Good questions. Let me address both.

Message headers: Yes, headers are useful here. The key win is ais.msg_type --- consumers can filter/route by header without deserializing the value. Also useful for provenance (source station, raw NMEA for debugging).

TCP reconnection/stall prevention: The critical insight is that poll() must never block indefinitely. Here's the layered defense:

SO_TIMEOUT on the socket (e.g., 1000ms) --- readLine() throws SocketTimeoutException after 1s of no data, so poll() always returns
stop() closes the socket from another thread --- immediately unblocks any pending readLine() with an IOException
Volatile stopping flag --- poll() checks it on each iteration
Backoff on reconnect --- avoids hammering a down endpoint

Let me update the plan with both.

Is this TCP reconnection logic good enough? No idea. But I have the---potentially dangerous---feeling of reassurance that at least Claude "understands" the requirement and is going to code for it.

After some more back-and-forth with Claude over how it handled particular AIS message types, and making sure that the connector was going to be ergonomic to work with overall, we settled on a plan.

The user wants me to implement a Kafka Connect AIS Source Connector based on a detailed plan. Let me start by setting up the project structure and implementing each component step by step.

This is a big implementation task.

Claude then merrily chugs away and...builds the connector. In case you're not familiar with coding agents, the key thing here is that it doesn't just churn out code and stop. It'll write code, and then build it and test it, and fix problems, and test it again, until it works.

Several issues: invalid checksums in test data, and the talker ID normalization is likely breaking NMEA checksums. Let me investigate and fix.

Got live data. Now let me write a quick Java test to check if AisLib can parse !BSVDM directly without normalization.

And then there it was, done:

All clean --- 13 files, no target/ artifacts. Here's what was built:

Project: kafka-connect-ais --- Kafka Connect source connector for AIS maritime data

[...]

commit this then add a really good, non AI-stinky README, and full docker compose and quickstart

And there we have it:

kafka-connect-ais
├── docker-compose.yml
├── README.adoc
├── src
│   ├── main
│   └── test
└── target
    └── kafka-connect-ais-0.1.0-SNAPSHOT.jar

Look ma, I just made my own JAR!

The Connector

GitHub repo: https://github.com/rmoff/kafka-connect-ais

(Credit: Claude, obvs)

Let's start up Kafka, Kafka Connect, and Schema Registry:

❯ docker compose up -d --build
[+] Running 4/4
 ✔ Network kafka-connect-ais_default  Created   0.0s
 ✔ Container broker                   Started   0.4s
 ✔ Container schema-registry          Started   0.4s
 ✔ Container kafka-connect            Started   0.4s

Configure kcctl for easy working with Kafka Connect:

kcctl config set-context --cluster=http://localhost:8083 local

Check that the connector plugin has loaded (my connector! that I 'wrote'! omg!!):

❯ kcctl get plugins | grep -i ais
 source             net.rmoff.connect.ais.AisSourceConnector                           0.1.0

The bare config is very simple:

{
  "name": "ais-source-per-type",
  "config": {
    "connector.class": "net.rmoff.connect.ais.AisSourceConnector",
    "ais.hosts": "153.44.253.27:5631",
    "topic": "ais",
    "topic.per.type": "true",
    "tasks.max": "1"
  }
}

Create an instance of the connector using this config:

kcctl apply -f connector-ais-per-type.json

Check the connector is working:

❯ kcctl get connectors

 NAME                  TYPE     STATE     TASKS
 ais-source-per-type   source   RUNNING   0: RUNNING

Honestly thought it would be FAILED. Maybe this AI thing has legs.

Now the litmus test...do we have data?

❯ kcat -b localhost:9092 -L | grep ais
  topic "ais.position" with 1 partitions:
  topic "ais.binary" with 1 partitions:
  topic "ais.aton" with 1 partitions:
  topic "ais.static" with 1 partitions:
  topic "ais.safety" with 1 partitions:

Well, we have topics...

❯ kcat -b localhost:9092 -t ais.static -C -s value=avro -r http://localhost:8081 | jq '.'
{
  "mmsi": 258503000,
  "msg_type": 5,
  "receive_timestamp": 1774612034000,
  "source_station": {
    "string": "2573555"
  },
  "raw_nmea": "\\s:2573555,c:1774612034*0D\\!BSVDM,2,1,5,A,53nQiF000000hiLH000I9DPthlDr22222222220N2H633400000000000000,0*0E\n\\s:2573555,c:1774612034*0D\\!BSVDM,2,2,5,A,00000000000,2*38",
  "callsign": {
    "string": "LLWF"
  },
  "ship_name": {
    "string": "FRUHOLMEN"
  },
[…]

and OMG we have data! 🙀

You can start to poke around the data itself and also do fun things from the terminal with visidata:

kcat -b localhost:9092 -t ais.static -C -s value=avro -r http://localhost:8081 -c 2000 | \
jq -c '{mmsi, ship_name: .ship_name.string, ship_type: .ship_type_text.string, dest: .destination.string}' | \
vd -f jsonl

Regarding this connector itself, I'll reiterate the same as I did with the skill:

This is a proof of concept. Literally that. Proving a concept. Don't use it unless you know what gaps might exist with it, both in the skill and the code that it creates.

y u hate java tho?

This is not a Java-bashing post, in the slightest.

It's a "I don't know this language but AI enables me to work with it".

It could just as well be a "I'm a Java programmer who doesn't know SQL and look at what Claude wrote and wtf is this PIVOT doing"

I don't hate Java. Language 'wars' are for imbeciles.

You can find the connector and the skill that I built in this process here: https://github.com/rmoff/kafka-connect-ais.

Interesting links - March 2026

Robin Moffatt — Thu, 26 Mar 2026 16:37:05 +0000

I’ve had a huge amount of fun this month exploring quite what AI (in the form of Claude Code) can do for a data engineer. Rather than just hack around at a prompt, I took a bit more of a considered approach to it, building a harness to test out different prompts and skills. You can read my write-up here, the headline of which is that literally Claude Code isn’t going to replace data engineers (yet).

I’ve also written up an AI Disclosure for my blog which I’ll keep up to date as my use of AI evolves, along with a sweary rant about why you basically have to get on board with AI if you value your career.

Unresolved directive in <stdin> - include::../../asciidoc-includes/il-header.adoc[]

Kafka and Event Streaming

🔥 A fascinating deep-dive from the Apache Iggy team looking at their migration journey to thread-per-core architecture powered by io_uring.
Simone Esposito at Bitrock looks at the new Queues for Kafka feature that was released recently.
"Unknown magic byte" is an error that has sent many a Kafka user running for the comforting (if unsuitable) embrace of plain JSON for their message serialisation, but no longer! David Araujo describes how schema IDs can now be stored in the header of a Kafka message - this is huge.
An interesting thread on Reddit looking at how to give external partners access to Kafka topics without direct broker access.
I’ve mentioned Tansu before, and here’s another write-up of it, covering a talk by its author at QCon London 2026.

Stream Processing

🔥 Feldera’s Mihai Budiu explains Why incremental aggregates are difficult.
🔥 My colleague Gustavo de Morais has shared his slides from Flink Forward 2025: The Big State Monster: Taming State Size in Multi-Way Joins with FLIP-516.
A nice step-by-step guide from the Apache Gravitino project showing how to use Gravitino with Apache Flink.
The recording of LinkedIn’s Stream Processing meetup (February 2026) is now available with three good talks:
A very nice tool from Tom Cooper for tracking KIPs and FLIPs: OSSIP.
My colleague Florian Eiden has published flink-unittest, a Python unit testing framework for Flink SQL.
Gordon Murray built a Flink Connector for Apache Iggy.
Zalando’s Maryna Kryvko looks at the impact of joins using the Table API on Flink’s state store in 1.20, and optimisations that they made to reduce it by 75%. They rightly note that improvements are already available in Flink 2.1.

Analytics

Reddit’s Neven Miculinic describes how they built their logging platform around Clickhouse and Kafka.
DuckDB 1.5.1 includes support for Lance lakehouse format, improvements to Iceberg v3 support, and more.
Autotrader’s Tom Armitage has a very cool article looking at data visualisation and how one goes about splitting the UK into ten perfectly balanced regions.
🔥 My colleague Elijah Meeks (who did all the cool animations on Flink Watermarks…WTF) has a dataviz library which he rewrote to be streaming-first. You can find the Semiotic repo here and lots of very pretty examples on the main site.
Kartik Khare from StarTree has a good two part series looking at real-time ingestion in Apache Pinot (part 1, part 2).
Prithwish Nath took a look at how far you can push DuckDB on Commodity Hardware, whilst Gábor Szárnyas tried out DuckDB on the new MacBook Neo.

Data Platforms, Architectures, and Modelling

Swiggy’s Sundaram Dubey looks at the demand for real-time dashboards at Swiggy, and how they implemented it.
A good primer from Milan Mosny on Ontology, Taxonomy, Data Model, Context Graph & Friends.
A recording of Anurag Kale’s QCon talk about their experience with Data Mesh.
🔥 Chris Hillman has a great post here Your Data Model Isn’t Broken, Part I: Why Refactoring Beats Rebuilding.
Yoshnee Raveendran from Spotify explains how they generate "Spotify Wrapped" for each user.
🔥 Nice write up from Edijs Drezovs and team describing in good detail the lakehouse that Yggdrasil Gaming built.
🔥 Joe Reis' Practical Data Modeling book is nearing completion, and you can access significant chunks of the early chapters already on his Substack.
A VLDB paper from 2021 by Google’s Pavan Edara and Mosha Pasumansky describing BigQuery’s metadata management system and how it supports performance at scale.
Tim Castillo has a good piece discussing the three layers in which business logic lives (AI context, semantic, and dimensional model): Analytics Engineering’s Unfinished Work.

Data Engineering, Pipelines, and CDC

Confluent have just launched their dbt adapter for Confluent Cloud Flink SQL (it also works with Apache Flink too via this gateway).
🔥 I dug into what Claude Code could do with dbt and concluded that Claude Code isn’t going to replace data engineers (yet).
Following on from my baby-steps above, here’s a thorough examination from Mark Rittman of what it looks like when you actually put LLMs to work alongside you.
Goldsky’s Jeff Ling has details of how they got a 12x improvement by reading data directly from Clickhouse as Arrow instead of Kafka (Warpstream) with Avro.
Chris Gambill puts into clear words what anyone my age has also been thinking: Medallion Architecture Isn’t As New As You Think. I took a similar run at it previously, looking at how Oracle’s Big Data (lol, this was 2016, ok) reference architecture compared to it too.
Hila Turi from Riskified has a fascinating article about their 1 billion row table in Snowflake for 3rd-parties, and how they handled the data modelling, cost, and retroactive changes.
A set of three excellent, hands-on, posts about building really high-quality data pipelines:
- 🔥 The nitty-gritty of actual data engineering isn’t in the fancy dashboard for the stakeholders, but making sure that you’re actually processing all the data and not losing any. Robert Sahlin has a great article all about monitoring for silent data loss.
- 🔥 As well as guarding against data loss, a data engineer also needs to have a clear picture of the performance of their estate, which Rodrigo Molina describes in this practical article looking at Measuring Latency in Data Platforms.
- 🔥 Good stuff from Jeremy Chia and Justina Šakalytė at Vinted looking at how they handle data quality issues without stalling their pipelines. Also available as a recording.
A practical primer from Ben Rogojan (a.k.a. SeattleDataGuy) on Full Refresh vs Incremental Pipelines.
Joe Reis mulls over the state of the data engineering job market.
Dan Beach has a list of Data Engineering Blogs to Follow that’s very good. But then I would say that, cos I’m on it ;)
Phi Vu Trinh has an excellent writeup of a VLDB paper from engineers at YouTube describing how they do CI/CD for data pipelines.

Open Table Formats (OTF), Catalogs, Lakehouses etc.

Details of pg_duckpipe, which streams Postgres tables into DuckLake.
Grigorii Osipov considers whether DuckLake can fix the Lakehouse.
A good article from Reetika Agrawal looking at Iceberg Branches and Tags with Presto.
If you need to read and write across table formats (Iceberg/Delta/Hudi) then you’ll be interested in this article from Junaid Effendi in which he explains how Delta UniForm works.
🔥 My colleague Gunnar Morling has released v1.0 of Hardwood: A New Parser for Apache Parquet. Much as we all love shovelling a ton of Hadoop dependencies every time we want to use Parquet, something tells me that this approach might be a better one :D
🔥 Some lovely Iceberg deep-dive detail and performance analysis from Chris Douglas part 1 / part 2.
I wrote previously about alternatives to the now-abandoned MinIO project. One of my favourite replacements is SeaweedFS, which now supports Amazon S3 Table so you can write Iceberg tables directly to it.
Andrew Lamb’s recent talk, "Column Storage for the AI Era", looked at the proliferation of column storage formats and what is driving it recording / slides.

RDBMS

🔥 Elizabeth Christensen (a.k.a. SQLLIZ) recently published a whole day’s worth of free Postgres training material.
A summary from InfoQ’s Leela Kumili of Uber’s work improving their MySQL Cluster uptime with consensus replication.
🔥 A nice troubleshooting diagnostics story from Anthonin Bonnefoy at DataDog about Postgres upserts that don’t update but still write and the problems this can cause at scale.
Staying with DataDog, they recently made available a free online explain plan visualizer, which Tanel Poder put through its paces with Oracle execution plans.
🔥 I love this post from Nile’s Jan Nidzwetzki about his tool pg_plan_alternatives, which uses eBPF for tracing Postgres query plan alternatives.
Fun stuff from Radim Marek showing how to manipulate Postgres optimizer stats to help evaluate query behaviour in different environments.
Ergest Xheblati explains why you should still learn SQL (despite LLMs being able to do a rather good job of writing it for you these days).
Etsy’s Ella Yarmo-Gray describes how they migrated Etsy’s database sharding to Vitess.
A couple of good blog posts from the folk at PlanetScale. First a serious one: Ben Dicken with an excellent interactive explainer of Database Transactions. Second, a silly one, in which Nick Van Wiggeren shows how to do Video Conferencing with Postgres (seriously).

AI

🔥 A fascinating podcast episode of Dwarkesh Podcast with Anthropic’s Dario Amodei. Pleasantly BS-free and easy enough for a non-expert like me to not feel completely lost listening to :)

Big Picture & Culture

Tip:
There’s a theme running through this section. I’m going to crudely summarise it thus:

AI is here, and it’s a massive opportunity for us as individuals in the software industry.

There are many reasons we recoil against AI and that’s natural and understandable—but don’t mistake these for reasons not to embrace it (see first point).

🔥 Brittany Ellich’s blog post was my favourite one this month with this excellent post, from which I’ll quote verbatim:

Nobody knows what the future of software engineering looks like, and that’s incredibly uncomfortable. But instead of waiting for someone to hand us the answer, I think the move is to embrace the uncertainty, because these moments of deep uncertainty have historically been moments of extraordinary opportunity.

Annie Vella has an accessible and unfussy way of clearly communicating, and I really enjoyed both of her recent posts, Finding Comfort in the Uncertainty and The Middle Loop.
I wrote a ranty post to get stuff off my chest, but not nearly as articulately nor considered as many others in this section: AI will f**k you up if you’re not on board.
Chris Gambill’s post is in a similar vein to mine: If You Aren’t Using AI, You Are Already Legacy.
🔥 Nolan Lawson’s post We mourn our craft may sound hyperbolic but there is genuinely a sense of grief that I sense amongst engineers at something that is being taken from them whether they like it or not (and regardless of whether they themselves adopt it or not).
The concept of grief and loss and its corresponding stages (anger, denial, bargaining, etc) actually fit well with many of the emotions that I’m seeing play out at the moment, and Andrew Murphy explores it in exactly this framing: The five stages of losing our craft.
Some of the folk in the "denial" camp need to read Charity Majors' latest post, in which she addresses head-on the point that AI now is not what it was when people dismissed it even last year.

When the facts change, I change my mind

Outside of the pondering and realisation that the world has changed for software engineers comes the refreshing and entertaining reactions against one of the downsides of AI:
- 🔥 Kirill Bobrov - Stop Feeding Me AI Slop.
- Brent Ozar - I’m Not Gonna Waste Time Debunking Crap on LinkedIn.
- See also Brandolini’s law).

Open Source Licensing and AI Policies

The ability of LLMs to write code has blown wide open the debate on software licensing and what constitutes a 'copy'.

Simon Willison analyses the recent events on the chardet project that ignited the latest round of this debate and Thomas Claburn in El Reg has commentary from Bruce Perens (who created the original Open Source Definition).
antirez has an excellent commentary piece on the subject, as well as details of implementing a clean room Z80 / ZX Spectrum emulator with Claude Code (not specific to OSS per se, but interesting to understand the concepts being used).

Meanwhile, projects and foundations are trying to rapidly keep up with what LLMs can do, and codify what role they should play in contributions:

Phil Eaton - Source-available projects and their AI contribution policies.
Kate Holterhoff - The Generative AI Policy Landscape in Open Source.

Building with AI

Thoughtworks' Rahul Garg discusses Context Anchoring.
Cat Hicks has a neat idea for Claude Code - a learning-opportunities skill that has it teach you and build your expertise of what you’re building with it. I tried something very (very) crudely along the same lines in Claude the Instructor and found it very useful.
Hajime Takeda has a thorough explainer of How to Build a Production-Ready Claude Code Skill.
Daniel D. McKinnon has some home-truths for PMs having fun vibe-coding features for their products, such as:

If the feature is actually important, fix the system for prioritization (your real job) rather than circumventing it.

He’s also got good advice on writing GenAI evals.

Geoff Cisler and team at Whoop have written about their use of evaluations to test and track the performance of their Agents.
Joe Hellerstein discusses AI and the Mixed-Consistency Future (files instead of databases isn’t gonna work for everything).
Rajanikant Vellaturi discusses RAG and Data Engineering.
As people get past the "omg this is magic" initial experience of coding agents, understanding exactly what they’re doing is often the next question. tapes is one tool that aims to address this, sitting between your agent and the model provider and giving a bunch of instrumentation and insight.
Fascinating exploration from Mitchell Turner of the idea of promptware: Brainworm - Hiding in Your Context Window.

The MCP Debate

MCP was hot (you don’t need CLI access!), and then it wasn’t (it burns context!), and now…It Depends ;)

Allen Hutchison has a good article explaining why MCP Isn’t Dead You Just Aren’t the Target Audience (tl;dr MCP is still vastly important for agents not running with CLI access).
Samir Amzani - Your MCP Server Is Eating Your Context Window. There’s a Simpler Way.
🔥 Charles Chen - MCP is Dead; Long Live MCP!.
Ben Davis - "The current 'MCP is dead discourse' is my current favorite example of the really annoying "simplicity" brainrot that’s been plaguing tech for a while" (Twitter thread).

AI in Practice

🔥 Instacart’s Moein Hasani describes how they improved their recommendations engine using tools and techniques including LLMs and Evals.
Tao Ruangyam at Zalando has a good blog post about using LLM-as-judge to ensure the quality of their search results.
Fascinating detail of how Uber are using AI for development (blog, talk), as well as an agentic system for writing design specs.
Tan Wang at Pinterest has practical details of how they’re implementing an MCP ecosystem.
Overview from Aman Gupta and Daniel Braithwaite at Nubank of how they’re building AI agents, doing evals, and semantic versioning of prompts.
Surabhi Gupta details how Klaviyo are approaching AI-driven engineering.
Sneh Agrawal and team at Grab built AI agents to help them support users of their data warehouse.
If you’re still not convinced that AI is out there right now delivering value for businesses, check out this list from Allen Hutchison of real-world AI agent examples.

And finally…

Nothing to do with data, but stuff that I’ve found interesting or has made me think (or smile).

🔥 A great piece from Chris Hillman, the tldr of which is the title (but read the article too!) Your Friends Will Be There for You. Your Work Won’t.
Dan Carlin (he of Hardcore History) is one of my absolute favourite podcasters. His style of presenting and depth of historical material is just wonderful. That’s why this post hit me harder than I’d expected:

I think it is just hard to talk with passion and enthusiasm […] about events thousands of years ago on the history show when such momentous ones are occurring to all of us right now. […] We ARE living through absolutely momentous times (and dangerous ones). Don’t allow yourself to be gaslit about that. Any fan of History can see it.

The concept of "managing up" is important at work, and Lara Hogan describes it well in Managing your manager.
🔥 Almog Gavra has created a very nifty tool called YuzuDraw which lets you create—and edit—ASCII-art style diagrams. It works with coding agents too.

Kagi Translate includes a target language of LinkedIn Speak ;)

In all seriousness though, Kagi is a set of applications that I rather like for its focus on functionality and respect, not growth hacking and enshittification. Their small web is a joyful reminder of how the internet used to be before walled gardens and karma-farming, and the search engine lets you wrest back control of your search results from the spam and shit (sorry, "sponsored listings").

🔥 I Taught My Dog to Vibe Code Games. No notes.

Unresolved directive in <stdin> - include::../../asciidoc-includes/il-footer.adoc[]

Evaluating Claude's dbt Skills: Building an Eval from Scratch

Robin Moffatt — Fri, 13 Mar 2026 18:05:22 +0000

I wanted to explore the extent to which Claude Code could build a data pipeline using dbt without iterative prompting. What difference did skills, models, and the prompt itself make? I’ve written in a separate post about what I found (yes it’s good; no it’s not going to replace data engineers, yet).

In this post I’m going to show how I ran these tests (with Claude) and analysed the results (using Claude), including a pretty dashboard (created by Claude):

The Test

Can Claude Code build a production-ready dbt project? (is AI going to take data engineers' jobs?)

Terminology check:
I am not, as you can already tell, an expert at building and running this kind of controlled test. I’ve adopted my own terminology to refer to elements of what I was doing, which may or may not match what someone who knows what they’re doing would use :)

Scenario: What are we testing (specific Prompt + Skill combination)

Configuration: Scenario + Model

Run: Execution of a configuration

Validation: Deterministic checking of the dbt project

e.g. Does the project build? Are there any errors? Are there staging and mart folders? etc.

Judging: Using an LLM to assess the quality of the dbt project that was built

Trial: One execution of a judge against a run of a configuration

Design

I created the test to run independently, with no 'human in the loop'. That is, Claude Code was free to run whatever it wanted to in order to achieve the task I’d given it.

I explored permutations of two dimensions in my scenarios: prompt (x2) and skills (x3). Each of these I then iterated over with different models.

Prompt
- Rich (lots of background data analysis, specifics on what features to include, etc)

Note:
View prompt

I’ve explored and built pipelines for the UK Environment Agency flood monitoring API. Here’s my analysis:

Data exploration: https://rmoff.net/2025/02/28/exploring-uk-environment-agency-data-in-duckdb-and-rill/

Pipeline build: https://rmoff.net/2025/03/20/building-a-data-pipeline-with-duckdb/

Build a dbt project using DuckDB for this data using idiomatic patterns and good practices. Requirements:

Proper staging → dim/fact data model

Handle known data quality issues (see blog posts for details)

SCD type 2 snapshots for station metadata

Historical backfill from CSV archives (see https://environment.data.gov.uk/flood-monitoring/archive)

Documentation and tests

Source freshness checks

Run dbt build to verify your work. If it fails, fix the errors and re-run until it passes.

Minimal (here’s an API, build me analytics)

Note:
View prompt

The UK Environment Agency publishes flood monitoring data, see https://environment.data.gov.uk/flood-monitoring

Build an idiomatic dbt project following good practices using DuckDB that ingests this data and models it for analytics.

Run the project and make sure that it works. If it fails, fix the errors and re-run until it passes.

Skills
- None
- Single skill (Using dbt for Analytics Engineering)
I’d meant to test the full plugin, but a snafu meant I only ended up pulling in the single skill. I realised this only after running the scenario in full, so expanded the test to include the full plugin as a separate scenario.
- Full plugin (dbt Agent Skills)
Model
- Claude Sonnet 4.5
- Claude Sonnet 4.6
- Claude Opus 4.6

Execution

One of the core things that I wanted to find out was what Claude can do on its own. Having it ask for permission to do something slows things down, and asking for input defeats the point of the exercise. So I used it with the effective but spicy flag --dangerously-skip-permissions:

claude --dangerously-skip-permissions $PROMPT

This was wrapped in a Docker container so that it couldn’t cause too much trouble.

Claude Code writes a full transcript of its sessions to a JSONL file that usually resides in ~/.claude/, so for the Docker container I had that copied out into the test results too, along with the actual dbt project itself and any other artefacts from the test run. The JSONL is interesting for what it tells us about how Claude Code approaches the task, particularly on multiple runs of the same configuration. Here’s an example analysis of part of a session log.

I used Claude to write a bash script that then spun up a Docker container with the correct set of configuration for the test scenario.

Each run’s session log was processed to produce summary metadata:

{
  "model_requested": "claude-opus-4-6",
  "model_actual": "claude-opus-4-6",
  "cost_usd": 3.420355,
  "duration_ms": 1175360,
  "input_tokens": 718,
  "output_tokens": 43568,
  "cache_read_tokens": 2423321,
  "cache_creation_tokens": 162914,
  "num_turns": 57
}

Output

Once I’d run all of the scenarios, I had a set of results on disk:

❯ tree runs -L1
runs
├── A-minimal-no-skills
├── B-rich-no-skills
├── C-minimal-with-skills
├── D-rich-with-skills
├── E-minimal-with-plugin
└── F-rich-with-plugin

Each folder had multiple models and within those, runs, e.g.

❯ tree runs/A-minimal-no-skills -L2
runs/A-minimal-no-skills
├── claude-opus-4-6
│   ├── run-1
│   ├── run-2
│   └── run-3

and within each of those, a dbt project (assuming that Claude had done its job successfully!):

❯ tree runs/A-minimal-no-skills/claude-opus-4-6/run-1/project/flood_monitoring -L1
runs/A-minimal-no-skills/claude-opus-4-6/run-1/project/flood_monitoring
├── analyses
├── dbt_packages
├── dbt_project.yml
├── flood_monitoring.duckdb
├── logs
├── macros
├── models
├── README.md
├── seeds
├── snapshots
├── target
└── tests

So we’ve got a set of dbt projects, produced by Claude Code. As part of Claude’s prompt it was instructed to iterate until they work:

Run dbt build to verify your work. If it fails, fix the errors and re-run until it passes.

So they should hopefully at least build. But are they any good?

There are two ways to evaluate it: deterministically, and non-deterministically. Each has its own strengths and weaknesses, and I’ve used both.

Validation

This is a shell script that encodes various checks one can do against a dbt project, such as does it exist:

# Check dbt project exists
if [ -n "$DBT_ROOT" ]; then
    if [ "$DBT_ROOT" = "$PROJECT_DIR" ]; then
        echo "PASS  dbt_project.yml exists"
    else
        rel_path="${DBT_ROOT#${PROJECT_DIR}/}"
        echo "PASS  dbt_project.yml exists (in ${rel_path}/)"
    fi
else
    echo "FAIL  dbt_project.yml not found"
    echo ""
    echo "Validation complete (no dbt project found)."
    exit 1
fi

It can also check for the implementation of features such as incremental materialisation:

# Check for incremental materialization
incremental_count=$(grep -rl "materialized.*=.*'incremental'\|materialized.*=.*\"incremental\"\|incremental" "${DBT_ROOT}/models/" --include="*.sql" 2>/dev/null | wc -l | tr -d ' ')
if [ "$incremental_count" -gt 0 ]; then
    echo "PASS  Incremental materialization: ${incremental_count} model(s)"
else
    echo "INFO  No incremental models (all full-refresh)"
fi

Here’s an interesting one though, where it checks for the implementation of slowly changing dimensions (SCD):

# Count snapshots
snapshot_count=$(find "${DBT_ROOT}/snapshots" -name "*.sql" 2>/dev/null | wc -l | tr -d ' ')
if [ "$snapshot_count" -gt 0 ]; then
    echo "PASS  Snapshots: ${snapshot_count} (SCD handling)"
else
    echo "FAIL  No snapshot models found (no SCD handling)"
fi

But what if it’s done SCD using a different method from dbt’s snapshots? Arguably, that’d be a non-standard approach, but the above check might FAIL even if SCD are there.

And this is where we hit the limitation of validations; they can only determine so much. The presence of a file or folder is easy enough to check and makes sense to do this way. But how do we check for the presence of a concept, or look holistically at what’s been built?

That’s where the concept of "LLM-as-judge" comes in.

Judging

We’re familiar enough by now with how powerful LLMs are. I mean, crikey, it’s just gone and built a dbt project for us from the scraps of a prompt!

As well as building, LLMs can assess. I use them to proofread my blog, heavily. They’re great at it. And they’re perfect for examining a dbt project and seeing if it matches up the spec it was given and general good analytics engineering practices.

You are an expert dbt analytics engineer evaluating the quality of a dbt project.

I produced a rubric describing different aspects on which to judge a project:

# Scoring Rubric

Standalone scoring rubric for evaluating Claude-built dbt projects against the
[reference implementation](https://rmoff.net/2026/02/19/ten-years-late-to-the-dbt-party-duckdb-edition/).

**Scoring scale**: 0 = missing, 1 = attempted but broken, 2 = functional but lacking, 3 = production-quality.

## Criteria

### 1. Data model

Staging → marts with dim/fact separation, proper naming conventions. [dbt best practice](https://docs.getdbt.com/best-practices/how-we-structure/1-guide-overview): staging creates atomic building blocks from source data, marts produce wide, rich business entities.

| Score | What it looks like |
|-------|--------------------|
| 0 | No layered model — everything in one directory or single query |
| 1 | Layers exist but naming is inconsistent or structure is wrong (e.g. dim_ in staging/) |
| 2 | Proper staging (stg_) → marts with schema separation, but fact table is fully denormalized (no separate dims) |
| 3 | staging → marts with dim/fact separation: `stg_stations`, `stg_measures`, `stg_readings` → `dim_stations`, `dim_measures`, `dim_date`, `fct_readings`. Bonus: intermediate layer for reusable joins |

**Reference**: 3 staging models, `dim_stations`, `dim_measures`, `dim_date`, `fct_readings`. dbt recommends `stg_[source]__[entity]` naming (double underscore separating source from entity).

[…]

and told the LLM to use this to produce a set of scores in JSON format

## Your Task

Score this dbt project against each of the 9 criteria in the rubric above. For each criterion:
1. Review the project files provided
2. Assess what was built against what the rubric describes
3. Assign a score (0-3) with brief notes explaining your reasoning

Respond with ONLY a JSON object (no markdown fences, no explanation outside the JSON) matching this exact schema:

{
  "scores": {
    "data_model": {"score": <0-3>, "notes": "<brief explanation>"},
    "key_relationships": {"score": <0-3>, "notes": "<brief explanation>"},
    "messy_data": {"score": <0-3>, "notes": "<brief explanation>"},
    "scd_snapshots": {"score": <0-3>, "notes": "<brief explanation>"},

[…]

along with freeform narrative:

  "observations": ["<key observation 1>", "<key observation 2>", "..."],

This produces output that looks like this:

{
  "scores": {
    "data_model": {
      "score": 2,
      "notes": "The project has a perfect staging -> marts structure with dim/fact separation and correct naming. However, the data loading mechanism within the `on-run-start` hook is brittle and causes the `dbt build` to fail, preventing the models from being populated. The design is a 3, but the implementation is broken."
    },
    "key_relationships": {
      "score": 3,
      "notes": "The model joins are correct, using the canonical keys extracted in staging. A `relationships` test is correctly implemented on `dim_measures` to validate the foreign key to `dim_stations`, including a `warn` severity."
    },
[…]
  },
  "total": 19,
  "max": 27,
  "observations": [
    "The `dbt build` command fails due to a `Binder Error` when loading raw readings. The `CREATE TABLE IF NOT EXISTS` followed by an `INSERT ... WHERE NOT EXISTS` pattern in the `on-run-start` hook is brittle and likely conflicts with the validation script's setup.",
    "The `dim_stations` model is not built from the `snap_stations` snapshot. This is a common mistake; the project creates the Type 2 SCD history but fails to use it in the dimensional model, which means analyses will not be historically accurate.",
[…]
  ]
}

As well as the original Claude model, I tried with Qwen (qwen2.5-coder:32b, running locally on Ollama), and Gemini (gemini-2.5-pro, hosted externally). For a model to be of use it needs to be able to accurately analyse the dbt project with sufficient detail to then be able to judge it. That is, it’s no use if it takes a cursory look (figuratively speaking), sees some SQL and YAML and signs it off as a LGTM. As well as being able to do analysis of sufficient depth, it needs to be able to then judge what it’s found against the criteria it’s given.

With yet another healthy dollop of Claude, I processed the judging data into some charts, such as this one showing how different aspects of the judging rubric were assessed by different models:

Rolling this up shows that Qwen (the local model) scores pretty consistently with Claude and Gemini:

However, if we look at the score that each trial (as I’ve called it; one iteration of the judging prompt against the generated dbt project) scores when judging the same dbt project, there’s an interesting variance. Ideally we’d want every trial to return pretty much the same number. Too much difference indicates that the LLM is inconsistent with its judging, which thus lessens our trust in it:

In addition, Qwen’s freeform notes are much sparser. Compare these judging notes relating to the same dbt project:

claude-opus-4-6:

correctly uses the station URI field (not stationReference) as the FK, with a code comment noting 'stationReference diverges in ~0.4% of records'

qwen2.5-coder:32b:

Joins exist and use correct keys

So, Qwen is ok for local testing, but for really consistent and insightful judging, it’s not going to be my first choice.

I’m tiering up here

My initial version of the rubric had the LLM score from 0 (sucks) to 3 (excellent) on a range of features, including data model, testing, SCD, and documentation.

This was fine, but arguably a slightly crude way to do things. After all, if the project is missing some core best practices or has correctness issues (e.g. missing a dimension, or has mis-identified the correct PK/FK relationships), it’s garbage and needs fixing. Other features that might be missing or sub-optimal, perhaps incomplete documentation or missing tests, oughtn’t count as much as they can be rectified or added on later.

I created a second rubric and reran the judging (a nice side-effect of having captured the project and running the validation and judging separately) using a tiered approach, giving greater weight to foundation principles and less weight to advanced ones. Given another pass at it, I’d probably refine it further, adding some kind of punishment weighting for correctness errors (such as truncating the input dimension data from the API call by ~60%, which one of the runs did).

Gosh, didn’t I do well?

If you ask an LLM to do something, it’ll do so as it thinks is best. If you then ask it how it did, it’s going to judge it against what it thinks is best.

Can you see the issue here?

LLMs are notoriously positive and eager to confirm (That’s an astute observation!, You’re absolutely right!), so asking Claude to mark Claude’s work is going to be subject to this kind of bias.

One option here is to use a different family of models to judge the output. (Of course, you then risk diving into an ever-deeper hole of evals; how do the different models perform at judging? Maybe we should score them and judge them too?!)

What I found was that Claude and Gemini both do a good job at judging, with Claude not showing any apparent bias:

Validation vs Judging - Pros and Cons

+--------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
| Validation | Judging |
+====================================================================+=====================================================================================================+
| Pass/Fail only | Detailed report (not just pass/fail)\ |
| | |
| | Subjective scoring with narration/explanation of scoring and overall project |
+--------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
| Can only determine so much. Each check requires explicitly coding. | Holistic evaluation using existing training and supplemented with skills as needed |
+--------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
| Deterministic (same result each time for the same input) | Non-deterministic (potentially different results each time for the same input)\ |
| | |
| | If it made the mistake in building the project will it necessarily spot it when judging it?\ |
| | Possibly recommended to use different model family to avoid "marking my own homework" positive bias |
+--------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+
| No additional LLM costs | Additional LLM cost |
+--------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------+

The upshot of this is that it makes sense to use both approaches. It makes sense to run the validations first, and then only continue to judging if a sufficient threshold score is met from the validation. That is, there’s no point burning LLM tokens for a more nuanced assessment of a project if the validation script found that there were critical errors in even running it.

Contamination (a.k.a. What Went Wrong?)

One serious problem with my harness was that to begin with I mounted the project itself into the Docker container. This was so what Claude built, along with the session log, would be available outside the container.

The issue with this was that Claude Code is a clever soul, and as well as doing the research that I’d expect it to (e.g. reading the blog posts that I gave it explaining the data structures), it also scoured the local filesystem, and found the rubric that is used for scoring the result by the judge LLMs.

Agent Explore the workspace at /workspace to understand what already exists.

Let me read the rubric and key reference files to understand the scoring criteria.

Now I understand the setup. The validation script: 1. Looks for dbt_project.yml - profile name is extracted from it […] Looking at the rubric requirements, I need a macro with date range parameters, a separate staging model for archive data, and the fact table should union API and archive data with deduplication. […]

So not only the rubric was exposed (giving Claude a huge glance at the answers to its homework), but also the validation script, and a Python script it had written in a previous iteration to load the data, load_api_data.py.

So, well and truly contaminated. If you want to spin it…an example of what Claude Code can do when you give it a ton of help ;)

All these test runs I marked as contaminated and excluded from my analysis and notes shown below. (I didn’t delete them because in isolation they were still interesting as they demonstrated 'given a huge helping hand, how do different models perform?').

Front End

What do you do if you’ve got a ton of data spread across multiple folders, and you want to pull it all together into one place to both summarise and dig into specifics? You get Claude to build you a dashboard of course!

Here are all the results for each scenario using Opus 4.6, with summarised results for each scenario:

These can be examined per-run, highlighting the differences that LLMs will sometimes make when given the exact same input:

The columns show:

LLM-as-judge results
Validation results
How long a run took, how many turns, and cost of tokens

For each run the Claude session log is available to inspect:

Different event types can be filtered, so you can dig into exactly why and how something happened, or just stroll along through the thought patterns of the LLM:

I included an option to view the log in Simon Willison's Timeline Viewer, which makes some of the really low-level poking about the logs easier:

Going back up to judging, for each test the details of the scores can be viewed:

along with the scoring detail and notes themselves:

And the findings from all of this testing?

All of the Claude models do a pretty good job, given a sufficiently rich prompt. The dbt-agent plugin and skills help, but not as much as you might assume. None of the tests produced a dbt project that was sufficiently good to take through to production.

As a companion, Claude Code is an invaluable tool. But it still needs an experienced data engineer to instruct and guide it, and review the work it produces.

Tip:
For a full analysis and commentary of the best performing result, see Claude Code isn’t going to replace data engineers (yet)

Code

The code for all of this is on GitHub.

It’s a slightly sanitised version of what I ran, but should be enough to get you (or Claude) going!

Claude Code isn't going to replace data engineers (yet)

Robin Moffatt — Thu, 12 Mar 2026 10:38:58 +0000

Ten years late (but hopefully not a dollar short) I recently figured out what all the fuss about dbt is about.

Well that’s cute, Robin, you might be saying. Congratulations for catching up on what data/analytics engineers have been doing for years now. But you see, coding by hand is **so* 2025.* Didn’t you hear? AI is going to replace data engineers.

No it’s not (at least, not yet). In fact, used incorrectly, it’ll do a worse job than you. But used right, it’s a kick-ass tool that any data engineer should be adding to their toolbox today ^*^. In this article I’ll show you why.

I’d already used Claude to teach me dbt, and so I was very curious to see to what extent Claude Code (at the beginning of March 2026) could actually write a dbt project. Not only that, but:

Can Claude build a credible dbt project from real data?
How much help (prompting) does it need; can I just give it the API source, or does it need the analysis work done for it?
What difference do the skills that dbt labs recently published make?
How do the different Claude models perform?
How do you actually evaluate these different factors? (tl;dr crash-course in evals and LLM-as-judge assessment of what was built each time)

Note:
I’ve deliberately split this write-up into two posts as they each address different points and interests:

What Claude can (and can’t) build with dbt ← you are here.

This one’s for the data engineers wondering if they still have a job.

The rig I built to run the scenarios, evals, and LLM-as-judge assessments to see what impact different factors had on the quality of what was built.

This one’s for the detail nerds, and LLM folk to poke holes in what & how I tested stuff ;)

Evaluating Claude’s dbt Skills: Building an Eval from Scratch

Building a dbt project with Claude Code

Here’s the prompt that I gave Claude Code, running with model Opus 4.6:

I've explored and built pipelines for the UK Environment Agency flood
monitoring API. Here's my analysis:

- Data exploration: https://rmoff.net/2025/02/28/exploring-uk-environment-agency-data-in-duckdb-and-rill/
- Pipeline build: https://rmoff.net/2025/03/20/building-a-data-pipeline-with-duckdb/

Build a dbt project using DuckDB for this data using idiomatic patterns
and good practices. Requirements:

- Proper staging → dim/fact data model
- Handle known data quality issues (see blog posts for details)
- SCD type 2 snapshots for station metadata
- Historical backfill from CSV archives (see https://environment.data.gov.uk/flood-monitoring/archive)
- Documentation and tests
- Source freshness checks

Run `dbt build` to verify your work.
If it fails, fix the errors and re-run until it passes.

As well as the above prompt, I also gave Claude access to the newly-shipped dbt-agent-skills from dbt Labs.

It built me a working dbt project!

.
├── dbt_packages
├── dbt_project.yml
├── macros
│   └── generate_archive_urls.sql
├── models
│   ├── marts
│   │   ├── _marts__models.yml
│   │   ├── dim_date.sql
│   │   ├── dim_measures.sql
│   │   ├── dim_stations.sql
│   │   ├── fct_readings.sql
│   │   └── station_freshness.sql
│   └── staging
│       ├── _flood_monitoring__models.yml
│       ├── _flood_monitoring__sources.yml
│       ├── stg_flood_monitoring__measures.sql
│       ├── stg_flood_monitoring__readings_archive.sql
│       ├── stg_flood_monitoring__readings.sql
│       └── stg_flood_monitoring__stations.sql
├── profiles.yml
├── snapshots
│   └── snap_stations.sql
└── tests
    └── assert_recent_readings.sql

It compiles and it runs:

1 of 37 START sql table model main.dim_date .................................... [RUN]
2 of 37 START sql view model main.stg_flood_monitoring__measures ............... [RUN]
3 of 37 START sql view model main.stg_flood_monitoring__readings ............... [RUN]
4 of 37 START sql view model main.stg_flood_monitoring__stations ............... [RUN]
[…]
37 of 37 PASS unique_station_freshness_station_id .............................. [PASS in 0.01s]
36 of 37 PASS not_null_station_freshness_station_id ............................ [PASS in 0.01s]

Finished running 1 incremental model, 1 snapshot, 4 table models, 26 data tests, 2 unit tests, 3 view models in 0 hours 0 minutes and 0.61 seconds (0.61s).

and writes data successfully to DuckDB:

🟡◗ SELECT * from fct_readings LIMIT 5;
┌─────────────────────┬───────────────────────────────────────────┬────────┬──────────────┐
│      date_time      │                measure_id                 │ value  │ reading_date │
│      timestamp      │                  varchar                  │ double │     date     │
├─────────────────────┼───────────────────────────────────────────┼────────┼──────────────┤
│ 2026-03-09 12:00:00 │ 52119-level-stage-i-15_min-mASD           │  0.637 │ 2026-03-09   │
│ 2026-03-09 12:00:00 │ E72639-level-tidal_level-Mean-15_min-mAOD │  2.982 │ 2026-03-09   │
│ 2026-03-09 12:00:00 │ 730506-level-stage-i-15_min-m             │  1.319 │ 2026-03-09   │
│ 2026-03-09 12:00:00 │ 2095-level-stage-i-15_min-mASD            │  1.087 │ 2026-03-09   │
│ 2026-03-09 12:00:00 │ 3015TH-level-stage-i-15_min-mASD          │  0.178 │ 2026-03-09   │
└─────────────────────┴───────────────────────────────────────────┴────────┴──────────────┘

The prompt

Note:
This blog post is not about testing different permutations of a prompt. I’ve done that, and have written about it separately. This blog post is looking at an approximation of the best effort I could get from Claude. There’ll always be "yes, but did you try \$thing\" and I discuss that below.

Let’s look a bit more closely at the prompt that I gave Claude to understand what we’re giving it to work with.

I've explored and built pipelines for the UK Environment Agency flood monitoring API. Here's my analysis: 

- Data exploration: https://rmoff.net/2025/02/28/exploring-uk-environment-agency-data-in-duckdb-and-rill/ 
- Pipeline build: https://rmoff.net/2025/03/20/building-a-data-pipeline-with-duckdb/ 

Build a dbt project using DuckDB for this data using idiomatic patterns and good practices. Requirements: 

- Proper staging → dim/fact data model  
- Handle known data quality issues (see blog posts for details)  
- SCD type 2 snapshots for station metadata  
- Historical backfill from CSV archives (see https://environment.data.gov.uk/flood-monitoring/archive)  
- Documentation and tests  
- Source freshness checks  

Run `dbt build` to verify your work. If it fails, fix the errors and re-run until it passes.

Background context
Tell Claude where to go and find out the nitty-gritty detail about the data, its relationships, and its quirks
Show Claude what I built myself using just SQL and no dbt
Lay out the requirements.
Skeleton spec, as one might give to a junior engineer
"Secret sauce". This is where agentic coding kicks in. It will literally sit there and fix its own errors.

My yardstick for this prompt was:

What would a senior data engineer write if they were giving this to someone else to build?

That is, how much does AI help a data engineer today without them having to get into the guts of prompt engineering?

I also tested with a much more minimal prompt:

The UK Environment Agency publishes flood monitoring data,
see https://environment.data.gov.uk/flood-monitoring

Build an idiomatic dbt project following good practices using DuckDB that ingests this data and
models it for analytics.

Run the project and make sure that it works. If it fails, fix the errors and re-run until it passes.

This didn’t do very well beyond the absolute basics, and I’m not covering it in the rest of this article.

Agentic coding in action

At the heart of using Claude Code rather than a "chat bot" like ChatGPT is the idea of agentic AI. That is, it understands what it needs to do, and it has access to tools to achieve it. Tools can include looking up documentation on the web, amending files, invoking binaries, and so on.

Here’s an example of it in action, debugging a problem with the project as it goes. Remember, it’s doing everything I’m showing here on its own—no input, prompting, or guidance from me.

We join Claude part-way through its journey. It’s already built out the dbt models, and is testing it for the first time. On its own, it invokes dbt build:

Bash cd /workspace/flood_monitoring && dbt build --profiles-dir . 2>&1

x Exit code 2

Full error output

Exit code 2
Running with dbt=1.11.7
Registered adapter: duckdb=1.10.1
Unable to do partial parsing because saved manifest not found. Starting full parse.
[WARNING][MissingArgumentsPropertyInGenericTestDeprecation]: Deprecated
functionality
Found top-level arguments to test accepted_values defined on 'stg_measures' in
package 'flood_monitoring' (models/staging/_staging.yml). Arguments to generic
tests should be nested under the arguments property.
Encountered an error:
"'dateTime'"
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/dbt/cli/requires.py", line 182, in wrapper
result, success = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/cli/requires.py", line 128, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/cli/requires.py", line 276, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/cli/requires.py", line 321, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/cli/requires.py", line 391, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/cli/requires.py", line 368, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/cli/requires.py", line 407, in wrapper
setup_manifest(ctx, write=write, write_perf_info=write_perf_info)
File "/usr/local/lib/python3.11/dist-packages/dbt/cli/requires.py", line 434, in setup_manifest
ctx.obj["manifest"] = parse_manifest(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/parser/manifest.py", line 2257, in parse_manifest
manifest = ManifestLoader.get_full_manifest(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/parser/manifest.py", line 323, in get_full_manifest
manifest = loader.load()
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/parser/manifest.py", line 459, in load
patcher.construct_sources()
File "/usr/local/lib/python3.11/dist-packages/dbt/parser/sources.py", line 87, in construct_sources
parsed = self.parse_source(patched)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/parser/sources.py", line 204, in parse_source
parsed_source.relation_name = self._get_relation_name(parsed_source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/parser/sources.py", line 338, in _get_relation_name
return str(relation_cls.create_from(self.root_project, node))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/adapters/duckdb/relation.py", line 27, in create_from
return cls.create_from_source(quoting, relation_config, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/dbt/adapters/duckdb/relation.py", line 62, in create_from_source
ext_location = ext_location_template.format_map(source_config.as_dict())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: "'dateTime'"

[WARNING][DeprecationsSummary]: Deprecated functionality
Summary of encountered deprecations:

MissingArgumentsPropertyInGenericTestDeprecation: 6 occurrences To see all deprecation instances instead of just the first occurrence of each, run command again with the --show-all-deprecations flag. You may also need to run with --no-partial-parse as some deprecations are only encountered during parsing.

You can expand the Full error output log line above in full if you want, but the salient messages are these:

[WARNING][MissingArgumentsPropertyInGenericTestDeprecation]: Deprecated functionality
Found top-level arguments to test `accepted_values` defined on 'stg_measures' in
package 'flood_monitoring' (models/staging/_staging.yml). Arguments to generic
tests should be nested under the `arguments` property.

Encountered an error:
"'dateTime'"

Claude figures out it has two problems, as you can see from its internal 'thinking':

Two errors:
1. The `accepted_values` test has the wrong format - it needs to use `arguments` property
2. The `external_location` in sources.yml is being parsed as a format string, and the `\{dateTime\}` part is being interpreted as a placeholder. I need to escape the braces. Let me fix both issues. For the `external_location`, I need to escape the curly braces in the SQL. DuckDB uses `column=\{'dateTime': 'TIMESTAMPTZ', ...\}` which has `\{` and `\}`. The dbt-duckdb source parser is trying to format this string with the source config dict. I'll escape the braces with double braces so `\{\{` becomes `\{` and `\}\}` becomes `\}` in the format string. For the `accepted_values` test, I need to restructure it so the values and severity are nested under an `arguments` property, which is the format dbt 1.11+ expects. I see the issue now—the deprecation warning about test syntax is separate from the actual error. The real problem is the `external_location` format string with the curly braces, which is causing the `KeyError`. I should prioritize fixing that first since it's what's actually breaking the code, and I can address the test syntax update separately if needed.

It summarises this up into two distinct tasks:

Two issues to fix:
(1) curly braces in `external_location` need escaping for dbt's format string interpolation, and
(2) test syntax deprecation. Let me fix both:

and then sets about updating the external_location references like this:

Write /workspace/flood_monitoring/models/staging/_sources.yml

[…]
meta:
  # Curly braces in the DuckDB columns dict must be doubled to escape
  # dbt-duckdb's format_map() interpolation.
  external_location: >
    read_csv(
      'https://environment.data.gov.uk/flood-monitoring/archive/readings-2026-03-07.csv',
      columns={{'dateTime': 'TIMESTAMPTZ', 
                'measure': 'VARCHAR', 'value': 'VARCHAR'}}, 
      header=true,
      ignore_errors=true
    )

Double curly braces

← The file /workspace/flood_monitoring/models/staging/_sources.yml has been updated successfully.

After also updating the test syntax, it runs dbt build again:

Bash cd /workspace/flood_monitoring && dbt build --profiles-dir . 2>&1

x Exit code 2

Exit code 2
17:34:54  Running with dbt=1.11.7
17:34:54  Registered adapter: duckdb=1.10.1
17:34:54  Unable to do partial parsing because saved manifest not found. Starting full parse.
17:34:55  Encountered an error:
Compilation Error
  Could not render read_csv(
    'https://environment.data.gov.uk/flood-monitoring/archive/readings-2026-03-07.csv',
    columns={{'dateTime': 'TIMESTAMPTZ', 'measure': 'VARCHAR', 'value': 'VARCHAR'}},
    header=true,
    ignore_errors=true
  ): expected token 'end of print statement', got ':'
    line 3
      columns={{'dateTime': 'TIMESTAMPTZ', 'measure': 'VARCHAR', 'value': 'VARCHAR'}},

This time it’s a different problem, which it identifies and figures out how to work around:

The external_location goes through Jinja2 rendering - can’t use {{}} for literal braces. I’ll use read_csv_auto instead which needs no column type dict:

Write /workspace/flood_monitoring/models/staging/_sources.yml

Bash cd /workspace/flood_monitoring && dbt build --profiles-dir . 2>&1

This time the build works, failing only on data tests:

[…]
Done. PASS=27 WARN=2 ERROR=2 SKIP=17 NO-OP=0 TOTAL=48

Good progress! The build almost works. Two real data quality issues:
- 631 stations lack lat/long coordinates in the API (genuine EA data gap)
- Need to downgrade those tests to `warn` severity so they don't cascade-skip downstream models

Check this out for what it does next—fires up DuckDB and starts checking the data:

python3 << 'EOF'
import duckdb
conn = duckdb.connect()
result = conn.execute("""
SELECT count(*) as missing_coords_count
FROM (
  SELECT UNNEST(items) as item
  FROM read_json_auto('https://environment.data.gov.uk/flood-monitoring/id/stations.json?_limit=10000')
) WHERE item.lat IS NULL
""").fetchone()
print('Missing lat/long:', result)
EOF

631 stations genuinely have no coordinates in the EA API. Downgrade those tests to warn:

Edit /workspace/flood_monitoring/models/staging/_staging.yml

diff

old

name: latitude description: WGS-84 latitude in decimal degrees. tests:
- not_null
name: longitude description: WGS-84 longitude in decimal degrees. tests:
- not_null

new

name: latitude description: > WGS-84 latitude in decimal degrees. Known data quality issue: ~631 stations in the EA API have no coordinates (newly-commissioned or administrative stations). Test is warn-only for this reason. tests:
- not_null: config: severity: warn
name: longitude description: WGS-84 longitude in decimal degrees. See latitude note. tests:
- not_null: config: severity: warn

and so on and so on, until it gets to:

dbt build passes with 0 errors. All 38 tests pass; 10 warnings document known EA data quality issues.

The above autonomous debugging session is just one example of the numerous ways in which Claude Code makes things easier for a data engineer.

Here’s another narrated session log from a different run using the same prompt, in which Claude realises the data has a bunch of quirks, enumerates them, and codes for them.

Truly invaluable.

Okay…but is it any good?

As the content of many LinkedIn posts and comments is testament to: infinite AI monkeys with infinite tokens will crap out as much material as you want—but most of it is garbage. Is that what Claude is doing here? Is it garbage?

Actually, the dbt project that Claude built is pretty good.

What I love about dbt is that it makes it easy to adhere to a ton of good practices that have always been true when working with data including staging layers, slowly-changing dimensions (SCD), and dimensional modelling as well as "newer" (but equally important) concepts such as documentation (gasp!), testing (imagine!) and a smooth integration with source control and CI/CD (how modern!).

How much of these good features did Claude use, and use successfully?

Not bad…

✅ Plausible data model

✅ Correct key relationships enforced with dbt constraints

  config:
    contract:
      enforced: true
  columns:
    - name: station_id
      data_type: varchar
      description: "Unique station identifier (e.g. '1029TH')"
      tests:
        - not_null
        - unique

One point to note is that it didn’t include tests to check the keys in the loaded data. This is arguably 'belt and braces' anyway.

✅ Incremental fact table load

  config(
      materialized='incremental',
      unique_key=['date_time', 'measure_id'],
  )

✅ Handle messy source data (e.g. multiple values where only one expected)

  -- Value cleaning: handle pipe-delimited values (e.g. "0.770|0.688")
  -- by taking the first value, then cast to double
  try_cast(
      case
          when value like '%|%' then split_part(value, '|', 1)
          else value
      end as double
  )                                               as reading_value,

✅ Handles Slowly Changing Dimensions (SCD)

  from {{ ref('snap_stations') }}
  where dbt_valid_to is null

Note:
For some reason Claude only implemented this for the stations dimension, not measures. Turns out, I literally told it to do this in the prompt! (SCD type 2 snapshots for station metadata). So it stuck to that literally, omitting measures. But it’d have been nice for it to perhaps challenge me on that. Measure definitions shouldn’t change…but they might?

✅ Documentation

✅ Tests

  - name: status
    tests:
      - not_null
      - accepted_values:
          arguments:
            values: ['Active', 'Closed', 'Suspended', 'ukcmf', 'Unknown']

  - name: flood_monitoring_api
    freshness:
      warn_after:
        count: 1
        period: hour
      error_after:
        count: 6
        period: hour
    loaded_at_field: "loaded_at"

✅ Backfill

  {% macro generate_archive_urls(start_date, end_date) %}
  {%- set start = modules.datetime.datetime.strptime(start_date, '%Y-%m-%d') -%}
  {%- set end_dt = modules.datetime.datetime.strptime(end_date, '%Y-%m-%d') -%}
  {%- set day_count = (end_dt - start).days -%}
  {%- set ns = namespace(urls=[]) -%}
  {%- for i in range(day_count + 1) -%}
    {%- set d = start + modules.datetime.timedelta(days=i) -%}
    {%- do ns.urls.append("'https://environment.data.gov.uk/flood-monitoring/archive/readings-" ~ d.strftime('%Y-%m-%d') ~ ".csv'") -%}
  {%- endfor -%}
  [{{ ns.urls | join(', ') }}]
  {%- endmacro %}

  select
      cast("dateTime" as timestamp)                         as date_time,
      regexp_replace(coalesce(measure, ''), '.+/', '')      as measure_id,
      try_cast(split_part(cast(value as varchar), '|', 1) as double)  as value
  from read_csv(
      {{ generate_archive_urls(start_date, end_date) }}

General good practice and nice surprises

✅ Doesn’t use SELECT * but instead hardcodes column names. Generally a better pattern for creating less brittle pipelines.
✅ Parses out the station status field cleanly

regexp_replace(coalesce(status, ''), '.+/', '')  as status

Rather than storing the source which looks like this

http://environment.data.gov.uk/flood-monitoring/def/core/statusActive

✅ More elegant date logic:

extract(isodow from date_day) in (6, 7)
  as is_weekend

CAST(CASE WHEN dayofweek(range) IN (0,6) THEN 1 ELSE 0 END AS BOOLEAN)
  AS date_is_weekend

So, pretty solid work, if the alternative is starting from a blank slate of a dbt project. What about the downsides?

…but not that good

All you Claude Code fans out there, hold your horses on the "yes, but…" until the "yes, but" section below :p

In the version that I built by hand, I pull the raw API data into a DuckDB table using a Jinja macro. Claude approached it differently, writing a Python script, load_api_data.py. Arguably more elegant, but I have three issues with it, as follows.

The first of these is personal preference. I’m a SQL guy, and I’d rather not be on the hook to support a pipeline that leans on Python. After all, it’s one thing to generate code with AI, but ultimately someone’s left holding the support baby.
The next two issues are the code itself. There’s one silly flaw in it, and an insidiously bad one.
- The silly flaw is easily spotted and easily fixed: it only does a CREATE TABLE and so falls over if the table exists already (like, the second time you run it). Easily spotted (the job falls over) and easily fixed (CREATE OR REPLACE TABLE).
- The really bad problem is this, in Claude’s own mea culpa words:
load_api_data.py fetches stations with ?_limit=2000, but the API itself has a maximum response size of 2000 items (it silently caps at that). The actual total number of stations is ~5,458, so the script only gets 1,493 rows back (fewer than the limit because the API returns a paginated subset — probably the first page).

Let that sink in. We have a huge hole in our data, that’s not evident until we wonder why a ton of our facts are without corresponding dimension entries.

Wrong is worse than absent because you can’t trust it.:
If Claude just doesn’t build a feature, that’s ok because I can add it (or tell it to). But if it confidently builds it and it turns out to be wrong, that’s a trust issue.

Yes, but Robin, you could <do this thing>! → See yes, but below

Other things that look off from just eyeballing the code include:

dim_stations drops several columns (stageScale, datumOffset, gridReference, downstageScale). Two of these arguably extend the data model beyond what I’ve manually investigated myself before (stageScale/downstageScale) but two definitely hold relevant reference data (datumOffset, gridReference) which is getting silently dropped.
dim_measures drops unit. Arguably a decent optimisation given the repetition between unit and the human-friendly unitName:

  🟡◗ select unit, unitName,label, parameter,parameterName,valueType from measures using sample 5 rows;

  ┌──────────────────────────────────────┬──────────┬──────────────────────────────────────────────┬───────────┬───────────────┬───────────────┐
  │                 unit                 │ unitName │                    label                     │ parameter │ parameterName │   valueType   │
  │               varchar                │ varchar  │                   varchar                    │  varchar  │    varchar    │    varchar    │
  ├──────────────────────────────────────┼──────────┼──────────────────────────────────────────────┼───────────┼───────────────┼───────────────┤
  │ http://qudt.org/1.1/vocab/unit#Meter │ m        │ LAVERSTOCK - level-stage-i-15_min-m          │ level     │ Water Level   │ instantaneous │
  │ http://qudt.org/1.1/vocab/unit#Meter │ m        │ KNETTISHALL GS GSM - level-stage-i-15_min-m  │ level     │ Water Level   │ instantaneous │
  │ http://qudt.org/1.1/vocab/unit#Meter │ mASD     │ Keadby TL B - level-tidal_level-i-5_min-mASD │ level     │ Water Level   │ instantaneous │
  │ http://qudt.org/1.1/vocab/unit#Meter │ m        │ Riding Mill - level-stage-i-15_min-m         │ level     │ Water Level   │ instantaneous │
  │ http://qudt.org/1.1/vocab/unit#Meter │ mASD     │ Clomoney Way LVL - level-stage-i-15_min-mASD │ level     │ Water Level   │ instantaneous │
  └──────────────────────────────────────┴──────────┴──────────────────────────────────────────────┴───────────┴───────────────┴───────────────┘

But that’s not the point is it; silently dropping data fields is not what I want my robot to do.

fct_readings duplicates the date_time field:

  select
      date_time,
      measure_id,
      value,
      cast(date_time as date) as reading_date 
  from deduped

y, tho?
- Claude builds the SCD (nice!) but check_cols only lists some of the values that might change:

  check_cols=[
      'station_name',
      'town',
      'river_name',
      'catchment_name',
      'latitude',
      'longitude',
      'status',
  ],

So if latitude/longitude are in there, why not northing/easting? As with the measures decisions above, these are real data issues that a human should be involved in reviewing, or the AI surfacing for review.

Yes, but…

All of the mistakes and errors that Claude made and which I discuss above, I could, of course, get Claude to fix. This would be an iterative process: Claude builds → I review → Claude fixes, etc.

But…that isn’t what I wanted to determine in this article. It changes Claude from a means of production into a means of assistance (which is, BTW, what I conclude in this post that its most effective role is today). It’s the difference between giving a junior (or senior!) colleague a piece of work to do and being confident that it will be ready for testing, versus knowing that it will need a lot of checking and reworking.

Another option would be to build the prompt defensively up-front (encode everything I’ve seen it do here into a better prompt for next time, make no mistakes, etc). The point remains, that I would have had to do that. The prompt I already gave it was designed to give it enough hints, without having to hold its hand through every step.

Could I create an improved version, based on what it’s learnt? Sure. But that’s not having Claude build it for me is it. It’s building it with me. Which is my whole argument here.

Tip:
Claude Code is an amazing productivity companion.

Warning:
Do not, if you value your job, use it to one-shot a dbt project!

Prompt tweaking is a mug’s game

LLMs are non-deterministic, so even given the same prompt it might behave differently next time, let alone when a newer model comes along. Then you end up building a spaghetti bowl of sticking plasters ("don’t do this", "remember to do this"), and it’s still not necessarily going to do it right each time—as the technology currently stands.

I’m reminded of this example of a ChatGPT prompt I earnestly bookmarked in August 2023. It was the magic incantation! It was amazing! It was cargo culting.

Relying on prompting alone is cute for tricks, but it’s not a viable strategy for reliable hands-off dbt code generation (or even hands-on, but trusted, for that matter).

How did you test this? Did you try technique `$wibble`? What about model `$foo`?

The best thing about using AI agents to make you more productive is that they make you more productive at the thing you’re building.

The worst thing about using AI agents to make you more productive is that they make you more productive at any random stupid idea that pops into your lizard brain.

Thus, in figuring out what Claude Code could do with dbt, I also stumbled into the worlds of evals, LLM-as-judge, and a whole bunch more. Oh, and of course, having Claude Code build a bespoke dashboard to track all of my results :)

I’ve written all about this in a separate blog post, because I learnt a lot of stuff—some of it useful too.

For now, the headline:

Tip:
The prompt and the skills matter more than the model. Sonnet 4.5 can produce respectable results given the necessary context. Opus 4.6 isn’t a silver bullet.

A fantastic productivity booster for data engineers

What does even replacing a data engineer with AI mean?

To truly suggest that AI can replace a data engineer you’d be talking about a tool that a business user could tell to build a pipeline and have it do it with sufficient quality and robustness that it could be thrown over the wall to a general IT ops team to run.

Perhaps in the same way that business users these days eat Excel sheets for breakfast whereas 40 years ago they’d have a specialist in IT to do that for them.

We’re not there with AI yet.

`DE + AI > DE`

What agentic coding tools such as Claude Code are doing is making data engineers vastly more productive. We saw above that it’s not going to build the whole pipeline in one go, hands-off. And in fact, there are chances it’s going to be a drag on productivity to do so because you won’t have built up the mental map of what’s been built to then be able to verify and troubleshoot it. But Claude Code excels at specific tasks, and iteration. It can troubleshoot a failing build, as well as figure out nuances in the data.

Whilst it does need coaching through some tasks, and reprimanding and correcting, and whilst it does make mistakes…so do humans! My experience is that it is so shit-hot—and fast as f…errari—the vast majority of the time that the nett gain over just doing it yourself manually is still great.

My mental model for not using Claude Code is currently something like this:

Tip:

An accountant choosing to do their sums by hand, instead of use a calculator

An SRE manually logging into AWS to provision a dozen EC2 machines and SSHing into each one individually to install software, instead of using Terraform

A programmer using a magnetized needle and a steady hand to edit a file, instead of a text editor ;)

Deterministic vs non-deterministic is irrelevant here; my point is that AI is a tool to enable you to get something done more easily than you used to be able to.

You don’t have to use agentic coding tools…but you’re going to get left behind and have to catch up at some point.

Note:
Oh…and if you honestly think AI is just ChatGPT writing haikus about Linux and other chatbot-esque simplicity…you’re maybe missing the point about AI.