The next box appeared on its own.
A transformation node. A suggested join. Half the logic was already filled in before anyone typed anything.
Nishant was watching a demo of Databricks Lakeflow Designer. A few nodes, a few connections, one pipeline running. No notebook. No Spark code. No moment where the screen said, "Now hand this to engineering."
In the comments, people were typing the same thing. "This replaces half my work."
He did not feel impressed. He felt something quieter and harder to name.
What the friction used to do
Before tools like this, building a data pipeline had a specific shape.
Raise a ticket. Wait for someone with bandwidth. Write the code. Debug when a dependency broke two weeks later. That sequence had inconvenience built into it, but the inconvenience was also doing something else.
It was deciding who could build.
You needed to know Spark. You needed to understand the infrastructure underneath. You needed to be comfortable in a notebook, reading errors, tracing what broke and where. That knowledge was the entry point. Without it, you waited for someone who had it.
Lakeflow Designer removes that entry point. UI plus prompt plus system suggestion is enough to get something running. Not a rough draft to hand off. Not a prototype to validate the concept. Something that behaves like production from the first connection.
How the shift moves without announcing itself
The steps come off one at a time, which is why nobody calls a meeting about it.
First you stop writing boilerplate. That feels like saved time. Then infrastructure management disappears from the job. That feels like an upgrade. Then the code itself becomes optional. That still feels like progress, until the question arrives: what exactly is the role now?
Each removal looks like a feature. Taken together, they are rearranging who the work belongs to.
The difficulty does not disappear. It relocates.
Writing the pipeline was hard in one direction. Now the hard part is somewhere else: deciding what should exist, trusting what the system generates, and catching the assumptions the interface buries inside its suggestions. A clean UI hides choices. Someone has to know which choices were made and whether they were right.
That is not easier than writing Spark. It is a different kind of hard, less legible, less teachable, and much easier to miss when something goes wrong downstream.
What Nishant actually watched
He thought he was watching a product demo.
What the demo was showing, without saying it directly, was a boundary moving.
Work that required an engineer last year requires an analyst today. Not because analysts got more technical. Because the technical layer is compressed into a surface that does not look technical anymore.
That changes who gets to build. It changes who gets credit for building. It changes where the judgment has to live, and who is held responsible when the pipeline produces something wrong.
None of this was in the demo. The demo showed boxes connecting cleanly and a pipeline running on the first try.
But control was shifting in the background, from one type of professional to another, without a handover, without a conversation, without anyone in the room saying what was actually happening.
It already felt normal. That was the part that stayed with him.
One Question Before You Go
If the system builds the pipeline for you, where does your responsibility actually begin?
And more importantly, would you know if the pipeline is correct, or just working?
I have been thinking about this shift, and the answer is not obvious. I would genuinely like to hear how you see it.
I will go first in the comments.
Your turn. 👇
Top comments (0)