DEV Community: tpf

Data fails first

Mariano Barcia — Thu, 21 May 2026 14:56:57 +0000

The strange thing was that the first real scaling problems had nothing to do with scale. Or at least, scale as in "infrastructure scale".

The application was still relatively small. Traffic was growing, yes, but nothing dramatic. CPUs were healthy, latency was acceptable, deployments remained pleasantly uneventful. From the outside, the system still looked exactly like what it technically was: a single Quarkus application running as a monolith, with pipeline steps invoking one another directly inside the same process.

And yet developers had started speaking about it differently.

Not formally. Nobody walked into a meeting and announced that “the architecture had changed.” It emerged indirectly, through the kind of comments engineers make when systems begin slipping away from them:

“Be careful changing that field.”
“I think another step depends on this.”
“We should probably keep the old format for compatibility.”
“I’m not entirely sure who consumes this anymore.”

The incidents themselves were subtle enough to avoid triggering panic. Customer support found two apparently identical shipments taking different routing paths. Finance noticed that premium customers occasionally missed SLA prioritization. A new carrier integration introduced a field named region, while another team added shipping_region, and for several months both quietly coexisted because somewhere in the middle of the pipeline a mapper handled the ambiguity without anybody realizing it.

Nothing crashed.

That was precisely the problem.

The failures were no longer technical failures. They were semantic failures. The system continued running while different parts of the application slowly stopped agreeing on what the data actually meant.

Marta recognized the smell immediately because she had seen it before in event-driven and serverless systems: payloads beginning life as wonderfully flexible JSON blobs and gradually hardening into undocumented distributed contracts. Except this time, it was happening entirely inside a monolith.

That realization bothered her more than any infrastructure issue they had faced so far because it exposed something most engineers rarely think about when they talk about distributed systems.

When people hear “distributed systems,” they usually imagine the visible mechanics:

remote calls
retries
eventual consistency
latency
concurrency
partitions

But those are often second-order effects. The deeper problem appears much earlier, when independently evolving parts of a system begin sharing meaning imperfectly.

Two teams interpreting the same field differently. A payload surviving longer than the assumptions under which it was created. Producers and consumers changing independently without realizing the semantic contract between them has drifted.

That is already a distributed systems problem, even if everything still runs inside one JVM.

And that was the moment Marta realized something important:

Distribution is not a deployment problem. It starts as a data problem.

The architecture diagrams had not changed yet, but the organizational reality already had. Different teams owned different stages of the pipeline. External integrations introduced their own semantics. Some transformations were deterministic and easy to reason about; others depended on databases, carrier APIs, fraud services, and infrastructure outside the team’s control. The application might still have been physically centralized, but conceptually it had already become a collection of independently evolving boundaries.

That was where TPF started becoming much more than a convenient orchestration framework.

The team was not using it as a generic workflow interpreter where loosely defined tasks execute dynamically at runtime. What TPF increasingly provided was something much closer to a typed application pipeline compiler: a system capable of understanding the boundaries between stages well enough to generate contracts, validate transformations, wire transports, expose endpoints, and fail builds when parts of the pipeline drifted out of alignment.

The distinction mattered enormously.

The pipeline steps themselves remained ordinary application code. Developers wrote Quarkus services, plain Java methods, some reactive Mutiny handlers where concurrency justified the complexity, some blocking implementations where simplicity mattered more than theoretical throughput. Nobody had to abandon the normal ergonomics of application development to “enter the workflow engine.”

But the boundaries between those steps became explicit.

Instead of passing anonymous maps through the system and hoping conventions would survive organizational growth, stages now declared typed inputs and outputs, mapper rules, cardinality expectations, operator contracts, and transport compatibility in ways the compiler could actually verify.

The effect on development was immediate and surprisingly human. Refactoring became less frightening. Teams stopped relying on tribal knowledge to understand dependencies. A mapper mismatch became a compilation error instead of a production incident discovered three weeks later through telemetry dashboards and apologetic Slack messages.

Peace of mind was alas, restored, once all the data contracts between steps were formalised, and the architecture stopped evolving accidentally.

The Lift and Shift

Mariano Barcia — Thu, 07 May 2026 18:12:31 +0000

This is the continuation of "The script that refused to stay small".

The conversation had already been happening for months: three data centers, rising costs, aging hardware, and a growing sense that the whole setup was one incident away from becoming a liability.

The decision came quickly after the "kalima" event: everything had to move.

What followed was not elegant, as the platform team scrambled to pull off a lift & shift to the cloud:

hypervisor VMs mapped 1:1 into cloud instances
network rules replicated (and occasionally guessed)
storage reattached, sometimes awkwardly
timelines optimistic, then revised, then ignored

Most applications... just came along for the ride whilst Marta watched all of this unfold.

Her service was, technically, just another VM in the inventory. It could have been copied over, like everything else, but the lift and shift was rushed and was accumulating tech debt almost by definition. So she made a different call.

Instead of following the 1:1 VM migration path, she opted into an early container platform the company had been experimenting with—something not yet standard, slightly under-documented, but good enough.

Not because she was trying to be innovative, but because her system made it easy.

Containerizing the application turned out to be almost trivial.

Quarkus already produced a lean runtime, with the pipeline living inside a single process with well-defined boundaries. There were no hidden dependencies on the host, no fragile startup scripts, no tight coupling to the underlying machine.

Marta wrote a Dockerfile, wired a couple of environment variables, and that was… mostly it.

The hardest part wasn’t the application, but everything around it:

getting the right network access
aligning with security policies
making sure it could talk to the same external systems as before

So while most systems were being translated from one VM to another, Marta’s was quietly being repackaged.

And once it ran, something became clear:

It was still a monolith
Steps still called each other directly, in-process
The pipeline still lived entirely inside a single runtime

Only the execution environment had changed.

That was the first quiet lesson The Pipeline Framework taught the team: you can move where something runs without changing how it behaves.

A few weeks later, someone asked Marta how long her migration had taken; she hesitated (because the honest answer sounded wrong: “About a day. Maybe two, if you count the networking issues.”

Photo credit: Javier Mediavilla E, CC BY-SA 2.5 https://creativecommons.org/licenses/by-sa/2.5, via Wikimedia Commons

The Script That Refused to Stay Small

Mariano Barcia — Thu, 09 Apr 2026 21:54:41 +0000

It started, as these things often do, with a single process running on a single machine in a server room nobody liked visiting. The system took in shipment requests, enriched them with a few heuristics, and spat out routing hints. Nothing fancy, just enough logic to save operations teams a few hours a day, and Marta built it on her own as a side project.

Marta wasn’t new to Java, she had spent a few months working in a microservices project in Spring and liked the serverless functional-style thinking: small transformations composed together. She had enough scar-tissue by now having experienced the inevitable issues with Spring and wanted to experiment with Quarkus anyway.

She picked TPF because, on top of featuring a canvas UI to design the pipeline, she got the complete app's scaffolding as a pure Quarkus project. She could keep developing on her IDE as usual, only with the addition of the Quarkus environment plugin. And, she got to use Dev Services which she also wanted to try.

Her choices for the script were deliberately boring:

a JVM runtime (a single process) with
no high availability
a traditional direct method invocation between the orchestrator and the pipeline steps
no DB, just basic HTML responses to human queries

Each step in the pipeline was just a function. Input came in as loosely structured key/value data. No rigid schema, no upfront modeling, no SQL persistence.

She didn’t yet have to think about

execution tracking or audits
fan-out/fan-in logic
retry semantics
error handling
availability
high throughput
latency
integrations with 3rd parties
monitoring
security

And because she had started using AI-assisted coding tools, even the tests came almost for free, as TPF’s step isolation made it trivial to generate meaningful unit tests for each transformation.
It was fast. It was understandable. It worked and, crucially, it already had shape—even if nobody called it that yet.

But during one summer's day, a sandstorm from the Sahara impacted Valencia and caused temperatures to rise (a phenomenon called kalima). The A/C unit in the server room failed and, by the next morning morning, alerts were firing, CPUs were throttling, and someone mentioned, only half jokingly, the possibility of actual fire.

Architectural mobility for stronger software

Mariano Barcia — Tue, 07 Apr 2026 19:37:36 +0000

There’s a concept in sports science that’s easy to overlook because it sounds so basic: “mobility”. Mobility is defined as “the ability of a joint to move actively through its full, functional range of motion with control, stability, and strength“.

If you look at a simplified “athlete performance” pyramid, the foundation isn’t power or strength; it’s mobility. Everything else builds on top of that, but we spend most of our time discussing the upper layers and very little attention is paid to what sits at the foundation of the pyramid. Take away mobility, and the strength will falter.

       / - \
      /     \
     / Power \    Athlete Performance Pyramid
    /_________\
   / Strength  \  
  /_____________\
 /   Mobility    \ 
/_________________\

Now, the reasons why mobility is so important are many and complex, but basically have to do with how the nervous system is able to recruit muscle fibres, whilst at the same time staying healthy (ie. by doing “reciprocal inhibition”).

Putting aside the inherent complexity of the muscles in the human body, I think software systems behave in a similar way, and we spend most of our time discussing the upper layers:

scalability
throughput

whilst paying very little attention to what sits underneath all of that.

Most software systems miss something fundamental: they cannot change shape without breaking themselves.

You move from containers to functions → everything rewrites
You switch from REST to gRPC → contracts ripple through the system
You change how remote work is executed → business logic gets entangled

The system might be “strong” in its current form, but it lacks mobility. It’s rarely stated explicitly, and runtime, transport, and protocol are treated as the same decision.

You deploy to containers → you expose REST → you serialise JSON.
Or you go serverless → you wire HTTP → you pass payloads around.

Nothing looks wrong and in fact, it feels coherent, but still:

your execution model has dictated your transport
your transport has dictated your protocol
and all three together have dictated your system shape

This affects “architectural mobility” because

You cannot change where things run without rewriting how they communicate
You cannot change how they communicate without rewriting what they mean
You cannot evolve protocols without locking yourself into a runtime

That mobility is what allows your business logic to evolve regardless of any external infrastructure decision, and will allow you to make these deliberate choices:

do your workloads belong in containers?
do your workloads belong in serverless functions?
do they benefit from gRPC, or REST?

In The Pipeline Framework, architectural mobility is enabled by decoupling the transport from the execution platform from the runtime topology, like so:

                     Transport
               (LOCAL / REST / gRPC / Proto-HTTP)
                           ↑
                           │
                           │
                           │
                           │
                           │
                           ●───────────────→ Execution Platform
                          /                (COMPUTE / FUNCTION)
                         /
                        /
                       /
                      ↓
             Runtime Topology
  (Monolith / Pipeline / Modular)

Photo credit: https://unsplash.com/es/@weareambitious