DEV Community: Dhruvi

What Developers Underestimate About Long-Running Workflows

Dhruvi — Wed, 24 Jun 2026 12:52:47 +0000

Long-running workflows look simple when you first build them.

Something happens.

A few systems exchange data.

Everything completes.

Done.

At least that's the expectation.

Reality is very different.

The biggest thing I underestimated was time.

Not execution time.

Elapsed time.

Because once workflows start running for hours, days, or continuously, strange things start happening.

APIs become temporarily unavailable
Data changes halfway through the process
Retries arrive much later than expected
Someone manually updates a record
Another system processes things in a different order

Nothing is broken.

But everything is slightly different from when the workflow started.

Early on, I assumed workflows were transactions.

Start.

Execute.

Finish.

Now I think of them as conversations between systems.

And conversations can get interrupted.

Another thing I underestimated:

State changes.

You might start processing an order that is "pending".

Ten minutes later, another system marks it as "cancelled".

An hour later, a retry comes in from an earlier step.

If your workflow only thinks about data, weird things happen.

Because the world has changed while the process was still running.

Long-running workflows also expose assumptions you didn't know you made.

Like:

this API will always respond quickly
data will arrive in order
users won't modify records manually
retries will happen immediately

Those assumptions survive in testing.

Production removes them quickly.

One thing that changed how I build these systems:

I stopped asking:

"Will this workflow finish?"

And started asking:

"What state will the world be in when it finishes?"

Because those are two very different questions.

Most problems in long-running systems aren't caused by one big failure.

They're caused by lots of small changes happening while the workflow is still alive.

And if you don't account for that, eventually the workflow finishes successfully and still produces the wrong outcome.

This is something we think about constantly at BrainPack while operating workflows that span multiple systems and AI layers. Long-running processes are less about moving data and more about managing changing state over time.

Why Retries Are More Dangerous Than Failures in Production Systems

Dhruvi — Fri, 19 Jun 2026 12:47:54 +0000

Failures are obvious.

Retries are sneaky.

When something fails, everyone notices.

An alert goes off.
A request errors out.
Someone starts investigating.

Retries are different.

They look harmless.

Most of the time, they save the system.

But sometimes, retries create bigger problems than the original failure.

Imagine an API call times out.

No problem.

The system retries.

But what if the first request actually succeeded and only the response was lost?

Now the retry creates:

duplicate orders
repeated emails
inconsistent records
workflows running twice

The failure happened once.

The retry multiplied it.

Another thing I've seen:

One slow dependency causes requests to pile up.

Retries start firing.

Those retries create even more traffic.

Which slows things down further.

Which triggers even more retries.

Suddenly, the system is spending more effort retrying than doing useful work.

Retries also hide problems.

A temporary issue gets retried five times and eventually succeeds.

Everything looks normal.

Meanwhile:

latency increases
queues grow
users experience delays Nothing technically failed.

But the system is getting less healthy.

What changed for me is that I stopped treating retries as free.

Every retry has a cost.

It consumes resources.

It increases load.

And if actions aren't designed carefully, retries can repeat side effects that should only happen once.

Now when I build something, I don't ask:

"What happens if this fails?"

I ask:

"What happens if this runs again?"

Because in production, things almost always run again.

And if the answer is "bad things happen," the retry mechanism isn't helping.

It's making things worse.

Failures are part of every system.

Retries are too.

The difference is that failures usually happen once.

Retries can turn one problem into hundreds if you don't design for them.

This is something we think about constantly at BrainPack when operating long-running workflows across multiple systems. AI and automation layers make retries even more common, which means making actions safe to repeat becomes just as important as handling failures themselves.

An Experience From the Deployment Process Worth Sharing

Dhruvi — Mon, 15 Jun 2026 13:23:49 +0000

One deployment taught me a lesson I keep coming back to.

Nothing was technically wrong.

The code worked.

Testing passed.

The deployment completed successfully.

And yet, a few hours later, things started behaving strangely.

Some workflows were running slower than expected.

A few records weren't updating.

Nothing was completely broken.

Just enough to make people question whether the system was working correctly.

Those are often the hardest situations.

Because there isn't a clear error pointing you in the right direction.

After digging through the flow, we found the issue.

The new deployment introduced a change that increased the number of requests between systems.

Not enough to cause failures.

But enough to create a small backlog.

That backlog slowly grew throughout the day.

The deployment itself was fine.

The operational impact wasn't.

What stuck with me was this:

We had tested functionality.

We hadn't tested behavior under real operating conditions.

Those are very different things.

Since then, I pay much more attention to questions like:

What will this change do to traffic patterns?
Will it create more retries?
Will it increase queue sizes?
How will it behave after running for several hours?

Those questions rarely come up during feature development.

But they matter a lot after deployment.

One thing I've noticed about production systems:

Problems often don't appear immediately.

They accumulate.

A small delay becomes a backlog.

A backlog becomes slower processing.

Slower processing creates more retries.

And suddenly you're debugging something that started hours earlier.

The experience changed how I think about deployments.

A successful deployment is not when the code reaches production.

A successful deployment is when the system continues behaving predictably after the change.

This is something we think about constantly at BrainPack while operating systems that run continuously across enterprise environments. The deployment is usually the easy part. Understanding how a change affects the system over the next few hours and days is where the real work begins.

Something I Learned Recently That Changed How I Approach a Problem

Dhruvi — Thu, 11 Jun 2026 12:49:58 +0000

One thing I learned recently:

Most production problems are not technical problems.

They're visibility problems.

For a long time, my instinct was:

Something breaks → find the bug → fix the code.

Seems reasonable.

But the more time I spend operating systems, the more I notice that many issues happen because we can't clearly see what's happening.

A workflow gets stuck.

Data stops syncing.

An automation behaves unexpectedly.

The first question isn't:

"Why did it fail?"

It's:

"Can we actually see where it failed?"

I've worked on issues where the fix took 15 minutes.

Finding the issue took several hours.

Not because the bug was complicated.

Because there wasn't enough visibility into the system.

That changed how I approach new work.

Now, before I think about features, I think about observability.

Questions like:

How will we know this is broken?
How will we know it's slow?
How will we know it's stuck?
How will we investigate it six months from now?

Those questions often end up being more important than the implementation itself.

The interesting thing is that adding visibility rarely feels urgent when you're building.

Everything is working.

Everything looks fine.

Until the day it isn't.

And that's usually when you realize how valuable those extra logs, status checks, and monitoring points actually are.

One pattern I've noticed:

Teams often spend more time locating problems than solving them.

So improving visibility doesn't just reduce downtime.

It makes engineering work faster.

Now when I build something, I try to leave behind enough information that future me doesn't have to guess what happened.

That's probably one of the highest-return investments I've found in software engineering.

This lesson comes up constantly at BrainPack while operating systems that run continuously across multiple platforms and workflows. AI systems can only be as reliable as your ability to understand what the underlying infrastructure is doing at any given moment.

What Documentation Looks Like in a Permanently Operated System

Dhruvi — Thu, 04 Jun 2026 12:39:15 +0000

I used to think documentation was mostly for onboarding.

A way to help new developers understand the system.

That's part of it.

But when you're operating systems continuously, documentation becomes something else entirely.

It's operational infrastructure.

The biggest misconception about documentation is that it's about explaining how things were built.

Most of the time, that's not what people need.

What they need is:

how does this actually work today?
what happens if it fails?
who depends on it?
what should happen next?

One thing I learned pretty quickly:

Nobody reads long documentation during an incident.

If something breaks, people need answers fast.

So the most useful documentation is usually the simplest.

Things like:

workflow diagrams
system dependencies
retry behavior
recovery steps
known failure points

Another thing that changes in long-running systems:

Documentation can't be static.

The system evolves.

Integrations change.

Business processes change.

Automations get added.

If documentation doesn't evolve too, it slowly becomes misleading.

And outdated documentation is often worse than no documentation at all.

The documentation I use most is rarely technical.

It's operational.

Questions like:

Why does this process exist?

What happens if this service is unavailable?

Which systems depend on this workflow?

Those answers save more time than implementation details.

One thing I appreciate now:

Good documentation reduces dependency on specific people.

Without it, knowledge gets trapped.

One person knows how something works.

One person knows how to recover it.

One person knows why it was built that way.

That's a risk.

In systems that run continuously, documentation is less about explaining code and more about preserving operational knowledge.

Because eventually, everyone forgets why a decision was made.

The documentation is what remains.

At BrainPack, a lot of the systems we operate involve multiple integrations, workflows, and AI layers running together. Good documentation helps turn individual knowledge into infrastructure that the entire team can rely on over time.I used to think documentation was mostly for onboarding.

What Building Software That Runs 24/7 Actually Means Day to Day

Dhruvi — Wed, 03 Jun 2026 13:04:16 +0000

When people hear that a system runs 24/7, they usually think about uptime.

Servers running.

Services responding.

No outages.

But day to day, that's not what I spend most of my time thinking about.

What I actually think about is:

What happens at 2:13 AM when something unexpected occurs?

Because eventually, it will.

A queue gets stuck.

A third party API slows down.

A workflow starts behaving differently.

A retry arrives hours later than expected.

The interesting part is that most problems aren't dramatic.

The system doesn't crash.

It keeps running.

Just slightly wrong.

And those are often the hardest issues to catch.

Building software that runs continuously means caring about things that demos never show:

recovery
observability
retries
data consistency
failure handling

Not because they're exciting.

Because they become important every single day.

One thing I learned pretty quickly:

The goal isn't building a system that never fails.

The goal is building a system that can recover without someone jumping in every time.

If a process gets stuck, can it restart?

If an API fails, can it retry safely?

If data arrives late, can the workflow still complete correctly?

Those questions matter more than most features.

Another reality is that software running 24/7 creates a different relationship with technical decisions.

Small shortcuts last a long time.

Small bugs eventually surface.

Small assumptions eventually get tested.

The system has a lot of time to find weaknesses.

What surprised me most is how much of the work is actually about predictability.

Not speed.

Not new features.

Predictability.

Knowing how the system behaves when things go right and when they don't.

Because people eventually start depending on that behavior.

Building software that runs continuously has changed how I think about engineering. The feature is only the beginning. The real work starts when the system has to keep doing its job reliably every hour of every day.

This is the reality of a lot of the systems we operate at BrainPack. Once enterprise workflows and AI automations are running continuously, reliability becomes less about uptime and more about making sure the system behaves predictably under real-world conditions.

The Hardest Part of Integrating with Legacy ERPs Is Not the API

Dhruvi — Mon, 01 Jun 2026 13:03:15 +0000

When people hear "ERP integration," they usually think the difficult part is the API.

Sometimes there isn't even an API.

But honestly, that's rarely the biggest problem.

The harder problem is understanding how the business actually uses the system.

Because what the ERP does and what the organization does are often two different things.

I've seen situations where:

a field means one thing in the documentation and something completely different in practice
a workflow exists because someone created a workaround five years ago
critical business rules live in spreadsheets instead of the ERP
important decisions happen outside the system entirely

Technically, the integration works.

Operationally, it's wrong.

This is where many integration projects get stuck.

The data moves successfully.

The API calls succeed.

The sync completes.

And yet users immediately tell you something is broken.

Because the integration followed the system.

Not the business process.

One thing I learned early on:

Never assume the ERP is the source of truth for how work gets done.

It's often just one piece of a much larger process.

The real workflow usually spans:

ERP systems
spreadsheets
emails
manual approvals
undocumented habits

The API is a technical challenge.

Understanding operational behavior is a people challenge.

And in my experience, the people challenge takes longer.

The best integrations I've worked on started with questions like:

Why does this process exist?

Who actually uses this field?

What happens if this step is skipped?

Those conversations usually uncover more than the technical documentation ever does.

The interesting part is that older systems are often doing exactly what they were designed to do.

The complexity comes from everything built around them over the years.

This is something we see regularly at BrainPack when connecting existing enterprise systems to modern workflows and AI capabilities. The technical connection is usually the easy part. Understanding how the organization actually operates is where most of the real integration work happens.

Something Honest About Being a Developer on This Kind of Team

Dhruvi — Thu, 28 May 2026 12:38:43 +0000

One thing I didn’t expect working on systems like this:

A lot of the job is uncertainty.

Not coding itself.

Uncertainty.

You’re constantly working across:

systems you didn’t build
workflows nobody fully documented
integrations that behave differently under production load
business logic hidden inside years of habits and manual processes Sometimes the hardest part is simply figuring out what is actually happening.

Another thing people don’t talk about enough:

You rarely get the satisfaction of “finished.”

Because the systems keep evolving.

You fix one workflow.

Then another dependency appears.

You stabilize one integration.

Then business priorities change and the flow changes again.

The system keeps moving underneath you.

There’s also a different kind of pressure when systems run continuously.

You know real operations depend on them.

If something breaks:

people stop receiving orders
workflows stop moving
teams lose visibility
data becomes unreliable

It changes how carefully you think about small decisions.

At the same time, this kind of work made me much calmer technically.

You stop panicking when things fail.

Because eventually you realize:
production systems always fail somewhere.

The goal is not perfection.

The goal is building systems that recover safely and predictably.

One thing I genuinely like about this work though:

You get very close to how businesses actually operate.

Not the clean diagrams.

The real workflows.

The weird edge cases.

The manual processes people invented just to keep things moving.

You learn quickly that software is usually less about code and more about understanding operational behavior.

Working at BrainPack exposed me to how complex enterprise environments actually are once multiple systems, teams, and AI workflows start interacting together. Most of the engineering work is not about building isolated features, it’s about making entire operational flows stable enough to trust long term.

A Technical Problem I Worked On This Week

Dhruvi — Fri, 22 May 2026 12:43:15 +0000

This week, I spent more time than expected debugging something that looked simple.

Data syncing between two systems.

One side said the record was updated.

The other side disagreed.

No errors.

No failed requests.

Everything looked normal.

Which usually means the problem is not obvious.

The issue ended up being timing.

One system updated immediately.

The other processed updates through a delayed workflow.

Most of the time it worked.

Sometimes updates arrived in a different order.

Which created small inconsistencies:

outdated values appearing temporarily
automation triggering from stale information
users seeing different states in different systems

The difficult part was that it only happened occasionally.

So reproducing it locally was almost impossible.

The fix itself was not complicated.

We changed how updates were processed.

Instead of assuming data arrives in the right order, we added validation around state changes before applying updates.

Small change.

Big difference.

One thing I keep learning working on systems that run continuously:

A lot of problems are not caused by failures.

They come from assumptions.

Assuming systems process things instantly.

Assuming updates arrive in order.

Assuming timing stays consistent.

Production environments break assumptions very quickly.

The interesting part about enterprise systems is that most technical problems are not isolated.

One small inconsistency spreads.

An automation behaves differently.

A report becomes inaccurate.

Another system trusts the wrong data.

Small problems travel.

This is something we deal with regularly at BrainPack while connecting systems that were never originally designed to operate together. AI workflows become much more reliable once the underlying data movement becomes predictable.

The Difference Between Building for Demo and Building for Production

Dhruvi — Wed, 20 May 2026 12:45:36 +0000

A lot of software looks great in demos.

Clean data.
Fast responses.
Perfect flow.

Production is where reality shows up.

A demo assumes everything behaves correctly.

Production assumes eventually something will break.

That changes how you build.

For demos:

You optimize for:

speed
presentation
showing capability

For production:

You optimize for:

failure recovery
retries
monitoring
stability
weird edge cases nobody planned for

A demo works when everything goes right.

Production works when things go wrong.

One thing I noticed early on:

Demo environments are predictable.

Production environments are messy.

You get:

duplicate events
incomplete data
slow third party systems
retries arriving late
users doing things you never expected

The code that looked perfect in testing suddenly behaves very differently.

Another difference:

Demo code usually answers:

"Can we do this?"

Production code answers:

"Can this keep working for months while real people depend on it?"

Very different problem.

One thing that changed how I build systems:

I stopped asking:

"Does this work?"

Now I ask:

"What happens when this fails?"

Because eventually it will.

The question is whether the system recovers safely.

A lot of engineering work happens after the feature already works.

Observability. Recovery. Reliability.

The things users never notice.

Until they stop existing.

This comes up constantly at BrainPack while operating systems that run continuously across enterprise environments. Layering AI on top becomes much easier once the underlying infrastructure is designed for production conditions instead of demo conditions.

A Small Fix That Helped a Live Deployment Immediately

Dhruvi — Tue, 19 May 2026 12:44:30 +0000

One of the most useful fixes I worked on recently was not complicated at all.

We added a queue between two systems that were talking to each other directly.

That was it.

Before that, everything worked fine most of the time.

Until traffic increased or one system slowed down for a few seconds.

Then things started piling up:

requests timing out
retries triggering
duplicate operations
random failures appearing across workflows

The problem was that both systems expected immediate responses from each other.

So when one slowed down, the other started failing too.

Classic cascading failure.

The fix was surprisingly small.

Instead of:
System A → direct request → System B

We changed it to:
System A → queue → System B

Now:

requests could wait safely
retries became manageable
temporary slowdowns stopped affecting the entire flow

The deployment stabilized almost immediately.

What I liked about this fix is that it changed the behavior of the system more than the complexity of the code.

No massive rewrite.
No new infrastructure layer.

Just removing the assumption that everything has to happen instantly.

A lot of production issues come from systems being too tightly coupled.

One delay becomes everybody’s problem.

Queues don’t remove failures.

They absorb pressure long enough for the rest of the system to keep operating normally.

One thing I learned working on live systems:

Performance issues are often really coordination issues.

The systems themselves are usually capable.

They just fail because everything depends on perfect timing.

This is something we run into constantly at BrainPack while integrating multiple enterprise systems and AI workflows together. A lot of stability comes from reducing tight coupling between systems so temporary failures don’t spread across the entire infrastructure.

How We Debug Issues That Only Happen Once Every Few Days

Dhruvi — Fri, 15 May 2026 12:45:23 +0000

The hardest bugs are not the ones that happen constantly.

The hardest ones are:

once every few days
under unknown conditions
with no obvious pattern Especially in systems that run continuously.

Because by the time you notice the issue, the original state is already gone.

Early on, I used to approach these bugs the wrong way.

I would immediately start reading logs and trying to reproduce the issue locally.

Most of the time, that went nowhere.

Because these problems usually depend on:

timing
retries
load
specific data states
interactions between systems

Things that almost never exist in your local environment the same way.

What changed for me was realizing:

The goal is not “find the bug immediately.”

The goal is:
make the system observable enough that the bug exposes itself next time.

So instead of guessing, we start adding visibility around the problem.

Things like:

tracking state transitions
storing retry history
recording execution timing
correlating events across systems

Not permanent debugging noise.

Just enough context to reconstruct what actually happened later.

Another thing I learned:

Rare bugs are often not random.

They usually happen when multiple small conditions align:

a delayed queue
a retry arriving late
stale data
another service slowing down

Individually, nothing breaks.

Together, something weird appears for 30 seconds and disappears again.

One mistake I made a lot before:

Trying to “fix” the issue too early.

When you don’t fully understand intermittent bugs, quick fixes usually just hide the symptom temporarily.

So now I spend more time understanding:

what sequence created the issue
what state the system was in
why recovery didn’t happen automatically

Only then do we change the flow.

The interesting part is that debugging these issues slowly changes how you design systems.

You stop building only for normal operation.

You start building for investigation too.

Because eventually, every long-running system develops behaviors you didn’t predict.

At BrainPack, a lot of debugging work involves understanding interactions between systems that only fail under very specific timing conditions. The more AI workflows and automations are layered on top, the more important observability and recoverability become.