DEV Community: Girish bari

Tricked Claude to Design the system, Not to Code (part 1)

Girish bari — Sat, 17 Oct 2026 11:15:12 +0000

I wanted to explore automation for a few days, and I came across
Minicor, a company working on the problem of automating legacy
software. The problem is simple, and I'm still figuring out whether it
is worth solving at this level. But I learned a lot while building this
project.

Let's start.

Folks who want to see and demonstration
github
live link
watch the demo

Understanding the pain

I can relate to this problem from my experience working with Tally back
in 2018. You simply couldn't automate many scenarios easily. Tally was
licensed software, and in practice, some people used pirated versions.
The software generally had to be installed on a particular system, and
there often wasn't an API available for the workflow you wanted to
automate. Add network restrictions and other environment-specific
limitations, and the problem becomes even harder.

There may be more integration options available today, but the broader
question remains:

How do we reduce manual work and make these workflows faster when the
software wasn't designed to be automated?

That's where Minicor caught my attention.

How I understand Minicor

As far as I understand it, Minicor breaks the problem into a few steps:

A business requests an automation for a particular scenario.
Minicor understands the requirements and how a human performs that task.
It creates and tests a workflow for that scenario.
Once the business approves the workflow, Minicor exposes an API.
The business can call that API whenever it needs the task performed.

workflow" width="800" height="425">

I took this idea and started building SmartPay: an automation that
creates an invoice on every API hit.

But how would we design this for more users and more legacy systems
while keeping the system reliable?

While working through distributed-system design, I realised something
important: before jumping into requirements and fancy features, we
should first track the events and understand how a payload moves from
one service to another.

Here is the flow I want to design:

Business demonstrates a scenario → Engine proposes a workflow →
Business approves it → API is exposed → Business calls the API →
Workflow succeeds, fails, or ends in an uncertain state.

That last part is important. What happens if the system fails and we
don't know whether the invoice was created?

Functional requirements

Let's derive the requirements from the flow above.

Record scenarios and workflows. We need to capture the business process that the automation is supposed to perform.
Store workflow definitions and parameter mappings. Each workflow should define the inputs it needs and how those inputs map to the actions being performed.
Publish workflows for business review. The business should be able to review the workflow name, parameters, outputs, and commit step before approving it.
Expose an API for each published workflow. Every API request must be validated against that workflow's rules.
Provide a monitoring dashboard. Business users should be able to inspect runs and take action when something needs attention.

Out of scope for now: AI-based repair when a screen changes, editing
workflow steps through the UI, tenant and user management, and billing
for the automation engine itself.

Non-functional requirements

Every non-functional requirement should support one or more of the
functional requirements above.

Before defining them, let's look at two important failure scenarios.

1. Graceful handling of saves

We need to understand what happens when a workflow is interrupted while
saving an invoice.

2. Uncertain outcomes

Consider two scenarios:

The business's system stops before the invoice is saved.
The business's system stops after the invoice is saved, but before our engine records the success.

In the second scenario, the invoice exists, but our system may not know
that. If we retry blindly, we could create a duplicate invoice and
potentially charge a customer twice.

This leads to an important requirement: we must prevent duplicate
invoice creation, even when the outcome of an execution is uncertain.

We need idempotency for each invoice operation. We also need to
distinguish a confirmed failure from an uncertain outcome. An uncertain
run should be reconciled before another attempt is allowed to create an
invoice.

In other words, we shouldn't blindly retry an API request just because
we didn't receive a successful response. The same request needs a stable
idempotency key, and the execution system must check whether the
intended effect already happened.

We also need to store the workflow definitions, parameters, mappings,
and rules in a structured format, such as JSON, so every API request can
be validated against the workflow it invokes.

What about response time?

How fast should an API request be?

Here's the catch: we can optimise everything on our side, but what if
the target application itself is slow?

We might be able to acknowledge a request in under 500 ms, but that does
not mean the invoice will be created in 500 ms. The legacy billing
application may take much longer to complete the operation.

So we need to distinguish between API response time and workflow
completion time.

This leads to another question: what happens when we receive 50,000
requests?

That's what I spent a good few hours discussing with Claude. Before
getting into the cost estimates, I found it useful to understand a few
basic concepts.

Cost estimation

Before going deeper, here are three questions that help us estimate
capacity and cost.

Concept	Question	What it helps estimate
Rate	How quickly do requests arrive?	API requests per second, database writes per second, and queue throughput
Concurrency	How many operations are in progress at once?	Worker slots, connections, and memory
Size	How much data does each operation produce?	Storage, bandwidth, and cost

A useful mental model is:

Rate = number of requests ÷ time window
Concurrency ≈ rate × average processing time
Storage = number of records × average record size

We also need to ask one more question: what is the limiting resource,
and where does that limit live?

Let's assume 50,000 requests

Suppose we need to process 50,000 invoices in four hours.

Four hours is 14,400 seconds, so the average arrival rate is:

50,000 ÷ 14,400 ≈ 3.5 requests per second

Now suppose the legacy application takes around 30 seconds to create
one invoice.

Using Little's Law:

Concurrency ≈ 3.5 requests/second × 30 seconds ≈ 104 concurrent runs

At first, that doesn't sound too bad. Around 104 runs in progress seems
manageable from an infrastructure perspective.

But there is another constraint.

The legacy system is the bottleneck

Let's assume each business has one legacy application instance, and that
instance can create only one invoice at a time.

If one business needs to process 500 invoices, and each invoice takes 30
seconds:

500 × 30 seconds = 15,000 seconds = 4 hours and 10 minutes

That already exceeds our four-hour target.

Adding more API servers or workers won't solve this problem. If the
target application can process only one invoice at a time, the
automation engine cannot make that individual instance process invoices
faster simply by adding infrastructure.

The real capacity limit is the concurrency supported by each business's
legacy application. Increasing it may require additional sessions,
application instances, or licences, depending on what the software and
the business environment support.

So the important question isn't simply, "How do we handle 50,000
requests?"

It is: How many requests can arrive, how long does each workflow take,
and how many workflows can each target system safely execute at once?

Do we need Kafka or SQS?

For this assumed workload, we don't necessarily need Kafka or SQS from
day one.

We could start with PostgreSQL as the durable job queue, keeping the
execution record and queue state in the same database. That reduces the
number of systems we need to operate and helps keep the execution state
consistent with the queued work.

Core entities

Before implementing the system, I want to identify the core entities and
how they relate to each other.

Entity	Meaning	Example
Tenant	The business using our platform	Sharma Traders
Client App	The application calling our API	A script or Postman during development
Target App	The software product being automated	Tally
Target Instance	A particular installation or instance of that software	Sharma Traders' Tally installation
Agent	The automation worker that performs the task	A worker operating the target application
Workflow	The task the business wants to automate	`create-invoice`
Demonstration	The recorded example from which a workflow version was created	A recording of a person creating an invoice
Run	One invocation of a workflow, including its input and outcome	A request to create a particular invoice

At a high level, the relationships look like this:

Tenant
├── Target Instances ──> Target App
│   └── Agents
├── Client Apps
│   └── Runs ──> Target Instance
└── Workflows
    └── Demonstrations

Workflow ──> Runs

Where I'll pause for now

I wanted to keep this part focused on understanding the problem, tracing
a request through the system, and identifying the first set of design
constraints. There is still a lot to explore, especially around queue
implementation, failure recovery, and cost estimation.

I'll leave the deeper implementation details for the next part. For now,
the main lesson is simple: before scaling the automation engine,
understand the workflow, track the request end to end, and identify the
system that actually limits throughput.

How $exists can become cuplrit in MONGODB?

Girish bari — Sun, 30 Aug 2026 12:14:10 +0000

From a long time I had a doubt that $exists makes the query inefficient, but I was not confident enough to conclude any points soon. Today I observed something which made me confident to not use $exists going forward in my queries until it is extremely (I mean extremely) needed.

Here is why?

[
  {
    $match: {
      project_id: ObjectId('XXXXXXXXXXXX'),
      source: "marketing",
      attempt_count: { $exists: true },
    },
  },
  { $project: { _id: 1 } },
]

Motive :- Get all the records from event_logs where attempt_count exists (docs which has field attempt_count: 1/2/3)

Even after giving the hint :- Query was inefficient

db["event_logs"].aggregate([
  {
    $match: {
      project_id: ObjectId('XXXXXXXXXXXX'),
      source: "marketing",
      attempt_count: { $exists: true },
    },
  },
  { $project: { _id: 1 } },
], { hint: "project_id_1_source_1_attempt_count_1" }).explain("executionStats")

The point here is why it is checking 16k totalkeys and totalDocs while the nreturned is 9k, the ideal case should be totalkeys = totalDocs = nreturned (making it a covered query)

Solution for above issue :-

Use range $gte, $gt, $lte, $lt

I used this query:

db["event_logs"].aggregate([
  {
    $match: {
      project_id: ObjectId('XXXXXXXXXXXX'),
      source: "marketing",
      attempt_count: { $gte: 1 },
    },
  },
  { $project: { _id: 1 } },
], { hint: "project_id_1_source_1_attempt_count_1" }).explain("executionStats")

Magic 🎉

CULPRIT:- $exists

Reason by LLM:

Point of $exists: It tells us the field presence.

-> Now according to our requirement it seems to be very much perfect to use it. What if I convert $exists into something as a range?

In layman terms: What I have understood is that $exists makes your query search the needed as well as not needed docs because $exists tells MongoDB cursor to check all the docs where there is no attempt_count and check all the docs where there is attempt_count exists and then it is only able to segregate between those sets of docs to give you the required nReturned.

Proof:

My needed docs were 9k and not 16k, but $exists made the query scan all 16k docs (around 8k extra).

When DSA Solved Our AWS Scaling Problem

Girish bari — Tue, 18 Aug 2026 15:33:26 +0000

When DSA Solved Our AWS Scaling Problem

In the age of AI, it feels like DSA is slowly disappearing from day-to-day development.

You can ask an AI to generate a queue, configure an AWS service, or write a retry mechanism in seconds.

But there is one thing AI cannot remove from engineering:

Constraints.

And when constraints appear, DSA suddenly becomes relevant again.

At our startup, we often solve problems in a scalable way. But scalability doesn't always mean adding another AWS service. Sometimes, the solution is simply using the existing components differently.

You usually don't recognize a DSA concept in a real-world system until someone asks:

"Why are we doing it this way?"

That question led us to a surprisingly simple solution to an AWS scheduling problem.

The Problem: We Were Running Out of Schedules

Imagine we have a large number of jobs.

Each job can fail and needs to be retried on subsequent days:

Parent Job
   │
   ├── Retry 1 → +1 day
   ├── Retry 2 → +2 days
   └── Retry 3 → +3 days

Initially, this seems straightforward.

We can simply create an EventBridge schedule for every retry.

But then scale enters the picture.

We encountered a situation where, for every 6,000 jobs, we needed to create 3 separate EventBridge schedules.

That means:

6,000 jobs
   ×
3 retry schedules
   =
18,000 schedules

Now imagine increasing the workload.

What happens if we have 10,000 jobs?

10,000 jobs
    ×
3 retries
    =
30,000 schedules

And what if this keeps growing?

We eventually run into EventBridge scheduling limits.

The obvious solution might be:

"Let's add another AWS component."

But our senior asked a different question:

"Can we solve this using a sliding window?"

That was the interesting part.

First, What Is a Sliding Window?

A sliding window is a technique where we don't process the entire dataset at once.

Instead, we maintain a smaller active window and continuously move it forward.

For example, imagine these jobs:

Day 1   Day 2   Day 3   Day 4   Day 5   Day 6   Day 7
 ├───────┼───────┼───────┼───────┼───────┼───────┼───────┤
         [──────────── Window ────────────]

As time moves forward:

Before:

Day 1   Day 2   Day 3   Day 4   Day 5   Day 6   Day 7
         [──────── Window ────────]


After:

Day 1   Day 2   Day 3   Day 4   Day 5   Day 6   Day 7
                 [──────── Window ────────]

The important idea is:

We don't need to keep scheduling everything forever.

We only care about jobs relevant to the current time window.

That observation changes how we think about the EventBridge problem.

What If We Don't Schedule Every Retry?

This was the key idea.

Instead of asking:

"When should this retry happen?"

and creating an EventBridge schedule for that future date...

we ask:

"Which retries are supposed to happen today?"

That small change removes the need to schedule every future retry individually.

The Original Approach

Let's say today is August 18.

A parent job runs today.

Its retries are:

August 18
   │
   ├── Retry 1 → August 19
   ├── Retry 2 → August 20
   └── Retry 3 → August 21

The traditional approach is:

              EventBridge
                  │
       ┌──────────┼──────────┐
       ▼          ▼          ▼
    Aug 19     Aug 20     Aug 21
    Retry 1    Retry 2    Retry 3

For every parent job, we create future schedules.

This is where the number of schedules grows rapidly.

The Sliding Window Approach

Now let's change the perspective.

Instead of scheduling:

Retry 1 → tomorrow
Retry 2 → day after tomorrow
Retry 3 → three days later

we maintain a daily execution window.

Every day, we ask:

"Which existing parent jobs have a retry due today?"

Then we execute those retries.

                    TODAY
                      │
                      ▼
             ┌─────────────────┐
             │   Sliding       │
             │    Window       │
             └────────┬────────┘
                      │
        ┌─────────────┼─────────────┐
        ▼             ▼             ▼
     Retry 1       Retry 2       Retry 3
      due today     due today     due today

EventBridge no longer needs to know about every individual retry.

It only needs to trigger the parent job.

But What About Jobs From Previous Days?

This is where the idea becomes interesting.

Suppose a parent job ran 3 days ago.

Let's say:

Parent Job
August 15
   │
   ├── Retry 1 → August 16
   ├── Retry 2 → August 17
   └── Retry 3 → August 18

Today is August 18.

The third retry is due today.

Under the traditional approach, we would have created a schedule on August 15 for the retry that should execute on August 18.

But we don't need to do that anymore.

Instead, today's sliding window discovers it.

August 15        August 16        August 17        August 18
Parent Job       Retry 1          Retry 2          Retry 3
    │                │                │                │
    └────────────────┴────────────────┴────────────────┘
                                                     ▲
                                                     │
                                                   TODAY

The system simply asks:

"What retries are due on August 18?"

And finds:

Parent Job: August 15
Retry:      3
Due:        August 18

Then it executes it.

The Trick

This is the entire trick:

Don't schedule the future retries.

Instead Conceptually:

                 ┌─────────────────────┐
                 │   Daily Trigger     │
                 └──────────┬──────────┘
                            │
                            ▼
                 ┌─────────────────────┐
                 │  Current Time      │
                 │     Window         │
                 └──────────┬──────────┘
                            │
                            ▼
              ┌──────────────────────────┐
              │ Find retries due today   │
              └────────────┬─────────────┘
                           │
                 ┌─────────┴─────────┐
                 ▼                   ▼
            Retry due             Not due
                 │                   │
                 ▼                   ▼
              Execute              Ignore

Why This Scales Better

The important difference is what we are scheduling.

Number of schedules grows
with the number of jobs

Now the number of EventBridge schedules is no longer directly proportional to:

Number of Jobs × Number of Retries

Instead, EventBridge becomes the mechanism that wakes up the scheduler/processor, while the application determines which work actually needs to happen.

The Bigger Lesson

The interesting part of this problem wasn't really EventBridge.

It was recognizing that we were using EventBridge to solve a problem that didn't necessarily belong there.

We initially thought:

"We need to schedule millions of future events."

The better question was:

"Do we actually need to schedule millions of future events?"

And the answer was no.

We only needed to know:

"What needs to happen today?"

That is where a simple DSA concept like sliding window becomes a practical system-design technique.

DSA Is Not Dead

This is why I don't think DSA is disappearing from software engineering.
Sometimes the answer is a sophisticated distributed system.

And sometimes...

it's a sliding window.

The important skill isn't memorizing DSA patterns.