From RPA to Data Thinking: Building a High-Scale Payment Processing System

Kunal Deshmukh — Thu, 16 Apr 2026 11:39:43 +0000

In my previous role, I worked on a problem that pushed me to think beyond automation scripts and start thinking in terms of systems and data.

The challenge was to support an end-to-end payment pipeline handling 86 lakh records and a total disbursement of ₹1,720 crore, where even a small mismatch could lead to major reconciliation issues.

This wasn’t just automation. It was a data problem.

The Real Challenge Was Data, Not Just Scale
At this scale, the biggest issues were not execution—they were data quality problems. There were missing fields in critical records, duplicate entries across files, name mismatches affecting validation, and rejected records that required careful reprocessing. These are classic data problems, but at this scale, even small inconsistencies could cascade into larger failures.

How I Approached It
Instead of treating it as a simple RPA workflow, I approached it as a data pipeline.

The first step was data ingestion, where structured XML and database inputs were collected and standardized for processing. This ensured that downstream stages received consistent and usable data.

The next step was data validation. Using SQL-backed checks, I ensured data completeness, consistency across records, and early detection of anomalies before they could affect processing.

Once the data was validated, it moved into the processing stage, where cleaned datasets were passed into the automation layer for execution.

The reconciliation layer was the most critical part of the system. Rejected records were isolated, errors were tracked in a database, and corrections were applied before reprocessing. This ensured that the system remained reliable and traceable even when issues occurred.

Finally, the reporting layer generated outputs with clear tracking of processed records, failed or retried records, and the final reconciliation status.

Final Thought
This project changed how I think about systems.

I started as an RPA developer, but this experience pushed me toward building systems where the focus is not just on execution, but on ensuring the data behind it is correct, traceable, and reliable.

DEV Community: Kunal Deshmukh

From RPA to Data Thinking: Building a High-Scale Payment Processing System