The Ghost That Moves Your Money: Making an RL Payment-Retry Agent Accountable

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

HRA-Pay is a reinforcement learning system that decides what to do after a payment fails: retry or give up, when to retry, and on which rail (UPI, credit, debit, netbanking, wallet).

Most retry systems use a supervised classifier that predicts a success probability. HRA-Pay treats the whole retry sequence as a Markov Decision Process. A branched Q-network learns a policy over timing and channel jointly, and a deterministic guardrail layer sits between that policy and any action that moves money.

How I read the theme: every learned policy is a ghost in the machine, an opaque decision-maker acting on your money that nobody can fully inspect. This project is about making that ghost accountable. The learned policy proposes, a rule-based guard (not AI, on purpose) disposes, and an audit trail records both what the ghost wanted to do and what it was allowed to do. I'll admit this project wasn't built for this theme. But the question the theme asks, "what do we do about the thing acting inside the system that we can't fully see?", is one the project spends most of its design effort on.

Code

23f3003672 / hra-pay

HRA-Pay

Hierarchical reinforcement learning for payment retry optimisation, with LLM-calibrated reward shaping.

Razorpay Buildathon 2026 — Track 03, AI Revenue Recovery

When a payment fails, a merchant faces a compound decision, not a single one retry or give up, when to retry, and on which rail. Production retry systems generally answer only the first two, with a supervised classifier that predicts a success probability. HRA-Pay treats the whole retry sequence as a Markov Decision Process and learns an explicit policy over timing and channel jointly — with a deterministic guardrail layer between the learned policy and any action that moves money.

python -m venv .venv && .venv\Scripts\activate     # Windows
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e ".[dev]"

python tasks.py test        # 91 tests
python tasks.py eval        # reproduces every number below
python tasks.py demo        # the dashboard

---

Results

1,000 episodes per run…

View on GitHub

python -m venv .venv && .venv\Scripts\activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e ".[dev]"

python tasks.py test    # 91 tests
python tasks.py eval    # reproduces every number in the README
python tasks.py demo    # the dashboard

How I Built It

Open-source stack: PyTorch (CPU), Gymnasium, NumPy, and Streamlit. Both learned agents are trained from scratch locally, with no pretrained weights and no hosted inference.

• Decision policy: a Branching Dueling Q-Network (Tavakoli et al., 2018) with three branches: macro action {RETRY, SWITCH_CHANNEL, ABANDON}, timing {NOW, +2h, +6h, +24h, +72h}, and channel. Output size grows additively (13 outputs vs 31 for a flat head). I compared it against a flat dueling DQN and two static baselines, all through one shared Policy interface, runner, guard, and audit log.
• Reward calibration: a language model reads each raw decline-reason string, and nothing else, and scores how much friction a retry carries. It runs once, offline, and the output is cached in a committed penalty_table.json. The numeric success probabilities never enter the prompt, and a test asserts it. The table refuses to load until a human reviews it, and the review overrode the model on three of the codes.
• Guardrails: PolicyGuard is deterministic. It hard-blocks suspected_fraud (compliance), blocks futile retries, caps attempts, and escalates retries that come too soon. No Q-value can override it.
• Environment: a seeded, synthetic simulator defined by fully commented YAML specs, plus a held-out spec that inverts the relationships an agent is most likely to have learned.

The language model used for calibration was Gemini (gemini-3.7-flash), a closed API. It's the one non-open component, but it's a one-time offline step and its output is a plain JSON file in the repo. Reproducing or replacing it with an open-weight model doesn't touch the training loop.

Results (1,000 episodes, learned agents averaged over 3 seeds):

policy	recovered ₹ (train)	recovered ₹ (held-out shift)
bdq (branched)	1,015,730 ± 41,732	765,171 ± 46,101
flat_dqn	1,003,601 ± 20,949	844,508 ± 22,809
static_with_switch	880,377	753,958
static_schedule	536,078	381,354

Issuer-risk exposure was zero in every run. I also report what counts against the system: the learned agents spend more retries, BDQ is slower to recovery than the naive baseline, and the BDQ-vs-flat ranking flips under distribution shift. On the training distribution the two architectures are statistically indistinguishable, and the comparison tool says so instead of declaring a winner.

Why Does Open Innovation Matter?

For a system that moves money, "trust me" is not an acceptable answer. Open innovation made three things possible that a closed API wouldn't have:

1. Inspectability end to end. The environment, the reward, the guard, and the metrics are all readable code and committed YAML. A reviewer can dispute the ground-truth assumptions directly, which matters when the main objection to this project is "you wrote both the environment and the agent."
2. Training and owning the policy. Because the decision model is a small network trained locally, I could checkpoint it, test it, run 3-seed comparisons, and stress-test it on an adversarial held-out distribution. A hosted model behind an API would give me none of that control.
3. Reproducibility as a feature. The environment and static baselines reproduce bit-identically across Windows and Linux. The neural parts don't, because torch seeding differs across builds, so the README reports mean ± std and states conclusions as gaps rather than digits.

It also shows in the one place I used a closed model. Because its output is cached and human-reviewed in the open, the closed API only touches the project where I could audit it.

The Spirit of It

The moment that made the "ghost" real for me was a number I almost skipped. In the results table, one baseline's mean time-to-recovery was exactly 24.00h. That's too round to be natural. It turned out the guard was silently crippling the baseline: the original failed authorisation was eating half of every channel's retry budget. I had handicapped the baseline my own agent had to beat, in the direction that flattered my agent.

Nothing crashed and no test failed. The system was just quietly wrong in my favour. That's the lesson I took from this project: the ghost isn't only the model. It's also the assumptions you wrote into the machine and stopped looking at. The repo keeps an ENGINEERING_LOG.md of this and other failures, including a flat-vs-branched comparison that was rigged by exploration and a comparison tool that confidently named the loser as the winner.