The Bug That Deleted Itself Every Night

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

KRA Auto-Checker is an automated ops tool I built and run on petrol station mini PCs in Kenya. It monitors whether fuel sale transactions have actually been submitted to KRA's eTIMS tax system — the government's real-time transaction reporting requirement. It runs silently in the background via Windows Task Scheduler, checks each transaction's QR link against the KRA portal, writes results to a shared Google Sheet, and can self-update remotely across every station running it. It's currently deployed on 24 stations.

Each transaction check lands in one of three states:

• 🟢 SUCCESS — submitted to KRA
• 🟡 ERROR — a temporary failure: KRA portal downtime, timeouts, DNS issues, connection resets, or just a slow station network
• 🔴 NOT SUBMITTED — checked and confirmed the sale never reached KRA

Bug Fix or Performance Improvement

Here's the bug: ERROR-status transactions were retried automatically, on a schedule pulled from a Global Config sheet. That part worked. But once the retry attempts were exhausted, the program deleted retry_transaction.json — regardless of whether those transactions had actually resolved.

The assumption baked into that design was that overnight failures were unlikely. In practice, KRA's servers have issues most of the time. So every batch of transactions that failed to resolve within the retry window just vanished from the recovery path — with no further attempt to check on them ever again.

NOT SUBMITTED transactions had the same problem: they were never saved for retry at all, even though a sale can still get transmitted later.

I caught this by watching the Report Sheet: ERROR and NOT SUBMITTED counts kept climbing with no way to recover them — meaning I had no visibility into whether real KRA compliance gaps were quietly piling up across all 24 stations.

Code

Full diffs:

• v2.4.5 — improve retry scheduling and cleanup
• v2.4.6 — retry persistence, daily recovery, NOT_SUBMITTED handling, force_retry flag, coverage dashboard
• Current version of kra_auto_checker.py

1. A separate, indefinitely-persisted store for exhausted transactions, kept apart from the active retry_transaction.json so a fresh day's failures can never overwrite yesterday's still-unresolved ones:

# ── Separate storage for daily KRA-recovery transactions ───────────
# Kept apart from retry_file so today's fresh scheduled-retry failures
# never overwrite transactions still waiting on KRA to recover.

def save_recovery(self, transactions: list, recovery_attempts: int = 0):
    data = {
        "status": "pending_kra_recovery",
        "recovery_attempts": recovery_attempts,
        "saved_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
        "transactions": [],
    }
    for tx in transactions:
        data["transactions"].append({
            "transaction_date": str(tx["TransDateTime"]),
            "qr_link": tx["QRLink"],
            "check_date": tx["CheckDate"],
            "payment_mode": tx.get("PaymentMode", "UNKNOWN"),
            "last_status": tx.get("LastStatus", "ERROR"),
        })
    with open(self.recovery_file, "w") as f:
        json.dump(data, f, indent=4)
    self.logger.info(
        f"Saved {len(data['transactions'])} transaction(s) for recovery "
        f"(attempt #{recovery_attempts})"
    )

""" Note: load_recovery() deliberately has no 24h staleness
check — unlike load() for scheduled retries, a recovery file
is supposed to live indefinitely until KRA actually resolves it
(that's the whole point of this feature). """
def load_recovery(self) -> Optional[Dict]:
    if not os.path.exists(self.recovery_file):
        return None
    with open(self.recovery_file) as f:
        return json.load(f)

def delete_recovery(self):
    if os.path.exists(self.recovery_file):
        os.remove(self.recovery_file)
        self.logger.info("Cleared recovery file — all recovery transactions resolved")

2. Retries feed recovery from the very first failure — not just after exhaustion — so a Task Scheduler outage can never lose a transaction silently:

if retry_count < len(retry_hours):
    next_retry_time = f"{retry_hours[retry_count]:02d}:00"
    retry_manager.save(failed_transactions, retry_count)
    retry_manager.schedule(next_retry_time)

    # Safety net: write to recovery immediately on every failure,
    # not just after exhaustion. If scheduled retry tasks fail to
    # run (Task Scheduler issues, machine off, wrong time), the
    # transaction is already in recovery and won't be lost.
    retry_manager.save_recovery(failed_transactions, recovery_attempts=0)
else:
    logger.warning(
        f"All scheduled retries exhausted. Retry tasks removed. "
        f"{len(failed_transactions)} transaction(s) moved to daily recovery."
    )
    retry_manager.remove_retry_tasks()
    retry_manager.save_recovery(failed_transactions, recovery_attempts=0)
    retry_manager.delete()  # active retry file cleared — these are recovery-only now

3. Every normal daily run processes kra_recovery.json first, before touching the database, updating existing report rows in place instead of appending duplicates:

recovery_data = retry_manager.load_recovery()

if recovery_data:
    recovery_attempts = recovery_data.get("recovery_attempts", 0) + 1
    logger.info(
        f"Found {len(recovery_data.get('transactions', []))} "
        f"pending recovery transaction(s) — attempt #{recovery_attempts}"
    )

    still_failed = []
    for transaction in recovery_list:
        status, entry = _check_transaction(kra, logger, transaction, source_label="[RECOVERY]")

        # Update the existing report row for this QR link instead of
        # appending a new one — same mechanism scheduled retries use.
        sheets.update_report_entry(
            qr_link=transaction["QRLink"],
            data=entry,
            retry_count=f"recovery#{recovery_attempts}"
        )

        if status in ("NOT_SUBMITTED", "ERROR"):
            still_failed.append(transaction)

    if still_failed:
        retry_manager.save_recovery(still_failed, recovery_attempts=recovery_attempts)
    else:
        retry_manager.delete_recovery()

My Improvements

The fix restructures how failed transactions are tracked and recovered:

• Separated the two failure states into two files. Active scheduled retries live in retry_transaction.json; transactions that exhausted all retries move into a new kra_recovery.json, tagged pending_kra_recovery. Keeping them apart means a new day's fresh failures can never overwrite yesterday's still-unresolved ones.
• Recovery is written on every failure, not just after exhaustion. If Task Scheduler itself fails to fire a retry (machine off, wrong time, scheduling error), the transaction is already safe in kra_recovery.json — it doesn't depend on the retry cycle completing to be protected.
• Recovery runs before fresh checks. At the start of every daily run, the checker processes kra_recovery.json first — before it even queries the database for new transactions.
• In-place updates, not duplicate rows. Recovery attempts update the existing report row (matched by QR link) instead of appending a new one, so the Report Sheet stays a single source of truth per transaction.

• Visible recovery history. The Retries column shows recovery#N to distinguish a recovery pass from a normal scheduled retry, and recovery_attempts increments with each daily pass — so I can see at a glance how long a transaction has been stuck.

• Self-clearing. kra_recovery.json only gets deleted once every transaction inside it has resolved successfully — nothing is dropped until it's actually confirmed.
• NOT_SUBMITTED now gets the same treatment as ERROR. Previously these were logged and forgotten. Now they enter the identical retry-then-recovery pipeline, since a sale can still get transmitted later.
• A manual override for when KRA comes back online. A force_retry flag in the Global Config sheet lets me flip a switch once KRA's servers recover, and every station's heartbeat monitor picks it up within 30 minutes and runs a --force-recovery pass — processing kra_recovery.json only, with no interference to the normal daily run. The flag clears itself after firing so it never double-triggers.
• A Daily Coverage dashboard. One row per station: last check date, whether it reported today, last status, and a Recovery Pending flag — so I can see gaps across all 24 stations without opening each one's report or logging in remotely.

The underlying design shift was treating "exhausted retries" as a persistent state to track, not an end condition to discard. Small change in framing, but it turned a silent data-loss bug into a system that now never loses track of an unresolved transaction — and gives me a single dashboard to confirm that's actually true, across every station.