Idempotency as a Product Feature: Run Keys, Leases, and Scheduling

In distributed systems, we often treat idempotency as a defensive implementation detail—a way to prevent database corruption when a network request retries. But after shipping 18 production applications across mobile, web, and desktop, I have come to view idempotency not as a safety net, but as a core product feature.

When building the architecture for Synapsis Medical Technologies, where I served as the first engineering hire, the stakes for "exactly once" execution were not just about data integrity; they were about clinical safety. If a HIPAA-aligned RAG pipeline processing patient vitals or an LLM-driven clinical summary triggers twice, the cost isn't just a duplicate log entry—it is potential medical misinformation or redundant billing.

Over eight years of engineering, I have learned that the "exactly once" delivery problem is fundamentally a scheduling problem. To solve it, we must move beyond simple unique constraints and toward a robust architecture involving run keys and distributed leases.

The Distributed Double-Tap Problem

The core issue is that in a networked environment, "failure" is ambiguous. If a Next.js frontend calls a NestJS API to process a wearable data sync and the request times out, the client has no way of knowing if the request failed to reach the server, failed while processing, or succeeded only for the response to vanish in transit.

Standard retry logic is the enemy here. If the client retries, and the server is not idempotent, you end up with duplicate records. In a HealthTech context, where I managed integrations for FHIR/HL7 and wearables, a duplicate record can break longitudinal patient data.

We often reach for UUID columns with UNIQUE constraints as a first pass. While effective for preventing duplicate rows, they are insufficient for complex workflows where a single request triggers a cascade of side effects—like sending a notification, charging a card, and triggering an LLM inference step. If the process dies halfway through, a simple database constraint won't help you recover the state.

Context: The Shift Toward Durable Execution

The industry is currently moving toward durable execution frameworks, as seen in the recent updates to Temporal and the rise of "serverless" orchestration. The community debate has shifted from "how do we prevent duplicates" to "how do we ensure progress."

In the React Native ecosystem, particularly as we look at the New Architecture’s focus on synchronous communication between JavaScript and Native layers, handling asynchronous side effects becomes even more critical. When I oversaw the architecture from 0 to 1 at Synapsis, we had to ensure that mobile clients—often operating on unstable hospital Wi-Fi—could reliably resume complex uploads without re-triggering expensive AI pipelines.

Architecture: Run Keys and Distributed Leases

To solve this, I implement a two-tier strategy: the Run Key (Idempotency Key) for identification and the Lease for execution control.

1. The Run Key

A Run Key is a client-generated, opaque string that uniquely identifies the intent of an action, not the action itself. Unlike a database ID, which is assigned by the server, the Run Key must be generated as close to the user action as possible.

2. The Lease

A lease is a time-bound lock on a specific Run Key. In the NestJS architectures I’ve designed, we use Redis to manage these leases. When a request arrives with a Run Key, the system attempts to acquire a lease.

• If the lease is held, the request is a duplicate currently in progress. The server should return a 409 Conflict or a 202 Accepted with a pointer to the status.
• If the lease is expired but the work is marked as "completed" in the persistence layer, the server returns the cached result.
• If the lease is available and no work is found, the server begins execution.

The Cost of Implementation: Lessons from the Field

When I scaled the engineering team at Synapsis from zero to 21 engineers in 13 months, one of the biggest challenges was teaching the team to think in terms of "idempotency by default."

We overhauled our CI/CD across five production systems, cutting release cycles from two days down to four hours. A major component of that speed was our ability to deploy with confidence, knowing that if a deployment interrupted a running process, our idempotency logic would allow the system to recover without manual intervention.

However, this comes with a trade-off: storage overhead and state management. You are no longer just storing patient data; you are storing the metadata of the attempt. In our HIPAA-aligned RAG pipeline, which maintained 99.9% uptime, we had to carefully manage the lifecycle of these keys to ensure we weren't bloating our primary databases with millions of expired idempotency tokens.

A Worked Example: The Clinical AI Pipeline

Consider a scenario where a clinician submits a voice note for AI summarization. The pipeline involves:

1. Transcribing audio (External API).
2. Mapping to FHIR resources (Internal Logic).
3. RAG-based clinical summary (LLM Pipeline).

Without a Run Key, a retry at step 3 would re-run steps 1 and 2, wasting money and time.

// A simplified NestJS guard for idempotency
@Injectable()
export class IdempotencyGuard implements CanActivate {
  constructor(private redis: RedisService) {}

  async canActivate(context: ExecutionContext): Promise<boolean> {
    const request = context.switchToHttp().getRequest();
    const key = request.headers['x-idempotency-key'];

    if (!key) return true; // Or enforce it based on policy

    const lock = await this.redis.set(key, 'processing', 'EX', 60, 'NX');
    if (!lock) {
      throw new ConflictException('Request already in progress');
    }

    return true;
  }
}

The real complexity lies in the "recovery" phase. If the LLM pipeline fails, the lease must be released or transitioned to a "failed" state so that the next retry can actually execute, rather than being blocked by a stale lock.

Practical Recommendations

Based on my experience shipping across mobile and web platforms, here are the rules I follow for building idempotent systems:

1. Client-Side Generation: In React Native, generate the Run Key at the moment the user taps the "Submit" button and persist it to AsyncStorage or SQLite before the network call is even attempted. This ensures that even if the app crashes, the retry uses the same key.
2. Deterministic Transformations: Ensure your internal logic is deterministic. If you are integrating wearables data, the transformation from raw JSON to FHIR should produce the same output every time for the same input.
3. Separate Side Effects: Use a transactional outbox pattern. Don't send the HL7 message to the hospital system in the same function that updates your database. Log the intent to send, and let a separate, idempotent worker handle the delivery.
4. Status Endpoints: Always provide a way for the client to query the status of a Run Key. A GET /status/:runKey endpoint is essential for mobile clients to recover after a hard crash.

Conclusion

Idempotency is not just a technical requirement for distributed systems; it is a product requirement for reliability. Whether you are scaling a team or building a zero-to-one architecture for a high-stakes field like HealthTech, the ability to guarantee that an action happens exactly once—or at least, that its side effects are controlled—is what separates a prototype from a production-grade system.

By treating every request as a scheduled task identified by a Run Key and protected by a lease, you move away from the chaos of "hopeful processing" and toward a system that is resilient by design. The goal is not to prevent failure, but to make failure predictable and recovery automatic.

---

Amit Chakraborty is a founding engineer and senior architect — React Native, AI/RAG systems and production architecture. Portfolio: www.amitchakraborty.dev · LinkedIn · GitHub. Open to senior and founding engineering roles, remote worldwide.