Sync Data Between Two APIs in Python (Idempotent, with Retries)

To safely sync data between two APIs without duplicating records or losing data during network drops, you must implement two core patterns: Retries with Exponential Backoff (to handle transient network or server errors) and Idempotency (to ensure duplicate requests do not create duplicate resources).

1. The Robust Retry & Idempotency Pattern

This solution uses Python's requests library along with urllib3's built-in Retry utility. It handles HTTP status codes 429 (Too Many Requests) and 5xx server errors, while using a unique record ID as an idempotency key in the destination header.

import logging
import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

# Configure logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")

def get_resilient_session() -> requests.Session:
    """
    Creates a requests Session configured with automatic retries,
    exponential backoff, and handling of rate limits (429).
    """
    session = requests.Session()
    retries = Retry(
        total=5,                          # Total number of retries
        backoff_factor=1,                 # Wait 1s, 2s, 4s, 8s, 16s between retries
        status_forcelist=[429, 500, 502, 503, 504], # Retry on these status codes
        raise_on_status=False             # Return response instead of throwing MaxRetryError
    )
    adapter = HTTPAdapter(max_retries=retries)
    session.mount("https://", adapter)
    session.mount("http://", adapter)
    return session

2. Complete Sync Script

The script below fetches records from a source API and pushes them to a destination API. It uses the source record's unique ID as the X-Idempotency-Key to guarantee that duplicate POST requests are safely ignored or updated by the destination server.

SOURCE_API_URL = "https://api.source.com/v1/records"
DEST_API_URL = "https://api.destination.com/v1/records"
API_TOKEN = "your_api_token_here"

def sync_records():
    session = get_resilient_session()
    headers = {
        "Authorization": f"Bearer {API_TOKEN}",
        "Content-Type": "application/json"
    }

    # Step 1: Fetch data from the source API
    try:
        logging.info("Fetching records from source API...")
        response = session.get(SOURCE_API_URL, headers=headers, timeout=10)
        response.raise_for_status()
        records = response.json().get("data", [])
    except requests.exceptions.RequestException as e:
        logging.error(f"Failed to fetch source records: {e}")
        return

    # Step 2: Sync each record to the destination API idempotently
    for record in records:
        record_id = record["id"]  # Unique business key from source

        # Inject the unique source ID as the Idempotency Key
        dest_headers = headers.copy()
        dest_headers["X-Idempotency-Key"] = str(record_id)

        try:
            logging.info(f"Syncing record {record_id}...")
            # Using POST with an Idempotency-Key header, or PUT to a specific ID endpoint
            dest_response = session.post(
                DEST_API_URL, 
                json=record, 
                headers=dest_headers, 
                timeout=10
            )

            if dest_response.status_code in [200, 201]:
                logging.info(f"Successfully synced record {record_id}")
            elif dest_response.status_code == 409:
                logging.warning(f"Record {record_id} already exists (Conflict). Skipped.")
            else:
                logging.error(f"Failed to sync record {record_id}: {dest_response.status_code} - {dest_response.text}")

        except requests.exceptions.RequestException as e:
            logging.error(f"Network error syncing record {record_id}: {e}")

if __name__ == "__main__":
    sync_records()

3. How Idempotency is Guaranteed

- **HTTP PUT Method (Alternative):** If your destination API supports `PUT` requests to a specific resource endpoint (e.g., `/records/{id}`), use `PUT` instead of `POST`. `PUT` is inherently idempotent; sending the exact same payload multiple times to the same URI will yield the same state.
- **Idempotency-Key Header:** If you must use `POST`, passing a unique transaction/record ID via a header like `X-Idempotency-Key` tells the destination server to save the initial response. If the server receives a second request with the same key, it returns the cached response instead of creating a duplicate.

4. Production Considerations

- **State Tracking (Cursor Sync):** Avoid fetching all records every time. Store the last successfully synced timestamp or ID in a local database (e.g., SQLite or PostgreSQL) and pass it as a query parameter (`?updated_since=...`) to the source API.
- **Dead Letter Queue (DLQ):** If a record fails to sync after all retries (e.g., due to a 400 Bad Request validation error), catch the exception, log the payload to a database table or queue, and proceed to the next record rather than crashing the sync process.

Need this done fast? order an integration on Kwork (https://kwork.com/scripting/52991008/integrate-your-services-api-webhooks-crm-and-payments).