Notes from a 2 AM On-Call: When Rclone's --checkers Parallelism Killed Our Transfer Pipeline

The Failure Symptom

2026-10-15T02:17:43Z ERROR : data/chunks/shard_0417.tar.gz: Failed to copy: RequestThrottled: Please reduce your request rate.
2026-10-15T02:17:43Z ERROR : data/chunks/shard_0891.tar.gz: Failed to copy: RequestThrottled: Please reduce your request rate.
...
2026-10-15T02:18:12Z INFO  : Transferred: 847.2 GiB / 2.1 TiB, 40%, 127 MiB/s, ETA 3h12m
2026-10-15T05:29:51Z NOTICE: Backup window exceeded. Aborting.

Our nightly 2 TB backup job from on-prem NAS to S3 missed its 6-hour SLA for the third time in a week. Rclone reported "success" but silently skipped 1,247 files due to S3 throttling. No exit code, no alert, just incomplete data.

The Red Herring

Standard rclone tuning guides suggest cranking --checkers and --transfers for maximum throughput:

rclone sync /mnt/data s3:backup-prod \
  --checkers=32 \
  --transfers=16 \
  --s3-upload-concurrency=8

This worked flawlessly in dev (100 GB test dataset, minimal S3 bucket history). In production, with 2.1 TB across 94,000 files and an S3 bucket serving live application reads, we hit AWS's undocumented per-prefix rate limits within 90 seconds.

The real problem: rclone's default retry logic silently degrades under sustained throttling. After the third 503 retry on a file, it logs an ERROR but continues processing the queue, leaving failed transfers invisible unless you parse logs.

The Root Cause

Rclone's --checkers flag spawns parallel goroutines that simultaneously:

1. List source and destination file trees
2. Compute hashes to detect changes
3. Queue transfers

With --checkers=32 and 94K files, we fired 32 concurrent S3 LIST operations plus metadata HEAD requests, instantly consuming our burst capacity quota. Once throttled, the retry exponential backoff competed with new requests from the still-running checkers, creating a congestion collapse.

S3 request pattern from CloudWatch:

02:17:40 - 02:17:50: 1,847 LIST requests/sec (limit: 5,500 per prefix)
02:17:51 - 02:18:20: 94% throttled, exponential backoff jitter spreading load

The Runnable Fix

1. Clamp Parallelism to Observed Limits

rclone sync /mnt/data s3:backup-prod \
  --checkers=8 \
  --transfers=4 \
  --s3-upload-concurrency=4 \
  --retries=10 \
  --low-level-retries=3 \
  --tpslimit=500 \
  --tpslimit-burst=1000 \
  --error-on-no-transfer

Key changes:

• --tpslimit=500: Hard cap transactions per second to stay under S3 prefix limits
• --error-on-no-transfer: Exit code 9 if zero bytes moved (catches silent failures)
• Reduced --checkers from 32 → 8 to prevent list operation stampedes

2. Add Distributed Rate Limiting via Redis

For multi-node rclone fleets (e.g., parallel backup jobs from different servers to the same S3 bucket), we deployed a centralized rate limiter:

import subprocess
import time
import redis

redis_client = redis.Redis(host='rate-limiter.internal', decode_responses=True)

def acquire_s3_slot(bucket: str, cost: int = 1) -> bool:
    """Sliding window rate limiter using Redis sorted sets."""
    key = f"ratelimit:s3:{bucket}"
    now = time.time()
    window = 1.0  # 1 second
    limit = 500   # requests per second per bucket

    pipe = redis_client.pipeline()
    pipe.zremrangebyscore(key, 0, now - window)
    pipe.zadd(key, {f"{now}:{time.time_ns()}": now})
    pipe.zcard(key)
    pipe.expire(key, int(window) + 1)
    results = pipe.execute()

    current_count = results[2]
    return current_count <= limit

def run_rclone_with_backpressure(src: str, dst: str):
    bucket = dst.split(':')[1].split('/')[0]

    while not acquire_s3_slot(bucket, cost=8):  # Reserve 8 slots for checkers
        time.sleep(0.1)

    result = subprocess.run([
        'rclone', 'sync', src, dst,
        '--checkers=8', '--transfers=4',
        '--error-on-no-transfer'
    ], capture_output=True, text=True)

    if result.returncode != 0:
        raise RuntimeError(f"Rclone failed: {result.stderr}")

This prevented the thundering herd when five backup nodes launched simultaneously at 02:00 UTC.

3. Observability Hook

rclone sync /mnt/data s3:backup-prod \
  --stats=10s \
  --stats-log-level=NOTICE \
  --use-json-log \
  | tee >(jq -r 'select(.level=="error") | .msg' >> /var/log/rclone-errors.log) \
  | curl -X POST https://monitoring.internal/v1/ingest \
         -H "Content-Type: application/x-ndjson" \
         --data-binary @-

Streaming JSON logs to our monitoring stack surfaces throttling in real time, triggering PagerDuty before the backup window closes.

Invariant Rule

Never trust default concurrency settings in cloud sync tools.

Before deploying any rclone job to production:

1. Measure actual S3 request rates under load with --stats-log-level=DEBUG
2. Subtract 30% safety margin from observed throttle threshold
3. Enforce --tpslimit at 70% of that margin
4. Always set --error-on-no-transfer to catch silent skips

After tuning, our backup job completes in 4.2 hours with zero throttling events and a clean exit code.

---

When benchmarking multi-cloud failover pipelines for our LLM inference gateway (primary provider → fallback via https://b-lost.com/v1 relay), we hit a similar thundering herd issue with connection pool exhaustion. The same Redis rate limiter pattern above, adapted with Lua atomic scripts for strict sliding windows, dropped P99 retry latency from 3.8s to 240ms.

The lesson: distributed systems fail in the same ways whether you're syncing terabytes to S3 or routing inference requests across API providers. Rate limiting is not optional infrastructure.