A CRM request fails during a brief service interruption. Retrying may recover it. Retrying every failure immediately can create duplicates, overload the service, and hide a configuration problem.
A retry policy should classify failures, limit attempts, and preserve uncertain outcomes for investigation.
Classify before retrying
Use the provider's documented response contract. The following categories are a starting design, not universal rules for every API.
| Failure | Typical next action |
|---|---|
| Invalid field or unsupported property | Correct the payload |
| Authentication failure | Repair credentials or refresh through the supported flow |
| Rate limit | Respect provider guidance and delay |
| Temporary service failure | Consider bounded retries |
| Timeout after a create request | Check idempotency or reconcile the outcome |
A timeout does not prove that the remote operation failed. It may have succeeded before the response was lost.
Use bounded backoff
Increase the delay between attempts and add jitter so many workers do not retry together. Set a maximum attempt count and a maximum elapsed retry window.
Honor a valid Retry-After response when the provider uses it. Account for its supported date or delay format and avoid retrying earlier than requested.
A sample internal policy might use several delayed attempts over a limited window. Its actual timing should reflect the provider's limits and the business's response expectations.
Persist the next attempt
Store attempt count, last controlled error category, next attempt time, and current processing state. A process restart should not erase retry history.
Use worker leases or equivalent coordination so multiple workers cannot execute the same pending delivery at once. Define recovery after a worker dies while holding a lease.
Protect remote creation
Reuse the same supported idempotency key or external enquiry reference across retries. Changing the identifier on each attempt defeats duplicate protection.
If the remote API cannot safely identify a prior creation, put ambiguous outcomes into review. Repeating a write blindly is not always the best recovery action.
Stop automatically when necessary
Exhausted retries need a visible review queue and an accountable owner. Preserve enough restricted operational context to repair and replay the delivery safely.
Before replaying, verify whether the record already exists remotely. Correcting credentials should not cause a backlog of duplicate creations.
Test the policy
Simulate rate limiting, invalid payloads, temporary failures, lost responses, and worker restarts. Verify actual attempt spacing and that permanent failures do not loop forever.
Retries are a recovery mechanism. They work best when the system can explain why another attempt is safe and when automatic recovery should stop.
Top comments (0)