Designing Fault-Tolerant Async Architectures

Building an asynchronous TTS endpoint sounds straightforward: accept some text, return the job ID, generate the audio in the background, and upload the result to a presigned URL.

The interesting part starts when you wonder what should you do:

• When things fail:
  • What happens if your messaging broker restarts while a worker is processing a job?
  • What happens if the service restarts?
  • Who should be responsible for retrying when the presigned URL expires?
• About responsibilities of each service?
• About retries?
• Around observability topic?

This post explores one deliberately simple design for answering those questions without turning a small asynchronous service into a distributed-systems science project 😉.

tl;dr

• POST /synthesize endpoint returns 202 HTTP status code.
  • This means "the request has been accepted for asynchronous processing".
  • This does NOT mean "this operation will definitely complete".
• RabbitMQ is used as durable, persistent job state, so we won't be losing any message.
• The client owns the output destination. It supplies the presigned URL; the TTS service simply uploads to it.
• Instead of client asking about a job, I reversed the data flow. In other words TTS service now is sending the state of a job to the client.
• The client is responsible for choosing a presigned URL with a suitable lifetime. The TTS service should measure how long TTS generation takes and whether uploads fail, so you can detect if the service is routinely taking longer than the URLs remain valid.

---

The Service

Let's start with a deliberately small API. The client sends:

POST /synthesize
Content-Type: application/json

{
  "text": "Hello world, this is a test of the asynchronous synthesis API.",
  "genUploadUrl": "https://your-server.example/generate-upload-url",
  "statusCallbackUrl": "https://your-server.example/synthesis-status"
}

The service immediately responds:

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "jobId": "550e8400-e29b-41d4-a716-446655440000",
}

HTTP 202 Accepted is specifically intended for requests that have been accepted for processing but whose processing has not completed yet. The important part is that 202 is intentionally non-committal.

Note

"Non-committal" does not mean that 202 is a bad or unreliable response.

It means that the HTTP response does not commit the server to a particular eventual outcome.

Server-Side Request Forgery Attack

Traffic between services could be intercepted/tampered on the way. That is exactly why you need to:

• Reject loopback/private/link-local after DNS resolution: if statusCallbackUrl/genUploadUrl is attacker-influenced in any way (or even just misconfigured), you must resolve the hostname to an IP before connecting and check the actual IP isn't 127.0.0.1, 169.254.x.x, 10.x.x.x, 192.168.x.x, etc.

Checking the hostname string alone isn't enough because DNS rebinding lets a public-looking hostname resolve to an internal IP. This matters because TTS service is about to make an outbound HTTP call to a URL it didn't choose. That's the textbook SSRF setup.

• Allow-list: since we can enumerate valid callback hosts ahead of time (it's a known service), just allow-list them and skip most of the DNS-resolution complexity.

Callback Authentication

This is about proving the callback is legitimate when TTS service calls whoever who is on the receiving end.

When client calls POST /synthesize, it includes a token in the Authorization header, TTS service adds it as an extra header to the message published on RabbitMQ.

So when TTS service later calls back client, it echoes that same token back. Client then checks the token matches what it issued for that job before trusting the callback payload.

Why this is simpler than PAT/token exchange?

PATs and token exchange solve a different problem: "how does Service B authenticate itself as an identity to call arbitrary APIs on Service A, with scoped permissions, revocability, audit trails tied to a principal".

That's real infrastructure for a general-purpose service-to-service trust relationship. What I need here is much narrower: client service needs to verify that a callback claiming to be about jobId=123 is actually about the job it submitted, not an attacker (or a misbehaving other tenant) spoofing a callback.

That's not identity, it's a per-job shared secret, more like a webhook signing secret than an OAuth token. This is exactly the same pattern every webhook-based system uses, and none require the receiver to know which IdP the client service uses.

System Boundaries

TTS service	Client
Accepts the TTS request	Owns the request ID
Generates the audio	Dictates the output destination
Reports success/failure	Generates the presigned URL
Tries to upload	Handles the retry logic

The benefits of this system design:

• The TTS service doesn't need to know anything about the client's storage architecture.
• Simpler than having an outbox/inbox pattern. Instead of saying we should NOT reprocess the same job twice we say we do NOT care. We process it anyway. But of course this is possible since we are offloading the whole logic of retry and what if we override a newer version of text with an older one to the client which does make sense to me.

Why RabbitMQ?

• DLQ feature.
• Easy to scale by utilizing KEDA.

Tip

In TTS service when you pick up a message from RabbitMQ to process you have to think about when to NOT retry and drop the message. So when we call the callbacks and get a:

• 4xx in general = client error = "this request is malformed and retrying it unchanged won't help." A 400 on our callback because the payload doesn't match client's schema, a 404 because the job was deleted on client's side, a 401/403 because the token is wrong. None of these get better by trying again in 30 seconds. Retrying them just burns our attempt budget on something that will fail identically three times.
• 408 (Request Timeout) and 429 (Too Many Requests) are the exceptions because they're not really about our request being wrong, they're the server saying "I couldn't process this in time" or "slow down, I'm rate-limiting you." Both are explicitly transient by design (429 typically comes with a Retry-After header for exactly this reason) worth honoring that header if present rather than always waiting our fixed 30s, since Service A is telling you exactly how long to back off.
• 5xx and network errors (timeouts, connection refused) are always retryable, these mean the server (or network) failed, not that our request was invalid, so retrying is the correct default behavior for all of them, no exceptions to enumerate the way there is for 4xx.

Presigned URL Expiration

A presigned URL is intentionally temporary. The client creates the URL, so the client is responsible for choosing an appropriate lifetime.

This way we have two benefits:

1. If our TTS service if takes too long to generate and upload the audio file we can look at the logs (OTel is crucial) and see the issue. For example it might be our service whom is too slow or maybe our code is implemented poorly.
2. Client gets to choose different TTL for each presigned URL based on how many words we have, load, etc. In other words we have more room to maneuver.

Observability

For an asynchronous service like this, I'd want traces and metrics around at least these phases:

request accepted
    │
    ▼
job queued
    │
    ▼
job started
    │
    ▼
TTS generation started
    │
    ▼
TTS generation finished
    │
    ▼
upload started
    │
    ▼
upload finished

And when I say I want traces and metrics I am talking about:

• Queue wait time.
• TTS generation duration.
• Upload duration.
• Total job duration.
• Job failures.
• Redis failures.
• Upload failures.
• Expired/rejected presigned URLs.

And this is where OTel really shines.

Trace-to-Log

So I wanted to have the ability to connect a log and trace. Thus I switched to Grafana stack: Tempo for traces, Loki for logs, Grafana for visualization and OTel collector for collecting and sending logs /traces to Grafana.

And man, I love the outcome:

What About Duplicate TTS Generation?

The client decides where the generated audio should go by providing the presigned URL. In short client handles where it wanted to store the final result.

So client can generate presigned URLs for the same job to upload them to the same place, effectively overriding the old results. But this is a decision that client has to make!

Tip

Overriding is fine even if you need a durable audit trail. I mean client can handle that on its own. The TTS service should NOT concern itself with that.

┌────────────────────────────┐
| more guarantees            |      ┌───────────────────────────┐
|        │                   |      | simpler system            |
|        ▼                   |      |       │                   |
| more state                 |      |       ▼                   |
| more coordination          |  VS. | occasional duplicate work |
| more complexity            |      |       │                   |
| more failure modes         |      |       ▼                   |
|        │                   |      | client-driven retry       |
|        ▼                   |      └───────────────────────────┘
| fewer duplicate operations |
└────────────────────────────┘

When This Design Stops Being Good Enough

This approach isn't universal. For example if:

• Duplicate work is unacceptable:
  • For example the TTS generation is expensive. Expensive can mean scenarios such as:
    • Jobs can run for hours.
    • We make external API calls.
    • It is computationally intensive.
  • Or imagine your are charging customers for each TTS generation, and if we do that for no reason they will soon enough realize it.
• Every accepted request must eventually complete.
• Clients cannot safely retry.
• The output operation has irreversible side effects.
• You need exactly-once-like business semantics.

GitHub

My piper-tts-rest-api project has this architecture implemented: https://github.com/kasir-barati/piper-tts-rest-api/compare/v3.0.0...v4.0.0