Short answer: treat an email API timeout as an unknown delivery state, not a failed send. Record an idempotency key before the first attempt, separate connection, request, and delivery deadlines, and move attachment generation out of the request path. For an e-commerce order report, that combination tells you where the delay happened without sending the same report twice.
The constraint that changes the design is the attachment. A report may be generated from many orders, while the checkout or admin request should finish quickly. Waiting for rendering, upload, acceptance, and mailbox delivery in one HTTP request creates an opaque failure. I want one small synchronous operation: enqueue a durable job. Everything after that can be retried and observed independently.
How Can Email API Troubleshooting Separate a Timeout From a Rejected Response?
Almost nothing by itself. The client did not receive a response before its deadline. The server may have rejected the request, accepted it and kept working, or accepted it and returned a response that disappeared on the network path. Those states have different remedies. HTTP 408 and HTTP 504 also describe different points in the path, so neither status should be treated as proof that the message was not accepted.
Start by giving each attempt a phase. A useful event record has the report ID, order range, attachment digest, idempotency key, attempt number, phase, elapsed milliseconds, and provider response when one exists. Keep the raw message body out of ordinary logs; order reports can contain customer data.
Use separate timers for DNS and connection establishment, request transmission, response headers, and the total operation. A connection timeout points at routing, DNS, or a saturated connection pool. A slow response after the request is fully sent points at remote processing or a large payload. A quick acceptance response says nothing about inbox placement. These are different measurements.
The first trap is retrying every timeout. That is how one report becomes three emails. A retry is safe only when the receiver supports idempotency, or when the application can query the outcome with a stable message key. Otherwise, put the job in an uncertain state and reconcile it before sending again.
Keep it finite.
1. Make the send boundary small
Generate the report in a worker, store it with a short-lived reference, and hand the mail submission step a bounded payload. The web request should return a report job ID, not wait for a mailbox. The worker then records each transition.
The interface can stay boring. Boring is good here.
type SendState =
| "queued"
| "rendering"
| "submitting"
| "accepted"
| "uncertain"
| "failed";
type ReportJob = {
jobId: string;
orderFrom: string;
orderTo: string;
attachmentSha256: string;
idempotencyKey: string;
state: SendState;
};
async function enqueueReport(job: ReportJob): Promise<{ jobId: string }> {
await durableQueue.publish({
type: "order-report-email",
job,
dedupeKey: job.idempotencyKey
});
return { jobId: job.jobId };
}
The dedupe key should describe the business action, not the process attempt. For example, use the account, report period, recipient, and report revision. An attempt counter belongs in telemetry. It must not change the identity of the message.
2. Split the timeout into evidence
A single 30-second setting is convenient and nearly useless. Pick deadlines from the workflow, then expose the phase that consumed them. A report renderer might have a longer budget than a small status request, while the submission client needs a finite upper bound so workers do not pile up. The exact values belong in configuration for the workload, but the names should be stable and explicit.
Capture at least these fields:
-
connect_msandtls_msfor network setup. -
upload_msfor the request and attachment bytes. -
first_byte_msfor the remote response. -
total_msand the client deadline. -
outcomeasaccepted,rejected,timeout, orunknown.
Do not turn the fields into a dashboard graveyard. A p95 increase in upload_ms after a report-size change is actionable. A vague “email latency” chart is not.
Attachments create failure modes upstream of email. A report can be empty because the order query used the wrong timezone. It can be too large for the receiving service or downstream mailbox. It can be regenerated on every retry, changing its hash and defeating deduplication. It can also be held open by a file handle while the HTTP client is waiting for a response. These are easy troubleshooting misses because the visible symptom still says “API timeout.”
Hash the final bytes once. Record the byte count and MIME type. Validate that the archive or document can be opened before submission. Then make the worker reuse the same immutable artifact for all reconciliation attempts. This turns “it timed out” into a smaller question: did the same bytes ever reach the submission boundary?
That boundary deserves its own alarm. Suppose the report query returns 18,000 orders and the rendered file crosses the attachment limit halfway through a retry. Re-rendering inside the submission handler makes the next timeout look like a network problem, while the real defect is an oversized artifact. Measuring the byte count before upload catches that class of failure early and gives support a useful fact to act on.
A temporary object should have an explicit retention policy. Keep enough time for reconciliation and support investigation, then remove it. Do not put a public object URL in a message unless its access and expiry rules match the report's sensitivity.
3. Reconcile unknown outcomes
After a timeout, the safe state is unknown. The worker should persist that fact and run a reconciliation task keyed by the business idempotency key. If the submission system exposes a stable message identifier, look it up. If it does not, preserve the uncertain state for an operator or a controlled compensating action. The application must not infer rejection from silence.
This is where queue semantics matter. At-least-once delivery is normal for durable workers, so the handler needs a dedupe record that survives process restarts. A short in-memory set is not a control; it is a lucky cache.
Backoff should include jitter, and the retry budget should be finite. A report that has been uncertain for too long needs a visible state in the admin surface. Quietly retrying forever hides a broken integration and keeps attachment storage alive.
4. Test the failure, not just the happy path
A useful test harness delays DNS, stalls after request upload, closes the connection after acceptance, returns a duplicate response, and corrupts the attachment. Run the same job through each case. Assert that the report has one business key, that state transitions are persisted, and that a timeout does not automatically create a second message.
Test restart points too. Kill the worker after rendering, after artifact storage, and after the submission response is received but before the state update. Recovery behavior is part of the email feature.
At scale, I would add a small reconciliation store, a redacted event stream, and a sampled trace linking the report job to the submission attempt. I would also cap attachment size before the worker starts expensive rendering. The trade-off is storage and operational code. That is still cheaper than making a user guess whether a customer received a duplicate order report.
The decision rule is straightforward: optimize for evidence at each boundary. A timeout should narrow the search, preserve the possibility of acceptance, and leave one safe path to reconciliation. If the integration cannot provide that evidence, isolate it behind a queue and treat delivery state as asynchronous.
Top comments (0)