DEV Community: Yusuke Hayashi

Building an open-source Resend alternative on AWS: making accepted emails recoverable

Yusuke Hayashi — Fri, 07 Aug 2026 03:40:24 +0000

When an application asks an email API to send a password-reset link, receipt, or alert, the user sees a simple result: the request was accepted or rejected.

Behind that response, however, several systems still have work to do. The API must store the request, place work on a queue, let a worker call an email provider, and record what happened. A failure between any two of those steps can leave the application believing that an email is on its way when no worker knows about it.

I am building HayaSend, an Apache-2.0, Resend-compatible transactional email platform. An application can use the official Resend Node SDK and point its baseUrl at HayaSend. The main difference is operational: the delivery provider and data plane remain in the user's cloud account.

The purpose of the work in this article was specific:

After HayaSend accepts an email request, preserve enough durable information to recover the work without asking the application to send the request again.

This article follows one email request through that recovery path. I will introduce each moving part before discussing the AWS and DynamoDB details.

HayaSend is still early beta. The evidence here covers one tested AWS recovery path, not the production readiness of the whole platform.

The system in one diagram

The normal path looks like this:

Application
    |  Resend-compatible API request
    v
HayaSend API
    |  stores the accepted message and recovery intent
    v
DynamoDB  <--- dispatcher looks for unpublished work
    |                    |
    |                    v
    |                  Queue
    |                    |
    |                    v
    +--------------->  Worker  --->  Email provider  --->  Recipient

The terms used below mean:

API: the endpoint the application calls to request an email.
Database: the durable record of the message, recipients, and delivery state. HayaSend's tested AWS path uses DynamoDB.
Queue: a buffer of jobs waiting for a worker. It lets the API return without waiting for the provider to finish.
Worker: code that takes a queued job and continues delivery.
Provider: the external service that ultimately accepts the email for delivery.
Dispatcher: HayaSend code that finds durable work in DynamoDB and publishes it to the queue.

The database answers “what did we accept?” The queue answers “what should a worker do next?” The reliability problem appears when those answers get out of sync.

The failure I wanted to prevent

A straightforward implementation performs two separate operations:

1. Save the email request in DynamoDB
2. Publish a send job to the queue

Now imagine a password-reset request:

DynamoDB successfully stores the message.
The API process stops before queue publication finishes.
The application has already received an accepted response, so it does not retry.
The database contains the request, but the queue contains no job.
No worker sends the reset email.

This is a dual-write problem: one logical operation depends on successful writes to two different systems.

A client idempotency key solves a related but different problem. It can stop repeated API requests from creating several logical messages. It cannot repair work when the application never repeats the request.

The pattern is established; the implementation choices are not

I did not invent the solution pattern. AWS Prescriptive Guidance already documents the transactional outbox pattern, including the dual-write problem, possible duplicate messages, ordering, and the need for idempotent consumers.

An outbox is a durable “work still needs to be published” record stored with the business data. Instead of trying to update DynamoDB and the queue as one impossible cross-service transaction, the API commits the message and its outbox item together in DynamoDB. A dispatcher can publish the queue job later.

The pattern gave me the starting point. It did not decide:

which HayaSend records must be committed together;
whether the public recipient limit fits in one DynamoDB transaction;
how repeat publications keep the same logical identity;
how multiple dispatchers compete safely for work;
what to record when an email provider may have accepted a request;
how to observe recovery without exposing email data.

Those are the implementation decisions I made and tested.

Step 1: make “accepted” mean “recoverable”

When HayaSend accepts a delivery, one DynamoDB TransactWriteItems operation writes:

one message metadata record;
one provider-neutral delivery record;
one record per recipient;
one optional idempotency claim;
one initial outbox item;
one durable backlog counter update.

These records either commit together or do not become visible at all.

For a request with HayaSend's public maximum of 50 recipients, a focused v0.3.11 test observed 55 transaction actions:

1 message + 1 delivery + 50 recipients + 1 idempotency claim
+ 1 outbox item + 1 backlog counter = 55 actions

Here, an “action” is one item-level operation inside the DynamoDB transaction. AWS currently allows up to 100 actions in TransactWriteItems, as documented in DynamoDB constraints and the TransactWriteItems API reference.

The limit itself is ordinary AWS knowledge. The useful project result was checking the real HayaSend record model at the real API limit, including the five non-recipient actions that are easy to forget.

This changes the meaning of API acceptance. It still does not mean “the recipient received the email.” It means “the message and the intent to continue delivery were durably committed together.”

Step 2: recover without replaying the API request

After the transaction commits, the dispatcher searches for due outbox items and publishes queue jobs.

If the API process stops before publication, the outbox item remains in DynamoDB. A later dispatcher can find it. The application does not need to repeat the original send request, and the API does not need to recreate the message from logs.

A simplified recovery sequence is:

API commits message + outbox item
             |
             X  process stops before queue publication

Later dispatcher finds the outbox item
             |
             v
        publishes queue job
             |
             v
     marks outbox item dispatched

The durable source of truth is the outbox item. A scheduled trigger can wake the dispatcher, but the schedule itself is not proof that the work still exists.

Step 3: give repeated publications one logical identity

The dispatcher can also fail after the queue has accepted a job but before DynamoDB records the successful publication:

1. Dispatcher acquires the outbox item
2. Queue accepts the job
3. Dispatcher stops before acknowledging the outbox item
4. The lease expires
5. Another dispatcher publishes the item again

The queue may therefore contain a duplicate. The transactional outbox does not provide exactly-once delivery.

HayaSend derives the job ID from the message, job type, and generation instead of creating a new random ID for each publication attempt. Both queue messages then describe the same logical job. Downstream code can use that identity in deduplication and conditional state changes.

This is a deliberately limited guarantee: duplicates may exist, but retries keep the same name.

Step 4: use an index to find work, not to decide ownership

Several dispatchers may run at the same time. They need to avoid treating the same outbox item as exclusively theirs.

HayaSend uses a DynamoDB global secondary index (GSI) to find items whose due time or lease expiry has passed. A GSI is an alternate lookup view of the table, but its contents can briefly lag behind the base table. AWS explains in its read consistency guide that GSI reads are eventually consistent.

For that reason, an index result is only a candidate. It does not grant ownership.

The dispatcher performs a conditional update on the base-table item to acquire a short-lived lease—a claim that says, in effect, “this dispatcher may work on the item until this time.” If another dispatcher has already changed the item, the conditional update fails.

On publication failure, HayaSend releases the lease and makes the item due again. On success, it acknowledges publication and adjusts the backlog together.

Step 5: admit when the provider result is unknowable

Repairing the database-to-queue boundary reveals another failure window later in the path:

Email provider accepts the request
            |
            X  network response is lost
            |
HayaSend cannot confirm the result locally

Automatically retrying may produce a duplicate email. Reporting success may hide a lost request. Reporting an ordinary failure incorrectly claims that the provider rejected it.

HayaSend records this result as ambiguous: the provider may have accepted the request, but the local system could not confirm and commit the outcome.

The outbox cannot remove this uncertainty. Recovery depends on the provider's own idempotency or correlation features. The important design choice was to preserve the uncertainty instead of converting it into a more convenient but unsupported answer.

Step 6: observe recovery without exposing email content

Operators need to know whether delivery work is stuck. They should not need recipient addresses, subject lines, or message bodies to answer that question.

HayaSend's default outbox diagnostics expose aggregate operational facts such as:

counts of due, leased, expired, and undispatched items;
the age of the oldest due item;
cumulative publication failures;
whether a bounded diagnostic query was truncated.

They exclude addresses, subjects, bodies, raw provider responses, queue endpoints, credentials, and signed URLs.

Recipient and attempt IDs are random opaque values rather than values derived from an email address. Provider events use an opaque event ID or a digest of normalized allowlisted fields instead of copying the raw provider payload.

The goal is not zero observability. It is enough observability to operate the recovery loop without making logs and metrics another store of customer email data.

What I actually tested on AWS

I checked the v0.3.11 implementation at three levels:

Question	Check	Observed result
Does the record model fit the transaction limit?	Maximum-recipient focused test	50 recipients produced 55 actions
Do the model, outbox, DynamoDB adapter, and workflow agree?	Six focused test files	43 tests passed
Can a committed item be recovered in a deployed AWS stack?	Public integration workflow	Outbox recovery and acknowledgement succeeded

The deployed evidence is GitHub Actions run 30498672002.

In its recovery step, the workflow:

used a delivery already created by the API probe;
made that delivery's durable outbox item due;
invoked the deployed dispatcher;
required the item to gain dispatched_at;
required its lease and pending-index fields to be removed.

The workflow did not replay the client's API request and was designed not to send an actual email through Amazon SES. It deployed an ephemeral stack in a dedicated account, used GitHub OIDC instead of a long-lived AWS access key, and removed the test resources afterward.

The run used commit 071c2a3, an ancestor of v0.3.11. I compared the run commit with the release tag: the relevant outbox implementation, focused tests, and recovery step were unchanged; the workflow changes were tool-version pins.

This evidence supports one narrow claim: after an API-created delivery was committed, the deployed AWS dispatcher could recover and acknowledge its outbox work without another client request.

It does not prove that every email is delivered, that every provider behaves identically, that every deployment pack has the same evidence, or that an early-beta platform is production-ready.

The practical takeaway

AWS's transactional outbox guidance answered the architectural question: how can database state and later queue publication survive a crash between systems?

Implementing it for an email API required more concrete answers:

Define exactly what “accepted” guarantees.
Count every transaction action at the public API limit.
Preserve one logical job identity across publication retries.
Let a conditional base-table write—not an eventually consistent index—grant a lease.
Represent possible provider acceptance as an explicit uncertain state.
Test the failure path on deployed infrastructure without sending customer email.
Make recovery observable without copying sensitive payloads.

If you operate a database-to-queue workflow, draw the normal path first, then mark every place where one system may have accepted work while the next system has not recorded it. Those gaps are where recovery state and honest guarantees matter most.

Which boundary is least explicit in your system today: API acceptance, queue publication, or provider acknowledgement?

HayaSend's provider-neutral state model is documented in delivery-model.md, and the deployed test is described in aws-integration-testing.md.

Disclosure: I used an AI assistant to organize public source material and edit this article. I reviewed the AWS documentation, HayaSend v0.3.11 implementation and focused tests, generated-contract check, source comparison, and public AWS run described above on August 7, 2026.

How Much Can the OpenTelemetry Collector's Persistent Queue Really Protect? Measuring SIGKILL and Lost ACKs

Yusuke Hayashi — Thu, 06 Aug 2026 19:10:35 +0000

The traces produced while your observability backend was down are exactly the ones you need
for the postmortem. But what happens if the OpenTelemetry Collector relaying that data
restarts at the same time as the backend?

The OpenTelemetry Collector (hereafter "Collector") is open source software that receives
traces, metrics, and logs, applies whatever processing you need, and forwards them to an
observability backend. It ships as a
binary implemented in Go
(architecture docs) and as a
container image. You wire three kinds of components together in YAML:

Receiver: takes data in from applications and other sources
Processor: batches, adds attributes, filters, and otherwise transforms data
Exporter: sends data to destinations such as an observability backend

In this article all data moves over OTLP (OpenTelemetry Protocol), the standard protocol for
sending OpenTelemetry traces, metrics, and logs. The experiments use OTLP/gRPC.

Exporters have a sending_queue that temporarily holds data that cannot be sent yet. You can
keep queue contents in memory only (a "memory queue") or write them to disk with
file_storage (a "persistent queue").

For this article I stopped the destination, let data pile up in the queue, and then either
gracefully or forcibly terminated the Collector. I also built a failure in which the
destination loses its success response (ACK) immediately after durably storing the data, and
counted loss and duplication by Span ID.

Conclusion first: what a persistent queue actually protects

In OpenTelemetry, the whole flow of one request through multiple operations is recorded as a
"trace", and each individual operation as a "span". The input for these experiments was:

What is counted	Count	What it means here
Traces	500	500 application request executions
Spans per trace	2	one parent span and one child span
Spans	1,000	the data actually checked for loss/duplicates
Spans per OTLP request	100	the unit sent to the Collector
OTLP requests	10	how the queue holds them: 10 requests

So "500 traces", "1,000 spans", and "10 requests" throughout this article are the same input
counted in different units. Each span carries a Span ID, and those IDs are what I use to count
loss and duplication.

There are two ways I stopped the Collector:

Signal	Meaning
`SIGTERM`	requests a graceful shutdown; the Collector can run shutdown logic
`SIGKILL`	no chance to shut down; the OS terminates the process immediately

I ran six ordinary conditions plus one ACK-loss condition, three times each. Counted at the
destination:

Queue	Failure condition	Span records stored	Unique Span IDs	Duplicates	Lost
Memory	destination stopped, then recovered	1,000	1,000	0	0
Memory	`SIGTERM` while destination is down	0	0	0	1,000
Memory	`SIGKILL` while destination is down	0	0	0	1,000
Persistent	`SIGTERM` while destination is down	1,000	1,000	0	0
Persistent	`SIGKILL` while destination is down	1,000	1,000	0	0
Persistent	queue files lost after `SIGKILL`	0	0	0	1,000
Persistent	stored, then died before ACKing the Collector	1,100	1,000	100	0

"Duplicates" is the number of extra records stored relative to unique Span IDs. In the last
condition, 100 Span IDs were each recorded twice, so the total record count came to 1,100.

The most surprising result was the memory queue under SIGTERM. The Collector logged
Shutdown complete. and exited with code 0 — a clean shutdown — but because the destination
was still down it could not drain the queue, and nothing arrived after the restart.

Three takeaways:

A memory queue survives a temporary destination outage, but nothing carries over when the Collector itself restarts.
A persistent queue can resend after a Collector restart, but only when it can reattach to the same stored data.
Losing the destination's success response triggers a resend, so you can get duplicates, not just loss.

A persistent queue widens the range of Collector restarts you can resend through. It is not an
end-to-end guarantee that everything the Collector accepted reaches the backend exactly once.

To see why these results came out this way, let's look at the setup, then pin down what
"successfully sent" actually refers to, and then walk through each failure.

What is under test: app → Gateway → backend

The same Collector binary can be deployed in different roles. The official docs call running it
as a central OTLP endpoint the
Gateway deployment pattern. The
gateway container in this article is just the official Collector image configured in YAML.

Persistent-queue resilience matters most in exactly that shape: applications send OTLP to a
central Gateway Collector, and the Gateway relays to the observability backend.

When backend maintenance or a network failure coincides with a Gateway rolling update, pod
eviction, or process crash, the question becomes whether traces accepted during the incident
survive into the next process.

Gateways are typically deployed as a Kubernetes Deployment or a service on a VM, and are used
for centralized concerns: credentials, egress consolidation, filtering, sampling. They are not
mandatory — small environments, or setups where each language SDK (the library that emits
OpenTelemetry data) sends straight to the backend, can skip them.

The subject under test is a single Collector with a queue and storage area shared with no
other replica. Load balancers, queue sharing across replicas, and SDK-side retries are out of
scope.

Here is how real components map to the test environment. telemetrygen is OpenTelemetry's
official tool for generating test traces and other signals.

Real system / subject	Test-environment stand-in	Not measured
Application / SDK	OTLP/gRPC from the official `telemetrygen` image	SDK batching, retries, language variation
Gateway Collector	official Collector image configured in YAML	multiple replicas, load balancing
Observability backend	`file` exporter, or a purpose-built Go sink	indexing, search, backend durability
Fault injection and tally	Docker Compose, POSIX shell, `curl`, `jq`	Kubernetes, host-level failures

Because the measurement starts once the Gateway has accepted OTLP/gRPC, the source language is
not fixed. Whether you send from Java, Go, or Python, the fault injection and counting done
here are unchanged. A setup that sends from the SDK directly to the backend has no Collector
queue at all, so you have to check that SDK's own queueing and retry behavior separately.

From here on, the Collector that receives from the application and forwards is the "Gateway",
and the Collector or Go server that stores what it receives is the "Sink". The Sink stands in
for the observability backend in this experiment.

  telemetrygen (app stand-in, 500 traces)
        |  OTLP/gRPC
        v
  Gateway Collector (single instance)
        |
        v
  Exporter sending_queue  ---- file_storage (persistent queue only)
        |
        x  (cannot send while the Sink is down)
        v
  Sink (backend stand-in)

That covers the components. Next, let's split up how far along the path data has to get before
you can call it "sent". That distinction is the foundation for understanding the duplicates
caused by a lost ACK later on.

Background: splitting "it was sent" into four boundaries

An application can succeed at sending without the data being stored in, or searchable from, the
backend. To keep those look-alike states apart, here are four boundaries along the delivery
path.

Boundary	Evidence measured	What it lets you claim
Gateway accepted	`otelcol_receiver_accepted_spans=1000`	the pipeline accepted 1,000 spans
Enqueued / in flight	`otelcol_exporter_queue_size=10`	10 requests occupied queue capacity
Send to Sink succeeded	`otelcol_exporter_sent_spans=1000`	1,000 logical spans obtained a downstream ACK
Sink wrote to file	500 unique Trace IDs / 1,000 unique Span IDs / 0 dupes	the spans are in the `file` exporter's JSON

Two different success responses (ACKs)

There are two OTLP success responses in this article that are easy to conflate. ACK is short for
acknowledgement — here, the success response returned by whoever received a request.

  telemetrygen --(1) upstream OTLP request--> Gateway
  Gateway      --(2) enqueue--------------->  sending_queue
  Gateway      <-(3) upstream ACK----------   (returned to telemetrygen)
  sending_queue--(4) downstream OTLP request-> Sink
  Sink         --(5) downstream ACK--------->  Gateway

Name	Direction	What success means in this experiment
Upstream ACK	Gateway → telemetrygen	the Gateway pipeline accepted the data and enqueued it
Downstream ACK	Sink → Gateway	the OTLP request from Gateway to Sink succeeded

I explicitly set the exporter helper's wait_for_result to false. With that setting, the
upstream request does not wait for the exporter to finish sending to the Sink. So an upstream
ACK does not mean the Sink stored, indexed, or can search the data.

What the later ACK-loss experiment destroys is the downstream ACK from Sink to Gateway. Even
when the Sink has already stored the data, a Gateway that never receives the downstream ACK will
resend the same request, and duplicates can result.

otelcol_exporter_sent_spans counts logical spans that eventually got a downstream ACK. It is
neither the total number of send attempts nor the total records the Sink stored. In the ACK-loss
experiment, sent_spans stayed at 1000 while the Sink's stored records reached 1,100 spans.

It matters not to conflate the
"accepted by the server" boundary in
the OTLP spec with everything downstream of it: storage, indexing, and searchability.

What the Sink actually measured The Sink's `file` exporter was used only for counting records. Its [stability for traces is alpha as of 0.157.0](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.157.0/exporter/fileexporter). The tally scripts are written to fail if the JSON structure changes. Nothing here measures a real backend's index build, searchable-at time, or durability.

With four delivery boundaries and two ACK directions established, let's pin down when each
failure was injected and which evidence decided success or failure.

Method: seven failure conditions and how they were judged

Testing ran 2026-07-25 to 2026-07-27. I used the official Contrib distribution, which bundles
many extra components including file_storage, pinned to
OpenTelemetry Collector 0.157.0.

The ordinary conditions inject failure in this order:

Do not start the Sink, so the Gateway cannot send.
Send 500 traces and wait until the Gateway's accepted count reaches 1,000 spans.
Wait until queue usage reaches 10 requests.
Depending on the condition, leave the Gateway running or stop it with SIGTERM / SIGKILL.
Recover the Sink and the Gateway, then count the stored Trace IDs and Span IDs.

For conditions judged as total loss, I waited 10 seconds after Sink recovery — longer than the
5-second maximum retry interval — and confirmed the destination was still empty. Rather than
waiting a fixed time before injecting a failure, each step proceeds only after the accepted count
and queue usage are confirmed.

Only the ACK-loss condition starts the Sink from the beginning. It stops right after storing the
first OTLP request, so no downstream ACK is returned.

This is a low-volume functional test on a single machine, not a performance test. 500 traces is a
number chosen to make loss and duplication easy to count, not an assumed production rate. No
external telemetry backend or credentials are required.

Measured results and hypothetical requirements are kept separate.

The Collector configuration, fault injection code, counts, logs, and CSVs all come from running
the published code locally. The production requirements used as examples later — "tolerate a
10-minute outage" and so on — are hypothetical, for illustration.

Test environment and what was not tested The [captured environment details](https://github.com/yhay81/otel-collector-resilience-lab/blob/main/results/2026-07-28-environment.txt) are saved as well. Prerequisites and the measured environment: | Category | Prerequisite / measured environment | | ------------------ | ------------------------------------------------------------------------------ | | Host | macOS 26.5.1 / Apple M2 Pro / 12 CPU / 32 GiB | | Docker VM | arm64 / 12 CPU / approx. 15.6 GiB | | Container platform | Docker Desktop 4.81.0 / Engine 29.6.1 / Compose v5.2.0 | | Host-side CLI | POSIX shell, `curl`, `jq` | | ACK-loss Sink | Go 1.25.12 / OTLP proto 1.11.0 | | Local resources | TCP 4317/8888 free, and the test code's `data/` writable | | Network (first run)| image pulls from GHCR and Docker Hub, Go module fetch for the ACK-loss Sink | Non-arm64 CPUs, Linux, and minimum CPU/memory/disk requirements were not tested.

The input is 500 traces, 1,000 spans, and 10 requests, with a queue limit of 100 requests.
telemetrygen's --child-spans 1 makes each trace two spans, and --batch-size 100 sends 100
spans at a time. Additional batching inside sending_queue is disabled; retries are unlimited
with a 5-second maximum interval.

The full reproduction set lives in the
public repository with the test code and measured results.
The scripts contain not only the experimental conditions but the tallying logic and failure
conditions too.

The six non-ACK-loss conditions and the ACK-loss condition run with:

git clone https://github.com/yhay81/otel-collector-resilience-lab.git
cd otel-collector-resilience-lab
./run.sh
./run-ack-loss.sh

Three trials each are recorded in the
ordinary-conditions CSV,
the ACK-loss CSV,
and the metrics and logs.
From here, the results follow this order: destination-only outage, Gateway also terminated, ACK
alone lost, and storage itself lost.

Result 1: a memory queue resent fine when only the destination stopped

First, the Gateway kept running and only the Sink was stopped. After sending 500 traces and
waiting for the internal metric otelcol_exporter_queue_size to reach 10, I started the Sink.
All 500 traces / 1,000 spans arrived, with zero duplicate Span IDs.

The memory queue configuration used ```yaml exporters: otlp_grpc/sink: endpoint: sink:4317 tls: insecure: true sending_queue: enabled: true sizer: requests queue_size: 100 num_consumers: 1 wait_for_result: false block_on_overflow: false batch: enabled: false retry_on_failure: enabled: true initial_interval: 1s max_interval: 5s max_elapsed_time: 0s ```

sending_queue acted as a temporary holding area while the destination was down, and
retry_on_failure resent with a gradually widening interval. Because the Gateway process kept
running, the 10 in-memory requests were still there.

So a memory queue prepares you for a backend restart or a brief network interruption. Stopping
the Collector itself is a different story.

Result 2: nothing carried over from the memory queue after the Collector stopped

With the same 10 requests in the memory queue, I sent SIGTERM to the Gateway — in practice
docker compose stop -t 10 gateway. The Collector received terminated, logged
Shutdown complete. within the same second, and exited with code 0. It did not fall through to
the forced kill after 10 seconds.

Even so, not a single trace arrived at the Sink. Shutdown stopped retrying, attempted to drain
the queue, and dropped the 100-span requests it could not deliver to the stopped Sink.

Gateway log on SIGTERM ```text Received signal from OS signal: terminated Starting shutdown... Exporting failed. Dropping data. dropped_items: 100 Shutdown complete. ```

Under SIGKILL in the same condition, no trace arrived either — the data awaiting retry vanished
along with the Collector process.

The conclusion here is not "SIGTERM always loses data". If the destination is up and the data
can be sent during shutdown, the outcome can differ. What this measures is that when you
terminate the Collector while the destination is still down, even a graceful shutdown cannot
carry the memory queue over to the next process.

The reason for the loss is that the queue existed only in the Collector process's memory. So
next, the same data is written to a file outside the process before the same shutdowns are
repeated.

Result 3: a persistent queue resent after both graceful and forced termination

file_storage is an extension that persists queue data to local files. Point an exporter's
sending_queue.storage at that extension and it uses a persistent queue instead of a memory one.

Configuration diff for switching to a persistent queue ```yaml extensions: file_storage/queue: directory: /var/lib/otelcol create_directory: true fsync: true exporters: otlp_grpc/sink: sending_queue: storage: file_storage/queue service: extensions: [file_storage/queue] ```

The endpoint, queue limit, send concurrency, batching, and retry settings are identical to the
memory queue. The only substantive change is where the queue lives.

The Gateway's /var/lib/otelcol was bind-mounted to a host directory, so recreating the Gateway
container still reads the same queue files. Under SIGTERM I confirmed exit code 0 and
Shutdown complete.; under SIGKILL, exit code 137. Both delivered all 500 traces / 1,000 spans
after restart, with zero duplicate Span IDs.

Queue files and logs at restart `file_storage` uses bbolt, an embedded database, internally. Just before `SIGKILL` there was a bbolt file holding the 10 requests. The restart logs show the persistent queue metadata being loaded and the one in-flight request being moved back to the queue. `readIndex: 1` does not mean only one request was stored: `itemsSize: 1000` is the number of retained spans, and `numberOfItems: 1` is the number of requests that had been handed to a consumer at crash time. ```text Loaded queue metadata readIndex: 1, writeIndex: 10, itemsSize: 1000 Moved items for dispatching back to queue numberOfItems: 1 ```

This resume behavior and the ID tallies were consistent across three trials, and match the
behavior documented in the
official file_storage README.

fsync: true asks the OS to sync to disk on every write, improving database consistency across
an interruption — at the cost of write performance. The default is false, so decide based on
the durability you need and measured performance under real traffic. What I compared here is
only SIGKILL against the process: no performance or loss comparison against fsync: false, and
no host power loss, Docker Desktop VM, filesystem, or storage-controller durability. fsync: true
alone is not grounds for claiming power-loss resilience. The file_storage extension itself is
also beta as of 0.157.0. Pin the version, and re-run the same fault injection tests against the
release notes when you upgrade.

So far, when the stored data survives, the Collector can resend after a restart. But no loss does
not imply no duplication. Next, the downstream ACK boundary gets broken.

Result 4: losing the downstream ACK duplicated 100 spans

The Gateway cannot know whether the Sink finished storing unless it receives the downstream ACK.
So this experiment destroys only the downstream ACK from Sink to Gateway.

The purpose-built ACK-loss Sink writes the 100 spans in the first OTLP request to a JSONL file
(one record per line) and fsyncs it. Then, before returning a success response, it exits with
code 23. Docker Compose restarts just the Sink, and the Gateway — having never received a
downstream ACK — resends the same request.

  Gateway --(1) OTLP request, 100 spans--> crash-before-ACK Sink
                                            |
                                            +--(2) append + fsync--> Sink storage file
  Gateway <-X(3) process exits before responding
  Gateway    (4) Unavailable / EOF: back to the queue, retry
  Gateway --(5) resend the same request---> Sink
                                            |
                                            +--(6) append the same 100 spans again
  Gateway <--(7) OTLP success---------------+

All three trials agreed:

Observation	Value (3 trials)
Logical spans the Gateway accepted from upstream	1,000
Logical spans for which the Gateway got a downstream ACK	1,000
Queue usage after recovery	0
Span records stored at the Sink	1,100
Unique Span IDs at the Sink	1,000
Span IDs recorded twice	100
Successful resends of the request holding the first 100 spans	1

All 1,000 unique spans arrived, so nothing was lost. But the first 100 spans were recorded twice
— before the crash and after the resend — bringing stored records to 1,100.

The Sink's logs show the same request hash appearing both in the pre-crash store and in the
post-restart ACK.

ACK-loss Sink log ```text persisted request sha256=0642...d96d2e spans=100; exiting before OTLP response ack-loss-sink ready on :4317 acknowledged request sha256=0642...d96d2e spans=100 ```

On the Gateway side, rpc error: code = Unavailable ... EOF was logged, followed by a retry about
1.1 seconds later. The exporter counted 1,000 spans as successful after the resend, but the Sink
processed 1,100 spans in total.

The OTLP spec is explicit that a disconnect before the ACK can produce duplicates via
retry. This experiment reproduces that
ambiguous window using an identical request hash and identical Span IDs. It is evidence of
duplication across a process crash in a purpose-built Sink — not proof of host power-loss
durability or of any real backend's deduplication.

Up to this point the queue data needed for the resend still existed. Next: losing that storage
itself.

Result 5: losing the stored data meant even a persistent queue could not resend

Despite the name "persistent", once the queue files themselves are gone there is nothing to
resend. In this condition I confirmed 10 requests in the persistent queue, sent SIGKILL to the
Gateway, then deleted the queue directory data/queue before restarting.

This models complete loss of the storage contents. Kubernetes is not involved, so nothing here
tests pod rescheduling to another node or PersistentVolume reattachment. It is not an experiment
showing "a node failure always means total loss". Whatever the reason for the failure, this shows
what happens when the original queue data cannot be read after a restart.

In all three trials, not a single trace arrived at the Sink. file_storage can resume delivery
because it re-reads the remaining queue files. Having persistence configured and being able to
reach that same persisted data after a failure are two different requirements.

On Kubernetes, post-failure behavior differs by storage type. A PersistentVolume (PV) is storage
with a lifetime independent of the pod.

Storage	Caveat
`emptyDir`	survives container restarts, but disappears with the pod
Local PV	bound to a specific node's disk, so unusable as-is on another node
Network PV	verify reattachment on another node, plus I/O latency and AZ failures for real
External queue such as Kafka	decouples storage from the Gateway, but adds a new operational surface

The range of failure types you intend to protect against is what I call the "failure boundary"
here. If the boundary you accept is process crashes, file_storage plus a reattachable volume is
a candidate. If you need to keep accepting data through node or AZ (availability zone) loss, a
Collector's own persistent queue is not enough.

Storage loss is one limit of persistent queues. There are others, where data is lost before
storage or during resend. Here is where the measured and unmeasured ranges separate.

Limits of this test: conditions a persistent queue does not cover

Reading these results as "file_storage means you never lose anything" is dangerous. What a
persistent queue protects is resuming, after a Collector process restart, a queue that was
written to durable disk. At least these four conditions are outside that protection.

Unprotected condition	What happens	Main countermeasure
Queue limit reached	with `block_on_overflow: false`, enqueue fails	size capacity; verify upstream failure and retry
Retry deadline exceeded	requests past `max_elapsed_time` are dropped	set the deadline against your recovery objective
Disk failure/exhaustion	the persistent queue cannot be written	monitor capacity, I/O, and errors
Stored data lost	the original queue files cannot be read after restart	give each pod a reattachable persistent volume

Of those four, only stored-data loss was measured. With this pipeline configuration, enqueue
failures are returned as errors on the upstream OTLP request. If upstream can resend, that is not
immediately data loss — but ignoring the error is. max_elapsed_time: 0s is likewise a setting
chosen to isolate restarts for comparison. Unlimited retention keeps old data around and can fill
the queue or disk first. The official default is 300 seconds.

Also untested:

Performance and loss compared against fsync: false, host power loss, filesystem corruption
Kubernetes pod deletion, node shutdown, PVC reattachment, CSI failover
Real backend acceptance, index build, searchable-at time, deduplication
Production-equivalent throughput, 10-minute outages, queue drain within 15 minutes, resource usage

So far this has mostly covered everything after the Gateway accepted the data. But the real
delivery path starts at the application. Let's look upstream.

The application side: from SDK to Gateway

Collector and PersistentVolume configuration may belong to a platform team. But you cannot think
about loss between the application and the backend without the SDK's behavior.

This experiment measures only what happens after the Gateway accepted the data. In a real system,
also verify the following between the application and the Gateway:

How the SDK batches spans, and how much it can hold in memory
Whether the SDK retries or drops when it cannot connect to the Gateway or gets an error back
Whether application shutdown waits for unsent data, and for how long
After a success response from the Gateway, when the data becomes searchable in the backend
How the backend displays and aggregates the same Span ID arriving more than once

Each language's OpenTelemetry SDK configures batching, queueing, retry, and shutdown differently.
Do not treat "the application's send API returned success" as the delivery-complete condition —
test the SDK and backend you actually use, together.

Once the application-side boundary is settled, the Gateway side turns "how much downtime do we
absorb" into concrete capacity. Logical capacity and disk capacity are separate.

Capacity planning: separate `queue_size` from disk capacity

Sizing a persistent queue happens in two stages. First derive the Collector setting queue_size,
then translate that into disk capacity for a PersistentVolume or similar.

Logical capacity: derive `queue_size` from the outage you must survive

Rather than copying a sample value like queue_size: 1000, decide first how long a destination
outage you want to survive. A rough formula:

required queue_size = peak enqueue units/sec x tolerated outage seconds x safety factor

What an "enqueue unit" is depends on the sizer setting.

`sizer`	What `queue_size` counts	Characteristics
`requests`	number of requests entering the queue	lowest computational cost
`items`	spans, data points, log records, etc.	easiest to match against record counts
`bytes`	serialized data size	direct size control, highest computation cost

The exporter helper 0.157.0
documentation likewise describes requests as the lightest and bytes as the most
computationally expensive option. Base the calculation not on the application's send count but on
the units that actually enter the queue after processors and exporters have done their work. If
you pick requests, measure spans-per-request as well.

Disk capacity: derive it from actual file growth

The queue_size from the formula above is a logical ceiling, not the byte count a
PersistentVolume needs. Even with sizer: bytes, bbolt's bookkeeping and space that remains
allocated after release mean the queue file size will not match exactly.

Accumulate production-like traces, metrics, and logs for the required duration and measure the
following to set disk capacity and free-space alert thresholds:

File growth before and after enqueueing
How much file size remains after the queue drains
Changes from file_storage compaction settings
I/O latency in steady state and during recovery
Safety margin including filesystem reserved space

Once the queue is in place, monitor at least:

otelcol_exporter_queue_size and otelcol_exporter_queue_capacity
Enqueue failures such as otelcol_exporter_enqueue_failed_spans
Send-attempt failures such as otelcol_exporter_send_failed_spans
Persistent volume usage, free space, I/O latency, and file_storage error logs

These Collector internal metrics are alpha as of 0.157.0; recheck names and attributes when
upgrading. Detect rising utilization together with send failures, before the queue fills. And do
not put the Collector's own metrics solely on the same delivery path — keep an independent health
check.

That covers failure boundaries, capacity, and monitoring — the inputs to an adoption decision.
Finally, let's turn these individual results into a production checklist you can pass or fail item
by item.

Adoption decision: turning measurements into a production checklist

As an example, take a hypothetical requirement: "survive a 10-minute destination outage and one
forced Gateway termination, and drain the queue within 15 minutes of recovery." Do not treat this
scaled-down experiment as production certification; verify the incomplete items below under
production-like conditions.

Check item	Pass condition	Evidence from this test
Temporary outage	no loss of accepted traces	memory and persistent both pass
Graceful shutdown	no loss when terminating/restarting while destination is down	persistent only
Process kill	no loss after `SIGKILL`	persistent only
Storage loss	declare out of scope, or decide an upstream recovery path	deleting stored data: total loss
Volume reattachment	the same queue is re-read after pod/node rescheduling	not tested
Capacity	holds 10 minutes at peak throughput plus a safety factor	needs production-rate testing
Recovery time	queue returns to normal within 15 minutes of recovery	needs production-rate testing
Security	permissions, encryption, and deletion of stored data verified	review per environment

The decision is not a binary "adopt or skip the persistent queue". Adoption judgment means
separating the process boundary measured here from the node and storage boundaries not yet
verified, and deciding who accepts the remaining risk.

Building a checklist is not proof that things work in a real environment. The last step is to
inject the failures you expect into a pre-production environment and confirm recovery matches
expectations.

Summary: what to confirm before adopting a persistent queue

Enabling a persistent queue does not by itself mean "we now have a configuration that never loses
data". What matters is confirming, in your actual setup, which failures keep the queue intact and
allow delivery to resume afterwards.

From these tests, four things to nail down before production:

A success response at the application or the Gateway does not mean the data is searchable in the backend at that moment. Verify acceptance and storage/search separately.
Restarting the Collector while the destination is down does not carry memory queue contents to the next process, even on a graceful shutdown.
With file_storage, do more than enable the setting: build a storage arrangement that can re-read the same queue data after a restart.
Resends can produce duplicates. Beyond loss, confirm how the backend handles duplicates and how you would detect them.

Once the configuration is settled, try each failure one at a time: destination down, SIGTERM,
SIGKILL, queue limit reached, disk exhaustion. Do not let testing end at "the application sent
successfully". Make the completion criterion for a delivery test the ability to actually search
traces generated during the incident after recovery, and to explain both whether anything was lost
and how duplicates appear.

References

Tested 2026-07-25 to 2026-07-27.

This article was originally published in Japanese on Zenn.

A SELECT-only prompt is not a sandbox: bounding agent-generated SQL

Yusuke Hayashi — Thu, 06 Aug 2026 18:16:52 +0000

Suppose an AI agent has one job: read package.json and return the package name and version.

You can put “use only SELECT” in the prompt. But if the SQL engine can still open another file, load an extension, run forever, or overwrite an output path, the prompt has not created a security boundary. It has only described one.

I ran into this distinction while building sqrail, a small DuckDB-based executor for analytical SQL over explicitly named files. The useful lesson was not about generating better SQL. It was about turning the agent's request into a process contract that remains true when the request is wrong.

Five promises a prompt cannot enforce

Prompt-level intention	Executor-level boundary
“Read only this dataset”	Canonical file allowlist plus disabled external access
“Only run a query”	One parser-validated `SELECT` statement
“Do not use too much”	Deadlines and row, byte, file, thread, memory, and spill limits
“Do not overwrite anything”	No-replace output commit
“Tell me what failed”	Stable JSON diagnostics and exit classes

The second column is where enforcement starts.

Bind data instead of giving SQL a filesystem

The agent refers to a logical table name, while the host supplies the path:

sqrail run -t pkg=package.json --timeout 5s "SELECT name, version FROM pkg"

On sqrail 0.3.4, that produced:

{"name":"sqrail-site","version":"0.3.4"}

The executor canonicalizes every bound path. Globs and partitioned Parquet directories are expanded, sorted, deduplicated, and required to be non-empty. After binding, only those exact files are allowlisted and DuckDB external access is disabled.

I then tried to bypass the binding and read the same file directly:

SELECT * FROM read_json_auto('package.json')

The v0.3.4 Windows release rejected it with exit 4 and a QUERY_FAILED diagnostic whose message began Permission Error: Cannot access file.

That generic error code is worth noting: the important guarantee is the denied access, not a specially branded “sandbox” message.

Parse the statement; do not pattern-match text

Checking whether a string begins with SELECT is not sufficient. Comments, multiple statements, and other syntax make text-prefix rules brittle.

The v0.3 contract accepts exactly one parsed statement whose type is SELECT. That includes VALUES and queries beginning with WITH. It rejects DDL, DML, COPY, ATTACH, INSTALL, LOAD, PRAGMA statements, and multiple statements. Extension autoloading, automatic installation, and community extensions are disabled before configuration is locked.

This probe:

SELECT 1; SELECT 2

returned exit 4 with MULTIPLE_STATEMENTS.

This is still not a general sandbox for a hostile SQL engine. It is a deliberately narrower contract: one read-only analytical statement over files the caller explicitly bound.

Give the agent a preflight lane

An agent often does not know the schema before writing its query. Executing a guess just to discover column names wastes budget and mixes planning with side effects.

sqrail separates the flow into three commands:

schema  -> inspect names and types
check   -> bind and plan without running the query
run     -> execute once

For the package query, check reported two VARCHAR result columns, one resolved input file, and a READ_JSON_AUTO physical plan. An orchestrator can inspect that JSON before spending the execution budget.

Bound the whole task, not only the query

A memory flag alone does not bound an agent task. Discovery, schema inference, planning, execution, spilling, result materialization, and output finalization all consume resources.

The v0.3 executor can limit threads, deadline, result rows, output bytes, input-file count, SQL bytes, memory, and temporary spill. Its timeout begins when command handling starts rather than when DuckDB finally begins executing.

I reran two failure probes against the Windows x86-64 release:

Probe	Exit	Diagnostic code
Three rows with `--max-rows 2`	4	`RESULT_LIMIT`
Large aggregation with `--timeout 1ms`	4	`QUERY_TIMEOUT`

The DuckDB memory setting is not a hard operating-system RSS limit. If the threat model requires hard containment, the host still needs a process, container, Windows job object, or equivalent OS boundary.

Treat file output as a commit

Streaming JSONL to stdout is useful, but it cannot be rolled back. A failure after several rows may leave the consumer with a valid partial prefix.

File output needs different semantics. With -o, sqrail writes a private same-directory temporary file and exposes it only after the result completes. POSIX uses a no-replace hard-link commit; Windows uses a no-replace, write-through move.

When I pointed v0.3.4 at an existing .jsonl destination, it returned exit 5 with OUTPUT_EXISTS; the destination was not replaced.

That distinction belongs in the tool contract:

use stdout when partial streaming is acceptable;
use a file destination when the consumer needs all-or-nothing output.

What I would require from any agent data tool

Before giving an agent a local SQL executor, I would ask:

Are all readable files explicitly bound and canonicalized?
Is the statement type checked by the parser?
Are extension loading and other escape hatches disabled?
Do discovery, planning, execution, spill, and output share bounded resources?
Can failed output become visible or overwrite an existing file?
Are failures stable enough for software to handle without scraping prose?
Which guarantees still require OS-level isolation?

A prompt can tell a model to be careful. An execution contract determines what happens when it is not.

If you expose a data tool to an agent, which boundary has failed first in your experience: input access, resource use, or output commit?

The complete v0.3 process contract is public in CONTRACT.md, with the measurement rules in BENCHMARKS.md.

Disclosure: I used an AI assistant to organize the public specification and edit this article. The commands and failure cases above were rerun on August 7, 2026 against the published sqrail 0.3.4 Windows x86-64 archive after its SHA-256 digest was matched to the release checksum.

Can a Python gettext app adopt t-strings safely? A tested Python 3.14 migration path

Yusuke Hayashi — Thu, 06 Aug 2026 11:27:41 +0000

If your Python application already uses gettext, its translations probably live in .po files. Source code supplies a message ID, gettext looks up that ID in the catalog, and the application renders the translation with the values for this request.

That workflow is mature, but interpolation is often held together by convention:

# A translator must preserve the %(name)s placeholder.
_('Hello %(name)s') % {'name': name}

Python 3.14 introduced template strings (t-strings). Unlike an f-string, a t-string retains its literal text, expressions, and evaluated values as structured data:

name = "Ada"
template = t"Hello {name}"

That is attractive for translation: the message and its values are no longer fused into one ordinary string before the i18n layer sees them. But a t-string does not automatically become a gettext catalog entry, and an existing application cannot replace thousands of calls in one release.

I built gettext-tstrings, an alpha library that bridges that gap. Its goal is deliberately narrow:

Let an existing Python gettext application adopt t-strings one call site at a time while keeping its normal .po/.pot catalog workflow.

This article explains the problem, separates standard Python/gettext behavior from what the library adds, and reports a Windows migration check against v0.1.0a8. It is not a claim that t-strings replace gettext, or that this alpha package is ready for every production system.

What needs to fit together

There are three independent pieces:

Python source code  ->  extraction  ->  .pot/.po catalog  ->  runtime lookup
       |                  |                  |                    |
  t"Hello {name}"   stable message ID   translator edits      render values

gettext is the established translation system. A .po file maps a message ID (msgid) to each language's text.
Babel is commonly used to extract translatable messages from Python source into a .pot template.
t-strings are Python 3.14 structured templates. They are not a replacement for a translation catalog.

gettext-tstrings adds a small bridge: it turns an accepted t-string shape into a stable msgid, keeps translations as catalog data rather than executable expressions, and validates placeholders before a damaged translation reaches a user.

The standard pieces are not new. The project-specific question was whether old gettext calls and this new bridge could coexist in one real catalog without a flag-day migration.

The migration target: old and new calls in one file

The intended change is local. Existing calls keep working while a maintainer converts a selected call site:

from gettext_tstrings import tr

name = "Ada"
count = 2

old_style = _("Legacy message")
new_style = tr(t"Hello {name}")
plural = ngettext("One legacy file", "{count} legacy files", count)

The important promise is not that the syntax is shorter. It is that extraction still produces one catalog containing all three forms.

For the t-string extractor, the Babel mapping is small:

[gettext_tstrings: **.py]
encoding = utf-8

I used the normal Babel command rather than a special migration tool:

uv run --no-sync pybabel extract `
  -F babel.cfg `
  -o .verification/messages.pot `
  examples

The resulting POT preserved the ordinary gettext entry:

#: examples/mixed_migration.py:6
msgid "Legacy message"
msgstr ""

It added the t-string as a standard brace-format message, with a marker that identifies the stronger contract:

#. gettext-tstrings
#: examples/app.py:31 examples/app.py:37 examples/mixed_migration.py:7
#, python-brace-format
msgid "Hello {name}"
msgstr ""

And it kept the existing plural in that same catalog:

#: examples/mixed_migration.py:8
#, python-brace-format
msgid "One legacy file"
msgid_plural "{count} legacy files"
msgstr[0] ""
msgstr[1] ""

That is the result I needed before considering a migration: no all-at-once replacement, no second catalog, and no change to how translators edit .po files.

What the library changes—and what it does not

The bridge intentionally accepts only simple named placeholders such as {name}. It does not try to serialize arbitrary Python expressions into translations.

That restriction makes the catalog entry readable and reviewable. It also allows the runtime to check whether a translator removed, added, or misspelled a placeholder. A caller can use a fallback behavior for a damaged catalog or request a strict failure, depending on the application's policy.

The library does not make a translation correct by itself. It does not replace Babel, GNU gettext tooling, translator review, locale testing, or a deployment team's own concurrency checks. Its role is smaller: preserve a structured template long enough to create and validate a conventional gettext message.

What I tested

I checked the exact released source below rather than describing an unpinned development checkout:

Component	Version or environment
`gettext-tstrings`	`0.1.0a8`, commit `3b65baebfc710a750b943073a3c11b6596e396e3`
Python	CPython 3.14.6, 64-bit
Extraction	Babel 2.18.0
OS	Windows 11 `10.0.26200`
CPU	AMD Ryzen 5 3600XT, 6 cores / 12 logical processors
Environment tool	uv 0.11.28 with the committed lockfile

I created that environment without allowing uv to download another Python interpreter:

uv sync `
  --python "C:\Users\yusuk\AppData\Local\Python\pythoncore-3.14-64\python.exe" `
  --no-python-downloads `
  --frozen

I then ran the suite with coverage:

uv run --no-sync pytest `
  --cov=gettext_tstrings `
  --cov-report=term-missing

The result on that machine was:

452 tests collected
449 passed, 3 skipped in 8.52s
905 statements, 310 branches
100% coverage

The skips are part of the result, not a footnote. All three required GNU gettext tools, which were not installed on this Windows machine. Runtime, extraction, checker, conformance, retention, and locale-binding tests passed there; GNU msgfmt integration was not demonstrated.

If your release process uses GNU gettext, keep that compiler check in CI. A green Python suite is not proof that every external tool in a translation pipeline is configured correctly.

What the Windows benchmark did—and did not—show

I ran benchmarks/runtime.py in five separate processes. These medians help set expectations on this machine:

Path	Median ns/op
f-string	55.7
`gettext(str).format`	316.2
`compiled.render`	304.1
`compile_template`	517.5
`tr` with one field	901.6
`tr` with two fields	1,271.4
`Translator` with one field	1,046.0
`ngettext` with one field	1,758.9

For the common one-field tr path, the five runs ranged from 896.0 to 960.9 ns/op. On this particular Windows/AMD system, validation and catalog rendering stayed below one microsecond for that path.

That is not a universal performance claim. An earlier Apple Silicon measurement was roughly 0.4 microseconds for a broadly similar operation; CPU, OS, Python patch release, and package version make that comparison unsuitable for capacity planning. Run the benchmark on the interpreter and hardware that will actually serve your users.

A practical adoption checklist

I would consider a gradual migration only when all of these are true:

The application can run on Python 3.14 or later.
The team can pin a released version or commit while the package is alpha.
Old calls and new t-string calls extract into one POT, and the catalog diff is reviewed.
Placeholder validation runs in both the Babel and GNU gettext stages that the release pipeline uses.
Broken translations are tested in fallback and strict modes.
Locale binding is tested under the application's actual thread, task, or request-concurrency model.
Runtime cost is measured in the deployment environment rather than copied from this article.

Wait if you need arbitrary expressions in translations, cannot move to Python 3.14, or have not validated the same catalog compiler and locale behavior that your release process depends on.

The useful conclusion

The valuable result was not a synthetic benchmark or a new syntax. It was a smaller operational claim that the evidence supports:

In the tested v0.1.0a8 Windows environment, a gettext codebase could extract ordinary gettext calls, plural calls, and supported t-string calls into one catalog while retaining explicit placeholder validation.

That gives a maintainer a reversible migration path: choose one call site, inspect the POT diff, test the affected locale behavior, and continue only if the existing workflow remains intact.

The exact contract is in SPEC.md, and the release tested here is v0.1.0a8.

If you maintain a gettext-based Python project, what would you verify before moving the first call site: extraction, placeholder compatibility, catalog compilation, or locale behavior under concurrency?

Disclosure: I used an AI assistant to organize the public source material, structure the migration test report, and edit the prose. I ran and reviewed the commands, source references, and results included above before publication.