DEV Community: Yusuke Hayashi

How Much Can the OpenTelemetry Collector's Persistent Queue Really Protect? Measuring SIGKILL and Lost ACKs

Yusuke Hayashi — Thu, 06 Aug 2026 19:10:35 +0000

The traces produced while your observability backend was down are exactly the ones you need
for the postmortem. But what happens if the OpenTelemetry Collector relaying that data
restarts at the same time as the backend?

The OpenTelemetry Collector (hereafter "Collector") is open source software that receives
traces, metrics, and logs, applies whatever processing you need, and forwards them to an
observability backend. It ships as a
binary implemented in Go
(architecture docs) and as a
container image. You wire three kinds of components together in YAML:

Receiver: takes data in from applications and other sources
Processor: batches, adds attributes, filters, and otherwise transforms data
Exporter: sends data to destinations such as an observability backend

In this article all data moves over OTLP (OpenTelemetry Protocol), the standard protocol for
sending OpenTelemetry traces, metrics, and logs. The experiments use OTLP/gRPC.

Exporters have a sending_queue that temporarily holds data that cannot be sent yet. You can
keep queue contents in memory only (a "memory queue") or write them to disk with
file_storage (a "persistent queue").

For this article I stopped the destination, let data pile up in the queue, and then either
gracefully or forcibly terminated the Collector. I also built a failure in which the
destination loses its success response (ACK) immediately after durably storing the data, and
counted loss and duplication by Span ID.

Conclusion first: what a persistent queue actually protects

In OpenTelemetry, the whole flow of one request through multiple operations is recorded as a
"trace", and each individual operation as a "span". The input for these experiments was:

What is counted	Count	What it means here
Traces	500	500 application request executions
Spans per trace	2	one parent span and one child span
Spans	1,000	the data actually checked for loss/duplicates
Spans per OTLP request	100	the unit sent to the Collector
OTLP requests	10	how the queue holds them: 10 requests

So "500 traces", "1,000 spans", and "10 requests" throughout this article are the same input
counted in different units. Each span carries a Span ID, and those IDs are what I use to count
loss and duplication.

There are two ways I stopped the Collector:

Signal	Meaning
`SIGTERM`	requests a graceful shutdown; the Collector can run shutdown logic
`SIGKILL`	no chance to shut down; the OS terminates the process immediately

I ran six ordinary conditions plus one ACK-loss condition, three times each. Counted at the
destination:

Queue	Failure condition	Span records stored	Unique Span IDs	Duplicates	Lost
Memory	destination stopped, then recovered	1,000	1,000	0	0
Memory	`SIGTERM` while destination is down	0	0	0	1,000
Memory	`SIGKILL` while destination is down	0	0	0	1,000
Persistent	`SIGTERM` while destination is down	1,000	1,000	0	0
Persistent	`SIGKILL` while destination is down	1,000	1,000	0	0
Persistent	queue files lost after `SIGKILL`	0	0	0	1,000
Persistent	stored, then died before ACKing the Collector	1,100	1,000	100	0

"Duplicates" is the number of extra records stored relative to unique Span IDs. In the last
condition, 100 Span IDs were each recorded twice, so the total record count came to 1,100.

The most surprising result was the memory queue under SIGTERM. The Collector logged
Shutdown complete. and exited with code 0 — a clean shutdown — but because the destination
was still down it could not drain the queue, and nothing arrived after the restart.

Three takeaways:

A memory queue survives a temporary destination outage, but nothing carries over when the Collector itself restarts.
A persistent queue can resend after a Collector restart, but only when it can reattach to the same stored data.
Losing the destination's success response triggers a resend, so you can get duplicates, not just loss.

A persistent queue widens the range of Collector restarts you can resend through. It is not an
end-to-end guarantee that everything the Collector accepted reaches the backend exactly once.

To see why these results came out this way, let's look at the setup, then pin down what
"successfully sent" actually refers to, and then walk through each failure.

What is under test: app → Gateway → backend

The same Collector binary can be deployed in different roles. The official docs call running it
as a central OTLP endpoint the
Gateway deployment pattern. The
gateway container in this article is just the official Collector image configured in YAML.

Persistent-queue resilience matters most in exactly that shape: applications send OTLP to a
central Gateway Collector, and the Gateway relays to the observability backend.

When backend maintenance or a network failure coincides with a Gateway rolling update, pod
eviction, or process crash, the question becomes whether traces accepted during the incident
survive into the next process.

Gateways are typically deployed as a Kubernetes Deployment or a service on a VM, and are used
for centralized concerns: credentials, egress consolidation, filtering, sampling. They are not
mandatory — small environments, or setups where each language SDK (the library that emits
OpenTelemetry data) sends straight to the backend, can skip them.

The subject under test is a single Collector with a queue and storage area shared with no
other replica. Load balancers, queue sharing across replicas, and SDK-side retries are out of
scope.

Here is how real components map to the test environment. telemetrygen is OpenTelemetry's
official tool for generating test traces and other signals.

Real system / subject	Test-environment stand-in	Not measured
Application / SDK	OTLP/gRPC from the official `telemetrygen` image	SDK batching, retries, language variation
Gateway Collector	official Collector image configured in YAML	multiple replicas, load balancing
Observability backend	`file` exporter, or a purpose-built Go sink	indexing, search, backend durability
Fault injection and tally	Docker Compose, POSIX shell, `curl`, `jq`	Kubernetes, host-level failures

Because the measurement starts once the Gateway has accepted OTLP/gRPC, the source language is
not fixed. Whether you send from Java, Go, or Python, the fault injection and counting done
here are unchanged. A setup that sends from the SDK directly to the backend has no Collector
queue at all, so you have to check that SDK's own queueing and retry behavior separately.

From here on, the Collector that receives from the application and forwards is the "Gateway",
and the Collector or Go server that stores what it receives is the "Sink". The Sink stands in
for the observability backend in this experiment.

  telemetrygen (app stand-in, 500 traces)
        |  OTLP/gRPC
        v
  Gateway Collector (single instance)
        |
        v
  Exporter sending_queue  ---- file_storage (persistent queue only)
        |
        x  (cannot send while the Sink is down)
        v
  Sink (backend stand-in)

That covers the components. Next, let's split up how far along the path data has to get before
you can call it "sent". That distinction is the foundation for understanding the duplicates
caused by a lost ACK later on.

Background: splitting "it was sent" into four boundaries

An application can succeed at sending without the data being stored in, or searchable from, the
backend. To keep those look-alike states apart, here are four boundaries along the delivery
path.

Boundary	Evidence measured	What it lets you claim
Gateway accepted	`otelcol_receiver_accepted_spans=1000`	the pipeline accepted 1,000 spans
Enqueued / in flight	`otelcol_exporter_queue_size=10`	10 requests occupied queue capacity
Send to Sink succeeded	`otelcol_exporter_sent_spans=1000`	1,000 logical spans obtained a downstream ACK
Sink wrote to file	500 unique Trace IDs / 1,000 unique Span IDs / 0 dupes	the spans are in the `file` exporter's JSON

Two different success responses (ACKs)

There are two OTLP success responses in this article that are easy to conflate. ACK is short for
acknowledgement — here, the success response returned by whoever received a request.

  telemetrygen --(1) upstream OTLP request--> Gateway
  Gateway      --(2) enqueue--------------->  sending_queue
  Gateway      <-(3) upstream ACK----------   (returned to telemetrygen)
  sending_queue--(4) downstream OTLP request-> Sink
  Sink         --(5) downstream ACK--------->  Gateway

Name	Direction	What success means in this experiment
Upstream ACK	Gateway → telemetrygen	the Gateway pipeline accepted the data and enqueued it
Downstream ACK	Sink → Gateway	the OTLP request from Gateway to Sink succeeded

I explicitly set the exporter helper's wait_for_result to false. With that setting, the
upstream request does not wait for the exporter to finish sending to the Sink. So an upstream
ACK does not mean the Sink stored, indexed, or can search the data.

What the later ACK-loss experiment destroys is the downstream ACK from Sink to Gateway. Even
when the Sink has already stored the data, a Gateway that never receives the downstream ACK will
resend the same request, and duplicates can result.

otelcol_exporter_sent_spans counts logical spans that eventually got a downstream ACK. It is
neither the total number of send attempts nor the total records the Sink stored. In the ACK-loss
experiment, sent_spans stayed at 1000 while the Sink's stored records reached 1,100 spans.

It matters not to conflate the
"accepted by the server" boundary in
the OTLP spec with everything downstream of it: storage, indexing, and searchability.

What the Sink actually measured The Sink's `file` exporter was used only for counting records. Its [stability for traces is alpha as of 0.157.0](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.157.0/exporter/fileexporter). The tally scripts are written to fail if the JSON structure changes. Nothing here measures a real backend's index build, searchable-at time, or durability.

With four delivery boundaries and two ACK directions established, let's pin down when each
failure was injected and which evidence decided success or failure.

Method: seven failure conditions and how they were judged

Testing ran 2026-07-25 to 2026-07-27. I used the official Contrib distribution, which bundles
many extra components including file_storage, pinned to
OpenTelemetry Collector 0.157.0.

The ordinary conditions inject failure in this order:

Do not start the Sink, so the Gateway cannot send.
Send 500 traces and wait until the Gateway's accepted count reaches 1,000 spans.
Wait until queue usage reaches 10 requests.
Depending on the condition, leave the Gateway running or stop it with SIGTERM / SIGKILL.
Recover the Sink and the Gateway, then count the stored Trace IDs and Span IDs.

For conditions judged as total loss, I waited 10 seconds after Sink recovery — longer than the
5-second maximum retry interval — and confirmed the destination was still empty. Rather than
waiting a fixed time before injecting a failure, each step proceeds only after the accepted count
and queue usage are confirmed.

Only the ACK-loss condition starts the Sink from the beginning. It stops right after storing the
first OTLP request, so no downstream ACK is returned.

This is a low-volume functional test on a single machine, not a performance test. 500 traces is a
number chosen to make loss and duplication easy to count, not an assumed production rate. No
external telemetry backend or credentials are required.

Measured results and hypothetical requirements are kept separate.

The Collector configuration, fault injection code, counts, logs, and CSVs all come from running
the published code locally. The production requirements used as examples later — "tolerate a
10-minute outage" and so on — are hypothetical, for illustration.

Test environment and what was not tested The [captured environment details](https://github.com/yhay81/otel-collector-resilience-lab/blob/main/results/2026-07-28-environment.txt) are saved as well. Prerequisites and the measured environment: | Category | Prerequisite / measured environment | | ------------------ | ------------------------------------------------------------------------------ | | Host | macOS 26.5.1 / Apple M2 Pro / 12 CPU / 32 GiB | | Docker VM | arm64 / 12 CPU / approx. 15.6 GiB | | Container platform | Docker Desktop 4.81.0 / Engine 29.6.1 / Compose v5.2.0 | | Host-side CLI | POSIX shell, `curl`, `jq` | | ACK-loss Sink | Go 1.25.12 / OTLP proto 1.11.0 | | Local resources | TCP 4317/8888 free, and the test code's `data/` writable | | Network (first run)| image pulls from GHCR and Docker Hub, Go module fetch for the ACK-loss Sink | Non-arm64 CPUs, Linux, and minimum CPU/memory/disk requirements were not tested.

The input is 500 traces, 1,000 spans, and 10 requests, with a queue limit of 100 requests.
telemetrygen's --child-spans 1 makes each trace two spans, and --batch-size 100 sends 100
spans at a time. Additional batching inside sending_queue is disabled; retries are unlimited
with a 5-second maximum interval.

The full reproduction set lives in the
public repository with the test code and measured results.
The scripts contain not only the experimental conditions but the tallying logic and failure
conditions too.

The six non-ACK-loss conditions and the ACK-loss condition run with:

git clone https://github.com/yhay81/otel-collector-resilience-lab.git
cd otel-collector-resilience-lab
./run.sh
./run-ack-loss.sh

Three trials each are recorded in the
ordinary-conditions CSV,
the ACK-loss CSV,
and the metrics and logs.
From here, the results follow this order: destination-only outage, Gateway also terminated, ACK
alone lost, and storage itself lost.

Result 1: a memory queue resent fine when only the destination stopped

First, the Gateway kept running and only the Sink was stopped. After sending 500 traces and
waiting for the internal metric otelcol_exporter_queue_size to reach 10, I started the Sink.
All 500 traces / 1,000 spans arrived, with zero duplicate Span IDs.

The memory queue configuration used ```yaml exporters: otlp_grpc/sink: endpoint: sink:4317 tls: insecure: true sending_queue: enabled: true sizer: requests queue_size: 100 num_consumers: 1 wait_for_result: false block_on_overflow: false batch: enabled: false retry_on_failure: enabled: true initial_interval: 1s max_interval: 5s max_elapsed_time: 0s ```

sending_queue acted as a temporary holding area while the destination was down, and
retry_on_failure resent with a gradually widening interval. Because the Gateway process kept
running, the 10 in-memory requests were still there.

So a memory queue prepares you for a backend restart or a brief network interruption. Stopping
the Collector itself is a different story.

Result 2: nothing carried over from the memory queue after the Collector stopped

With the same 10 requests in the memory queue, I sent SIGTERM to the Gateway — in practice
docker compose stop -t 10 gateway. The Collector received terminated, logged
Shutdown complete. within the same second, and exited with code 0. It did not fall through to
the forced kill after 10 seconds.

Even so, not a single trace arrived at the Sink. Shutdown stopped retrying, attempted to drain
the queue, and dropped the 100-span requests it could not deliver to the stopped Sink.

Gateway log on SIGTERM ```text Received signal from OS signal: terminated Starting shutdown... Exporting failed. Dropping data. dropped_items: 100 Shutdown complete. ```

Under SIGKILL in the same condition, no trace arrived either — the data awaiting retry vanished
along with the Collector process.

The conclusion here is not "SIGTERM always loses data". If the destination is up and the data
can be sent during shutdown, the outcome can differ. What this measures is that when you
terminate the Collector while the destination is still down, even a graceful shutdown cannot
carry the memory queue over to the next process.

The reason for the loss is that the queue existed only in the Collector process's memory. So
next, the same data is written to a file outside the process before the same shutdowns are
repeated.

Result 3: a persistent queue resent after both graceful and forced termination

file_storage is an extension that persists queue data to local files. Point an exporter's
sending_queue.storage at that extension and it uses a persistent queue instead of a memory one.

Configuration diff for switching to a persistent queue ```yaml extensions: file_storage/queue: directory: /var/lib/otelcol create_directory: true fsync: true exporters: otlp_grpc/sink: sending_queue: storage: file_storage/queue service: extensions: [file_storage/queue] ```

The endpoint, queue limit, send concurrency, batching, and retry settings are identical to the
memory queue. The only substantive change is where the queue lives.

The Gateway's /var/lib/otelcol was bind-mounted to a host directory, so recreating the Gateway
container still reads the same queue files. Under SIGTERM I confirmed exit code 0 and
Shutdown complete.; under SIGKILL, exit code 137. Both delivered all 500 traces / 1,000 spans
after restart, with zero duplicate Span IDs.

Queue files and logs at restart `file_storage` uses bbolt, an embedded database, internally. Just before `SIGKILL` there was a bbolt file holding the 10 requests. The restart logs show the persistent queue metadata being loaded and the one in-flight request being moved back to the queue. `readIndex: 1` does not mean only one request was stored: `itemsSize: 1000` is the number of retained spans, and `numberOfItems: 1` is the number of requests that had been handed to a consumer at crash time. ```text Loaded queue metadata readIndex: 1, writeIndex: 10, itemsSize: 1000 Moved items for dispatching back to queue numberOfItems: 1 ```

This resume behavior and the ID tallies were consistent across three trials, and match the
behavior documented in the
official file_storage README.

fsync: true asks the OS to sync to disk on every write, improving database consistency across
an interruption — at the cost of write performance. The default is false, so decide based on
the durability you need and measured performance under real traffic. What I compared here is
only SIGKILL against the process: no performance or loss comparison against fsync: false, and
no host power loss, Docker Desktop VM, filesystem, or storage-controller durability. fsync: true
alone is not grounds for claiming power-loss resilience. The file_storage extension itself is
also beta as of 0.157.0. Pin the version, and re-run the same fault injection tests against the
release notes when you upgrade.

So far, when the stored data survives, the Collector can resend after a restart. But no loss does
not imply no duplication. Next, the downstream ACK boundary gets broken.

Result 4: losing the downstream ACK duplicated 100 spans

The Gateway cannot know whether the Sink finished storing unless it receives the downstream ACK.
So this experiment destroys only the downstream ACK from Sink to Gateway.

The purpose-built ACK-loss Sink writes the 100 spans in the first OTLP request to a JSONL file
(one record per line) and fsyncs it. Then, before returning a success response, it exits with
code 23. Docker Compose restarts just the Sink, and the Gateway — having never received a
downstream ACK — resends the same request.

  Gateway --(1) OTLP request, 100 spans--> crash-before-ACK Sink
                                            |
                                            +--(2) append + fsync--> Sink storage file
  Gateway <-X(3) process exits before responding
  Gateway    (4) Unavailable / EOF: back to the queue, retry
  Gateway --(5) resend the same request---> Sink
                                            |
                                            +--(6) append the same 100 spans again
  Gateway <--(7) OTLP success---------------+

All three trials agreed:

Observation	Value (3 trials)
Logical spans the Gateway accepted from upstream	1,000
Logical spans for which the Gateway got a downstream ACK	1,000
Queue usage after recovery	0
Span records stored at the Sink	1,100
Unique Span IDs at the Sink	1,000
Span IDs recorded twice	100
Successful resends of the request holding the first 100 spans	1

All 1,000 unique spans arrived, so nothing was lost. But the first 100 spans were recorded twice
— before the crash and after the resend — bringing stored records to 1,100.

The Sink's logs show the same request hash appearing both in the pre-crash store and in the
post-restart ACK.

ACK-loss Sink log ```text persisted request sha256=0642...d96d2e spans=100; exiting before OTLP response ack-loss-sink ready on :4317 acknowledged request sha256=0642...d96d2e spans=100 ```

On the Gateway side, rpc error: code = Unavailable ... EOF was logged, followed by a retry about
1.1 seconds later. The exporter counted 1,000 spans as successful after the resend, but the Sink
processed 1,100 spans in total.

The OTLP spec is explicit that a disconnect before the ACK can produce duplicates via
retry. This experiment reproduces that
ambiguous window using an identical request hash and identical Span IDs. It is evidence of
duplication across a process crash in a purpose-built Sink — not proof of host power-loss
durability or of any real backend's deduplication.

Up to this point the queue data needed for the resend still existed. Next: losing that storage
itself.

Result 5: losing the stored data meant even a persistent queue could not resend

Despite the name "persistent", once the queue files themselves are gone there is nothing to
resend. In this condition I confirmed 10 requests in the persistent queue, sent SIGKILL to the
Gateway, then deleted the queue directory data/queue before restarting.

This models complete loss of the storage contents. Kubernetes is not involved, so nothing here
tests pod rescheduling to another node or PersistentVolume reattachment. It is not an experiment
showing "a node failure always means total loss". Whatever the reason for the failure, this shows
what happens when the original queue data cannot be read after a restart.

In all three trials, not a single trace arrived at the Sink. file_storage can resume delivery
because it re-reads the remaining queue files. Having persistence configured and being able to
reach that same persisted data after a failure are two different requirements.

On Kubernetes, post-failure behavior differs by storage type. A PersistentVolume (PV) is storage
with a lifetime independent of the pod.

Storage	Caveat
`emptyDir`	survives container restarts, but disappears with the pod
Local PV	bound to a specific node's disk, so unusable as-is on another node
Network PV	verify reattachment on another node, plus I/O latency and AZ failures for real
External queue such as Kafka	decouples storage from the Gateway, but adds a new operational surface

The range of failure types you intend to protect against is what I call the "failure boundary"
here. If the boundary you accept is process crashes, file_storage plus a reattachable volume is
a candidate. If you need to keep accepting data through node or AZ (availability zone) loss, a
Collector's own persistent queue is not enough.

Storage loss is one limit of persistent queues. There are others, where data is lost before
storage or during resend. Here is where the measured and unmeasured ranges separate.

Limits of this test: conditions a persistent queue does not cover

Reading these results as "file_storage means you never lose anything" is dangerous. What a
persistent queue protects is resuming, after a Collector process restart, a queue that was
written to durable disk. At least these four conditions are outside that protection.

Unprotected condition	What happens	Main countermeasure
Queue limit reached	with `block_on_overflow: false`, enqueue fails	size capacity; verify upstream failure and retry
Retry deadline exceeded	requests past `max_elapsed_time` are dropped	set the deadline against your recovery objective
Disk failure/exhaustion	the persistent queue cannot be written	monitor capacity, I/O, and errors
Stored data lost	the original queue files cannot be read after restart	give each pod a reattachable persistent volume

Of those four, only stored-data loss was measured. With this pipeline configuration, enqueue
failures are returned as errors on the upstream OTLP request. If upstream can resend, that is not
immediately data loss — but ignoring the error is. max_elapsed_time: 0s is likewise a setting
chosen to isolate restarts for comparison. Unlimited retention keeps old data around and can fill
the queue or disk first. The official default is 300 seconds.

Also untested:

Performance and loss compared against fsync: false, host power loss, filesystem corruption
Kubernetes pod deletion, node shutdown, PVC reattachment, CSI failover
Real backend acceptance, index build, searchable-at time, deduplication
Production-equivalent throughput, 10-minute outages, queue drain within 15 minutes, resource usage

So far this has mostly covered everything after the Gateway accepted the data. But the real
delivery path starts at the application. Let's look upstream.

The application side: from SDK to Gateway

Collector and PersistentVolume configuration may belong to a platform team. But you cannot think
about loss between the application and the backend without the SDK's behavior.

This experiment measures only what happens after the Gateway accepted the data. In a real system,
also verify the following between the application and the Gateway:

How the SDK batches spans, and how much it can hold in memory
Whether the SDK retries or drops when it cannot connect to the Gateway or gets an error back
Whether application shutdown waits for unsent data, and for how long
After a success response from the Gateway, when the data becomes searchable in the backend
How the backend displays and aggregates the same Span ID arriving more than once

Each language's OpenTelemetry SDK configures batching, queueing, retry, and shutdown differently.
Do not treat "the application's send API returned success" as the delivery-complete condition —
test the SDK and backend you actually use, together.

Once the application-side boundary is settled, the Gateway side turns "how much downtime do we
absorb" into concrete capacity. Logical capacity and disk capacity are separate.

Capacity planning: separate `queue_size` from disk capacity

Sizing a persistent queue happens in two stages. First derive the Collector setting queue_size,
then translate that into disk capacity for a PersistentVolume or similar.

Logical capacity: derive `queue_size` from the outage you must survive

Rather than copying a sample value like queue_size: 1000, decide first how long a destination
outage you want to survive. A rough formula:

required queue_size = peak enqueue units/sec x tolerated outage seconds x safety factor

What an "enqueue unit" is depends on the sizer setting.

`sizer`	What `queue_size` counts	Characteristics
`requests`	number of requests entering the queue	lowest computational cost
`items`	spans, data points, log records, etc.	easiest to match against record counts
`bytes`	serialized data size	direct size control, highest computation cost

The exporter helper 0.157.0
documentation likewise describes requests as the lightest and bytes as the most
computationally expensive option. Base the calculation not on the application's send count but on
the units that actually enter the queue after processors and exporters have done their work. If
you pick requests, measure spans-per-request as well.

Disk capacity: derive it from actual file growth

The queue_size from the formula above is a logical ceiling, not the byte count a
PersistentVolume needs. Even with sizer: bytes, bbolt's bookkeeping and space that remains
allocated after release mean the queue file size will not match exactly.

Accumulate production-like traces, metrics, and logs for the required duration and measure the
following to set disk capacity and free-space alert thresholds:

File growth before and after enqueueing
How much file size remains after the queue drains
Changes from file_storage compaction settings
I/O latency in steady state and during recovery
Safety margin including filesystem reserved space

Once the queue is in place, monitor at least:

otelcol_exporter_queue_size and otelcol_exporter_queue_capacity
Enqueue failures such as otelcol_exporter_enqueue_failed_spans
Send-attempt failures such as otelcol_exporter_send_failed_spans
Persistent volume usage, free space, I/O latency, and file_storage error logs

These Collector internal metrics are alpha as of 0.157.0; recheck names and attributes when
upgrading. Detect rising utilization together with send failures, before the queue fills. And do
not put the Collector's own metrics solely on the same delivery path — keep an independent health
check.

That covers failure boundaries, capacity, and monitoring — the inputs to an adoption decision.
Finally, let's turn these individual results into a production checklist you can pass or fail item
by item.

Adoption decision: turning measurements into a production checklist

As an example, take a hypothetical requirement: "survive a 10-minute destination outage and one
forced Gateway termination, and drain the queue within 15 minutes of recovery." Do not treat this
scaled-down experiment as production certification; verify the incomplete items below under
production-like conditions.

Check item	Pass condition	Evidence from this test
Temporary outage	no loss of accepted traces	memory and persistent both pass
Graceful shutdown	no loss when terminating/restarting while destination is down	persistent only
Process kill	no loss after `SIGKILL`	persistent only
Storage loss	declare out of scope, or decide an upstream recovery path	deleting stored data: total loss
Volume reattachment	the same queue is re-read after pod/node rescheduling	not tested
Capacity	holds 10 minutes at peak throughput plus a safety factor	needs production-rate testing
Recovery time	queue returns to normal within 15 minutes of recovery	needs production-rate testing
Security	permissions, encryption, and deletion of stored data verified	review per environment

The decision is not a binary "adopt or skip the persistent queue". Adoption judgment means
separating the process boundary measured here from the node and storage boundaries not yet
verified, and deciding who accepts the remaining risk.

Building a checklist is not proof that things work in a real environment. The last step is to
inject the failures you expect into a pre-production environment and confirm recovery matches
expectations.

Summary: what to confirm before adopting a persistent queue

Enabling a persistent queue does not by itself mean "we now have a configuration that never loses
data". What matters is confirming, in your actual setup, which failures keep the queue intact and
allow delivery to resume afterwards.

From these tests, four things to nail down before production:

A success response at the application or the Gateway does not mean the data is searchable in the backend at that moment. Verify acceptance and storage/search separately.
Restarting the Collector while the destination is down does not carry memory queue contents to the next process, even on a graceful shutdown.
With file_storage, do more than enable the setting: build a storage arrangement that can re-read the same queue data after a restart.
Resends can produce duplicates. Beyond loss, confirm how the backend handles duplicates and how you would detect them.

Once the configuration is settled, try each failure one at a time: destination down, SIGTERM,
SIGKILL, queue limit reached, disk exhaustion. Do not let testing end at "the application sent
successfully". Make the completion criterion for a delivery test the ability to actually search
traces generated during the incident after recovery, and to explain both whether anything was lost
and how duplicates appear.

References

Tested 2026-07-25 to 2026-07-27.

This article was originally published in Japanese on Zenn.

A SELECT-only prompt is not a sandbox: bounding agent-generated SQL

Yusuke Hayashi — Thu, 06 Aug 2026 18:16:52 +0000

Suppose an AI agent has one job: read package.json and return the package name and version.

You can put “use only SELECT” in the prompt. But if the SQL engine can still open another file, load an extension, run forever, or overwrite an output path, the prompt has not created a security boundary. It has only described one.

I ran into this distinction while building sqrail, a small DuckDB-based executor for analytical SQL over explicitly named files. The useful lesson was not about generating better SQL. It was about turning the agent's request into a process contract that remains true when the request is wrong.

Five promises a prompt cannot enforce

Prompt-level intention	Executor-level boundary
“Read only this dataset”	Canonical file allowlist plus disabled external access
“Only run a query”	One parser-validated `SELECT` statement
“Do not use too much”	Deadlines and row, byte, file, thread, memory, and spill limits
“Do not overwrite anything”	No-replace output commit
“Tell me what failed”	Stable JSON diagnostics and exit classes

The second column is where enforcement starts.

Bind data instead of giving SQL a filesystem

The agent refers to a logical table name, while the host supplies the path:

sqrail run -t pkg=package.json --timeout 5s "SELECT name, version FROM pkg"

On sqrail 0.3.4, that produced:

{"name":"sqrail-site","version":"0.3.4"}

The executor canonicalizes every bound path. Globs and partitioned Parquet directories are expanded, sorted, deduplicated, and required to be non-empty. After binding, only those exact files are allowlisted and DuckDB external access is disabled.

I then tried to bypass the binding and read the same file directly:

SELECT * FROM read_json_auto('package.json')

The v0.3.4 Windows release rejected it with exit 4 and a QUERY_FAILED diagnostic whose message began Permission Error: Cannot access file.

That generic error code is worth noting: the important guarantee is the denied access, not a specially branded “sandbox” message.

Parse the statement; do not pattern-match text

Checking whether a string begins with SELECT is not sufficient. Comments, multiple statements, and other syntax make text-prefix rules brittle.

The v0.3 contract accepts exactly one parsed statement whose type is SELECT. That includes VALUES and queries beginning with WITH. It rejects DDL, DML, COPY, ATTACH, INSTALL, LOAD, PRAGMA statements, and multiple statements. Extension autoloading, automatic installation, and community extensions are disabled before configuration is locked.

This probe:

SELECT 1; SELECT 2

returned exit 4 with MULTIPLE_STATEMENTS.

This is still not a general sandbox for a hostile SQL engine. It is a deliberately narrower contract: one read-only analytical statement over files the caller explicitly bound.

Give the agent a preflight lane

An agent often does not know the schema before writing its query. Executing a guess just to discover column names wastes budget and mixes planning with side effects.

sqrail separates the flow into three commands:

schema  -> inspect names and types
check   -> bind and plan without running the query
run     -> execute once

For the package query, check reported two VARCHAR result columns, one resolved input file, and a READ_JSON_AUTO physical plan. An orchestrator can inspect that JSON before spending the execution budget.

Bound the whole task, not only the query

A memory flag alone does not bound an agent task. Discovery, schema inference, planning, execution, spilling, result materialization, and output finalization all consume resources.

The v0.3 executor can limit threads, deadline, result rows, output bytes, input-file count, SQL bytes, memory, and temporary spill. Its timeout begins when command handling starts rather than when DuckDB finally begins executing.

I reran two failure probes against the Windows x86-64 release:

Probe	Exit	Diagnostic code
Three rows with `--max-rows 2`	4	`RESULT_LIMIT`
Large aggregation with `--timeout 1ms`	4	`QUERY_TIMEOUT`

The DuckDB memory setting is not a hard operating-system RSS limit. If the threat model requires hard containment, the host still needs a process, container, Windows job object, or equivalent OS boundary.

Treat file output as a commit

Streaming JSONL to stdout is useful, but it cannot be rolled back. A failure after several rows may leave the consumer with a valid partial prefix.

File output needs different semantics. With -o, sqrail writes a private same-directory temporary file and exposes it only after the result completes. POSIX uses a no-replace hard-link commit; Windows uses a no-replace, write-through move.

When I pointed v0.3.4 at an existing .jsonl destination, it returned exit 5 with OUTPUT_EXISTS; the destination was not replaced.

That distinction belongs in the tool contract:

use stdout when partial streaming is acceptable;
use a file destination when the consumer needs all-or-nothing output.

What I would require from any agent data tool

Before giving an agent a local SQL executor, I would ask:

Are all readable files explicitly bound and canonicalized?
Is the statement type checked by the parser?
Are extension loading and other escape hatches disabled?
Do discovery, planning, execution, spill, and output share bounded resources?
Can failed output become visible or overwrite an existing file?
Are failures stable enough for software to handle without scraping prose?
Which guarantees still require OS-level isolation?

A prompt can tell a model to be careful. An execution contract determines what happens when it is not.

If you expose a data tool to an agent, which boundary has failed first in your experience: input access, resource use, or output commit?

The complete v0.3 process contract is public in CONTRACT.md, with the measurement rules in BENCHMARKS.md.

Disclosure: I used an AI assistant to organize the public specification and edit this article. The commands and failure cases above were rerun on August 7, 2026 against the published sqrail 0.3.4 Windows x86-64 archive after its SHA-256 digest was matched to the release checksum.

Migrating gettext one t-string at a time: what extraction, 449 tests, and Windows benchmarks showed

Yusuke Hayashi — Thu, 06 Aug 2026 11:27:41 +0000

My first article about gettext-tstrings explained the design: turn a Python 3.14 t-string into a stable gettext message ID, keep translations as data rather than code, and validate placeholders before a damaged catalog reaches users.

That article left a more practical question unanswered:

Can an existing gettext codebase adopt t-strings one call site at a time without replacing its catalog workflow?

I tested the current v0.1.0a8 release on Windows with a mixed old/new example, the full test suite, and the repository's microbenchmark. This is not another t-string introduction. It is a migration check with commands and limitations.

The environment

gettext-tstrings 0.1.0a8 at commit 3b65baebfc710a750b943073a3c11b6596e396e3
CPython 3.14.6, 64-bit
Babel 2.18.0
Windows 11 (10.0.26200)
AMD Ryzen 5 3600XT, 6 cores / 12 logical processors
uv 0.11.28 with the committed lockfile

I created the environment without allowing uv to download another Python:

uv sync `
  --python "C:\Users\yusuk\AppData\Local\Python\pythoncore-3.14-64\python.exe" `
  --no-python-downloads `
  --frozen

A deliberately mixed source file

The migration sample contains ordinary gettext calls and one t-string call:

from gettext_tstrings import tr

name = "Ada"
count = 2

old_style = _("Legacy message")
new_style = tr(t"Hello {name}")
plural = ngettext("One legacy file", "{count} legacy files", count)

The Babel mapping remains small:

[gettext_tstrings: **.py]
encoding = utf-8

I then ran the normal extraction command:

uv run --no-sync pybabel extract `
  -F babel.cfg `
  -o .verification/messages.pot `
  examples

The same POT contained all three forms. The ordinary message remained ordinary:

#: examples/mixed_migration.py:6
msgid "Legacy message"
msgstr ""

The t-string received the library marker and Babel's standard brace-format flag:

#. gettext-tstrings
#: examples/app.py:31 examples/app.py:37 examples/mixed_migration.py:7
#, python-brace-format
msgid "Hello {name}"
msgstr ""

The legacy plural was extracted into the same catalog:

#: examples/mixed_migration.py:8
#, python-brace-format
msgid "One legacy file"
msgid_plural "{count} legacy files"
msgstr[0] ""
msgstr[1] ""

This is the migration property I wanted to verify: the extractor does not require a flag day. Existing string calls and new t-string calls can share one POT while individual call sites move gradually.

What the full suite covered on this machine

I ran:

uv run --no-sync pytest `
  --cov=gettext_tstrings `
  --cov-report=term-missing

The result was:

452 tests collected
449 passed, 3 skipped in 8.52s
905 statements, 310 branches
100% coverage

The three skips matter. All were toolchain tests requiring GNU gettext tools, which were not installed on this Windows machine. I am not treating a skip as proof. Runtime, extraction, checker, conformance, retention, and binding tests passed here; the GNU msgfmt integration still needs a machine with that tool installed.

That limitation is useful migration information by itself. A green Python suite does not demonstrate every external catalog compiler in your delivery pipeline. Keep the GNU check in CI if your translators or release process depend on it.

The measured runtime cost

I ran the repository's benchmarks/runtime.py in five separate processes. These are the medians:

Path	Median ns/op
f-string	55.7
`gettext(str).format`	316.2
`compiled.render`	304.1
`compile_template`	517.5
`tr` (1 field)	901.6
`tr` (2 fields)	1,271.4
`Translator` (1 field)	1,046.0
`ngettext` (1 field)	1,758.9

For tr with one field, the five runs ranged from 896.0 to 960.9 ns/op.

These numbers are not universal. My earlier article included an Apple Silicon measurement around 0.4 microseconds; this Windows/AMD result is about 0.90 microseconds for the same broad operation. Different CPU, OS, Python patch release, and package version make a direct comparison misleading.

The useful conclusion is narrower: on this machine, validation and catalog rendering remained below one microsecond for the common one-field tr path. Measure it again on your own interpreter and hardware before using the number in a capacity decision.

What I would require before migrating production code

This experiment changed my migration checklist into something concrete:

Pin a released version or commit while the package is alpha.
Extract old and new calls into one POT and review the diff.
Keep placeholder validation in both Babel and GNU gettext stages used by your pipeline.
Test damaged translations in fallback and strict modes.
Test locale binding under the concurrency model your application actually uses.
Benchmark on the deployment interpreter instead of copying a number from a README or article.

The most valuable result was not the benchmark. It was seeing the old and new calls coexist in an ordinary catalog with an explicit marker on messages that require the stricter t-string contract.

The reproducible contract is documented in SPEC.md, and the exact release tested here is v0.1.0a8.

If you maintain a gettext-based Python project, I would be interested in the migration failure you would test first. Catalog extraction, placeholder compatibility, thread/context behavior, or something else?

Disclosure: I used an AI assistant to organize the test plan, run the documented checks on my development machine, and edit this draft. I reviewed the commands, source references, and results included above before publication.

Python 3.14 t-strings finally fix gettext i18n — stdlib said no, so I built the bridge

Yusuke Hayashi — Tue, 28 Jul 2026 05:32:10 +0000

For twenty years, internationalizing Python code has meant writing this:

_("Hello %(name)s") % {"name": name}          # % style
_("Hello {name}").format(name=name)           # or .format style

And for twenty years, the same bugs have shipped: .format() applied inside _() so the catalog never matches; a translator's typo in %(name)s crashing production in exactly one locale; a placeholder that can't move to where Japanese or German grammar needs it. The string being translated and the values being interpolated are glued together by convention and hope.

Python 3.14's template strings (PEP 750) change the raw material. A t-string is not a string — it's a structured object that carries the static text, the expressions, and the evaluated values separately:

name = "Ada"
template = t"Hello {name}"   # Template(strings=('Hello ', ''), interpolations=...)

That structure is exactly what an i18n layer always wanted. So the obvious question: can _(t"Hello {name}") just... work?

The gap nobody owned

It turns out there was no answer for what happens between a t-string and a .po file. CPython considered and declined t-string support in stdlib gettext — with a core developer noting that a convention with this many design choices belongs on PyPI, where it can iterate. Babel has an open issue for native support that's been stalled since 2025. Meanwhile Python 3.14 is climbing past 8% of PyPI downloads.

The missing piece is small but load-bearing: a convention for turning a t-string into a gettext msgid, plus enforcement that translations stay compatible with it. That's what gettext-tstrings is — not a new i18n framework, just the bridge.

import gettext
from gettext_tstrings import Translator

_ = Translator(gettext.translation("messages", localedir="locales", languages=["ja"]))

name = "Ada"
print(_(t"Hello {name}"))                     # -> "Adaさん、こんにちは"

n = 3
print(_.ngettext(t"One file", t"{n} files", n))  # plural rules per language

The catalog receives the complete sentence Hello {name}. The Japanese translation {name}さん、こんにちは reorders the placeholder — the thing %-style positional formatting never allowed.

The convention: boring on purpose

The whole msgid derivation fits in a sentence: literal braces are escaped, each interpolation becomes {name}, and only simple variable names are allowed. This is valid:

tr(t"Total: {amount:,.2f}")    # msgid: "Total: {amount}" — format spec stays in code

These are rejected, at extraction time and at runtime, with the same rule:

tr(t"Hello {user.name}")         # attribute access
tr(t"Hello {get_name()}")        # function call

Compute the value first: name = user.display_name(). Why so strict? Because any scheme that derives names from expressions (user.name to user_name?) becomes permanent — change it later and every existing catalog breaks. And because a translation is data, never code: a .po file can reorder and repeat {name}, but {name.__class__}, {name!r}, and {name:>9999} in a translation are all rejected. The catalog can't execute anything, can't reformat anything, can't be an expression language. The full convention is written down as a versioned contract in SPEC.md, so other tools can implement it and produce identical catalogs.

Your existing toolchain already validates this

Here's the part I didn't expect. Babel automatically marks these msgids with the standard python-brace-format flag:

#, python-brace-format
msgid "Hello {name}"
msgstr "Bonjour {wrong}"

That one flag lights up the entire existing toolchain. GNU msgfmt refuses to compile the broken translation above:

$ msgfmt --check-format -o /dev/null ja.po
ja.po:24: a format specification for argument 'name' doesn't exist in 'msgstr'
msgfmt: found 1 fatal error

Weblate runs its Python-brace-format check as translators type. Crowdin, Transifex, and POEditor flag placeholder mismatches in their QA. Zero configuration. Your translation vendor's pipeline from 2015 validates your t-string catalogs today. Extraction is the normal Babel workflow (pybabel extract with an entry-point extractor); PO and MO files are completely standard.

The parts that make it production-shaped

A broken catalog never crashes a render. If a translation's placeholders don't match, you get the source text and a logged warning — mirroring gettext's own contract. strict=True restores fail-loud for CI.
Per-request language via a contextvar: with use_translations(ja): ... — safe under concurrency, built for web frameworks.
Lazy translation for module-level strings: SAVE = lazy_gettext(t"Save changes") renders in whatever language is active when it's used, like Django's gettext_lazy.
Sub-microsecond overhead: ~0.4us per rendered message on CPython 3.14 (Apple Silicon). The cached render path is actually faster than a naive gettext(...).format(...). The difference from a bare f-string buys you validation and safety — this optimizes safety per nanosecond, not benchmarks.

Honest limitations

Python 3.14+ only, by nature. Extraction requires Babel (xgettext and pygettext can't parse t-strings — the SPEC exists partly so they someday can). Only simple {name} placeholders, by design. And it's an alpha: the core contract is small and tested (94 tests, 3 OSes), but the API deserves real-world bruises before 1.0.

If you maintain a Python project with existing .po catalogs and you're eyeing 3.14 — this migrates one call site at a time, and the extractor handles mixed _("old style") and _(t"new style") codebases in one pass. I'd love issues, especially disagreements with the SPEC while it's still cheap to change.

pip install gettext-tstrings

Links: GitHub · PyPI · SPEC · related: CPython discussion, Babel #1206