How Much Can the OpenTelemetry Collector's Persistent Queue Really Protect? Measuring SIGKILL and Lost ACKs

The traces produced while your observability backend was down are exactly the ones you need
for the postmortem. But what happens if the OpenTelemetry Collector relaying that data
restarts at the same time as the backend?

The OpenTelemetry Collector (hereafter "Collector") is open source software that receives
traces, metrics, and logs, applies whatever processing you need, and forwards them to an
observability backend. It ships as a
binary implemented in Go
(architecture docs) and as a
container image. You wire three kinds of components together in YAML:

• Receiver: takes data in from applications and other sources
• Processor: batches, adds attributes, filters, and otherwise transforms data
• Exporter: sends data to destinations such as an observability backend

In this article all data moves over OTLP (OpenTelemetry Protocol), the standard protocol for
sending OpenTelemetry traces, metrics, and logs. The experiments use OTLP/gRPC.

Exporters have a sending_queue that temporarily holds data that cannot be sent yet. You can
keep queue contents in memory only (a "memory queue") or write them to disk with
file_storage (a "persistent queue").

For this article I stopped the destination, let data pile up in the queue, and then either
gracefully or forcibly terminated the Collector. I also built a failure in which the
destination loses its success response (ACK) immediately after durably storing the data, and
counted loss and duplication by Span ID.

Conclusion first: what a persistent queue actually protects

In OpenTelemetry, the whole flow of one request through multiple operations is recorded as a
"trace", and each individual operation as a "span". The input for these experiments was:

What is counted	Count	What it means here
Traces	500	500 application request executions
Spans per trace	2	one parent span and one child span
Spans	1,000	the data actually checked for loss/duplicates
Spans per OTLP request	100	the unit sent to the Collector
OTLP requests	10	how the queue holds them: 10 requests

So "500 traces", "1,000 spans", and "10 requests" throughout this article are the same input
counted in different units. Each span carries a Span ID, and those IDs are what I use to count
loss and duplication.

There are two ways I stopped the Collector:

Signal	Meaning
SIGTERM	requests a graceful shutdown; the Collector can run shutdown logic
SIGKILL	no chance to shut down; the OS terminates the process immediately

I ran six ordinary conditions plus one ACK-loss condition, three times each. Counted at the
destination:

Queue	Failure condition	Span records stored	Unique Span IDs	Duplicates	Lost
Memory	destination stopped, then recovered	1,000	1,000	0	0
Memory	SIGTERM while destination is down	0	0	0	1,000
Memory	SIGKILL while destination is down	0	0	0	1,000
Persistent	SIGTERM while destination is down	1,000	1,000	0	0
Persistent	SIGKILL while destination is down	1,000	1,000	0	0
Persistent	queue files lost after SIGKILL	0	0	0	1,000
Persistent	stored, then died before ACKing the Collector	1,100	1,000	100	0

"Duplicates" is the number of extra records stored relative to unique Span IDs. In the last
condition, 100 Span IDs were each recorded twice, so the total record count came to 1,100.

The most surprising result was the memory queue under SIGTERM. The Collector logged
Shutdown complete. and exited with code 0 — a clean shutdown — but because the destination
was still down it could not drain the queue, and nothing arrived after the restart.

Three takeaways:

1. A memory queue survives a temporary destination outage, but nothing carries over when the Collector itself restarts.
2. A persistent queue can resend after a Collector restart, but only when it can reattach to the same stored data.
3. Losing the destination's success response triggers a resend, so you can get duplicates, not just loss.

A persistent queue widens the range of Collector restarts you can resend through. It is not an
end-to-end guarantee that everything the Collector accepted reaches the backend exactly once.

To see why these results came out this way, let's look at the setup, then pin down what
"successfully sent" actually refers to, and then walk through each failure.

What is under test: app → Gateway → backend

The same Collector binary can be deployed in different roles. The official docs call running it
as a central OTLP endpoint the
Gateway deployment pattern. The
gateway container in this article is just the official Collector image configured in YAML.

Persistent-queue resilience matters most in exactly that shape: applications send OTLP to a
central Gateway Collector, and the Gateway relays to the observability backend.

When backend maintenance or a network failure coincides with a Gateway rolling update, pod
eviction, or process crash, the question becomes whether traces accepted during the incident
survive into the next process.

Gateways are typically deployed as a Kubernetes Deployment or a service on a VM, and are used
for centralized concerns: credentials, egress consolidation, filtering, sampling. They are not
mandatory — small environments, or setups where each language SDK (the library that emits
OpenTelemetry data) sends straight to the backend, can skip them.

The subject under test is a single Collector with a queue and storage area shared with no
other replica. Load balancers, queue sharing across replicas, and SDK-side retries are out of
scope.

Here is how real components map to the test environment. telemetrygen is OpenTelemetry's
official tool for generating test traces and other signals.

Real system / subject	Test-environment stand-in	Not measured
Application / SDK	OTLP/gRPC from the official telemetrygen image	SDK batching, retries, language variation
Gateway Collector	official Collector image configured in YAML	multiple replicas, load balancing
Observability backend	file exporter, or a purpose-built Go sink	indexing, search, backend durability
Fault injection and tally	Docker Compose, POSIX shell, curl, jq	Kubernetes, host-level failures

Because the measurement starts once the Gateway has accepted OTLP/gRPC, the source language is
not fixed. Whether you send from Java, Go, or Python, the fault injection and counting done
here are unchanged. A setup that sends from the SDK directly to the backend has no Collector
queue at all, so you have to check that SDK's own queueing and retry behavior separately.

From here on, the Collector that receives from the application and forwards is the "Gateway",
and the Collector or Go server that stores what it receives is the "Sink". The Sink stands in
for the observability backend in this experiment.

  telemetrygen (app stand-in, 500 traces)
        |  OTLP/gRPC
        v
  Gateway Collector (single instance)
        |
        v
  Exporter sending_queue  ---- file_storage (persistent queue only)
        |
        x  (cannot send while the Sink is down)
        v
  Sink (backend stand-in)

That covers the components. Next, let's split up how far along the path data has to get before
you can call it "sent". That distinction is the foundation for understanding the duplicates
caused by a lost ACK later on.

Background: splitting "it was sent" into four boundaries

An application can succeed at sending without the data being stored in, or searchable from, the
backend. To keep those look-alike states apart, here are four boundaries along the delivery
path.

Boundary	Evidence measured	What it lets you claim
Gateway accepted	otelcol_receiver_accepted_spans=1000	the pipeline accepted 1,000 spans
Enqueued / in flight	otelcol_exporter_queue_size=10	10 requests occupied queue capacity
Send to Sink succeeded	otelcol_exporter_sent_spans=1000	1,000 logical spans obtained a downstream ACK
Sink wrote to file	500 unique Trace IDs / 1,000 unique Span IDs / 0 dupes	the spans are in the file exporter's JSON

Two different success responses (ACKs)

There are two OTLP success responses in this article that are easy to conflate. ACK is short for
acknowledgement — here, the success response returned by whoever received a request.

  telemetrygen --(1) upstream OTLP request--> Gateway
  Gateway      --(2) enqueue--------------->  sending_queue
  Gateway      <-(3) upstream ACK----------   (returned to telemetrygen)
  sending_queue--(4) downstream OTLP request-> Sink
  Sink         --(5) downstream ACK--------->  Gateway

Name	Direction	What success means in this experiment
Upstream ACK	Gateway → telemetrygen	the Gateway pipeline accepted the data and enqueued it
Downstream ACK	Sink → Gateway	the OTLP request from Gateway to Sink succeeded

I explicitly set the exporter helper's wait_for_result to false. With that setting, the
upstream request does not wait for the exporter to finish sending to the Sink. So an upstream
ACK does not mean the Sink stored, indexed, or can search the data.

What the later ACK-loss experiment destroys is the downstream ACK from Sink to Gateway. Even
when the Sink has already stored the data, a Gateway that never receives the downstream ACK will
resend the same request, and duplicates can result.

otelcol_exporter_sent_spans counts logical spans that eventually got a downstream ACK. It is
neither the total number of send attempts nor the total records the Sink stored. In the ACK-loss
experiment, sent_spans stayed at 1000 while the Sink's stored records reached 1,100 spans.

It matters not to conflate the
"accepted by the server" boundary in
the OTLP spec with everything downstream of it: storage, indexing, and searchability.

What the Sink actually measured The Sink's `file` exporter was used only for counting records. Its [stability for traces is alpha as of 0.157.0](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.157.0/exporter/fileexporter). The tally scripts are written to fail if the JSON structure changes. Nothing here measures a real backend's index build, searchable-at time, or durability.

With four delivery boundaries and two ACK directions established, let's pin down when each
failure was injected and which evidence decided success or failure.

Method: seven failure conditions and how they were judged

Testing ran 2026-07-25 to 2026-07-27. I used the official Contrib distribution, which bundles
many extra components including file_storage, pinned to
OpenTelemetry Collector 0.157.0.

The ordinary conditions inject failure in this order:

1. Do not start the Sink, so the Gateway cannot send.
2. Send 500 traces and wait until the Gateway's accepted count reaches 1,000 spans.
3. Wait until queue usage reaches 10 requests.
4. Depending on the condition, leave the Gateway running or stop it with SIGTERM / SIGKILL.
5. Recover the Sink and the Gateway, then count the stored Trace IDs and Span IDs.

For conditions judged as total loss, I waited 10 seconds after Sink recovery — longer than the
5-second maximum retry interval — and confirmed the destination was still empty. Rather than
waiting a fixed time before injecting a failure, each step proceeds only after the accepted count
and queue usage are confirmed.

Only the ACK-loss condition starts the Sink from the beginning. It stops right after storing the
first OTLP request, so no downstream ACK is returned.

This is a low-volume functional test on a single machine, not a performance test. 500 traces is a
number chosen to make loss and duplication easy to count, not an assumed production rate. No
external telemetry backend or credentials are required.

Measured results and hypothetical requirements are kept separate.

The Collector configuration, fault injection code, counts, logs, and CSVs all come from running
the published code locally. The production requirements used as examples later — "tolerate a
10-minute outage" and so on — are hypothetical, for illustration.

Test environment and what was not tested The [captured environment details](https://github.com/yhay81/otel-collector-resilience-lab/blob/main/results/2026-07-28-environment.txt) are saved as well. Prerequisites and the measured environment: | Category | Prerequisite / measured environment | | ------------------ | ------------------------------------------------------------------------------ | | Host | macOS 26.5.1 / Apple M2 Pro / 12 CPU / 32 GiB | | Docker VM | arm64 / 12 CPU / approx. 15.6 GiB | | Container platform | Docker Desktop 4.81.0 / Engine 29.6.1 / Compose v5.2.0 | | Host-side CLI | POSIX shell, `curl`, `jq` | | ACK-loss Sink | Go 1.25.12 / OTLP proto 1.11.0 | | Local resources | TCP 4317/8888 free, and the test code's `data/` writable | | Network (first run)| image pulls from GHCR and Docker Hub, Go module fetch for the ACK-loss Sink | Non-arm64 CPUs, Linux, and minimum CPU/memory/disk requirements were not tested.

The input is 500 traces, 1,000 spans, and 10 requests, with a queue limit of 100 requests.
telemetrygen's --child-spans 1 makes each trace two spans, and --batch-size 100 sends 100
spans at a time. Additional batching inside sending_queue is disabled; retries are unlimited
with a 5-second maximum interval.

The full reproduction set lives in the
public repository with the test code and measured results.
The scripts contain not only the experimental conditions but the tallying logic and failure
conditions too.

The six non-ACK-loss conditions and the ACK-loss condition run with:

git clone https://github.com/yhay81/otel-collector-resilience-lab.git
cd otel-collector-resilience-lab
./run.sh
./run-ack-loss.sh

Three trials each are recorded in the
ordinary-conditions CSV,
the ACK-loss CSV,
and the metrics and logs.
From here, the results follow this order: destination-only outage, Gateway also terminated, ACK
alone lost, and storage itself lost.

Result 1: a memory queue resent fine when only the destination stopped

First, the Gateway kept running and only the Sink was stopped. After sending 500 traces and
waiting for the internal metric otelcol_exporter_queue_size to reach 10, I started the Sink.
All 500 traces / 1,000 spans arrived, with zero duplicate Span IDs.

The memory queue configuration used ```yaml exporters: otlp_grpc/sink: endpoint: sink:4317 tls: insecure: true sending_queue: enabled: true sizer: requests queue_size: 100 num_consumers: 1 wait_for_result: false block_on_overflow: false batch: enabled: false retry_on_failure: enabled: true initial_interval: 1s max_interval: 5s max_elapsed_time: 0s ```

sending_queue acted as a temporary holding area while the destination was down, and
retry_on_failure resent with a gradually widening interval. Because the Gateway process kept
running, the 10 in-memory requests were still there.

So a memory queue prepares you for a backend restart or a brief network interruption. Stopping
the Collector itself is a different story.

Result 2: nothing carried over from the memory queue after the Collector stopped

With the same 10 requests in the memory queue, I sent SIGTERM to the Gateway — in practice
docker compose stop -t 10 gateway. The Collector received terminated, logged
Shutdown complete. within the same second, and exited with code 0. It did not fall through to
the forced kill after 10 seconds.

Even so, not a single trace arrived at the Sink. Shutdown stopped retrying, attempted to drain
the queue, and dropped the 100-span requests it could not deliver to the stopped Sink.

Gateway log on SIGTERM ```text Received signal from OS signal: terminated Starting shutdown... Exporting failed. Dropping data. dropped_items: 100 Shutdown complete. ```

Under SIGKILL in the same condition, no trace arrived either — the data awaiting retry vanished
along with the Collector process.

The conclusion here is not "SIGTERM always loses data". If the destination is up and the data
can be sent during shutdown, the outcome can differ. What this measures is that when you
terminate the Collector while the destination is still down, even a graceful shutdown cannot
carry the memory queue over to the next process.

The reason for the loss is that the queue existed only in the Collector process's memory. So
next, the same data is written to a file outside the process before the same shutdowns are
repeated.

Result 3: a persistent queue resent after both graceful and forced termination

file_storage is an extension that persists queue data to local files. Point an exporter's
sending_queue.storage at that extension and it uses a persistent queue instead of a memory one.

Configuration diff for switching to a persistent queue ```yaml extensions: file_storage/queue: directory: /var/lib/otelcol create_directory: true fsync: true exporters: otlp_grpc/sink: sending_queue: storage: file_storage/queue service: extensions: [file_storage/queue] ```

The endpoint, queue limit, send concurrency, batching, and retry settings are identical to the
memory queue. The only substantive change is where the queue lives.

The Gateway's /var/lib/otelcol was bind-mounted to a host directory, so recreating the Gateway
container still reads the same queue files. Under SIGTERM I confirmed exit code 0 and
Shutdown complete.; under SIGKILL, exit code 137. Both delivered all 500 traces / 1,000 spans
after restart, with zero duplicate Span IDs.

Queue files and logs at restart `file_storage` uses bbolt, an embedded database, internally. Just before `SIGKILL` there was a bbolt file holding the 10 requests. The restart logs show the persistent queue metadata being loaded and the one in-flight request being moved back to the queue. `readIndex: 1` does not mean only one request was stored: `itemsSize: 1000` is the number of retained spans, and `numberOfItems: 1` is the number of requests that had been handed to a consumer at crash time. ```text Loaded queue metadata readIndex: 1, writeIndex: 10, itemsSize: 1000 Moved items for dispatching back to queue numberOfItems: 1 ```

This resume behavior and the ID tallies were consistent across three trials, and match the
behavior documented in the
official file_storage README.

fsync: true asks the OS to sync to disk on every write, improving database consistency across
an interruption — at the cost of write performance. The default is false, so decide based on
the durability you need and measured performance under real traffic. What I compared here is
only SIGKILL against the process: no performance or loss comparison against fsync: false, and
no host power loss, Docker Desktop VM, filesystem, or storage-controller durability. fsync: true
alone is not grounds for claiming power-loss resilience. The file_storage extension itself is
also beta as of 0.157.0. Pin the version, and re-run the same fault injection tests against the
release notes when you upgrade.

So far, when the stored data survives, the Collector can resend after a restart. But no loss does
not imply no duplication. Next, the downstream ACK boundary gets broken.

Result 4: losing the downstream ACK duplicated 100 spans

The Gateway cannot know whether the Sink finished storing unless it receives the downstream ACK.
So this experiment destroys only the downstream ACK from Sink to Gateway.

The purpose-built ACK-loss Sink writes the 100 spans in the first OTLP request to a JSONL file
(one record per line) and fsyncs it. Then, before returning a success response, it exits with
code 23. Docker Compose restarts just the Sink, and the Gateway — having never received a
downstream ACK — resends the same request.

  Gateway --(1) OTLP request, 100 spans--> crash-before-ACK Sink
                                            |
                                            +--(2) append + fsync--> Sink storage file
  Gateway <-X(3) process exits before responding
  Gateway    (4) Unavailable / EOF: back to the queue, retry
  Gateway --(5) resend the same request---> Sink
                                            |
                                            +--(6) append the same 100 spans again
  Gateway <--(7) OTLP success---------------+

All three trials agreed:

Observation	Value (3 trials)
Logical spans the Gateway accepted from upstream	1,000
Logical spans for which the Gateway got a downstream ACK	1,000
Queue usage after recovery	0
Span records stored at the Sink	1,100
Unique Span IDs at the Sink	1,000
Span IDs recorded twice	100
Successful resends of the request holding the first 100 spans	1

All 1,000 unique spans arrived, so nothing was lost. But the first 100 spans were recorded twice
— before the crash and after the resend — bringing stored records to 1,100.

The Sink's logs show the same request hash appearing both in the pre-crash store and in the
post-restart ACK.

ACK-loss Sink log ```text persisted request sha256=0642...d96d2e spans=100; exiting before OTLP response ack-loss-sink ready on :4317 acknowledged request sha256=0642...d96d2e spans=100 ```

On the Gateway side, rpc error: code = Unavailable ... EOF was logged, followed by a retry about
1.1 seconds later. The exporter counted 1,000 spans as successful after the resend, but the Sink
processed 1,100 spans in total.

The OTLP spec is explicit that a disconnect before the ACK can produce duplicates via
retry. This experiment reproduces that
ambiguous window using an identical request hash and identical Span IDs. It is evidence of
duplication across a process crash in a purpose-built Sink — not proof of host power-loss
durability or of any real backend's deduplication.

Up to this point the queue data needed for the resend still existed. Next: losing that storage
itself.

Result 5: losing the stored data meant even a persistent queue could not resend

Despite the name "persistent", once the queue files themselves are gone there is nothing to
resend. In this condition I confirmed 10 requests in the persistent queue, sent SIGKILL to the
Gateway, then deleted the queue directory data/queue before restarting.

This models complete loss of the storage contents. Kubernetes is not involved, so nothing here
tests pod rescheduling to another node or PersistentVolume reattachment. It is not an experiment
showing "a node failure always means total loss". Whatever the reason for the failure, this shows
what happens when the original queue data cannot be read after a restart.

In all three trials, not a single trace arrived at the Sink. file_storage can resume delivery
because it re-reads the remaining queue files. Having persistence configured and being able to
reach that same persisted data after a failure are two different requirements.

On Kubernetes, post-failure behavior differs by storage type. A PersistentVolume (PV) is storage
with a lifetime independent of the pod.

Storage	Caveat
emptyDir	survives container restarts, but disappears with the pod
Local PV	bound to a specific node's disk, so unusable as-is on another node
Network PV	verify reattachment on another node, plus I/O latency and AZ failures for real
External queue such as Kafka	decouples storage from the Gateway, but adds a new operational surface

The range of failure types you intend to protect against is what I call the "failure boundary"
here. If the boundary you accept is process crashes, file_storage plus a reattachable volume is
a candidate. If you need to keep accepting data through node or AZ (availability zone) loss, a
Collector's own persistent queue is not enough.

Storage loss is one limit of persistent queues. There are others, where data is lost before
storage or during resend. Here is where the measured and unmeasured ranges separate.

Limits of this test: conditions a persistent queue does not cover

Reading these results as "file_storage means you never lose anything" is dangerous. What a
persistent queue protects is resuming, after a Collector process restart, a queue that was
written to durable disk. At least these four conditions are outside that protection.

Unprotected condition	What happens	Main countermeasure
Queue limit reached	with block_on_overflow: false, enqueue fails	size capacity; verify upstream failure and retry
Retry deadline exceeded	requests past max_elapsed_time are dropped	set the deadline against your recovery objective
Disk failure/exhaustion	the persistent queue cannot be written	monitor capacity, I/O, and errors
Stored data lost	the original queue files cannot be read after restart	give each pod a reattachable persistent volume

Of those four, only stored-data loss was measured. With this pipeline configuration, enqueue
failures are returned as errors on the upstream OTLP request. If upstream can resend, that is not
immediately data loss — but ignoring the error is. max_elapsed_time: 0s is likewise a setting
chosen to isolate restarts for comparison. Unlimited retention keeps old data around and can fill
the queue or disk first. The official default is 300 seconds.

Also untested:

• Performance and loss compared against fsync: false, host power loss, filesystem corruption
• Kubernetes pod deletion, node shutdown, PVC reattachment, CSI failover
• Real backend acceptance, index build, searchable-at time, deduplication
• Production-equivalent throughput, 10-minute outages, queue drain within 15 minutes, resource usage

So far this has mostly covered everything after the Gateway accepted the data. But the real
delivery path starts at the application. Let's look upstream.

The application side: from SDK to Gateway

Collector and PersistentVolume configuration may belong to a platform team. But you cannot think
about loss between the application and the backend without the SDK's behavior.

This experiment measures only what happens after the Gateway accepted the data. In a real system,
also verify the following between the application and the Gateway:

• How the SDK batches spans, and how much it can hold in memory
• Whether the SDK retries or drops when it cannot connect to the Gateway or gets an error back
• Whether application shutdown waits for unsent data, and for how long
• After a success response from the Gateway, when the data becomes searchable in the backend
• How the backend displays and aggregates the same Span ID arriving more than once

Each language's OpenTelemetry SDK configures batching, queueing, retry, and shutdown differently.
Do not treat "the application's send API returned success" as the delivery-complete condition —
test the SDK and backend you actually use, together.

Once the application-side boundary is settled, the Gateway side turns "how much downtime do we
absorb" into concrete capacity. Logical capacity and disk capacity are separate.

Capacity planning: separate queue_size from disk capacity

Sizing a persistent queue happens in two stages. First derive the Collector setting queue_size,
then translate that into disk capacity for a PersistentVolume or similar.

Logical capacity: derive queue_size from the outage you must survive

Rather than copying a sample value like queue_size: 1000, decide first how long a destination
outage you want to survive. A rough formula:

required queue_size = peak enqueue units/sec x tolerated outage seconds x safety factor

What an "enqueue unit" is depends on the sizer setting.

sizer	What queue_size counts	Characteristics
requests	number of requests entering the queue	lowest computational cost
items	spans, data points, log records, etc.	easiest to match against record counts
bytes	serialized data size	direct size control, highest computation cost

The exporter helper 0.157.0
documentation likewise describes requests as the lightest and bytes as the most
computationally expensive option. Base the calculation not on the application's send count but on
the units that actually enter the queue after processors and exporters have done their work. If
you pick requests, measure spans-per-request as well.

Disk capacity: derive it from actual file growth

The queue_size from the formula above is a logical ceiling, not the byte count a
PersistentVolume needs. Even with sizer: bytes, bbolt's bookkeeping and space that remains
allocated after release mean the queue file size will not match exactly.

Accumulate production-like traces, metrics, and logs for the required duration and measure the
following to set disk capacity and free-space alert thresholds:

• File growth before and after enqueueing
• How much file size remains after the queue drains
• Changes from file_storage compaction settings
• I/O latency in steady state and during recovery
• Safety margin including filesystem reserved space

Once the queue is in place, monitor at least:

• otelcol_exporter_queue_size and otelcol_exporter_queue_capacity
• Enqueue failures such as otelcol_exporter_enqueue_failed_spans
• Send-attempt failures such as otelcol_exporter_send_failed_spans
• Persistent volume usage, free space, I/O latency, and file_storage error logs

These Collector internal metrics are alpha as of 0.157.0; recheck names and attributes when
upgrading. Detect rising utilization together with send failures, before the queue fills. And do
not put the Collector's own metrics solely on the same delivery path — keep an independent health
check.

That covers failure boundaries, capacity, and monitoring — the inputs to an adoption decision.
Finally, let's turn these individual results into a production checklist you can pass or fail item
by item.

Adoption decision: turning measurements into a production checklist

As an example, take a hypothetical requirement: "survive a 10-minute destination outage and one
forced Gateway termination, and drain the queue within 15 minutes of recovery." Do not treat this
scaled-down experiment as production certification; verify the incomplete items below under
production-like conditions.

Check item	Pass condition	Evidence from this test
Temporary outage	no loss of accepted traces	memory and persistent both pass
Graceful shutdown	no loss when terminating/restarting while destination is down	persistent only
Process kill	no loss after SIGKILL	persistent only
Storage loss	declare out of scope, or decide an upstream recovery path	deleting stored data: total loss
Volume reattachment	the same queue is re-read after pod/node rescheduling	not tested
Capacity	holds 10 minutes at peak throughput plus a safety factor	needs production-rate testing
Recovery time	queue returns to normal within 15 minutes of recovery	needs production-rate testing
Security	permissions, encryption, and deletion of stored data verified	review per environment

The decision is not a binary "adopt or skip the persistent queue". Adoption judgment means
separating the process boundary measured here from the node and storage boundaries not yet
verified, and deciding who accepts the remaining risk.

Building a checklist is not proof that things work in a real environment. The last step is to
inject the failures you expect into a pre-production environment and confirm recovery matches
expectations.

Summary: what to confirm before adopting a persistent queue

Enabling a persistent queue does not by itself mean "we now have a configuration that never loses
data". What matters is confirming, in your actual setup, which failures keep the queue intact and
allow delivery to resume afterwards.

From these tests, four things to nail down before production:

1. A success response at the application or the Gateway does not mean the data is searchable in the backend at that moment. Verify acceptance and storage/search separately.
2. Restarting the Collector while the destination is down does not carry memory queue contents to the next process, even on a graceful shutdown.
3. With file_storage, do more than enable the setting: build a storage arrangement that can re-read the same queue data after a restart.
4. Resends can produce duplicates. Beyond loss, confirm how the backend handles duplicates and how you would detect them.

Once the configuration is settled, try each failure one at a time: destination down, SIGTERM,
SIGKILL, queue limit reached, disk exhaustion. Do not let testing end at "the application sent
successfully". Make the completion criterion for a delivery test the ability to actually search
traces generated during the incident after recovery, and to explain both whether anything was lost
and how duplicates appear.

References

• OpenTelemetry Collector: Resiliency
• OTLP specification: Request acknowledgements and duplicates
• telemetrygen traces v0.157.0
• Kubernetes: Volumes

Tested 2026-07-25 to 2026-07-27.

This article was originally published in Japanese on Zenn.