DEV Community: AdmilsonCossa

Your Worker Crashed After the Side Effect. Should It Run Again?

AdmilsonCossa — Tue, 01 Sep 2026 11:52:21 +0000

A process uploads a batch and crashes after chunk 417 completes. When the job
starts again, it must decide whether chunk 417 should run.

Running it twice may duplicate a side effect. Skipping it without durable
evidence is not much better.

An activity boundary gives the operation an explicit identity, hashes its input
and stores its terminal result:

import { run } from "@workit/core";
import {
  createFileActivityStore,
  runActivity,
} from "@workit/core/activity";

const store = createFileActivityStore({
  dir: ".workit-activities",
});

// Application-owned provider boundary. The provider idempotency key remains
// separate from WorkIt's local activity record.
declare function uploadToProvider(
  chunk: number,
  opts: { signal: AbortSignal; idempotencyKey: string },
): Promise<{ etag: string }>;

const uploadChunk = runActivity(
  store,
  {
    activityId: "upload:batch-42:chunk-417",
    input: {
      batchId: "batch-42",
      chunk: 417,
      checksum: "6d7fce9f",
    },
    name: "upload chunk 417",
    version: "v1",
  },
  async (ctx) => {
    return uploadToProvider(417, {
      signal: ctx.signal,
      idempotencyKey: "batch-42:chunk-417",
    });
  },
);

const result = await run.scope(async (scope) => {
  return scope.spawn(uploadChunk);
});

After completion, a new process using the same store can submit the same
activity id and input. WorkIt returns the stored result without invoking the
body again.

That is terminal activity replay. It is not transparent workflow replay.

The application chooses the durable boundary

WorkIt does not persist arbitrary closures, JavaScript stacks or scheduler
state. The caller decides which segment has a durable identity.

The activity contract records:

activity id
activity version
canonical input hash
started timestamp
terminal status
result or bounded error evidence
typed cancellation reason when applicable

This explicit boundary is useful because external side effects rarely have
universal retry semantics. Uploading a chunk, charging a card and sending an
email need different idempotency and repair policies.

The same id with different input is a conflict

An activity id cannot safely mean two things.

import {
  ActivityConflictError,
  createMemoryActivityStore,
  runActivity,
} from "@workit/core/activity";

const memoryStore = createMemoryActivityStore();

const first = runActivity(
  memoryStore,
  { activityId: "export:42", input: { page: 1 } },
  async () => "page one",
);

await run.scope((scope) => scope.spawn(first));

const changed = runActivity(
  memoryStore,
  { activityId: "export:42", input: { page: 2 } },
  async () => "page two",
);

try {
  await run.scope((scope) => scope.spawn(changed));
} catch (error) {
  if (!(error instanceof ActivityConflictError)) throw error;
}

The second body does not run. WorkIt compares canonical input hashes and fails
at the boundary.

Inputs that cannot produce a stable JSON representation are rejected. This
includes cyclic objects, functions, symbols, bigint, non-finite numbers and
undefined.

Completed records replay; uncertain records do not

An activity record can be started, completed, failed or cancelled. Only a
completed record returns its stored result on a later invocation.

Started, failed and cancelled records fail closed. WorkIt does not silently
rerun them because their external effects may be uncertain. A provider could
have accepted a request just before the process stopped, or a cancellation
could have arrived after a remote commit.

The application can inspect the record and choose a repair policy. That may
mean querying the provider by an idempotency key, compensating a partial effect
or authorizing a new activity id.

The runtime preserves evidence instead of guessing.

What restart evidence proves

The release evidence runs the activity, discards the first store instance and
opens a fresh file-store instance over the same directory. A second execution
receives the saved result while the activity body remains at one invocation.

This proves persistence across a store-shaped restart after a completed
terminal write. The current evidence does not claim multi-process coordination,
recovery after process termination or recovery of arbitrary in-flight work.

The application still owns external correctness

Activity records do not replace provider idempotency keys, database
transactions, distributed locks or compensation logic.

The reliable composition is:

application chooses activity identity
provider receives its own idempotency key
WorkIt records terminal activity evidence
restart reuses completed evidence
uncertain states go through an explicit repair policy

This division keeps local lifecycle ownership separate from remote side-effect
authority.

Executable evidence

The relevant proofs are:

CORR-012 explicit activity boundary and conflict detection
LIFE-008 file activity store restart replay

Run them with:

npm run test:evidence
npm run test:coverage
npm run verify

The unit suite also covers input canonicalization, corrupt records, safe file
names, terminal error evidence and cancellation reason persistence.

The useful promise is intentionally bounded: for an explicit activity id and
matching input, WorkIt can reuse a completed terminal result after restart.

Sources

I Built HaloLog, a Go Logger That Measured 23.9 ns/op. Then I Tried to Break the Claim

AdmilsonCossa — Tue, 01 Sep 2026 09:42:52 +0000

Part 1 of Engineering HaloLog: making high-performance telemetry claims
reproducible and falsifiable.

23.9 ns/op is a dangerous number.

It is precise enough to be repeated and incomplete enough to mislead. Without
the work unit, output contract, clock semantics, sink, concurrency model,
hardware, toolchain, and competing configurations, it is not yet an engineering
result. It is only a number with good typography.

I built HaloLog, a structured logger for Go, around a zero-allocation hot path.
On the original benchmark host, its bare-message path encoded and dispatched a
complete JSON record in 23.9 ns. That reciprocal is roughly 42 million records
per second on one goroutine.

The second sentence needs more care than the first. It is arithmetic derived
from a microbenchmark, not a sustained ingestion test. The benchmark writes to
io.Discard; it does not measure disk, network, batching, backpressure, or a
production log collector. And the 23.9 ns value itself is being re-frozen after
a fairness review changed how Zap should be represented.

That review is the reason for this article.

The interesting part of HaloLog is not that one benchmark produced a low
number. It is the attempt to make performance claims executable: allocation
guards fail CI, two encoding architectures are checked against the same byte
contract, optimized paths become ineligible when transforms need structured
data, and competitor configurations remain open to correction.

This is how that system works, where it is strong, and where its claims stop.

First define the operation

The comparative microbenchmark has a narrow job. Each logger must emit one
newline-terminated structured JSON record containing:

a timestamp;
a level;
a message;
the same scenario-specific field names, types, and values.

The measured operation ends when those bytes have been encoded and dispatched
to io.Discard. Every comparative row uses one goroutine. Setup that an
application would perform once, such as constructing a logger or binding
request context, happens before the timed loop. Dynamic call-site fields remain
inside it.

That definition makes the result useful for one question:

What is the CPU and allocation cost of constructing and dispatching this log
record through each public logging API, excluding destination I/O?

It does not answer:

How many records will my service persist through a particular file,
collector, network, or storage stack under concurrent load?

Those questions need different harnesses. Combining them in one number would
hide more than it reveals.

The comparison currently includes HaloLog, phuslu/log, zerolog, Zap, slog,
and logrus, with dependency versions pinned in the benchmark module. It uses
typed fields where the library provides them. Caller and stack capture are not
enabled for one logger while disabled for another. No comparative row receives
a level-gated or disabled-output shortcut.

Even after making those choices explicit, one assumption remained too weak:
the benchmark documented output equivalence, but did not execute an output
contract.

The maintainer feedback that changed the harness

I opened a narrow discussion with the Zap community. The question was not
whether Zap was “fast enough” or whether HaloLog had won. It was this:

For dynamic fields available only at the call site, is
Logger.Info(msg, zap.Int(...)) the idiomatic public hot path Zap should be
represented by? If not, is there another public pattern with the same
timestamp, level, message, and dynamic-field semantics but less per-call
work?

A Zap maintainer replied that the usage looked basic but idiomatic. The useful
part of the reply was elsewhere: Zap's own benchmark
configuration overrides the production time encoder with
EpochNanosTimeEncoder, which is cheaper than the default EpochTimeEncoder.
The maintainer also suggested asserting the actual output so differences such
as time encoding could not pass unnoticed.

That was a valid criticism.

The committed comparison used zap.NewProductionEncoderConfig(). Its default
timestamp is a floating-point count of Unix seconds. Zap's benchmark override
emits integer Unix nanoseconds. Both records still contain a timestamp, level,
message, and the same dynamic fields, but they do not have the same byte
representation.

I measured the encoder change instead of arguing that it would be too small to
matter. In an exploratory Windows A/B, the change was material in the one-field
and request-scoped cases, small in the ten-field case, and too noisy to call in
the twenty-field case. Allocation counts did not change.

I am deliberately not publishing those exploratory percentages as a corrected
cross-platform result. The final comparison must come from the upgraded,
tagged suite with the same sample count and evidence policy on Linux and
Windows. A local result can justify more investigation; it cannot silently
become a public leaderboard.

The correct response is not to overwrite the old Zap cell under the same name.
Changing the time encoder changes the wire representation. The corrected suite
will therefore report two named configurations:

Zap production-default timestamp: fractional Unix seconds;
Zap benchmark timestamp: integer Unix nanoseconds.

The reader can see both the idiomatic production default and the faster variant
used by Zap's own benchmark. Nothing is silently substituted after the fact.

Fairness became a test, not a paragraph

The first new test exercises the same constructors as the benchmark. For each
logger it captures a representative one-field operation and verifies:

exactly one newline-terminated record was emitted;
the record is valid JSON;
timestamp and level are present;
the expected message is present;
the field value is a JSON number, not an accidentally quoted string.

This is a semantic contract. It deliberately does not require every logger to
use the same key names for built-in metadata or the same timestamp format.
Requiring byte identity across different public logger formats would be a fake
form of fairness.

The Zap encoder comparison has a second, stronger test. It installs a fixed
clock and compares the complete bytes against two golden records. That makes
the timestamp difference visible:

{"level":"info","ts":1700000000.1234567,"msg":"request completed","status":200}
{"level":"info","ts":1700000000123456789,"msg":"request completed","status":200}

Same semantic fields. Different wire representation. Both facts matter.

The test does not prove that every benchmark scenario across every library is
byte-equivalent. It proves a representative output contract and pins the exact
Zap difference that triggered the review. Extending that contract is now an
explicit benchmark task, not an assumption buried in methodology prose.

Where the HaloLog hot path saves work

HaloLog's measured path is not one clever instruction. Most of the gain comes
from deciding when work must happen and who owns the resulting bytes.

1. Time is cached, with an explicit accuracy trade-off

The default clock is refreshed by a background goroutine every 10 ms. A log
call reads the cached Unix-nanosecond value instead of acquiring wall-clock time
on every line.

For the default whole-second JSON format, HaloLog caches the entire prefix for
each (second, level) pair:

{"time":"2026-09-01T10:42:17Z","level":"INFO"

On a steady-state cache hit, rendering that prefix is an atomic pointer load
and one append/copy into the line buffer. When the second changes, the next call
rebuilds the immutable entry and atomically publishes it.

This optimization is not free semantics. A cached clock can lag wall time by
approximately its refresh interval, and scheduler pauses can extend that lag.
Whole-second timestamps intentionally discard sub-second detail. HaloLog offers
millisecond, microsecond, and nanosecond formatting, but those settings do not
make the global 10 ms clock fresher. The current public logger configuration
does not inject a different clock. A workload requiring tighter timestamp
freshness should not inherit the default-path claim; configurable clock
freshness is a limitation to resolve and benchmark separately.

2. Request context is encoded when it becomes context

Service logs often repeat the same fields for every line in a request:

reqLog := logger.With().
    WithString("service", "payments").
    WithString("region", "eu-west-1").
    WithString("request_id", requestID).
    WithInt("shard", 7).
    WithString("version", "1.4.2").
    Logger()

reqLog.Typed().WithInt("status", 200).Info("request completed")

The context builder keeps two representations. It stores structured fields for
paths that need transforms, and it encodes their final ,"key":value bytes for
the direct path. After Logger() creates the child, a direct log line appends
the complete bound byte slice once. It does not re-run five key scans and five
value encoders on every record.

Construction is not free. Building the context allocates and copy-on-appends to
keep branched builders from sharing writable backing. HaloLog caps bound context
at 32 fields. Those costs occur when the child logger is derived, outside the
per-line benchmark. A fair evaluation should report both construction cost and
steady-state line cost; they answer different lifecycle questions.

The measured request-scoped scenario uses primitive typed values. Arbitrary
Any values, mutable references, duplicate keys, and transform-heavy
configurations deserve separate ownership and semantic tests; they do not get
to borrow the primitive direct-path result.

3. The fast path is an eligibility decision

With one raw-capable JSON adapter and no masking or sampling, HaloLog can encode
typed fields directly into a pooled byte buffer. The line becomes:

cached header + escaped message + bound context + call-site fields + "}\n"

There is no intermediate LogEntry on that path.

But masking and sampling need structured data. If either is configured, or if
the output shape cannot use the direct JSON encoder, the logger selects the
capture path at construction. Bound context is then prepended as structured
fields, the transform sees the complete record, and the formatter emits it
afterward.

This is the security property I care about: the optimized path is not asked to
remember to call the masker. It is ineligible when masking exists.

That does not prove every regular expression is a sufficient PII policy, that
arbitrary values cannot contain secrets, or that encryption and retention are
solved. It proves a narrower architectural invariant: configuring the shipped
masking transform prevents this direct encoder from bypassing it.

4. The string scanner optimizes the common case and retains an oracle

JSON strings only need special handling for control bytes, quotes, and
backslashes. On the little-endian targets covered by the current implementation,
HaloLog's scanner examines eight bytes at a time with SWAR (SIMD Within A
Register) bit tricks. A clean word advances without eight separate table
checks. A possible hit falls back to the exact scalar escape path. Other
architectures must earn their own correctness and performance result; this
article does not infer one from amd64.

The optimization is guarded by two different kinds of evidence:

exhaustive placement of every possible byte value across SWAR lanes and boundary positions;
structured cases and a Go fuzz target compared against the retained scalar implementation.

“Exhaustive” applies to the enumerated byte-position test, not to the infinite
space of possible strings. Fuzzing explores additional combinations; it is not
a proof by itself.

An experimental portable SIMD implementation produced a surprising result on
one Go version and machine: it lost to the shipped SWAR scanner at typical log
string sizes. That result is interesting, but it is not evidence that SIMD is
inherently slower. It belongs to a separately reproducible experiment with the
exact branch, experiment flag, reductions, size distribution, and crossover
points. The full analysis will be a later article rather than a paragraph that
generalizes past its data.

Zero allocation is a regression gate, not an adjective

Benchmark output saying 0 B/op once is weak evidence. A compiler update, an
interface escape, or a seemingly harmless refactor can bring the allocation
back.

HaloLog currently has eight TestZeroAlloc* guards covering selected paths:

message-only logging;
interface-style field chains;
real-adapter dispatch rather than only a discard shortcut;
direct message encoding;
bound-context direct and capture paths;
typed inline and pooled field builders;
the direct-path field API;
the level-first line API.

They use testing.AllocsPerRun and fail when the protected operation allocates.
The benchmark workflow treats those tests as a hard gate while reporting
latency without failing on noisy shared-runner nanoseconds.

This claim has a boundary too. Context construction allocates. Configuration
allocates. Long or hostile values can outgrow retained buffers. Some adapters,
arbitrary-value formatting, transforms, and asynchronous ownership have
different cost models. “Eight protected hot paths remain at zero allocations”
is defensible. “The library never allocates” is not.

Allocation tests are not concurrency evidence. That is a separate gate. The
main CI calls a reusable workflow pinned by commit and runs the root module
with the race detector on both Linux and Windows. The otelbridge module runs
its own race-enabled suite. This still does not establish linear scalability or
prove the absence of all concurrency bugs; it means the committed test
executions are race-instrumented on those runners.

Two encoders, one tested byte contract

The direct path is optimized for the simplest eligible JSON configuration. The
capture path exists because real logging systems need transforms, multiple
adapters, sampling, metrics, and less specialized values.

Maintaining two implementations normally creates semantic drift. A quote is
escaped in one but not the other. A negative timestamp crosses an epoch
boundary differently. A bound field appears before call-site fields in one
path and after them in another. A security transform sees only one
representation.

HaloLog keeps a reference-style formatter and compares the direct path against
it. Golden tests pin complete output segments. Context tests normalize only the
timestamp portion and compare the remaining bytes. Key-prefix and string
oracles retain the older scalar form and compare optimized output against it.

This is differential testing applied inside one library. It does not prove that
the chosen JSON contract is universally ideal. It makes unintended divergence
between the two implementations observable.

The same discipline applies to halologgen, the included schema generator. It
regenerates the committed example into a temporary directory and byte-compares
the result. A misspelled field method or wrong value type becomes a compile-time
problem; nondeterministic generator output becomes a failing test. That is a
different thesis from runtime encoding performance, so it receives its own
article in this series.

Why there is no corrected ranking in this article

The repository retains the original published comparison and its methodology.
I am not pasting that table here and quietly changing one competitor's
configuration underneath it.

The corrected evidence release will upgrade Zap, name both timestamp encoders,
and report Linux and Windows separately. It will use at least ten samples per
row, retain raw benchmark output, publish benchstat summaries, and treat cells
inside the declared noise threshold as ties. A delta measured on one operating
system will not be projected onto another.

Until that release is frozen, the defensible result is methodological:

The maintainer feedback identified a faster, semantically different Zap
timestamp configuration. The difference was large enough in exploratory
measurement to require a separately named result, and the harness now has
executable output checks that make the distinction visible.

That conclusion survives even if HaloLog's final median or ranking changes.

What this article does not prove

It does not prove that HaloLog is the best logger for every Go service.
Performance is one selection criterion. API stability, ecosystem maturity,
operational familiarity, integrations, support horizon, and the cost of
changing an existing logging stack may matter more.

It does not measure storage throughput or durability. io.Discard removes the
destination so encoding and dispatch remain visible. A file, socket, collector,
or congested pipeline can dominate the total cost.

It does not show multi-goroutine scalability or p99 caller latency. Those need a
macro harness with controlled sinks, concurrency, queue capacity, drop policy,
and backpressure. That work is planned separately.

It does not prove universal zero allocation. It identifies and guards exact
paths on a named Go toolchain.

It does not prove SIMD is slow, that a 10 ms cached clock is suitable for every
system, or that matching four semantic fields makes all logging formats
interchangeable.

And a new public project with a small external user base has not earned the
ecosystem evidence of Zap, zerolog, or the standard library. Microbenchmarks can
measure code. They cannot manufacture operational history.

Reproduce it, then try to invalidate it

The repository pins dependency versions, test sources, and the historical
methodology. The core checks are ordinary Go tests:

go test ./core -run 'TestZeroAlloc|TestDirectPath|TestContext_' -count=1

go test ./adapters/formatters/json \
  -run 'TestSWAR|TestDirectMatchesFormat|TestAppendKeyPrefix|TestFormatterGolden|TestHeaderZeroAlloc' \
  -count=1

go test -race ./... -count=1

The corrected comparative run uses the normal Go benchmark toolchain. The Zap
timestamp A/B remains a named benchmark instead of being hidden inside a
generic Zap row:

cd benchmarks
go test -run '^$' \
  -bench 'Benchmark(Info|OneField|TenFields|TwentyFields|Context|ZapTimeEncoder)$' \
  -benchmem -benchtime=1s -count=10 . | tee results.txt
benchstat results.txt

There are several useful ways to break the claim:

produce a record that violates the declared output contract;
find a protected path that allocates on the release toolchain;
demonstrate a race or ownership failure;
show that a competitor configuration performs equivalent work with less overhead;
reproduce a ranking change on a documented machine;
find a masking, escaping, timestamp, or context case where the two paths diverge.

Open an issue with the smallest counterexample and the complete environment.
That is more valuable than a benchmark screenshot.

The result I want to preserve

The original result was a low latency number. The more important result is a
process that can survive having that number challenged.

A maintainer found a configuration difference. The benchmark changed. Output
assumptions became tests. The faster competitor path will be published beside
the production default instead of hidden or used to overwrite history.

That is the standard I want HaloLog to meet:

Fast paths may be specialized. Claims about them may not be vague.

The next articles will go deeper into benchmark contracts, bind-once request
context, byte-identity testing, allocation guards, SWAR versus portable SIMD,
compile-checked telemetry generation, and the gap between encoding
microbenchmarks and logging under load.

For now, the invitation is simpler: run the evidence, inspect the output, and
find the condition I missed.

Repository: Go-Gen-Ecosystem/halolog

Benchmark source: comparison_bench_test.go

Request-context benchmark: context_bench_test.go

Published historical methodology: comprehensive_comparison.md

Zap fairness discussion: uber-go/zap discussion #1576

The next article will isolate benchmark fairness itself: timestamp semantics,
output contracts, maintainer feedback, and what belongs inside the timed loop.

AI Agents Are Making an Old Distributed Systems Bug More Expensive

AdmilsonCossa — Fri, 07 Aug 2026 12:14:13 +0000

The dual-write problem existed long before LLMs. Now a duplicate event can trigger fraud investigations, payment holds, refunds, or financial execution.

At 11:57 PM on Black Friday, a global commerce platform approves a €2.4 million payout to a merchant.

The ledger commits the transaction. The application publishes PayoutApproved. Then the request times out.

Nothing is necessarily down. The database is healthy. The broker may also be healthy. The network recovered milliseconds later. Every dashboard is still green.

Yet the system has lost something more dangerous than availability.

It has lost certainty.

Did the broker receive the event?

If the producer does nothing, the risk system may never analyze the payout. If the producer retries — and the first publication actually succeeded — the event may be delivered twice. If downstream execution is not idempotent, the same business operation may be applied twice.

A few milliseconds of uncertainty can become a multimillion-euro question:

Did the operation fail, or did we merely fail to observe that it succeeded?

This is not fundamentally a Kafka problem, a NATS problem, or an AI problem. It is a distributed systems problem.

Modern AI agents make it more expensive because event consumers are becoming more capable. A duplicate event no longer has to mean a duplicated projection or a repeated log line. It may wake an agent that assesses fraud, recommends a payment hold, launches an investigation, requests a refund, or calls a tool connected to an external financial system.

The delivery semantics did not get worse. What sits behind the event became more powerful.

The architecture needs answers to six questions:

Where does the event become authoritative?
How is publication recovered after failure?
How do consumers tolerate redelivery?
How is ordering preserved where the business requires it?
How does each irreversible effect receive a stable identity, survive executor crashes, and remain reconcilable when its outcome is unknown?
How does a probabilistic assessment become one authoritative, deterministic decision?

1. Your Broker Does Not Solve the Dual-Write Problem

A service that approves a payout typically performs two operations that conceptually belong together but live in two separate transactional systems: the database and the broker.

There is no automatic atomic transaction spanning both systems, which creates three distinct failure states:

Failure	Consequence
Database commits, event publish fails	Business state exists, downstream systems never learn about it
Event published, database transaction rolls back	Consumers observe something that never became true
Broker accepts event, ACK is lost, producer retries	Possible duplicate delivery

This is the dual-write problem. A broker gives you decoupling, buffering, durability, fan-out, replay, and failure isolation. What it does not give you is atomicity between an event and the database state that caused that event to exist. That distinction is the foundation for everything that follows.

2. Broker Guarantees Stop at Their Transaction Boundary

The useful question is not “which broker is best?” It is:

Which state transitions can this product make atomic, and where does that atomicity stop?

Kafka and Redpanda can provide transactional guarantees across supported broker operations. Kafka Streams, for example, can atomically combine consumed offsets, state-store updates, and produced records within the Kafka transaction boundary.

RabbitMQ provides durable queues, publisher confirms, acknowledgements, quorum queues, and routing mechanisms. NATS JetStream provides durable streams, acknowledgements, redelivery, and time-bounded publication deduplication.

These are valuable guarantees, but they belong to different boundaries:

Boundary	Guarantee that may be available
Broker record → broker record	Broker transaction or atomic stream-processing operation
Producer → broker	Confirmed or acknowledged publication
Broker → consumer	Durable delivery with acknowledgement and redelivery
Application database → broker	Not atomic without an Outbox or equivalent mechanism
Broker → external provider	Not atomic
Internal ledger → bank transfer	Not atomic without a provider-level uniqueness and reconciliation contract

A broker transaction cannot atomically commit an unrelated Postgres transaction, a payment-provider request, or a bank transfer.

Choose the broker from the workload. Design correctness from the transaction boundary.

3. AI Agents Don't Create the Problem — They Increase Its Blast Radius

The dual-write problem predates LLMs by decades. Retries, duplicate messages, lost acknowledgements, and reordered events are established distributed systems failure modes. What's changed is what a consumer can now do with a duplicate.

If the old projection consumer processes a duplicate twice, and the projection is idempotent, the impact may be negligible. If the same duplicate now traverses an agentic path, it can trigger a second fraud investigation, a second account block, or a second refund instruction.

That doesn't make AI agents inherently unsafe. It means the architecture around them must assume messages arrive more than once, agents can be wrong, tools can time out, models can change, and external systems can return ambiguous outcomes.

Once an agent participates, the architecture must distinguish three identities:

eventId — the immutable identity of a fact that happened;
decisionRequestId — the identity of one request for an assessment or decision;
operationId — the identity of one intended external business effect.

These identifiers are related through causation, but they are not interchangeable. One event may legitimately trigger multiple decisions, and one approved decision may produce multiple external operations. Reusing eventId for every boundary can suppress legitimate work or accidentally collapse distinct effects into one.

An AI agent should never move money simply because it generated a plausible answer. The safer boundary separates reasoning from execution:

AI recommends. Policies decide. Systems execute.

The agent analyzes, classifies, explains, and recommends. A deterministic service evaluates limits, permissions, segregation of duties, and compliance. Only then does an idempotent executor perform the external effect.

A retry of the same decisionRequestId should return the already-recorded assessment. A reevaluation caused by new evidence, a new aggregate version, or an explicit review request receives a new identity. Idempotency here does not require a probabilistic model to generate identical tokens; it prevents a transport retry from creating a second authoritative decision.

4. CDC Is More Than a Legacy Escape Hatch

Enterprise systems rarely start clean. Banks, insurers, and telecoms often depend on ERPs and settlement engines built years before event-driven architecture became mainstream. Rewriting them to emit domain events may be too risky.

Change Data Capture — commonly Debezium — observes transactional database logs and captures committed changes without touching the legacy application.

The trap: a row mutation is not automatically a domain event. STATUS = 'A' might mean approved, awaiting review, or an internal transition with no meaning outside the legacy system. If downstream services must understand physical tables and cryptic status codes, the database schema has silently become a public contract — and a fragile one.

A translation layer protects the rest of the organization from the legacy system's physical representation. CDC is also strategic beyond migration: it can feed search indexes, warehouses, read models — and most importantly, it can capture an Outbox table, transporting events the application deliberately designed rather than technical noise.

CDC also inherits the ordering and transaction semantics of the source log. A translation layer must preserve transaction identity and commit order where the downstream business invariant depends on them; row-level change order alone is not automatically domain-event order.

5. Transactional Outbox Fixes the Birth of the Event

A new payment service knows two facts at the same instant: the payout changed state, and an event describing that change must eventually be published. Instead of two independent writes, it records both facts in one local transaction.

BEGIN;

WITH advanced_payout AS (
    UPDATE payouts
    SET status = 'APPROVED',
        version = version + 1
    WHERE id = 'pay_7f9q'
      AND version = 16
    RETURNING id, version
)
INSERT INTO outbox (
    event_id,
    aggregate_id,
    aggregate_version,
    event_type,
    payload,
    occurred_at
)
SELECT
    'evt_01K...',
    id,
    version,
    'PayoutApproved',
    '{"payoutId":"pay_7f9q"}',
    CURRENT_TIMESTAMP
FROM advanced_payout
RETURNING aggregate_version;

-- The application must require exactly one returned row.
-- Zero rows means that the expected version was stale.

COMMIT;

The version is advanced on the aggregate row inside the same transaction that records the Outbox event. It is not allocated from a standalone database sequence.

That distinction matters. Database sequences provide unique values, not gapless committed aggregate history. In PostgreSQL, a value obtained from nextval() is not reclaimed when the surrounding transaction rolls back. A consumer waiting for every sequence value could therefore wait forever for an event that can never exist.

The schema should protect both identities:

CREATE UNIQUE INDEX outbox_event_id_uq
    ON outbox(event_id);

CREATE UNIQUE INDEX outbox_aggregate_version_uq
    ON outbox(aggregate_id, aggregate_version);

The first constraint protects global event identity. The second prevents two committed events from claiming the same position in one aggregate's history.

If the transaction fails, neither the state transition nor the event exists. If it commits, both exist with the same aggregate version. Publication becomes a recoverable asynchronous process rather than a second point of failure.

This solves the birth of the event. It does not guarantee that publication succeeds before retention expires, that a consumer applies the event, or that an external provider executes the intended effect.

6. Outbox Does Not Mean Exactly-Once

The duplicate here isn't evidence of a broken broker — it's the correct consequence of recovering safely from uncertainty.

Do not build correctness on the assumption that delivery happens exactly once. Build consumers so repeated delivery does not repeat the business effect.

The working invariant:

Record intent atomically. Retry delivery until it is durably observed. Give every business effect a stable identity. Reconcile every outcome that cannot be proven.

Kafka and Redpanda offer exactly-once semantics inside defined transactional boundaries. NATS offers publication deduplication within a configured window. None of that automatically makes POST https://external-bank/pay exactly-once — the broker doesn't control the bank.

Even “at least once” depends on operational assumptions: retention must not expire, durable storage must remain available, incompatible schemas must not block the consumer indefinitely, and operators must not discard unresolved records. Correctness therefore requires monitoring the age of the oldest unpublished and unprocessed intent, not merely counting successful messages.

7. Inbox Deduplication Is Atomic Only With Local Effects

Outbox solves one source of duplication, not all of them. If a client retries POST /payouts after losing the response, the service may create two payouts before the Outbox becomes relevant. Ingress commands therefore need a stable idempotency key and a stored result.

At consumption time, an Inbox prevents repeated delivery from repeating a local effect — but only when the Inbox row and that effect commit in the same database transaction.

BEGIN;

WITH accepted AS (
    INSERT INTO inbox (
        consumer_name,
        event_id,
        received_at
    )
    VALUES (
        'payout-projection',
        'evt_01K...',
        CURRENT_TIMESTAMP
    )
    ON CONFLICT (consumer_name, event_id) DO NOTHING
    RETURNING event_id
)
UPDATE payout_projection
SET status = 'APPROVED',
    source_version = 17
WHERE payout_id = 'pay_7f9q'
  AND EXISTS (SELECT 1 FROM accepted);

COMMIT;

-- Acknowledge the broker message only after COMMIT succeeds.

The uniqueness scope includes consumer_name because different consumers may legitimately process the same event.

The crash behavior is now safe:

Crash point	Result
Before commit	Both the Inbox insert and projection update roll back; the broker redelivers
After commit	Both the Inbox row and projection update exist; redelivery is a local no-op

Inbox retention must also cover the maximum broker replay and recovery horizon. Deleting deduplication rows while the corresponding events can still be replayed reopens the duplicate window.

An Inbox row is therefore a completion record only when the business effect lives inside the same local transaction.

An HTTP request, bank transfer, email, or agent tool invocation cannot join that transaction.

8. External Effects Need a Durable Operation Ledger

For an external effect, “event observed” and “business operation completed” must be separate durable facts.

If a consumer commits an Inbox row and then crashes before calling the provider, redelivery may find the event already recorded and suppress work that never happened. Inbox conflicts stay normal, the dead-letter queue remains empty, and the missing effect can look like a healthy no-op.

The consumer must atomically record recoverable work before acknowledging the event:

BEGIN;

WITH accepted AS (
    INSERT INTO inbox (
        consumer_name,
        event_id,
        received_at
    )
    VALUES (
        'payment-executor',
        'evt_01K...',
        CURRENT_TIMESTAMP
    )
    ON CONFLICT (consumer_name, event_id) DO NOTHING
    RETURNING event_id
)
INSERT INTO external_operations (
    operation_id,
    source_event_id,
    aggregate_id,
    operation_type,
    status,
    idempotency_key,
    attempt_count,
    created_at
)
SELECT
    'payout:pay_7f9q:release:decision-12',
    'evt_01K...',
    'pay_7f9q',
    'RELEASE_PAYOUT',
    'PENDING',
    'payout:pay_7f9q:release:decision-12',
    0,
    CURRENT_TIMESTAMP
FROM accepted
ON CONFLICT (operation_id) DO NOTHING;

COMMIT;

-- Acknowledge the broker event only after both durable records exist.

The accepted dependency matters: it makes creation of the external operation conditional on this consumer accepting the event. operationId identifies the intended business effect. It should not automatically be the same as eventId: one event may legitimately produce more than one effect, while several redeliveries may refer to the same effect.

A separate worker claims durable operations using an expiring lease:

IN_FLIGHT is not allowed to remain permanently owned by a dead worker. When its lease expires, the operation is recovered by a scanner. If dispatch may already have started, the scanner must reconcile or retry with the same provider idempotency key — never assume that the first attempt failed.

The recovery contract must cover every crash window:

Crash point	Durable state	Recovery
Before the operation transaction commits	No operation exists	Broker redelivery recreates it
After commit, before provider dispatch	`PENDING` or an unstarted expired claim	Worker claims and dispatches it
After dispatch may have started, before local confirmation	`IN_FLIGHT` with an ambiguous outcome	Query the provider or retry with the same idempotency key
Provider response times out	`UNKNOWN`	Reconciliation resolves the outcome
After `CONFIRMED`	Final result exists	Redelivery and repeated claims are no-ops

A timeout means UNKNOWN, not FAILED. Retrying an UNKNOWN operation without provider-level uniqueness may duplicate the external effect.

A provider idempotency key is useful only within the provider's documented scope and retention window. The application's retry and reconciliation horizon must fit that contract. Some providers return the previously stored result for a repeated key; that is a provider guarantee, not a general HTTP property.

If an irreversible provider supports neither idempotent execution nor lookup by stable business identity, the caller cannot guarantee exactly-once execution. The safe choices are to require manual resolution, introduce a compensating control, or reject that provider for autonomous financial execution.

9. Ordering Requires Gap Semantics, Not Just Version Numbers

“Messages are ordered” is incomplete. The useful question is:

Ordered with respect to which aggregate, which committed history, and which consumer contract?

Processing v17 → v19 → v18 may produce an invalid state even when every message eventually arrives. An aggregateVersion makes stale events and possible gaps detectable, but the number does not enforce ordering by itself.

A consumer can observe at least three different kinds of gap.

A temporary delivery gap

v19 arrives before committed event v18. A bounded reorder buffer may wait briefly for v18.

A permanent allocation gap

If versions come from a non-transactional database sequence, a rolled-back transaction may consume v18. The next committed event is v19, and v18 will never exist.

Aggregate versions used for contiguous ordering should therefore be advanced on the aggregate row or event stream inside the same transaction that records the event.

A legitimate consumer gap

An aggregate may produce:

v17 PayoutApproved
v18 BeneficiaryVerified
v19 PayoutReleased

A consumer subscribed only to payout lifecycle events may observe v17 → v19. That does not prove that v18 is missing or delayed.

A consumer may require contiguous aggregate versions only when it is guaranteed to observe every committed event in that aggregate stream. A filtered consumer must instead do one of the following:

consume all aggregate events and no-op the irrelevant types while advancing its observed version;
use a consumer-specific sequence;
fetch the authoritative aggregate state when it observes a jump;
treat versions as monotonic stale-event protection without assuming contiguity.

A reorder buffer must always have explicit bounds:

MAX_GAP_AGE
MAX_BUFFERED_EVENTS_PER_AGGREGATE
MAX_BUFFERED_BYTES_PER_AGGREGATE

When a bound is exceeded:

1. Stop applying new events for that aggregate.
2. Do not block unrelated aggregates on the same partition or shard.
3. Fetch the missing event or an authoritative snapshot.
4. Rebuild or reconcile the consumer state.
5. Escalate when the authoritative state cannot be established.

A Kafka partition is a storage and ordering boundary. A NATS subject is a routing address. In either case, records for the same ordered aggregate must be routed deterministically to the same serial execution boundary.

finance.payouts.eu.s042.pk_7f9q.approved

Parallel workers may preserve arrival order while violating completion order. If an external effect depends on order, the operation ledger must serialize or reject conflicting operations for the same aggregate.

Versions expose ordering violations. Serialization, bounded buffering, authoritative reads, and reconciliation preserve the business invariant.

10. Edge and Multiregion Systems Make Authority Explicit

A branch, regional gateway, or edge deployment that must survive a disconnected network needs a local database and local Outbox that synchronizes once connectivity returns.

A NATS leaf node does not automatically turn a local database into a conflict-free offline store. The architecture must still define: which system is authoritative per entity, how conflicts resolve, how gaps are detected, what happens when local disk fills, and how reconciliation completes.

If two regions may concurrently mutate the same aggregate, aggregateVersion can detect the conflict but cannot resolve it. The design still needs a single-writer rule, explicit ownership transfer, or a domain-specific merge protocol. “Last write wins” is not a safe default for financial state.

Local availability with eventual global convergence — not global consistency independent of the network.

11. The Reference Architecture

The essential building blocks are: an authoritative transactional source, an atomic Outbox, recoverable publication, durable transport, immutable event identity, transactional Inbox processing for local effects, bounded ordering recovery, stable decision identity, deterministic policy boundaries, a durable external-operation ledger, provider-level idempotency where available, and reconciliation where certainty cannot be obtained synchronously.

12. Event Envelopes Are Part of the Contract

{
  "eventId": "evt_01K...",
  "eventType": "PayoutApproved",
  "aggregateId": "pay_7f9q",
  "aggregateVersion": 17,
  "schemaVersion": 3,
  "occurredAt": "2026-08-07T10:57:01Z",
  "correlationId": "corr_8x7b",
  "causationId": "cmd_9c3a"
}

eventId establishes the immutable identity of this fact. aggregateVersion identifies its committed position in the aggregate history and makes stale delivery or a possible gap detectable; it does not enforce ordering by itself.

correlationId groups a wider business flow. causationId identifies the command or event that directly caused this event. Neither should be used as an idempotency key without first defining the business operation being deduplicated.

The architecture should maintain separate identities for separate boundaries:

Identity	Meaning
`eventId`	One immutable fact
`decisionRequestId`	One request for an assessment or decision
`operationId`	One intended external business effect
`correlationId`	A wider end-to-end business flow
`causationId`	The direct cause of this record

schemaVersion also requires an explicit compatibility and migration policy. A version number without rules for backward compatibility, replay, and unsupported consumers is only metadata.

13. AI Decisions Need Stable Identity and Evidence

“Payout blocked” is not an adequate decision record. A serious system must preserve enough evidence to reconstruct why an assessment existed and how it became an authoritative action.

{
  "decisionRequestId": "dec_req_01K...",
  "sourceEventId": "evt_01K...",
  "aggregateId": "pay_7f9q",
  "aggregateVersion": 17,
  "evidenceSnapshotId": "evidence_42",
  "evidenceSnapshotHash": "sha256:...",
  "modelVersion": "risk-model-2026-08",
  "promptVersion": 4,
  "toolsetVersion": 3,
  "policyVersion": 12,
  "assessment": {
    "riskLevel": "HIGH",
    "confidence": 0.91
  },
  "createdAt": "2026-08-07T10:57:03Z"
}

The first completed assessment for a decisionRequestId becomes the recorded result. Redelivery of the same request returns that result instead of invoking the model again.

This is decision idempotency:

Retry of the same logical request
    → return the recorded assessment

New evidence, new aggregate state, or explicit reevaluation
    → create a new decisionRequestId

The goal is not to pretend that a probabilistic model is deterministic. Exact replay may be impossible when hosted models, retrieval indexes, tools, or safety systems change. The goal is to reconstruct the inputs, versions, evidence, recorded output, policy evaluation, approvals, and executed operation.

The AI assessment is evidence, not financial authority. A deterministic policy service evaluates limits, permissions, segregation of duties, compliance rules, and approval state. Only an authorised policy result may create an external operation.

Agent tools should follow the same boundary. Read-only evidence tools may be called during assessment. Tools that create irreversible effects must be represented as authorised commands and executed through the external-operation ledger.

14. Observability Must Detect Missing Effects, Not Just Failures

A dashboard with green CPU, healthy brokers, and an empty dead-letter queue can still hide a missing payment. The most dangerous failure may look like a successful deduplication no-op.

Correctness observability needs to measure both backlog and age:

oldest_pending_outbox_age
oldest_unprocessed_event_age
oldest_pending_operation_age
oldest_in_flight_operation_age
expired_operation_claim_count
unknown_operation_age
reconciliation_backlog
reconciliation_mismatch_count
gap_buffer_oldest_age
gap_buffered_events
authorised_decisions_without_operation
confirmed_operations_without_provider_match
provider_effects_without_internal_confirmation

Backlog size measures load. Oldest-item age measures whether the system is making progress.

Thresholds should come from business SLOs, provider idempotency windows, settlement deadlines, and the maximum acceptable uncertainty period. A universal threshold such as “five minutes” is meaningless without those contracts.

The system should periodically reconcile four sets:

Examples of correctness violations include:

an authorised decision with no durable external operation;
a PENDING operation older than its dispatch SLO;
an expired IN_FLIGHT claim that no worker recovered;
an UNKNOWN operation older than the reconciliation SLO;
an internally confirmed payment missing from the provider;
a provider-side payment with no matching internal operation;
an aggregate gap held beyond its configured bound.

The most dangerous state is not merely:

service = DOWN

It is:

business_effect = UNKNOWN

UNKNOWN must be a first-class, queryable business state with ownership, age, reconciliation policy, and escalation — not a log line inside a generic error counter.

15. Security Must Survive the Event Pipeline

Event systems copy information widely, which makes careless identifiers expensive. Don't embed account numbers, card details, or national IDs in topic or subject names — they leak into logs, metrics, traces, and dashboards.

❌ finance.account.447819002343.payment
✅ finance.payouts.eu.s042.pk_7f9q.approved

An authenticated event is not permanent authorization to repeat an action. Replays and delayed deliveries must pass the current execution contract or an explicitly versioned historical authorization contract.

Agents should not hold unrestricted provider credentials. The execution worker should receive narrowly scoped authority to perform one approved operationId, with amount, currency, beneficiary, policy decision, and validity window bound to that operation.

Encryption in transit and at rest, fine-grained access control, schema governance, controlled replay, dead-letter handling, and data residency controls are non-negotiable. Durability without governance only makes mistakes survive longer.

16. Choose the Mechanism From the Invariant

The architecture should follow the invariant. The invariant should not follow the product logo.

Required invariant	Mechanism
Business state and publication intent are born together	Transactional Outbox
A local projection is not applied twice	Inbox and local effect in one transaction
An external effect survives executor crashes	Durable external-operation ledger
Retried provider calls do not repeat an effect	Stable `operationId` and provider idempotency contract
Ambiguous provider outcomes become resolvable	`UNKNOWN` state and reconciliation
Aggregate transitions do not complete out of order	Deterministic routing and serial execution per aggregate
A missing version cannot block forever	Bounded buffer and authoritative reconciliation
A transport retry does not create another AI decision	Stable `decisionRequestId` and stored assessment
Probabilistic reasoning cannot directly move money	Deterministic policy and approval boundary
Regional autonomy does not create silent conflict	Explicit authority, ownership transfer, or merge protocol
Replay does not bypass governance	Versioned authorization, schema, retention, and access policy

Products implement parts of these mechanisms. They do not define the business invariant.

17. Exactly-Once Requires an Explicit Boundary

“Does this broker support exactly-once?” is rarely the useful question.

The useful questions are:

Exactly once inside which transaction boundary?
Which system assigns the business effect its identity?
Which component rejects a repeated identity?
How long is that identity retained?
Can an ambiguous result be queried?
What happens when certainty cannot be recovered automatically?

Kafka can provide exactly-once processing across supported Kafka operations because the relevant offsets, state, and produced records participate in the Kafka transaction boundary. That guarantee does not automatically include an unrelated database or external provider.

For irreversible effects, correctness contains both safety and liveness:

Safety:
The same operationId must not create two business effects.

Liveness:
Every authorised operation must eventually become
CONFIRMED, FAILED_TERMINAL, or explicitly escalated.

Deduplication without liveness is not sufficient. An Inbox tombstone that suppresses work which never reached the provider may prevent a duplicate while permanently losing the intended effect.

The end-to-end contract is therefore:

Record each business intent atomically.
Deliver until the intent is durably observed.
Create one stable identity per intended effect.
Use destination-level idempotency where available.
Never interpret timeout as definitive failure.
Reconcile every ambiguous outcome.
Escalate when the architecture cannot prove the result.

Exactly-once is not merely a transport property. It is a claim that must name its boundary, identity, enforcement point, retention window, and recovery protocol.

If an irreversible provider offers neither idempotent execution nor lookup by stable business identity, the architecture must not claim exactly-once execution across that boundary.

18. What AI Changes

AI does not repeal distributed systems. Agents still run on networks. Tools still time out. Databases still commit independently. Messages can still be duplicated.

AI also introduces a probabilistic decision boundary between fact and effect. That boundary needs its own durable identity. Reprocessing an event must not silently create another authoritative assessment, while a legitimate reevaluation based on new evidence must remain possible and auditable.

What AI changes is the distance between information and action and the number of probabilistic steps inside that distance.

As agents gain operational authority, distributed-systems correctness becomes more important, not less. Agent reliability is not primarily a prompt-engineering problem. The decisive guarantees still live in atomicity, stable identity, bounded recovery, deterministic authorization, least privilege, reconciliation, and auditability.

The model is one component. It is not the transaction boundary, the source of financial truth, or the enforcement point for an irreversible effect.

Conclusion

Transactional Outbox makes business state and publication intent atomic at the source. A relay or CDC process transports that intent, while durable delivery and transactional Inbox processing make local consumers tolerant of redelivery.

Those guarantees stop at the local database boundary. External effects require their own durable operation identity, recoverable claim state, provider-level uniqueness contract, and reconciliation. Aggregate versions expose stale events and possible gaps; deterministic routing, bounded buffering, and authoritative recovery protect the ordering invariant. AI assessments require stable decision identity, evidence snapshots, and deterministic policy approval before they can create an external operation.

AI agents do not create these distributed systems problems. They make ignoring them more expensive.

A duplicate event can now travel farther. A retry can activate more capable software. An uncertain result can trigger decisions that affect customers, accounts, infrastructure, or money.

That is why modern agentic architecture should not begin by asking which model is smartest or which broker is fastest. It should begin with the business invariant:

Every financial movement must have a stable operation identity, originate from an authorised and versioned business state, survive publication and executor failures, tolerate redelivery, detect ordering violations, pass deterministic policy controls, use destination-level uniqueness where available, and remain reconcilable when certainty cannot be obtained synchronously.

Or, simply:

Record intent once. Give every effect an identity. Reconcile uncertainty.

Then ask the question that matters when everything appears healthy and the acknowledgement never arrives:

Can your system prove that the same payment will not leave twice?

And can it also prove that an authorised payment will not disappear behind a successful deduplication record?

A correct design needs both guarantees: no duplicate effect and no silently lost intent.

Not whether Kafka stayed online. Not whether the database stayed online. Not whether your AI agent produced the right explanation.

Whether the architecture preserved both safety and liveness when the network stopped being able to tell you what happened — and whether every unresolved outcome remained visible until reality could be established.

Revision note — August 2026: Expanded to cover recoverable external-effect claims and bounded reorder buffers after thoughtful reader feedback.

Technical References

Your Retry Policy Can Break the SLO Before the First Attempt Starts

AdmilsonCossa — Wed, 05 Aug 2026 15:46:56 +0000

A retry policy can look reasonable on its own and still be impossible inside the request that owns it.

Suppose one provider attempt may take 800 milliseconds. The call allows four attempts with increasing backoff, while the request has two seconds left. The individual numbers are valid, yet the composition cannot fit.

@workit/core/time-policy evaluates that shape before task execution:

import { planTimePolicy } from "@workit/core/time-policy";

const plan = planTimePolicy({
  type: "timeout",
  timeout: "2s",
  policy: {
    type: "retry",
    attempt: {
      type: "attempt",
      duration: "800ms",
    },
    retry: {
      times: 4,
      initialDelay: "100ms",
      factor: 2,
      jitter: false,
    },
  },
});

if (!plan.valid) {
  console.error(plan.warnings);
}

The planner does not call the provider. It computes a conservative upper bound from the declared policy and reports typed warnings when the composition cannot fit.

Runtime policy and planning policy have different jobs

run.retry() owns execution. It invokes the task, observes cancellation,
sleeps with the task signal and decides whether another attempt may start.

planTimePolicy() owns pre-execution analysis. It works with declared attempt costs and composition rules:

attempt
retry
hedge
timeout
deadline
series
parallel

Keeping those responsibilities separate matters. A planner should not produce side effects, while a runtime should not pretend it can predict provider latency that the caller never declared.

A deadline is part of the task context

WorkIt 0.5.0 exposes the earliest effective absolute deadline as ctx.deadlineAt.

import { run } from "@workit/core";

const deadlineAt = Date.now() + 2_000;

const result = await run.group(async (task) => {
  return task(run.deadline(async (ctx) => {
    return {
      deadlineAt: ctx.deadlineAt,
      remainingMs: Math.max(0, ctx.deadlineAt! - Date.now()),
    };
  }, deadlineAt));
});

If a task has both an inherited scope deadline and a wrapper deadline, it sees the earlier value. Retry, fallback and hedge compositions preserve that effective deadline for their task bodies.

The value is introspection, not preemption. The task must still cooperate with ctx.signal, and external I/O must receive that signal when the client supports abort.

A retry count is not a shared admission policy

Per-operation retry limits prevent one wrapper from running forever. They do not stop several sibling operations from consuming too many retries together.

Version 0.5.0 adds a shared retry budget:

import { createBudget, run } from "@workit/core";

const ProviderRetries = createBudget("ProviderRetries", {
  unit: "retries",
});

// Application-owned provider boundary.
declare function callProvider(ctx: {
  signal: AbortSignal;
}): Promise<{ answer: string }>;

const operation = run.retry(callProvider, {
  times: 4,
  initialDelay: "100ms",
  retryBudget: ProviderRetries,
});

const result = await run.context.with(
  ProviderRetries,
  { spent: 0, limit: 5, unit: "retries" },
  async () => run.group(async (task) => {
    const first = task(operation);
    const second = task(operation);
    return Promise.all([first, second]);
  }),
);

The initial invocation is not a retry, so it is not charged. Before each
additional attempt is admitted, the wrapper atomically consumes one unit from the scope-visible budget. When the budget is exhausted, the next retry body does not start.

This turns “three retries per call” into a policy that can also say “no more than five additional provider invocations across this request.”

Check aggregate retry demand

The planner can evaluate that shared policy when the caller supplies a runtime snapshot:

import {
  planTimePolicy,
  type RetryBudgetSnapshot,
} from "@workit/core/time-policy";

const retryBudgets: RetryBudgetSnapshot[] = [{
  key: ProviderRetries,
  state: {
    spent: 1,
    limit: 5,
    unit: "retries",
  },
}];

const providerPolicy = {
  type: "retry" as const,
  attempt: {
    type: "attempt" as const,
    duration: "800ms" as const,
  },
  retry: {
    times: 3,
    initialDelay: "100ms" as const,
    retryBudget: ProviderRetries,
  },
};

const plan = planTimePolicy({
  type: "parallel",
  policies: [
    providerPolicy,
    providerPolicy,
  ],
}, {
  retryBudgets,
});

For every referenced budget, the result reports required retries, remaining capacity and one of three statuses:

admissible
exceeded
unverified

An absent snapshot produces retry_budget_snapshot_missing. Insufficient
capacity produces retry_budget_exceeded. Both make the plan invalid because the declared composition cannot be admitted from the supplied state.

What the upper bound means

For fixed retry delays, the planner includes attempt duration and the waits between failed attempts. For parallel policies it distinguishes critical-path time from aggregate parallel work. Timeout and deadline nodes can truncate the outer bound while retaining a warning that inner work exceeds it.

Jitter, dynamic backoff, event-loop stalls and provider latency add uncertainty. The planner reports its bounded contract instead of presenting the result as an exact wall-clock forecast.

Executable evidence

The release connects these claims to:

CORR-009 retry upper bounds
CORR-010 infeasible deadline warning
CORR-016 bounded time-policy cost model
CORR-021 nested composition model
CORR-024 effective runtime deadline
CORR-025 shared retry admission budget
CORR-027 aggregate retry budget planning

Run the proofs with:

npm run test:evidence
npm run test:coverage
npm run verify

The bounded model currently checks 640 generated policies, while the nested composition evidence checks 1,516 generated policy trees. These are executable finite models, not theorems over arbitrary TypeScript or real provider timing.

Planning does not remove runtime uncertainty. It catches policies that are already impossible before that uncertainty begins.

Sources

Github: https://github.com/WorkRuntime/workit
Article source: https://github.com/WorkRuntime/workit/blob/main/articles/09-plan-retries-before-they-run.md

NPM: https://www.npmjs.com/package/@workit/core

Your Async Job Finished. Can You Prove What Actually Ran?

AdmilsonCossa — Wed, 29 Jul 2026 15:23:37 +0000

An async operation can return successfully while leaving an awkward question behind: what actually happened inside it?

For a small function, the return value may be enough. For a provider call with retries, cancellation and cleanup, it is not. Operators need to know which attempts ran, why the operation stopped, whether cleanup timed out and whether owned work remained pending.

WorkIt receipts preserve those lifecycle facts as data.

import { run } from "@workit/core";
import { createReceiptRecorder } from "@workit/core/replay";

// Application-owned provider boundary.
declare function callProvider(ctx: {
  signal: AbortSignal;
}): Promise<{ answer: string }>;

let observedScope;
let recorder;

await run.scope(async (scope) => {
  observedScope = scope;
  recorder = createReceiptRecorder(scope, {
    receiptId: "answer:request-42",
  });

  await scope.spawn(run.retry(callProvider, {
    times: 3,
    initialDelay: "100ms",
  }), {
    name: "provider.answer",
    kind: "llm",
  });
});

const receipt = recorder.build(observedScope.status());
recorder.unsubscribe();

The receipt includes the terminal outcome, normalized lifecycle events, a final scope snapshot and a summary of cleanup or leaked-task evidence. In WorkIt 0.5.0 it can also derive one record for every retry attempt admitted by the outer task boundary.

Evidence, not deterministic replay

The word “replay” is overloaded. Deterministic replay records enough scheduling and nondeterministic input to execute a program again with the same decisions.
That requires control over clocks, random values, I/O, scheduling and usually the runtime itself.

WorkIt does something narrower. It records what the scope observed:

typed task and scope events
final scope snapshot
terminal outcome
cancellation reason
cleanup failures and timeouts
retry attempt outcomes
telemetry drop and truncation counts

You can store and inspect that evidence later. You cannot use it to rerun an arbitrary JavaScript program. The distinction is useful because it keeps the receipt contract small enough to verify.

Attempts belong to the task lifecycle

Before 0.5.0, receipts could include retrying events. Those events explained that another attempt was planned, but they did not provide a terminal outcome for every invocation.

The task:attempt event closes that gap:

scope.onEvent((event) => {
  if (event.type !== "task:attempt") return;

  process.stdout.write(JSON.stringify({
    taskId: event.taskId,
    attempt: event.attempt,
    outcome: event.outcome,
    durationMs: event.durationMs,
  }) + "\n");
});

An attempt ends as succeeded, failed or cancelled. Nested retry wrappers do not produce competing generic histories for the same task. The outer retry boundary owns the task-level attempt sequence, while an application can add more specific provider or activity evidence when it needs it.

That ownership rule prevents a receipt from presenting two incompatible
answers to “how many task attempts ran?”

Add metadata at the boundary that knows it

The runtime knows the task id, attempt number, timing and outcome. It does not know whether a particular invocation targeted a primary provider, a regional replica or a billing-sensitive activity.

createAttemptRecorder() lets the caller add that context explicitly:

import { createAttemptRecorder } from "@workit/core/replay";

const attemptRecorder = createAttemptRecorder({
  maxAttempts: 20,
  maxMetadataBytes: 1_024,
});

// `callProvider` is the application-owned provider function from the previous
// example.
const callPrimary = attemptRecorder.wrap(callProvider, {
  metadata: {
    provider: "primary",
    operation: "answer",
  },
  reasonCode: (error) =>
    error instanceof TypeError ? "transport_error" : "provider_error",
});

const operation = run.retry(callPrimary, {
  times: 3,
  initialDelay: "100ms",
});

Reason codes must be bounded slugs. Metadata must be a JSON object and must fit the configured byte limit. Common secret fields are redacted before the record is retained.

This is deliberately caller-owned enrichment. Inferring provider policy from an arbitrary error object would turn a lifecycle recorder into a second routing engine.

Redaction belongs before storage

Progress data is often valuable during an incident, although it may contain fields that should never reach a durable ledger.

Receipt redaction has conservative defaults for common secret names, and the caller can add its own policy:

import { redactReceipt } from "@workit/core/replay";

const publicReceipt = redactReceipt(receipt, {
  removeFields: ["privateNote"],
  redactFields: ["authorization", "tenantToken"],
});

Redaction is not a substitute for data minimization. The safest private payload is still the one that was never attached to an event. It does, however, provide a clear boundary between local lifecycle evidence and a receipt intended for storage or publication.

What a receipt can establish

A captured receipt can support claims about the WorkIt lifecycle it observed:

the scope reached a terminal state;
no owned tasks were pending in the final snapshot;
cancellation carried a typed reason;
cleanup failure or timeout events were present;
admitted retry invocations had terminal outcomes;
event truncation or telemetry drops were counted.

It cannot establish that a remote provider stopped billing, that an external transaction was semantically correct or that an uncaptured event occurred.

Those limitations are not footnotes. They define where runtime evidence ends and application or provider evidence begins.

Executable evidence

The relevant release proofs are:

LIFE-004 completed scope receipt
LIFE-005 typed cancellation reason
LIFE-012 admitted attempt evidence and default secret redaction

They run through:

npm run test:evidence
npm run test:coverage
npm run verify

The tests cover completed and cancelled receipts, cleanup evidence, bounded event windows, redaction, attempt outcomes, metadata limits and installed package consumers.

The practical result is modest but important: when WorkIt owns an async
lifecycle, it can leave behind a typed account of what it observed.

Sources

Your AI Agent Finished. Why Are Its Tools Still Running?

AdmilsonCossa — Wed, 17 Jun 2026 15:25:52 +0000

Five articles built the runtime. The sixth made it observable. This one introduces the agent primitive: runAgent plus AgentScope, with budgets, replayable events, and structured cancellation in the box.

The whole loop:

import { runAgent, AgentToolCalls, OpenAITokens } from "@workit/core/ai";
import { CostBudget, run } from "@workit/core";

const { result, events } = await runAgent(async (agent, ctx) => {
  const plan = await agent.tool("plan", goal, planLLM,
    { tokens: 600, cost: 0.001, retry: 2 });

  for (const step of plan.steps) {
    await agent.tool(step.tool, step.input, tools[step.tool],
      { tokens: 1_200, toolCalls: 1, timeout: "10s" });
  }

  return await agent.tool("synthesize", workspace, synthesizeLLM,
    { tokens: 2_000, cost: 0.004 });
});

That call returns two things — result (whatever the body returned) and events (the complete, ordered, type-discriminated trace of the run). No external tracing setup. No DSL. The body is plain async/await, the tools are plain functions, and every agent.tool(...) call is a typed primitive whose budget, retry, and timeout policy live in the call site.

This is the practical lifecycle primitive between "I wired up an LLM call and a tool router" and "I can explain, bound, cancel, and replay the run."

The contract `agent.tool(name, input, fn, opts)`

interface AgentScope {
  readonly id: string;
  readonly events: readonly AgentEvent[];
  tool<I, O>(
    name: string,
    input: I,
    fn:   (input: I, ctx: TaskContext) => O | Promise<O>,
    opts: AgentToolOptions,
  ): Promise<O>;
}

interface AgentToolOptions {
  tokens:    number;       // charged against OpenAITokens budget
  cost:      number;       // charged against CostBudget budget
  toolCalls: number;       // charged against AgentToolCalls budget
  retry:     number | RetryOpts;
  timeout:   Duration;
}

Five things to notice:

The tool function is a plain (input, ctx) => Promise<O>. No generators. No effect type. No "tool description JSON schema" to feed an LLM -- that's your application's job, not the runtime's.
Budgets are charged before the call returns. Overrun rejects synchronously and cancels the owning scope with CancelReason { kind: "budget", budgetKey, limit, spent }. Runtime budget accounting stops at the cap.
retry/timeout are per-tool, composing with the same engine described in articles 02 and 05.
ctx.signal inside the tool body is linked to the parent scope. Client disconnects, deadline fires, sibling fails — all aborts propagate into the tool body so its fetch / db.query / provider.call aborts at the I/O boundary.
agent.events is a readonly buffer that mirrors the event stream. After the run, events is a replayable log of the whole loop.

A 50-cent agent with a hard tool-call cap

import { runAgent, AgentToolCalls, OpenAITokens } from "@workit/core/ai";
import { CostBudget, run } from "@workit/core";

await run.context.with(CostBudget,      { spent: 0, limit: 0.50,    unit: "USD" },
() => run.context.with(OpenAITokens,    { spent: 0, limit: 100_000, unit: "tokens" },
() => run.context.with(AgentToolCalls,  { spent: 0, limit: 20,      unit: "tool_calls" },
  () => runAgent(async (agent) => reactLoop(agent, goal)),
)));

Three caps, three reasons:

Budget	What it bounds	What overrun does
`CostBudget`	Aggregate USD across the whole run	Rejects with `BudgetExceededError` and cancels the owning scope. The 32 inflight LLM/tool calls see the abort on `ctx.signal`; provider-side billing depends on the provider honoring cancellation.
`OpenAITokens`	Total tokens across all LLM calls	Same shape. Use a dedicated key per provider when you want separate caps.
`AgentToolCalls`	Total tool calls -- fan-out limiter	Stops a runaway agent from invoking tools forever. Bench 19-B caps it at 1 and the second tool call fails closed.

Bench 19-agent-scope.mjs. Five scenarios -- measured.

# Scenario Result

A Tool events bracket execution Single agent.tool("calc", 3, x => x*x) call -> 4 events [agent:started, agent:tool_started, agent:tool_succeeded, agent:completed], sequential seq: [1,2,3,4], monotonic at, stable agentId.

B AgentToolCalls cap hit limit: 1. Second call rejects with BudgetExceededError, budgetKey: "AgentToolCalls", limit: 1.

C OpenAITokens charged via opts { tokens: 50 } then { tokens: 25 } -> final spent: 75 exactly.

D Parent scope cancel during tool ctx.scope.cancel({ kind: "manual", tag: "user-stop" }) mid-tool -> tool body's ctx.signal aborts, outer settles CancellationError with reason.kind: "manual", tag: "user-stop".

E Replayable log, 3-tool run 8 events: started -> (tool_started -> tool_succeeded) x 3 -> completed. Seq [1..8]. Same agentId. Tool names captured in order.

#	Scenario	Result
A	Tool events bracket execution	Single `agent.tool("calc", 3, x => x*x)` call -> 4 events `[agent:started, agent:tool_started, agent:tool_succeeded, agent:completed]`, sequential `seq: [1,2,3,4]`, monotonic `at`, stable `agentId`.
B	`AgentToolCalls` cap hit	`limit: 1`. Second call rejects with `BudgetExceededError`, `budgetKey: "AgentToolCalls"`, `limit: 1`.
C	`OpenAITokens` charged via opts	`{ tokens: 50 }` then `{ tokens: 25 }` -> final `spent: 75` exactly.
D	Parent scope cancel during tool	`ctx.scope.cancel({ kind: "manual", tag: "user-stop" })` mid-tool -> tool body's `ctx.signal` aborts, outer settles `CancellationError` with `reason.kind: "manual"`, `tag: "user-stop"`.
E	Replayable log, 3-tool run	8 events: `started -> (tool_started -> tool_succeeded) x 3 -> completed`. Seq `[1..8]`. Same agentId. Tool names captured in order.

Replayable events -- the typed trace

type AgentEvent =
  | { type: "agent:started";        seq: number; agentId: string; at: number }
  | { type: "agent:tool_started";   seq: number; agentId: string; tool: string; at: number }
  | { type: "agent:tool_succeeded"; seq: number; agentId: string; tool: string; at: number }
  | { type: "agent:tool_failed";    seq: number; agentId: string; tool: string; error: string; at: number }
  | { type: "agent:tool_cancelled"; seq: number; agentId: string; tool: string; reason: CancelReason; at: number }
  | { type: "agent:completed";      seq: number; agentId: string; at: number }
  | { type: "agent:failed";         seq: number; agentId: string; error: string; at: number };

Seven variants. Discriminated by type. Every variant carries seq and at. Cancelled events carry the typed CancelReason.

What you can do with that:

Pivot a dashboard on tool x type for failure heatmaps without parsing logs.
Replay a run in a test by walking the events array -- you have the order, the names, the timing.
Audit a charge by reconstructing the budget timeline from tool_succeeded events tagged with the tokens / cost charged at the call site.
Diff two runs on the event sequence to see exactly which tool path diverged.

The events array on the AgentRunResult is readonly and mirrors the same event stream that flows through scope.onEvent(...) -- so live observers see the same shape the post-run audit log sees.

How does this compare

Stack	Tool primitive	Budget primitive	Replayable event log	Scope cancellation	Bundle
WorkIt `runAgent`	yes typed `(input, ctx) => O`	yes `CostBudget` / `OpenAITokens` / `AgentToolCalls` / `createBudget(...)` composable	yes `AgentRunResult.events` typed union	yes `ctx.signal` aborts each tool body	included in `@workit/core/ai` (~8 KB gzip with the rest of `/ai`)
LangChain agents	yes but typed loosely; many tools as JSON	no no first-class budget primitive	partial via callbacks	no no scope tree	~hundreds of KB
Vercel AI SDK	yes tool schemas	no no first-class budget	events on stream	yes via `AbortSignal`, no scope tree	medium
Mastra	yes generators-based	partial	yes trace store	yes	medium
Roll-your-own with `for`-loop + `fetch`	yes, by definition	DIY	DIY	DIY	minimal but you wrote the runtime

The design point: the agent primitive composes with the same CancelReason, ctx.signal, defer, budget, and scope.tree() machinery from articles 01-06. There is no second runtime. You don't choose between "the agent loop's lifecycle" and "the rest of your app's lifecycle" -- they share one tree.

A complete, runnable example

import { runAgent, AgentToolCalls, OpenAITokens } from "@workit/core/ai";
import { CostBudget, run, renderTree } from "@workit/core";

const tools = {
  search: async ({ q }, ctx) =>
    fetch(`https://api.search.dev/q=${q}`, { signal: ctx.signal }).then(r => r.json()),

  fetchPage: async ({ url }, ctx) =>
    fetch(url, { signal: ctx.signal }).then(r => r.text()),

  summarize: async ({ text }, ctx) =>
    openai.chat({ messages: [{ role: "user", content: `tl;dr: ${text}` }] },
                { signal: ctx.signal }),
};

const { result, events } = await run.context.with(
  CostBudget, { spent: 0, limit: 0.50, unit: "USD" },
  () => run.context.with(
    AgentToolCalls, { spent: 0, limit: 12, unit: "tool_calls" },
    () => runAgent(async (agent) => {
      const hits = await agent.tool("search",
        { q: "structured concurrency typescript" }, tools.search,
        { toolCalls: 1, timeout: "5s", retry: 2 });

      const docs = await Promise.all(hits.slice(0, 3).map((hit, i) =>
        agent.tool(`fetchPage[${i}]`,
          { url: hit.url }, tools.fetchPage,
          { toolCalls: 1, timeout: "10s" })));

      return await agent.tool("summarize",
        { text: docs.join("\n\n") }, tools.summarize,
        { tokens: 4_000, cost: 0.02, toolCalls: 1, timeout: "30s" });
    }),
  ),
);

console.log(result);
console.log(events.map(e =>
  `${e.seq.toString().padStart(2)} ${e.type}${"tool" in e ? ` (${e.tool})` : ""}`,
).join("\n"));

That's an agent that searches, fetches three pages, summarises, and stops at 50 cents or 12 tool calls -- whichever comes first. Cancel the parent scope and every in-flight fetch and LLM stream aborts at the TCP layer. No manual AbortController plumbing. No "did I forget to thread the signal." No try/catch around the agent loop.

Receipts

node benchmarks/articles/19-agent-scope.mjs           # 5 contract scenarios
node benchmarks/articles/run-all.mjs                  # full 19-bench suite

Production-side gates that back the same surface:

Claim	Evidence
Tool events bracket execution with monotonic seq	`19-agent-scope.mjs` A verifies four ordered events, sequential `seq`, stable `agentId`, and monotonic `at`.
`AgentToolCalls` overflow rejects with `BudgetExceededError`	Bench 19 B sets `limit: 1`; the second tool call throws with `budgetKey: "AgentToolCalls"`.
`OpenAITokens` consumed via `{ tokens: N }`	Bench 19 C verifies the final token budget `spent` is exactly `75`.
Parent scope cancel propagates into tool body	Bench 19 D verifies the tool body observes abort and the outer scope settles with the original manual reason.
Replayable, ordered, typed event log	Bench 19 E verifies eight events, sequential `seq`, monotonic `at`, and tool names in call order.
Tool failure surfaces as `agent:tool_failed`	Unit coverage verifies tool errors propagate and are captured in the typed event log.

Closing The Series

The important part is not that WorkIt has an agent helper. The important part is
that the agent helper is not a second runtime. Tool calls, token budgets,
timeouts, retries, cancellation, progress events, and cleanup all use the same
ownership tree as the rest of the library.

The public claims behind this series are tracked in
evidence/claims.json, exercised by
npm run test:evidence, and benchmarked by npm run bench:articles. The prose
is intentionally not the evidence store; it is the readable path through the
engineering tradeoffs.

Source, Benchmarks, And Evidence

NPM: https://www.npmjs.com/package/@workit/core
Source: https://github.com/WorkRuntime/workit
Article source: https://github.com/WorkRuntime/workit/blob/main/articles/07-agent-scope-and-tool-lifecycles.md
Reproduce: npm run bench:articles and npm run test:evidence

Inspect an AI Agent Run Without Paying for Logs You'll Never Read — Telemetry Shouldn't Be Your Second Biggest Bill

AdmilsonCossa — Wed, 10 Jun 2026 08:50:28 +0000

Last time we put hard budgets on cost and guaranteed cleanup. This time we make sure you can inspect an agent run without making telemetry volume the default cost center.

Take a real number. An agent does 200 tool calls per run. Each tool emits 5 events. That's 1,000 events per run. 100K runs/day = 100 million events/day.

At CloudWatch Logs ingestion ($0.50/GB) with 500-byte structured JSON, that is roughly $9,125/year of telemetry that may not be needed on successful runs. Datadog APM, Application Insights, and Honeycomb have different pricing models, but the engineering issue is the same: unbounded event volume turns observability into a cost surface.

WorkIt's core has zero networking imports.

$ grep -E "node:http|node:https|fetch" dist/index.js

That's not a feature claim. That's a gate.

Bench 15-core-zero-network.mjs. Walks the published dist/ tree (excluding the explicit observability, otel, and worker subpaths), greps every .js/.cjs/.mjs for node:http, node:https, raw http/https imports, and fetch(...).

Metric Result

Files scanned 14

Forbidden imports found 0

Excluded subpaths observability, otel, worker

assert.equal(hits.length, 0) passed

Metric	Result
Files scanned	14
Forbidden imports found	0
Excluded subpaths	`observability`, `otel`, `worker`
`assert.equal(hits.length, 0)`	passed

If a PR adds a networking import to core, the production gate npm run check:no-network fails. The bench above verifies the same property in the artifact a consumer installs from npm.

Layer 1: Local-first by default (cost: $0)

import { run, renderTree } from "@workit/core";

const result = await run.scope(async (scope) => {
  // your agent code
  console.log(renderTree(scope.status()));
  return await doWork(scope);
});

Zero network calls. Zero log lines. Zero telemetry export cost. scope.status() returns a snapshot. renderTree(...) prints an ASCII tree. That is the built-in observability surface; exporters are opt-in.

When you do want telemetry, you opt in:

import { attachTelemetryExporter } from "@workit/core/observability";

const attachment = attachTelemetryExporter(scope, async (event) => {
  await otlp.write(event);
}, {
  sampling:       { mode: "errors_and_slow", slowThresholdMs: 2_000 },
  circuitBreaker: { failureThreshold: 3, openForMs: 60_000 },
  sanitize:       (event) => stripPII(event),
});

Four words: sampled, aggregated, budgeted, circuit-broken.

Layer 2 — Sampling: errors_and_slow is the production default

Same workload, same agent. With errors_and_slow (slow threshold 2 seconds) and 95% of runs completing fast and successful:

Workload	Without sampling	With `errors_and_slow` (2 s)	Reduction
100K runs/day, 5% slow/errored	100K x 1,000 ev x 500 B = 50 GB/day	5K x 1,000 ev x 500 B = 2.5 GB/day	20x
CloudWatch Logs ingestion	$25/day = $9,125/year	$1.25/day = $456/year	$8,669/yr saved

The intended debugging signal is preserved for slow and failing runs. A passing run rarely needs full trace inspection -- you need it when something breaks or hangs, which is exactly what this policy keeps.

Bench 16-sampling-and-aggregation.mjs. 100 root scopes x 5 child tasks each. 5% slow, 2% errored. Both modes attach attachTelemetryExporter to the same workload.

sampling.mode Exported events Reduction factor

"all" 1,300 baseline

"errors_and_slow" (slowThresholdMs: 55) 36 ~36x

The bench asserts >= 5x to stay tolerant of jitter. The measured ratio came out higher than the article's nominal 20x because the synthetic workload concentrates slow/errored scopes; the savings table above uses the conservative ratio.

`sampling.mode`	Exported events	Reduction factor
`"all"`	1,300	baseline
`"errors_and_slow"` (slowThresholdMs: 55)	36	~36x

Sampling modes — how to choose:

Mode	Use case
`"off"`	Local dev, high-volume tests -- same shape as Layer 1 ($0)
`"errors_and_slow"`	Production default -- keep failing and slow traces, drop the rest
`"head"`	Random sampling at scope start -- cheap, no buffering, good for high-throughput service tracing
`"all"`	Debugging session for one run -- opt-in firehose

A child scope cannot upgrade itself to "kept" if its root was sampled out. This is the rule that keeps traces causally intact.

Layer 3 — Aggregation, not enumeration

By default, an aggregated exporter receives one summary record per scope, not per task:

interface ScopeSummary {
  scopeId: string;
  parentId: string | null;
  durationMs: number;
  outcome: "completed" | "errored" | "cancelled";
  taskCounts: {
    started: number; succeeded: number; failed: number;
    cancelled: number; retried: number; cleanupFailed: number;
  };
  droppedTelemetryEvents: number;
}

200 tasks succeed -> 1 summary record exported. Not 200. Not 1,000.

import { attachScopeSummaryExporter } from "@workit/core/observability";

attachScopeSummaryExporter(scope, async (summary) => {
  await otlp.writeAggregate(summary);
}, { /* same circuit breaker / queue / sampling options */ });

Aggregation level	Records per scope	When to use
Scope summary (default)	1 per closed scope	Production -- cost-efficient
Hybrid (summary + per-task for failures/slow)	summary + N	Investigating a specific failure pattern
Per-task firehose	one per event	Short opt-in debug session

The summary aggregator is exercised end-to-end by npm run check:exporter-stress -- 100,000 events with bounded queue and drop-front under back-pressure. The bench in this folder skips the summary path on purpose: attachScopeSummaryExporter needs the scope:opened event, which fires before user code can attach inside run.scope. Companion packages wire it earlier; the production gate covers it.

Layer 4 — Telemetry budget (the safety net)

import { TelemetryBudget, run } from "@workit/core";

await run.context.with(
  TelemetryBudget,
  { spent: 0, limit: 100_000, unit: "events" },
  () => agentRun(),
);

The exporter checks the budget before emitting each event. When a scope's event count would exceed the limit, the event is dropped silently. The first overrun emits one warning. Subsequent overruns are counted into the next scope summary's droppedTelemetryEvents field. Tasks continue executing normally.

Telemetry overrun must never affect application behaviour. The companion withOTel wrap sets a default budget at the wrap-call boundary, so the floor is opt-out, not opt-in.

Cardinality Discipline And Cost Control

Every exported field is classified bounded or unbounded. Unbounded fields are never emitted as metric labels.

Field	Bound	In metric labels
`scope.name`	dev-chosen	yes
`task.kind`	5 fixed values (`io`/`llm`/`tool`/`cpu`/`custom`)	yes
`outcome`	3 values	yes
`cancelReason.kind`	9 values	yes
`error.name`	finite by class	yes
`attempt`	bounded by retry limit	yes (bucketed)
`task.id`	unbounded UUID	no
`error.message`	unbounded text	no
`meta.*`	user-controlled	explicit opt-in only

Wrap your metric exporter with createCardinalitySafeMetricExporter and pass an allowedLabels allow-list -- anything outside is rejected at runtime.

Bench 17-cardinality-safe-metrics.mjs. Five candidate metric points, allow-list ["task.kind", "outcome", "scope.name"].

Point Labels Outcome

1 { "task.kind": "io", "outcome": "succeeded" } yes emitted

2 { "task.kind": "llm", "outcome": "failed" } yes emitted

3 { "task.kind": "tool", "task.id": "uuid-abc" } no rejected -- Metric label "task.id" is not in the allowed label set

4 { "task.kind": "io", "error.message": "EHOSTUNREACH at 10.0.0.42 retrying" } no rejected -- Metric label "error.message" is not in the allowed label set

5 { "task.kind": "evil" } yes emitted (label-key check only -- out-of-enum value validation is the OTel-adapter's job)

Point	Labels	Outcome
1	`{ "task.kind": "io", "outcome": "succeeded" }`	yes emitted
2	`{ "task.kind": "llm", "outcome": "failed" }`	yes emitted
3	`{ "task.kind": "tool", "task.id": "uuid-abc" }`	no rejected -- `Metric label "task.id" is not in the allowed label set`
4	`{ "task.kind": "io", "error.message": "EHOSTUNREACH at 10.0.0.42 retrying" }`	no rejected -- `Metric label "error.message" is not in the allowed label set`
5	`{ "task.kind": "evil" }`	yes emitted (label-key check only -- out-of-enum value validation is the OTel-adapter's job)

The wrapper rejects unbounded label keys at runtime. Out-of-enum value rejection (taskKind: "evil" failing in OTel) is enforced by the adapter contract, while the wrapper keeps cardinality control at the label-key boundary.

Correct usage:

task(() => fetch(url),       { kind: "io" });     // yes
task(() => llm.call(prompt), { kind: "llm" });    // yes
task(() => runTool(input),   { kind: "tool" });   // yes
task(() => heavyCalc(),      { kind: "cpu" });    // yes
task(() => custom(),         { kind: "custom" }); // yes

Exporter circuit breaker -- the OOM defence

{
  circuitBreaker: { failureThreshold: 10, openForMs: 5 * 60_000 },
  queue:          { maxItems: 10_000, maxBytes: 10 * 1024 * 1024 },
}

OTLP backend goes down. The exporter sees N consecutive failures -> opens for openForMs -> events drop (counted, not queued). The bounded queue uses drop-front when full, so you keep the most recent context. After openForMs elapses -> half-open -> trial export -> close on success.

Result: process memory growth bounded under 50 MB through 1,000 scopes against a backend that returns 503 for every request. Tracked under tests/perf/exporter-failure.test.ts. This is the feature that prevents the most embarrassing observability incident -- the telemetry agent eating all process memory because the backend is unreachable. Datadog had this. New Relic had this. We design it out.

`scope.tree()` — the print statement for agents

agent-run
|- ok planLLM (243ms)
|- retry fetchTool (attempt 2/3)
|  `- [running] retry-delay (120ms)
|- [running] summarize (running, 45% -- "embedding chunks")
`- failed auditLog (TimeoutError)

5 tasks * 1 ok * 1 failed * 2 running * 1 retrying * deadline in 12s

Print this when something breaks. Print this in your test runner. Print this from a SIGUSR1 dump.

Icon	Meaning
`ok`	succeeded (durationMs)
`failed`	failed (error.name)
`cancelled`	cancelled (reason.kind)
`[running]`	running (elapsed -- message)
`retry`	retrying (attempt N/total)
`pending`	pending

import { renderTree } from "@workit/core";
console.log(renderTree(scope.status()));

Progress is a typed event, not a log line

Inside any task body, ctx.report({ pct, message, data }) emits a typed task:progress event tagged with the task's stable id and name. Your exporter, your dashboard, your test assertion all pivot on the same shape. No console.log. No string parsing. No grep.

// samples/progress-parallel.sample.js
const TARGET = "embed.batch.7";

await run.scope(async (scope) => {
  scope.onEvent((event) => {
    if (event.type === "task:progress" && taskNames.get(event.taskId) === TARGET) {
      targetProgress.push({ pct: event.pct, message: event.message });
    }
  });

  const handles = Array.from({ length: 16 }, (_, i) => scope.spawn(async (ctx) => {
    if (i === 7) {
      for (const step of [1, 2, 3, 4]) {
        ctx.report({ pct: step / 4, message: `chunk-${step}` });
        await sleep(1, ctx.signal);
      }
    } else { await sleep(8, ctx.signal); }
  }, { name: `embed.batch.${i}`, kind: "llm" }));

  return await Promise.all(handles);
});

// Asserted by the sample:
//   targetProgress.map(e => e.pct) === [0.25, 0.5, 0.75, 1]
//   maxActive === 16

16 parallel embeddings. One of them is embed.batch.7. We filter the event stream to that task's id and watch the progress sequence land -- [0.25, 0.5, 0.75, 1], with the messages chunk-1..chunk-4 attached. The other 15 siblings run concurrently and don't interleave their reports into our channel because every event carries the typed taskId.

npm run sample:progress

That's the difference between "tail the log file and hope" and "subscribe to a typed event stream and pivot on the shape that the type system already validated."

Snapshots: the stable view of a live runtime

The hot runtime exposes stable snapshots, not live references. Pull one whenever you need to inspect -- the engine doesn't mutate it, you can hand it to anything (logger, diagnostics, test assertion, JSON wire).

import type { ScopeSnapshot, TaskSnapshot } from "@workit/core";

const snapshot: ScopeSnapshot = scope.status();

interface ScopeSnapshot {
  id: ScopeId;
  name: string;
  status: "running" | "cancelling" | "closed";
  startedAt: number;
  deadlineAt: number;
  pendingCount: number;
  completedCount: number;
  failedCount: number;
  cancelledCount: number;
  tasks:  TaskSnapshot[];     // every task currently owned by this scope
  scopes: ScopeSnapshot[];    // every child scope, recursively
}

interface TaskSnapshot {
  id: TaskId;
  name: string;
  kind: TaskKind;             // "io" | "llm" | "tool" | "cpu" | "custom"
  status: "pending" | "running" | "succeeded" | "failed" | "cancelled";
  attempt: number;
  startedAt: number;
  durationMs: number;
  progress: ProgressReport;
  meta: Record<string, unknown>;
}

Three properties matter:

Snapshot is immutable -- taking one twice gives you two independent objects. The runtime never mutates one you already hold.
Snapshot is recursive -- scopes is the same shape as the root, all the way down. Tools that walk it (the diagnoser, the renderer, your test assertions) share one shape.
Snapshot is the only public surface -- renderTree consumes it, diagnoseSnapshot consumes it, you consume it. The hot runtime owns the live state; rich tooling lives outside core.

That's the architectural boundary that keeps core small. Anything that wants to inspect the runtime asks for a snapshot.

`@workit/core/diagnostics` -- the stuck-task detector

import { diagnoseSnapshot } from "@workit/core/diagnostics";

const report = diagnoseSnapshot(scope.status(), {
  staleTaskMs: 30_000,
  events:      recentEvents,
});

if (report.findings.length > 0) {
  console.warn("agent stalled:", report.findings);
}

Diagnoses live, on demand, against an existing snapshot. Subpath-only so the root runtime stays small.

Bench 18-diagnostics-finding-codes.mjs. Five hand-crafted snapshots, one per finding code.

Scenario report.status Finding codes emitted

Healthy snapshot ok (none)

Task running > 30 s needs_attention old_pending_task

Scope status cancelling needs_attention scope_cancelling

Pending child scope needs_attention pending_child_scope + recursive old_pending_task

task:cleanup_timeout event in window needs_attention cleanup_timeout

Scenario	`report.status`	Finding codes emitted
Healthy snapshot	`ok`	(none)
Task running > 30 s	`needs_attention`	`old_pending_task`
Scope status `cancelling`	`needs_attention`	`scope_cancelling`
Pending child scope	`needs_attention`	`pending_child_scope` + recursive `old_pending_task`
`task:cleanup_timeout` event in window	`needs_attention`	`cleanup_timeout`

Wire it to a SIGUSR1 handler in production:

process.on("SIGUSR1", () => {
  console.error(renderTree(rootScope.status()));
  console.error(JSON.stringify(diagnoseSnapshot(rootScope.status(), { staleTaskMs: 30_000 }), null, 2));
});

Something hangs at 3 a.m. -> one signal gives you the live tree and a list of suspicious tasks. No APM. No external service. Stderr.

OpenTelemetry -- opt-in, with optional peer

import { attachOpenTelemetry } from "@workit/core/otel";

const detach = attachOpenTelemetry(scope, { tracer, meter });

@opentelemetry/api is an optional peer. The root WorkIt package stays at zero runtime dependencies. Install OTel only when you need it:

npm install @opentelemetry/api

If the peer is missing, attachOpenTelemetry throws a message that names exactly the install command -- no cryptic "cannot resolve module" trace.

Three canonical configurations

// Local development -- default. No setup needed.
await run.scope(async (scope) => { /* ... */ });
// Inspect: console.log(renderTree(scope.status()))

// Production -- sampled, aggregated, budgeted, circuit-broken.
await withOTel(
  {
    exporter:       otlpExporter,
    sampling:       { mode: "errors_and_slow", slowThresholdMs: 2_000 },
    aggregation:    "per_scope",
    eventBudget:    50_000,
    redact:         ["context.user.email"],
    circuitBreaker: { failureThreshold: 10, openForMs: 300_000 },
  },
  () => agentRun(),
);
// Typical bill, 100K runs/day: ~$456/year.

// Full trace debugging: one-off, manual, full event stream.
await withOTel(
  { exporter: otlpExporter, sampling: { mode: "all" }, aggregation: "per_task" },
  () => problemReproduction(),
);

Receipts

node benchmarks/articles/15-core-zero-network.mjs       # 0 hits in 14 dist files
node benchmarks/articles/16-sampling-and-aggregation.mjs # 1,300 -> 36 events
node benchmarks/articles/17-cardinality-safe-metrics.mjs # 2 of 5 unbounded labels rejected
node benchmarks/articles/18-diagnostics-finding-codes.mjs # 4 finding codes proven
node benchmarks/articles/run-all.mjs                    # full article suite

Production-side gates that back the same surface:

Claim	Evidence
Core has zero networking imports	Static gate finds no `node:http`/`node:https`/`fetch` in `dist/index.js`. Reproduced by `15-core-zero-network.mjs` over the full published `dist/` tree minus the explicit network-bridge subpaths.
Sampling reduction (`errors_and_slow` @ slowThreshold)	100 root scopes / 5 children, >= 5x reduction asserted; measured ~36x. Production exporter stress test runs 100,000 events.
Aggregation collapses N tasks -> 1 record	`npm run check:exporter-stress` exercises the full summary path with bounded queue.
Telemetry budget never throws	Property test: any budget x any event volume -> tasks complete normally.
Cardinality enforcement at runtime	`17-cardinality-safe-metrics.mjs` verifies unbounded label keys are rejected at the wrapper boundary; adapter coverage owns enum-value validation.
Circuit breaker memory bound under 503 backend	`tests/perf/exporter-failure.test.ts`: < 50 MB heap growth across 1,000 scopes with backend down.
Diagnostics finding codes	`18-diagnostics-finding-codes.mjs` verifies healthy snapshots, old pending tasks, cancelling scopes, pending child scopes, and cleanup timeout findings.
OTel optional peer	Missing peer throws explicit install message; not a cryptic resolver error.

What's coming

You now have an agent that holds itself to a budget, releases its connections, and tells you what's happening inside without sending a byte to the cloud unless you ask.

Tomorrow: the final article. Agent scopes and tool lifecycles.

The same ownership tree that cancels streams and bounds observability also
governs agent tool calls, token budgets, progress events, and replayable
execution logs. The point is not a new agent framework; it is one lifecycle
contract for the agent loop and the rest of the application.

Source, Benchmarks, And Evidence

Source: https://github.com/WorkRuntime/workit
NPM: https://www.npmjs.com/package/@workit/core
Article source: https://github.com/WorkRuntime/workit/blob/main/articles/06-observability-without-core-bloat.md
Reproduce: npm run bench:articles and npm run test:evidence

Most AI Agents Have Infinite Ambition and Zero Budget Control

AdmilsonCossa — Tue, 02 Jun 2026 15:38:56 +0000

Resource Safety And Budgeted Work

Last time we showed backpressure with channels and work().stream() — pausing the producer the moment the consumer slows down. This article puts hard boundaries on cost and guarantees that cleanup always runs, even when the user hits Ctrl-C, the deadline fires, or a sibling throws.

Three primitives. One ownership tree.

run.bracket — open, use, release. Always release.
run.uncancellable — short critical sections that survive cancellation.
Budgets — hard caps on cost, tokens, or any metric, enforced atomically across parallel work.

No try/finally you'll forget to write. No "did the connection close" post-mortem.

`run.bracket` -- acquire, use, release. Always release.

import { run } from "@workit/core";

const rows = await run.scope(async (scope) => scope.spawn(run.bracket(
  async () => db.connect(),                                                   // acquire
  async (conn, ctx) => conn.query("select 1", { signal: ctx.signal }),        // use
  async (conn) => conn.close(),                                                // release
  { timeout: "5s" },                                                           // bounded cleanup
)));

release runs once on every exit path: success, throw, parent cancel, timeout, sibling failure. The release receives the resource. The release also receives cleanupCtx.signal so it can give up if the cleanup itself hangs. Nested brackets release LIFO.

Bench 12-bracket-vs-try-finally.mjs. Five scenarios -- measured.

#	Scenario	Result
A	Success path	order: `[acquire, use, release:RES-A]`, release ran exactly once
B	`use` throws	order: `[acquire, use, release:RES-B]`, release runs with the resource, error propagates
C	`acquire` throws	order: `[acquire]`, release does NOT run, error propagates
D	Parent cancel during `use`	order: `[acquire, use, release:RES-D]`, outer settled `CancellationError` with `kind: "manual"`
E	Hanging release	Native `try/finally` with a non-resolving cleanup is still pending after 250 ms (would deadlock forever). `run.bracket(..., { timeout: "150ms" })` settles at t=157 ms and emits `task:cleanup_timeout`.

Bundle cost: +58 B min, +15 B gzip on public-api. Effectively free.

When to use bracket — anything with a single resource that must be closed exactly once: database connections, file handles, distributed locks, HTTP client sessions, ML model contexts.

`run.uncancellable` — receipts that always commit

const receipt = run.uncancellable(async (ctx) => {
  const intent = await stripe.confirmIntent({ signal: ctx.signal });
  await db.recordReceipt({ id: intent.id }, { signal: ctx.signal });
  return intent.receipt_url;
}, { timeout: "2s" });

const url = await run.scope(async (scope) => scope.spawn(receipt));

The user hits Ctrl-C. The parent scope cancels. The deadline fires. Inside the shielded body, none of those are visible — ctx.signal is a fresh signal local to the shield. The body runs to completion (or to its own timeout). When the shield finishes, if the parent had cancelled during the shield, the original CancellationError rethrows after the body completes.

Cancellation is delayed, not hidden.

This is the line that lets you write a Stripe webhook handler, a distributed-lock release, or a database commit without relying on ordinary task cancellation to preserve the critical section. Bench 08-uncancellable-shield.mjs (article 03) measured the body running 95 ms past a parent cancel before the original reason was rethrown.

When to use uncancellable — short, critical sections that must finish atomically: Stripe charges, audit log flushes, idempotency-key writes, distributed-lock release.

bracket vs uncancellable decision rule:

Use run.bracket when there is a resource you opened and must close. Cleanup is the contract; the body is just what runs in between.
Use run.uncancellable when there is a critical section that must run to the end even if the parent cancels. There may be no resource.
Use both together when a critical section needs a resource: spawn a run.bracket whose use body is a run.uncancellable.

run.uncancellable is cooperative. It cannot stop a non-cooperative CPU loop inside the body. For that, see article 03 — offload with worker termination.

Budgeted Agent Work: A 50-Cent Ceiling

import { CostBudget, group, run } from "@workit/core";

const answer = await run.context.with(
  CostBudget, { spent: 0, limit: 0.50, unit: "USD" },
  () => group(async (task) => reactLoop(task, goal)),
);

Inside any task body, charge with ctx.consumeCost:

async function callLLM(ctx, prompt) {
  const res = await openai.chat({ messages: [{ role: "user", content: prompt }] }, { signal: ctx.signal });
  ctx.consumeCost(res.usage.total_cost);   // throws + cancels owning scope on overrun
  return res;
}

ctx.consumeCost(amount) is atomic. Concurrent charges across siblings serialize through the budget cell. Overrun throws BudgetExceededError and cancels the owning scope — the scope that set the budget, even if the charge happened five levels deeper. The cancel reason is typed: CancelReason { kind: "budget", budgetKey, limit, spent }.

Built-in budgets:

import { CostBudget, TokenBudget, OpenAITokens, AgentToolCalls } from "@workit/core";

Custom budgets:

import { createBudget } from "@workit/core";
const Anthropic = createBudget("anthropic-tokens", { unit: "tokens" });

Bench 13-budget-atomicity-and-cancel.mjs. Three rules, measured.

Rule	Bench observation
Atomic concurrent charges	100 sibling tasks each consume 0.01 from a 1.00 cap -> final spent = 1.0000... exactly
Owning scope cancellation	Budget set at depth 0; overrun attempted at depth 5; outer scope cancelled with `kind: "budget"`
Caller-object immutability	After 0.5 of charges, the caller's input object stays `{ spent: 0, limit: 1, unit: "USD" }` (engine clones); live snapshot reflects the actual spend

Three more rules complete the contract (each tracked in the production suite):

Rule	Where it's enforced
Inner scope can shadow parent budget	Evidence coverage verifies an inner budget cell can charge independently while the outer budget remains unchanged.
Live read via `run.context.budget(key)`	Returns a fresh snapshot. Mutating the snapshot does not affect future reads.
Snapshots are `Readonly<BudgetState>` at the type	Consumer cannot mutate; engine routes mutation through `ctx.consume()` only

100,000 documents under a token cap

import { run, group } from "@workit/core";
import { OpenAITokens, embedAll } from "@workit/core/ai";

await run.context.with(
  OpenAITokens, { spent: 0, limit: 1_000_000, unit: "tokens" },
  () => group(() => embedAll(documents, {
    concurrency: 32,
    countTokens: (doc) => doc.tokens,
    async embed(doc, ctx) {
      return openai.embed(doc.text, { signal: ctx.signal });
    },
  })),
);

embedAll is a thin helper built on work().inParallel() from article 02 plus ctx.consume(OpenAITokens, count) per item. Hit the cap mid-stream -> scope cancels with CancelReason { kind: "budget", limit: 1_000_000, spent: 1_000_000 }. The 32 inflight embeddings see the abort on their ctx.signal. Provider calls that honor the signal cancel at the transport boundary. Partial results return and no additional budget is consumed after the cap.

Tracked: sample:embed100k runs the full 100,000-document pipeline against a deterministic provider fixture in CI. Asserts maxActive <= concurrency, finalBudget.spent === total, output.results.length === total.

The Context Overlay Speedup

Budgets, cancellation reasons, request scopes, idempotency keys, agent identity, deadlines — every cross-cutting concern lives in ContextBag. The first-pass implementation cloned the underlying Map on every .with() call. That's quadratic when you have a deep agent stack.

The fix: an overlay-based context. Think of it as a linked list of single-key deltas pointing at the parent bag. .with(key, value) returns a child that stores one entry and points at its parent. Lookup walks up the chain. Memory and cost per .with() are O(1).

Bench 14-context-overlay-perf.mjs. 100 .with() calls over a 5,000-key bag.

Implementation	Wall time	Per call
Naïve clone-on-`with` (inline baseline)	32.6 ms	~0.33 ms
WorkIt overlay context	0.011 ms	~0.0001 ms

The representative run shows a large constant-factor improvement over the inline clone baseline. Same public API. Same lookup result. The CI gate npm run check:context-performance asserts the overlay completes the workload in < 10 ms and the bench additionally asserts the inline baseline is at least 10x slower.

Evidence coverage verifies a deep shadow chain still resolves correctly and child shadows do not leak into the parent.

How WorkIt compares on resource safety

Pattern	Cancel-aware	Cleanup runs on every exit	Bounded cleanup time	Notes
WorkIt `run.bracket`	yes	yes	yes via `CleanupOpts.timeout` + `task:cleanup_timeout` event	release receives the resource and a cleanup signal
WorkIt `run.uncancellable`	yes (delayed rethrow)	n/a (it's the body, not a release)	yes via shield `timeout`	for short critical sections, not resource cleanup
`try { } finally { }`	no — finally cannot run after a hard cancel; cannot be bounded	partial — runs only if the awaiter completes settlement	no — a hanging cleanup deadlocks	bench 12-E: still pending after 250 ms
ES2024 `using` / `await using`	no — disposal hooks have no signal awareness	yes on scope exit	no — no timeout	best when the resource has an `[Symbol.dispose]` and no cleanup timeout is required
Effect-TS `acquireRelease`	yes	yes	partial (no built-in timeout on release in the public surface)	richer, but inside the Effect DSL

WorkIt is the only row that provides cancel-aware cleanup, guaranteed-to-run release, and a bounded timeout for the cleanup itself, surfaced as a typed event.

Receipts

node benchmarks/articles/12-bracket-vs-try-finally.mjs        # 5 bracket scenarios
node benchmarks/articles/13-budget-atomicity-and-cancel.mjs   # atomic + owning + immutable
node benchmarks/articles/14-context-overlay-perf.mjs          # 32.6 ms vs 0.011 ms
node benchmarks/articles/run-all.mjs                          # full article suite

Production-side gates that back the same primitives:

Claim	Evidence
`run.bracket` scenarios	`12-bracket-vs-try-finally.mjs` covers success, throw, cancel, timeout, hanging cleanup, and bounded release.
`run.uncancellable` scenarios	`08-uncancellable-shield.mjs` covers parent cancel during body, shield timeout, nested shields, and signal isolation.
Budget atomicity	Property test: 100 concurrent charges of 0.01 -> spent = 1.00 exactly. Reproduced by `13-budget-atomicity-and-cancel.mjs`.
Budget snapshot immutability	`tests/evidence/correctness/runtime-contracts.mjs` verifies caller objects remain unchanged and snapshots are read-only views of budget state.
Budget owning-scope cancellation	Charge attempted at depth 5 cancels the owning scope at depth 0 with `kind: "budget"`.
Context overlay perf	`npm run check:context-performance` asserts < 10 ms; bench records a representative ~0.01 ms run with a large speedup over the inline baseline.
100K embeddings sample	`sample:embed100k`: 100,000 docs, concurrency 32, token budget enforced, in-CI assertion.

What's coming

Now you can build an agent that costs 50 cents max, holds a database connection that always closes, and confirms a Stripe charge through a user disconnect.

Tomorrow: observability with bounded telemetry cost.

scope.tree() as a print statement for agents. The four-layer cost-control architecture — sampling, batching, summarization, budgeting — that takes a 100K-runs/day workload from $9,125/year of CloudWatch ingestion down to $456/year while preserving slow/error traces. 20x less data. One config object.

The headline: a structured-concurrency runtime where observability is sampled, batched, summarized, and budgeted by default — and you opt out of cost protection, not in.

Source, Benchmarks, And Evidence

NPM: https://github.com/WorkRuntime/workit
Source

workruntime.github.io
Article source: https://github.com/WorkRuntime/workit/blob/main/articles/05-resource-safety-and-budgeted-work.md
Reproduce: npm run bench:articles and npm run test:evidence

Don't let a billion RAG docs drown your 25-result pipeline

AdmilsonCossa — Mon, 25 May 2026 10:17:41 +0000

Backpressure For Streaming Pipelines

Last time we showed how to terminate non-cooperative CPU work at the worker boundary. This article stays cooperative but adds the missing piece: backpressure, the runtime contract that lets a producer pause the moment the consumer can't keep up.

A RAG ingest pipeline has a billion candidate documents. You only need the 25 that match a downstream filter. A naive promise collection can materialize far more work than the consumer needs; a hand-rolled async iterator can still fill a prefetch buffer before the first result arrives. With WorkIt:

import { work } from "@workit/core";

async function* billionDocuments() {
  for (let i = 0; i < 1_000_000_000; i++) yield { id: i, text: `doc ${i}` };
}

const results = [];
for await (const processed of work(billionDocuments())
  .inParallel(16)
  .map(async (doc, ctx) => enrich(doc, { signal: ctx.signal }))
  .stream()) {
  results.push(processed);
  if (results.length === 25) break;
}

Two things to notice:

work() accepts an async iterable directly. No .from(), no Readable.from(...) shim. The signature is Iterable<I> | AsyncIterable<I> -> WorkBuilder<I, I>.
.map().stream() is the streaming pipeline form. .do(fn) returns a Promise<WorkOutput<R>> (full batch result). .map(fn) returns a new builder; .stream() on a builder returns an AsyncIterable<O> that respects backpressure. Both terminals exist; you pick by what the consumer is doing.

What the producer actually does:

Bench 09-stream-1b-lazy.mjs. 1,000,000,000-row generator. inParallel(16). Consumer takes 25, breaks.

Implementation Consumed Items pulled from the generator maxActive In-flight after break

Naïve eager prefetch buffer (256-deep) 25 281 1 0 (all let to settle)

work().inParallel(16).map().stream() 25 40 1 0 (cancelled at break)

Implementation	Consumed	Items pulled from the generator	maxActive	In-flight after break
Naïve eager prefetch buffer (256-deep)	25	281	1	0 (all let to settle)
`work().inParallel(16).map().stream()`	25	40	1	0 (cancelled at break)

These are representative captured values. The bench asserts the invariant: produced items stay bounded by TAKE + CONCURRENCY. The naïve baseline pulled 281 items because once the prefetch buffer is full it doesn't pause the producer -- it pauses the worker pool, which is a different question.

That's backpressure: the producer pauses when the consumer slows down or stops, not when the worker pool fills.

`work().stream()` -- bounded, lazy, cancellable

for await (const summary of work(documents)
  .inParallel(8)
  .withRetry(2)
  .withTimeout("15s")
  .map(async (doc, ctx) => summarize(doc, { signal: ctx.signal }))
  .stream()) {
  ui.append(summary);
}

Properties the runtime guarantees:

inParallel(N) is a hard cap. maxActive never exceeds N. Property test runs 1..20 wide x 1..100 items, asserts the cap holds across every shape.
stream() is lazy. The producer iterator pulls only when an inflight slot is free.
break is cancellation. The remaining inflight tasks abort with CancelReason { kind: "manual", tag: "stream_consumer_closed" }. Their ctx.defer runs. The producer iterator's return() runs.
A throw inside the body triggers CancelReason { kind: "manual", tag: "stream_failed" } for siblings -- typed, distinguishable from the consumer-break path on a dashboard.
Slow consumer pauses producer. Tracked under check:stream-memory: 1,000,000 logical items, slow consumer, bounded heap growth, and no unbounded producer advance.

Bench 10-stream-slow-consumer.mjs. 5,000-item source, inParallel(16), consumer ~5 ms per item, take 200.

Metric Value

Consumed 200

Produced 215

Producer overshoot 15 (bound: CONCURRENCY + 1 = 17)

maxActive 1

In-flight after break 0

Wall time ~3,108 ms

Metric	Value
Consumed	200
Produced	215
Producer overshoot	15 (bound: `CONCURRENCY + 1` = 17)
maxActive	1
In-flight after break	0
Wall time	~3,108 ms

The interesting detail: even with inParallel(16), maxActive stayed at 1 because the consumer was the bottleneck. The runtime didn't speculatively saturate the worker pool -- it paced the producer to consumer demand. That is what "backpressure" actually means. A pool that always runs at capacity isn't backpressure; it's a pool.

Streaming map: stop after 12, produce only what demand requires

The most practical reader-facing form of the same property -- a real summarizer pipeline, the size of a real prompt:

// samples/streaming-summarizer.sample.js
const TAKE = 12;
const CONCURRENCY = 5;

for await (const summary of work(documents())
  .inParallel(CONCURRENCY)
  .withRetry(2)
  .withTimeout("500ms")
  .map(async (doc, ctx) => `summary:${doc.id}`)
  .stream()) {
  summaries.push(summary);
  if (summaries.length === TAKE) break;
}

// Asserted by the sample:
//   summaries.length === TAKE
//   produced     <= TAKE + CONCURRENCY - 1
//   maxActive    === CONCURRENCY
//   active       === 0       // all in-flight cancelled cleanly on break

50-doc generator. Consume 12. Producer never advances past 16. Concurrency cap exact. Active count zero after break. Retry and timeout policy attached without breaking the pull cadence.

npm run sample:stream

Defaults that don't surprise

Setting	Default	Why
`inParallel`	`1` (sequential)	Auto-concurrency surprises rate-limited APIs. Sequential is correct.
`withRetry`	none	Retrying non-idempotent ops silently is a footgun.
`withTimeout`	none	Cancelling work the user didn't ask to cancel is worse than no timeout.
`onError`	`"fail"`	Matches `Promise.all` intuition. The discriminated `WorkOutput<R>` return type forces explicit handling on the others.

You opt into resilience. Nothing is implicit.

CSP-style channels -- `@workit/core/channel`

work().stream() is the right shape when the producer-consumer relationship is one fluent pipeline. When the producer and consumer are independent tasks running side by side -- fan-in, fan-out, work-queue -- you want a channel.

import { createChannel } from "@workit/core/channel";
import { group } from "@workit/core";

const orders = createChannel<Order>({ capacity: 100 });

await group(async (task) => {
  task(async (ctx) => {
    for await (const o of orderSource()) {
      await orders.send(o, { signal: ctx.signal });
    }
    orders.close();
  });

  task(async (ctx) => {
    for await (const o of orders) {
      await processOrder(o, { signal: ctx.signal });
    }
  });
});

Channel contract, all five rows verified by 11-channel-contract.mjs:

#	Scenario	Bench observation
A	`send` blocks when the channel is full	On a `capacity: 2` channel, the third `send` is still pending after a microtask turn and completes only after a `receive` frees a slot
B	`close()` drains buffered values	`[1, 2, 3]` delivered, then iteration ended cleanly
C	Pending `send` after `close(reason)` rejects	`ChannelClosedError` with `reason: { tag: "shutdown" }`
D	A `signal` cancels a pending `receive`	Pending receive rejects when the controller aborts
E	Capacity validation	`0`, `-1`, `0.5`, `NaN`, `Infinity` all rejected with `RangeError` at `createChannel(...)`

Cancellation composes with the parent scope. If the consumer task throws inside group, sibling cancellation aborts the producer's pending send. The producer's for await exits cleanly through the rejection. No orphaned sends, no leaked consumers, no half-drained buffer.

This is Go's chan with structured-concurrency parents. Kotlin's Channel without coroutines. It fills the gap between "raw async iterator" and "RxJS observable" for owned producer-consumer work.

Bad-batch bisection -- one rotten document doesn't poison the embedding

A real RAG pipeline failure mode: the provider returns 400 for a mixed batch because one of the documents is malformed. With Promise.all, the whole batch fails, the budget is spent on nothing, and the next 99 documents get re-embedded on retry.

WorkIt ships embedAllBisection that splits the failed batch and recovers the good vectors:

// samples/embed-bisection.sample.js
const result = await group(
  async () => embedAllBisection(["alpha", "bad-doc", "gamma"], {
    async embedBatch(inputs) {
      if (inputs.includes("bad-doc")) throw new BadBatchError("provider rejected mixed batch");
      return inputs.map((input) => [input.length]);
    },
  }, {
    batchSize:   3,
    onError:     "continue",
    countTokens: (input) => input.length,
  }),
  { context }
);

// Asserted by the sample:
//   result.results contains the vectors for "alpha" and "gamma"
//   result.errors  contains exactly one entry pointing at "bad-doc"
//   tokensSpent reflects only the successful work

BadBatchError is the contract. Throw it from embedBatch and the helper bisects: split the batch in halves, retry each half, isolate the rotten document, keep the good vectors. Token budget accounting follows the actual successful work -- you don't pay for the failed mixed batch twice.

npm run sample:bisection

This is the difference between "batch job dies at 2 a.m. and the on-call resyncs the warehouse" and "batch job logs the bad ID and keeps going."

Streaming STT with disconnect cleanup (revisited)

Article 1 showed this. Now you can read the backpressure underneath it:

import { transcribeStream } from "@workit/core/ai";

for await (const text of transcribeStream(microphone, {
  async transcribe(chunk, ctx) {
    return provider.transcribe(chunk, { signal: ctx.signal });
  },
}, { signal: socket.signal })) {
  socket.send(text);
}

When the user closes their laptop:

socket.signal aborts.
transcribeStream propagates the abort to the inflight transcribe() body.
The provider's HTTP request aborts at the AbortSignal boundary.
The async generator's finally runs, closing the microphone source.
The for await loop exits.

Tracked sample: sample:stt-disconnect -- disconnects mid-second-chunk, asserts the provider was cancelled, the source was closed, and the cancel reason kind is manual.

How WorkIt's streaming primitives compare

Library	Backpressure	Cancellation	Structured concurrency	Note
WorkIt `work().stream()`	yes producer pauses on consumer	yes via `ctx.signal` and `break`	yes scope-owned	Backpressure between producer and consumer in one pipeline
WorkIt `createChannel`	yes blocking `send`/`receive`	yes via signal + scope cancel	yes scope-owned	Backpressure between independent tasks
Node.js `Readable` stream	yes via `highWaterMark`	partial via `destroy()`	no no scope	No structured cancel propagation
RxJS observable	no by default; pressure operators are opt-in	yes on `unsubscribe`	per-subscription, not per-scope	Different model: events, not owned tasks
`p-queue`	partial (concurrency limit)	no	no	Bounds in-flight, not producer pull
Async generator (raw)	yes pull-based	partial via `return()`	no	No bounded concurrency without manual scaffolding

WorkIt's streaming and channel primitives are the only ones in the table that tie backpressure to ownership -- cancel the scope, the channel closes, the in-flight work aborts, and cleanup runs.

Receipts

node benchmarks/articles/09-stream-1b-lazy.mjs        # naive 281 vs WorkIt 40
node benchmarks/articles/10-stream-slow-consumer.mjs  # producer overshoot 15 vs bound 17
node benchmarks/articles/11-channel-contract.mjs      # 5 channel scenarios
node benchmarks/articles/run-all.mjs                  # full article suite

Production-side gates that back the same primitives:

Claim	Evidence
1 B virtual stream consumed = 25	`sample:1b` produces <= TAKE+CONCURRENCY items, asserted in CI. Reproduced by `09-stream-1b-lazy.mjs`.
1 M item slow-consumer gate	`check:stream-memory` -- heap growth bounded, max active capped, and producer pull remains demand-limited.
Channel backpressure on capacity 2	`11-channel-contract.mjs` verifies the third send blocks until the first receive.
Channel close + drain	`tests/evidence/correctness/runtime-contracts.mjs` verifies buffered values drain before `done: true`.
Channel cancel via signal	Channel contract coverage verifies pending receives reject with the cancel reason.
Channel composes with `group()`	Channel contract coverage verifies producer/consumer pipelines deliver values in order.
`work().inParallel(N)` cap	Property test (`fast-check`): for any (N, total), `maxActive <= N`.
STT disconnect	`sample:stt-disconnect`: provider cancelled, source closed, reason kind = `manual`.

Run them:

npm run sample:1b
npm run sample:stream
npm run sample:embed100k
npm run sample:bisection
npm run sample:stt-disconnect

What's coming

Now you have a producer that paces itself to the consumer, a channel that closes when its scope cancels, and a stream that exits cleanly when the user closes the tab.

Tomorrow we add the next ownership primitive on top: the budget.

A $0.50 CostBudget. A 100,000-token OpenAITokens. A 5-tool-call AgentToolCalls. Atomic across all parallel children. Inheritable through scope context. Shadowed by inner scopes for sub-budgets. Overrun cancels with CancelReason { kind: "budget" } and partial results stay.

The runtime change underneath this is context overlay lookup: 100 .with() calls over a 5,000-key context bag moved from tens of milliseconds in the inline clone baseline to well under the 10 ms gate, without changing a line of public API. The bench in the next article shows the representative timing.

The point is not simply "we have budgets." Many frameworks expose budgets. The stronger claim is budgets that compose with cancellation, race, retry, hedge, fallback, channels, and streams under one ownership tree.

Source, Benchmarks, And Evidence

Source: https://github.com/WorkRuntime/workit
Article source: https://github.com/WorkRuntime/workit/blob/main/articles/04-backpressure-for-streaming-pipelines.md
Reproduce: npm run bench:articles and npm run test:evidence

Cancelling a Task Doesn’t Stop the CPU Work — Killing Non-Cooperative Jobs in worker_threads

AdmilsonCossa — Thu, 21 May 2026 09:04:21 +0000

Last time we showed nine composables that cancel siblings, retry with signal-aware backoff, and hedge tied requests. That's cooperative cancellation – it works when the body checks the signal and the I/O it makes is signal-aware. This article answers the hard question: what happens when code does not cooperate?

Drop this in a worker module:

// benchmarks/articles/lib/spinner.mjs – ignores every signal you throw at it
import { writeFileSync } from "node:fs";

export function spin({ durationMs, markerPath }) {
  const start = Date.now();
  while (Date.now() - start < durationMs) {
    Math.sqrt(Math.random() * 1e6);
  }
  writeFileSync(markerPath, "late-marker-written-by-worker");
  return { completed: true, elapsedMs: Date.now() - start };
}

This is the canonical "non-cooperative" body. Tight loop, no await, no signal.aborted check. In a single-threaded JS runtime, cooperative cancellation cannot stop it from the outside. AbortController can record cancellation, but the callback cannot run until the event loop yields.

import { offload } from "@workit/core/worker";
import { run } from "@workit/core";

await run.scope(async (scope) => scope.spawn(
  offload(spinnerURL, "spin", { durationMs: 5_000, markerPath }, { timeout: "200ms" }),
));

200 ms later: TimeoutError. The worker thread is terminated by the host. The late-marker file does not exist on disk. We stat() for it in CI and fail the gate if it does.

That's the only honest answer to "can JS forcibly stop work". The answer is no on the main thread – but you can move the work to a worker and have the host kill the thread.

WorkIt has both layers. They are labeled honestly.

Bench 07-worker-hard-kill.mjs. 5,000 ms spin loop. 200 ms timeout. Late-marker file written after the loop completes. Timings are representative; the invariant is the marker-file result.

Implementation Settled at Late-marker file on disk

Main-thread AbortController.abort() t=5,001 ms (full duration) yes exists – the abort callback never even fired; the event loop was starved

offload({ timeout: "200ms" }) t=206 ms with TimeoutError no does not exist – and stays absent through an 800 ms grace window

Implementation	Settled at	Late-marker file on disk
Main-thread `AbortController.abort()`	t=5,001 ms (full duration)	yes exists – the abort callback never even fired; the event loop was starved
`offload({ timeout: "200ms" })`	t=206 ms with `TimeoutError`	no does not exist – and stays absent through an 800 ms grace window

The native baseline is the receipt for "AbortController cannot preempt a CPU loop." The body completed all 5 seconds and wrote the marker, and the setTimeout that was supposed to fire controller.abort() at 200 ms could not be delivered because the event loop never returned. The abort callback never ran.

Layer 1 -- Cooperative cancellation, in-process

async function callTool(input, ctx) {
  ctx.signal.throwIfAborted();                              // explicit checkpoint
  const res = await fetch(url, { signal: ctx.signal });     // signal-aware
  ctx.signal.throwIfAborted();                              // checkpoint after I/O
  return res.json();
}

Cooperative cancellation works when the body checks the signal at safe points and the I/O it makes is signal-aware. WorkIt handles the checkpoints automatically through await boundaries inside run.retry, run.timeout, run.race, and the work() builder. You just have to thread ctx.signal into the I/O calls.

This works for 95% of code: HTTP, database, filesystem, streams, child processes, sleeps, channel sends. They all take an AbortSignal.

It does not work for the 5% that does CPU loops, sync crypto, sync JSON.parse of a 200 MB string, or a fitness test in a genetic algorithm.

For that 5%, you need Layer 2.

Layer 2 -- Hard kill at the worker boundary

import { offload } from "@workit/core/worker";

const transcoded = offload(
  new URL("./ffmpeg-transcode.js", import.meta.url),
  "transcode",
  { input, format: "webm" },
  { timeout: "30s" },
);

await run.scope(async (scope) => scope.spawn(transcoded));

offload(...) returns a TaskFn. Spawn it into a scope. The named export runs in a Worker thread. When the timeout fires the worker is terminated -- not signalled, not asked nicely. The host process keeps running. The promise rejects with TimeoutError. If the parent scope cancels first, the worker is terminated with a CancellationError carrying the parent's reason.

What offload accepts:

Local file URLs (new URL("./mod.js", import.meta.url)).
Named export only.
Structured-cloneable input: primitives, arrays, plain objects, Map, Set, Date, RegExp, ArrayBuffer, SharedArrayBuffer, typed array views.

What offload rejects, before the worker spins up:

Remote and inline URL schemes (https:, data:, blob:).
Path traversal segments.
Functions, symbols, class instances, custom-prototype objects – including buried inside Map values, Set members, or cycles.

The worker boundary is covered by unit tests and by tests/evidence/security/worker-boundary.mjs. The two interesting subtleties: Object.create(null) is accepted (a null-prototype object is "plain enough"), and a class with a clean-looking shape is rejected at deep walk because the prototype check runs on the cloneable graph, not just the top level.

Worker offload -- the happy path

The hard-kill is the headline, but the everyday use of offload is mundane CPU work. The repo ships a sample that runs two Fibonacci computations on real worker threads through run.pool:

// samples/worker-offload.sample.js
const moduleURL = new URL("./cpu-worker.sample-worker.js", import.meta.url);

const results = await run.pool(2, [
  offload(moduleURL, "fibonacci", 20),
  offload(moduleURL, "fibonacci", 21),
]);

// Asserted by the sample:
//   results.map(r => r.value) === [6_765, 10_946]
//   results.every(r => r.threadId > 0)

Different OS thread per task. Both results returned. No try/catch around Worker. No parentPort plumbing. Just offload(modURL, "fnName", input) composed through the same run.pool you saw in article 02. The same primitive that terminates a CPU spinner at the worker boundary is also the one you use to take a heavy sync transform off the event loop.

npm run sample:worker

The shield: `run.uncancellable`

Some code must run to completion even when the parent scope is being cancelled. Database commit. Stripe webhook receipt. Distributed lock release. Audit log flush.

import { run } from "@workit/core";

const commit = run.uncancellable(async (ctx) => {
  await db.commit({ signal: ctx.signal });
  await flushReceipt({ signal: ctx.signal });
}, { timeout: "2s" });

await run.scope(async (scope) => scope.spawn(commit));

Inside the shielded body, ctx.signal is a fresh signal local to the shield -- the parent's cancel does not propagate in. The shield has its own bounded lifetime (timeout: "2s"). When the shield finishes, if the parent had cancelled during the shield, the original CancellationError rethrows after the body completes. Cancellation is delayed, not hidden.

What this is not: run.uncancellable is cooperative. It cannot stop a non-cooperative CPU loop inside the shielded body. For that, use offload.

Bench 08-uncancellable-shield.mjs. Three scenarios -- measured.

Scenario What we measure Result

A. Parent cancel mid-body Body started t=1 ms, parent cancelled at t=41 ms, body sleeping 120 ms Body completed naturally at t=136 ms (outlived cancel by 95 ms), bodyObservedAbort: false, outer settled cancelled with reason.kind === "manual"

B. Shield timeout while body runs Shield { timeout: "100ms" }, body sleeping 2,000 ms Body observed abort at ~100 ms, bodyAbortReasonClass === "TimeoutError", outer settled TimeoutError

C. Nested shields, outer scope cancels Inner sleep 80 ms, outer scope cancels at t=20 ms Inner completed at t=92 ms, outer-shield body completed at t=92 ms, outer settled cancelled at t=93 ms with reason.kind === "manual" -- preserved through both shields

Scenario	What we measure	Result
A. Parent cancel mid-body	Body started t=1 ms, parent cancelled at t=41 ms, body sleeping 120 ms	Body completed naturally at t=136 ms (outlived cancel by 95 ms), `bodyObservedAbort: false`, outer settled `cancelled` with `reason.kind === "manual"`
B. Shield timeout while body runs	Shield `{ timeout: "100ms" }`, body sleeping 2,000 ms	Body observed abort at ~100 ms, `bodyAbortReasonClass === "TimeoutError"`, outer settled `TimeoutError`
C. Nested shields, outer scope cancels	Inner sleep 80 ms, outer scope cancels at t=20 ms	Inner completed at t=92 ms, outer-shield body completed at t=92 ms, outer settled `cancelled` at t=93 ms with `reason.kind === "manual"` -- preserved through both shields

A is the "delayed cancel" contract. B is "the shield is bounded by its own timeout, which the body sees as a TimeoutError on its local signal". C is "nested shields don't lose the outer cancel reason."

Cancellation reasons are typed, not strings

type CancelReason =
  | { kind: "user"; message: string }
  | { kind: "deadline"; deadlineAt: number; elapsedMs: number }
  | { kind: "timeout"; timeoutMs: number }
  | { kind: "parent_failed"; error: unknown }
  | { kind: "sibling_failed"; siblingId: TaskId; error: unknown }
  | { kind: "race_lost"; winnerId: TaskId }
  | { kind: "budget"; budgetKey: string; limit: number; spent: number }
  | { kind: "scope_ended" }
  | { kind: "manual"; tag: string; data: unknown };

Every cancellation in WorkIt carries one of these. You can pivot a metric on cancelReason.kind. You can route a runbook on tag. You can build a "why did my agent stop" dashboard with seven buckets and an exhaustive switch. TypeScript will tell you when you forgot a case.

Compare:

controller.abort("user_clicked_stop");                  // string. lossy. arbitrary.
controller.abort(new DOMException("...", "AbortError")); // class with "Abort" name. that's it.

AbortSignal.reason was a stringly-typed escape hatch that won. WorkIt closes it with a discriminated union and tests that every kind is exercised in the suite.

How do other libraries handle non-cooperative work

Library	Cooperative cancellation	Hard kill (CPU loops)	Mechanism
WorkIt	yes signal-aware	yes built-in	`offload({ timeout })` terminates the worker thread
Effection	yes generator ops	no	bring your own worker
Effect-TS	yes fibers	no	bring your own worker
Native `AbortController`	yes	no	the event loop is single-threaded

If you need to kill a sync CPU loop today and you're not on WorkIt, you're hand-rolling worker management – module URL validation, structured-clone classification, timeout-driven termination, parent-cancel propagation, error propagation back to the host. WorkIt's offload is ~50 lines of public surface and the runtime contract is in CI.

Receipts

Two layers. Two benches. One evidence path per claim.

node benchmarks/articles/07-worker-hard-kill.mjs       # main-thread vs offload
node benchmarks/articles/08-uncancellable-shield.mjs   # 3 shield contracts
node benchmarks/articles/run-all.mjs                   # full article suite

Production-side gates that back the same contracts:

Claim	Evidence
Worker hard-kill on CPU spinner	`07-worker-hard-kill.mjs` runs `offload({ timeout: "200ms" })` against the spinner module, asserts bounded rejection, and verifies the late-marker file does not exist.
Worker hard-kill on parent cancel	`tests/evidence/security/worker-boundary.mjs` verifies parent cancellation terminates worker-owned CPU work.
5 concurrent offloads	Worker unit coverage exercises mixed fast and spinning workers without cross-talk between results.
Input validation	`tests/evidence/security/worker-boundary.mjs` verifies remote and executable worker URLs are rejected; unit coverage exercises structured-clone classification.
`run.uncancellable` semantics	`08-uncancellable-shield.mjs` covers parent cancel during body, shield timeout, nested shields, signal isolation, and reason preservation.
`CancelReason.kind` coverage	Every kind in the discriminated union has at least one tracked test that produces it.

The ergonomic version of cooperative cancellation:

await sleep(ms, ctx.signal);             // signal-aware sleep
await fetch(url, { signal: ctx.signal }); // signal-aware fetch

The ergonomic version of hard cancellation:

await scope.spawn(offload(modUrl, "fn", input, { timeout: "Xs" }));

That's the API. Two layers. Honest labels.

What's coming

Tomorrow: backpressure.

You're consuming a billion-row source you'll never materialize. You want to read 25 from the front, run them through a 16-wide map, and have the producer pause when the consumer can't keep up. You want a transcription stream that exits cleanly when the user closes the tab. You want CSP-style channels for the part of your pipeline that's actually a pipeline.

The slow-consumer memory gate runs a million items through a paused consumer in CI and asserts the heap doesn't move. That's the next bench.

Source, Benchmarks, And Evidence

Source: https://github.com/WorkRuntime/workit
Article source: https://github.com/WorkRuntime/workit/blob/main/articles/03-cancellation-and-worker-boundaries.md
Reproduce: npm run bench:articles and npm run test:evidence

Your AI Agent Failed. Why Are Its Retries, Timeouts, and Tool Calls Still Running?

AdmilsonCossa — Thu, 14 May 2026 10:00:15 +0000

They fail across concurrency, retries, timeouts, queues, tools, streams, and provider calls.

That is why the JavaScript ecosystem has huge demand for separate async primitives:

Package	Weekly Downloads
`p-limit`	~204M
`p-map`	~53M
`p-timeout`	~36M
`p-retry`	~38M
`async-retry`	~24M
`p-queue`	~23M
`bottleneck`	~10M

These libraries are good.

But the production pain is deeper:

You want concurrency → add p-limit

You want retries → add p-retry

You want timeouts → add p-timeout

You want queues → add p-queue

You want rate limits → add bottleneck

Now each primitive owns a different part of the lifecycle.

None coordinate cancellation together.

When an AI agent fails mid‑flight:

Who stops the retry?

Who clears the timeout?

Who drains the queue?

Who cleans up the tool call?

Who prevents the losing provider from continuing to bill?

WorkIt explores a different model:

work(items)
  .inParallel(8)
  .withRetry(3)
  .withTimeout("5s")
  .do(fn)

One scope. One owner. One cancellation path. One cleanup model.

Concurrency, retry, timeout – under the same ownership tree.

👉 Read the full article:

Concurrency, Retry, and Timeout Under One Owner

npm install @workit/core

AI agents do not fail in one place. They fail across retries, timeouts, tools, and provider calls.

AdmilsonCossa — Mon, 11 May 2026 14:43:53 +0000

Concurrency, Retry, And Timeout Under One Owner.

Last time we showed work(items).inParallel(8).withRetry(3).withTimeout("5s").do(fn) — the one-line fluent surface for processing a list. That handles the 80% case.

Check Owned Async Work in TypeScript — Promise.race does not cancel your work and Your AI Is Still Billing After the User Closed the Tab in case you missed them.

This article is about the other 20%: orchestrating heterogeneous tasks that race, fall back, hedge, and retry — with ownership.

Open package.json in enough AI codebases and you'll see combinations like these – or completely different helpers – depending on who wrote the stack:

"p-limit":      "^5.0.0",
"p-map":        "^7.0.0",
"p-retry":      "^6.2.0",
"p-timeout":    "^6.1.2",
"p-queue":      "^8.0.1",
"bottleneck":   "^2.19.5",
"async-retry":  "^1.3.3"

Six libraries is just one example. Some teams roll their own Promise.race loops. Others use async helpers from lodash or bluebird. The pattern is the same: scattered concurrency primitives with no shared ownership.

When a sibling throws, when a timeout fires, when the user hits stop, you have to stitch together queue state, retry delay, timeout wrapper, underlying I/O, cleanup, and error shape yourself.

That is the comparison in this article: not "those tools are useless", but "they are separate primitives." WorkIt's claim is ownership and composition. The runnable benches at the end of each section verify the WorkIt invariants on your machine.

WorkIt has five core composables, all sharing one runtime contract:

run.all      // Promise.all that actually cancels losers on first failure.
run.race     // Promise.race that actually cancels losers.
run.any      // Promise.any that actually cancels remaining tasks.
run.pool     // p-limit + p-map, but children belong to the scope.
run.series   // sequential, with shared cancellation.

Plus four more that compose with them:

run.retry    // backoff with signal-aware sleep.
run.timeout  // deadline that returns a TaskFn.
run.fallback // primary -> secondary, type-safe.
run.hedge    // bounded speculative execution for tail-latency control.

Same familiar names. Different runtime contract: everything below the call belongs to a scope, and the scope owns the cancel.

The composition property under all nine: every WorkIt resilience helper takes a TaskFn<T> and returns a TaskFn<T>. That makes the algebra closed – run.timeout(run.retry(callProvider, 3), "5s") is just function composition. Promise helpers usually return promises or independent wrapper functions, so crossing from timeout to retry to race means you own the glue and the signal threading.

Why `run.all` improves on Promise.all’s safety

import { run } from "@workit/core";

const [profile, plan, sources] = await run.all([
  (ctx) => fetchProfile({ signal: ctx.signal }),
  (ctx) => planLLM(question, { signal: ctx.signal }),
  (ctx) => retrieveContext(question, { signal: ctx.signal }),
]);

Promise.all rejects on first failure and leaves the other two requests running unless each branch has its own cancellation wiring. Their .then handlers can fire after your error handler already returned a 500, producing completion events that are no longer attached to the owning request.

run.all rejects on first failure and cancels the other two. ctx.signal aborts. defer cleanups run. The reason is typed: CancelReason { kind: "sibling_failed", siblingId, error }.

You can pivot a dashboard on that. You cannot pivot on Error: AggregateError.

Bench 01-run-all-vs-promise-all.mjs. A succeeds at 50 ms. B fails at 30 ms. C succeeds at 100 ms.

Implementation Outer rejected A still ran past reject C still ran past reject Defer ran for losers

Promise.all t=35 ms +16 ms +79 ms n/a

run.all t=32 ms 0 ms (cancelled at +1 ms) 0 ms (cancelled at +1 ms) yes before outer reject

Implementation	Outer rejected	A still ran past reject	C still ran past reject	Defer ran for losers
`Promise.all`	t=35 ms	+16 ms	+79 ms	n/a
`run.all`	t=32 ms	0 ms (cancelled at +1 ms)	0 ms (cancelled at +1 ms)	yes before outer reject

`run.race` – the race that actually races

const winner = await run.race([callOpenAI, callAnthropic, callGemini]);

Six tokens you wrote with Promise.race. Different runtime contract:

Each body receives a ctx.signal linked to the race.
First settlement cancels the rest at the AbortSignal boundary, before TCP completes.
Each loser sees CancelReason { kind: "race_lost", winnerId } — typed, exhaustively narrowed.
await run.race(...) returns only after losers have finished cleaning up.

Bench 02-run-race-vs-promise-race.mjs. Anthropic at 10 ms, OpenAI at 50 ms, Gemini at 80 ms.

Implementation Winner at OpenAI loser still ran Gemini loser still ran Loser reason

Promise.race t=14 ms +47 ms (61 ms total) +77 ms (91 ms total) none

run.race t=17 ms 0 ms (cancelled at t=16 ms) 0 ms (cancelled at t=16 ms) race_lost

Implementation	Winner at	OpenAI loser still ran	Gemini loser still ran	Loser reason
`Promise.race`	t=14 ms	+47 ms (61 ms total)	+77 ms (91 ms total)	none
`run.race`	t=17 ms	0 ms (cancelled at t=16 ms)	0 ms (cancelled at t=16 ms)	`race_lost`

That loser runtime x N parallel agents x P requests per second is the line on your invoice that nobody wrote.

`run.any` – first success, rest cancelled

const cheapest = await run.any([callExpensive, callCheap, callCheaper]);

Promise.any resolves with the first success and ignores the rest. The slower siblings keep running. The faster failing ones got logged and forgotten. run.any does the same – except the slower siblings actually stop.

Bench 03-run-any-vs-promise-any.mjs. A fails at 30 ms. B succeeds at 50 ms. C succeeds at 100 ms.

Implementation Resolved at C kept running Defer ran for C

Promise.any t=61 ms +47 ms (108 ms total) n/a

run.any t=65 ms 0 ms (cancelled at t=65 ms) yes

Implementation	Resolved at	C kept running	Defer ran for C
`Promise.any`	t=61 ms	+47 ms (108 ms total)	n/a
`run.any`	t=65 ms	0 ms (cancelled at t=65 ms)	yes

`run.pool` – bounded concurrency that cancels

const results = await run.pool(8, files.map((file) => async (ctx) => {
  return uploadOne(file, { signal: ctx.signal });
}));

p-limit(8) is a semaphore. That's useful, and current versions can clear pending queue items when you ask them to. What it is not is a structured scope: it does not automatically turn a sibling failure into in-flight cancellation, typed cancel reasons, cleanup, and a partial-result contract.

run.pool(8, tasks) is a semaphore + a scope. Default policy is Promise.all-style fail-fast: first throw cancels queued and in-flight. Results are positionally indexed regardless of completion order. Switch policy with one line and the return type changes so you can't ignore failures:

const out = await work(files).inParallel(8).onError("collect").do(uploadOne);

if (out.mode === "collect") {
  for (const r of out.results) {
    if (r.status === "rejected") logFailure(r.reason);
  }
}

WorkOutput<R> is a discriminated union – mode: "fail" | "continue" | "collect". Change .onError("continue") and the return type forces you to handle errors[]. The compiler is your audit log.

Bench 04-pool-vs-semaphore.mjs. 10 items, concurrency 4. Item 3 throws at 20 ms; the rest take 100 ms each.

Implementation Outer rejected Started Fulfilled AFTER rejection Cancelled Never started Longest post-rejection run

local pLimitLike(4) semaphore baseline t=31 ms 10 9 0 0 +295 ms

run.pool(4, ...) t=33 ms 4 0 3 6 0 ms

Implementation	Outer rejected	Started	Fulfilled AFTER rejection	Cancelled	Never started	Longest post-rejection run
local `pLimitLike(4)` semaphore baseline	t=31 ms	10	9	0	0	+295 ms
`run.pool(4, ...)`	t=33 ms	4	0	3	6	0 ms

295 ms of post-rejection work, multiplied across a fleet, becomes avoidable runtime and provider cost.

`run.retry` – composable, cancel-aware backoff

const callWithRetry = run.retry(callProvider, {
  times: 4,
  backoff: "exponential",
  initialDelay: "200ms",
  maxDelay: "5s",
  jitter: true,
  retryIf: (err) => isTransient(err),
});

const answer = await callWithRetry(ctx);

Three things WorkIt makes part of the retry contract:

Stop retrying on scope cancellation. When the parent scope cancels mid-attempt, run.retry does not enqueue another attempt. The task settles as cancelled, not failed.
Validate input at the boundary. run.retry({ times: 1e9 }) would create an unbounded retry policy. run.retry rejects it: RangeError: retry attempts must be an integer between 1 and 1000. Bound is MAX_RETRY_ATTEMPTS.
Sleep with the scope signal. Backoff sleep is interruptible – abort the signal, the sleep rejects, the loop exits. The benchmark below compares against a signal-unaware retry loop; current retry libraries may expose their own abort hooks, but they still do not own WorkIt's scope tree, cleanup, and cancel-reason contract.

Bench 05-retry-on-cancel.mjs. Body throws on every attempt. External cancel fires around t=50 ms. Up to 8 retries with 50 ms backoff.

Implementation Cancel observed Outer settled Cancel latency Extra attempts after cancel Settled as

signal-unaware retry loop t=63 ms t=701 ms 638 ms 7 rejected

run.retry t=61 ms t=61 ms 0 ms 0 cancelled (kind: manual)

Implementation	Cancel observed	Outer settled	Cancel latency	Extra attempts after cancel	Settled as
signal-unaware retry loop	t=63 ms	t=701 ms	638 ms	7	`rejected`
`run.retry`	t=61 ms	t=61 ms	0 ms	0	`cancelled` (kind: `manual`)

638 ms of wasted retry work after the user already cancelled. Per request. Multiply by the agent fan-out.

`run.timeout`– composes with retry, race, and pool

const fastest = await run.race([
  run.timeout(callPrimary,   "800ms"),
  run.timeout(callSecondary, "800ms"),
]);

run.timeout(task, "800ms") returns a TaskFn. It composes. You can wrap it in run.retry. You can put it inside run.race. You can hand it to run.pool. The signature is closed under composition.

Promise timeout helpers return promises or decorated promises. Some expose AbortSignal support. They still do not return a WorkIt TaskFn, so crossing timeout, retry, race, pool, and cleanup means you own the composition boundary.

`run.fallback` – primary, secondary, type-safe

const callWithFallback = run.fallback(
  run.retry(callProvider, 3),
  callBackupProvider,
);

Primary fails (after retries) -> secondary runs. Same ctx.signal. Same scope. Same cancel reason if the parent stops. No nested try/catch. No "did I forget to await the fallback" Slack message at 2 a.m.

`run.supervise` – restart policy for long-lived work

run.retry is for one operation that may fail transiently and then succeed. run.supervise is for a long-lived task – a heartbeat, a queue consumer, a connection watcher, an agent keep-alive – that may need restart semantics with bounded backoff.

import { run } from "@workit/core";

const result = await run.supervise(async () => {
  attempts++;
  if (attempts < 3) throw new Error("transient worker failure");
  return "stable";
}, {
  restartOn:    "error",
  maxRestarts:  3,
  backoff:      () => 1,
});

// samples/supervision.sample.js – asserted in CI:
//   result === "stable"
//   attempts === 3

The supervised body fails twice, restarts each time under the policy, and stabilises on the third attempt. The parent scope can still cancel everything at once, and the cancel reason carries down through the supervision wrapper. Restart policies cap at maxRestarts per resetWindow so a permanently broken body doesn't infinite-loop.

npm run sample:supervise

The decision rule: use run.retry for a single call that can hiccup; use run.supervise for a process that should keep running.

`run.hedge` – bounded speculative requests

const ranked = await run.hedge(
  (ctx) => reranker.rank(question, sources, { signal: ctx.signal }),
  { after: "2s", max: 2 },
);

If the first call hasn't returned in 2 seconds, fire a second one. First success wins; the rest cancel. Bounded by max, this is a measured way to reduce tail latency without paying for every speculative fan-out.

Bench 06-hedge-tied-requests.mjs. Two scenarios, opts { after: "50ms", max: 3 }.

Scenario Body latency Attempts fired (timestamps) Winner Losers cancelled Cancel reason

slow 200 ms 3 (t=2 ms, 62 ms, 107 ms) id=1 at 217 ms 2 race_lost

fast 30 ms 1 (no hedge fired) id=1 at 31 ms 0 n/a

Scenario	Body latency	Attempts fired (timestamps)	Winner	Losers cancelled	Cancel reason
slow	200 ms	3 (t=2 ms, 62 ms, 107 ms)	id=1 at 217 ms	2	`race_lost`
fast	30 ms	1 (no hedge fired)	id=1 at 31 ms	0	n/a

The fast path doesn't pay for hedging at all. The slow path bounded by max. Every loser tagged with race_lost.

Side-by-side – who actually cancels

// 3 tasks. B fails at 30 ms. A succeeds at 50 ms. C succeeds at 100 ms.

await Promise.all([A, B, C]);     // rejects at 30 ms. A and C keep running for ~16/63 ms.
await Promise.race([A, B, C]);    // rejects at 30 ms. A and C keep running.
await Promise.any([A, B, C]);     // resolves at 50 ms. C keeps running for ~44 ms.

await run.all([A, B, C]);         // rejects at 30 ms. A and C cancelled in 1 ms, defer ran.
await run.race([A, B, C]);        // rejects at 30 ms. A and C cancelled in 0-1 ms.
await run.any([A, B, C]);         // resolves at 50 ms. C cancelled in 0 ms, defer ran.

Same shape. Different contract. Native primitives return a value. WorkIt primitives own the tree underneath the value.

How do other libraries compare

Tool	Cancels on sibling failure	Signal-aware retry	Composable timeout (returns `TaskFn`)	Hedged requests	Bundle
WorkIt	yes	yes	yes	yes built-in	14,175 B / 4,835 B gz for all nine composables
`Promise.all` / `race` / `any`	no	n/a	n/a	n/a	0
`p-limit` + `p-retry` + `p-timeout`	partial/manual wiring	partial/manual wiring	separate abstractions	no	three deps
`RxJS`	yes on `unsubscribe`	partial via operators	yes via operators	no	large
Effection	yes structured (generator ops)	yes	yes	no	medium
Effect-TS	yes structured (fibers + typed `Cause`)	yes	yes	no	large

For simple array processing where nothing else can fail, p-limit is fine. For full-stack apps where you want a broader effect or operation model, Effection and Effect-TS are solid. WorkIt's distinction is narrower: structured-concurrency composition without leaving async/await, with nine composables in one ownership tree and the bundle size shown above.

Receipts

The WorkIt runtime claims above are verified by either npm run verify (the production gate) or npm run bench:articles (the side-by-side suite that produced the representative timing tables in this article).

npm run bench:articles
# full article suite: 19 passed, 0 failed

This article cites benches 01-06 from the full suite. Each bench script is ~100 lines, zero external dependencies, and asserts the WorkIt invariant in-line. Timings are representative captured runs; the assertions guard semantic invariants, not exact milliseconds. The folder has its own package.json so the published package's dependency graph stays empty. Read the README at benchmarks/articles/ for how the promise-helper baselines stay honest.

Production-side gates that back the same composables:

Claim	Evidence
Cancellation safety, all composables	Benches 01-06 plus `tests/evidence/lifecycle/owned-work.mjs` verify parent cancellation, sibling failure, retry cancellation, race loser cleanup, and owned background work.
`run.retry` validation	`RangeError` for `times` <= 0, > 1000, NaN, Infinity, fractional. Identical rejection on numeric and object form.
`run.race` / `run.any` loser cleanup	LIFO `defer` blocks observed in test for every loser; outer promise does not resolve until cleanup completes
Bundle of all nine composables	Included in 14,175 B min / 4,835 B gzip core-group-import. Tree-shaken if unused.

What's next

Nine composables share one engine. Tomorrow we open the engine.

We'll look at what cooperative cancellation can do, what it cannot do, and where the hard boundary starts. Then we put a CPU spin loop that ignores every signal in front of offload({ timeout: "200ms" }) and verify that worker termination prevents a late marker file from appearing. The CI gate runs stat() on it.

AbortController cannot preempt a CPU loop. WorkIt cannot change that language boundary, but a worker thread can be terminated by its host.

Source, Benchmarks, And Evidence

Source: https://github.com/WorkRuntime/workit
Article source: https://github.com/WorkRuntime/workit/blob/main/articles/02-concurrency-retry-timeout.md
Reproduce: npm run bench:articles and npm run test:evidence

DEV Community: AdmilsonCossa

Your Worker Crashed After the Side Effect. Should It Run Again?

The application chooses the durable boundary

The same id with different input is a conflict

Completed records replay; uncertain records do not

What restart evidence proves

The application still owns external correctness

Executable evidence

Sources

I Built HaloLog, a Go Logger That Measured 23.9 ns/op. Then I Tried to Break the Claim

First define the operation

The maintainer feedback that changed the harness

Fairness became a test, not a paragraph

Where the HaloLog hot path saves work

1. Time is cached, with an explicit accuracy trade-off

2. Request context is encoded when it becomes context

3. The fast path is an eligibility decision

4. The string scanner optimizes the common case and retains an oracle

Zero allocation is a regression gate, not an adjective

Two encoders, one tested byte contract

Why there is no corrected ranking in this article

What this article does not prove

Reproduce it, then try to invalidate it

The result I want to preserve

AI Agents Are Making an Old Distributed Systems Bug More Expensive

The dual-write problem existed long before LLMs. Now a duplicate event can trigger fraud investigations, payment holds, refunds, or financial execution.

1. Your Broker Does Not Solve the Dual-Write Problem

2. Broker Guarantees Stop at Their Transaction Boundary

3. AI Agents Don't Create the Problem — They Increase Its Blast Radius

4. CDC Is More Than a Legacy Escape Hatch

5. Transactional Outbox Fixes the Birth of the Event

6. Outbox Does Not Mean Exactly-Once

7. Inbox Deduplication Is Atomic Only With Local Effects

8. External Effects Need a Durable Operation Ledger

9. Ordering Requires Gap Semantics, Not Just Version Numbers

A temporary delivery gap

A permanent allocation gap

A legitimate consumer gap

10. Edge and Multiregion Systems Make Authority Explicit

11. The Reference Architecture

12. Event Envelopes Are Part of the Contract

13. AI Decisions Need Stable Identity and Evidence

14. Observability Must Detect Missing Effects, Not Just Failures

15. Security Must Survive the Event Pipeline

16. Choose the Mechanism From the Invariant

17. Exactly-Once Requires an Explicit Boundary

18. What AI Changes

Conclusion

Record intent once. Give every effect an identity. Reconcile uncertainty.

Technical References

Your Retry Policy Can Break the SLO Before the First Attempt Starts

Runtime policy and planning policy have different jobs

A deadline is part of the task context

A retry count is not a shared admission policy

Check aggregate retry demand

What the upper bound means

Executable evidence

Sources

Your Async Job Finished. Can You Prove What Actually Ran?

Evidence, not deterministic replay

Attempts belong to the task lifecycle

Add metadata at the boundary that knows it

Redaction belongs before storage

What a receipt can establish

Executable evidence

Sources

Your AI Agent Finished. Why Are Its Tools Still Running?

The contract agent.tool(name, input, fn, opts)

A 50-cent agent with a hard tool-call cap

Replayable events -- the typed trace

How does this compare

A complete, runnable example

Receipts

Closing The Series

Source, Benchmarks, And Evidence

Inspect an AI Agent Run Without Paying for Logs You'll Never Read — Telemetry Shouldn't Be Your Second Biggest Bill

Layer 1: Local-first by default (cost: $0)

Layer 2 — Sampling: errors_and_slow is the production default

Layer 3 — Aggregation, not enumeration

Layer 4 — Telemetry budget (the safety net)

The contract `agent.tool(name, input, fn, opts)`

`scope.tree()` — the print statement for agents

`@workit/core/diagnostics` -- the stuck-task detector

`run.bracket` -- acquire, use, release. Always release.

`run.uncancellable` — receipts that always commit

`work().stream()` -- bounded, lazy, cancellable

CSP-style channels -- `@workit/core/channel`

The shield: `run.uncancellable`

Why `run.all` improves on Promise.all’s safety

`run.race` – the race that actually races

`run.any` – first success, rest cancelled

`run.pool` – bounded concurrency that cancels

`run.retry` – composable, cancel-aware backoff

`run.timeout`– composes with retry, race, and pool

`run.fallback` – primary, secondary, type-safe

`run.supervise` – restart policy for long-lived work

`run.hedge` – bounded speculative requests