DEV Community: Jude

Our tool gateway says max_retries=2. It calls your tool exactly once.

Jude — Fri, 18 Sep 2026 16:41:06 +0000

The short answer

How many times does one tool call actually execute in our agent runtime?

Exactly once — while the policy gateway's constructor says max_retries: int = 2.

Those two facts do not contradict each other. Three independent reasons stack up:

That 2 is unreachable in the shipped wiring. Every construction site passes a trace_writer, so every call gets an idempotency key, and the gateway clamps the attempt count to 1 whenever it sees one.
Even if it were reachable, it does not mean "retry twice." The shared helper hands the number to tenacity's stop_after_attempt, which counts attempts.
Even if a retry happened, almost nothing would trigger it. All three tool adapters swallow their exceptions into ToolResponse(success=False), and tenacity only retries on exceptions that are actually raised.

What really keeps a side effect from happening twice is none of the above. It is the
layer below: a durable tool-call ledger with a lease and a column called
outbound_started_at.

For contrast, the other boundary in the same repository — outbound events — defaults
to at-least-once, up to 64 attempts.

This is not a bug list. Most of it is deliberate. The naming and the documentation
are the parts that genuinely do not line up, and they are all in the second-to-last
section.

1. This started as a reply to a comment

On 2026-09-07, under our post about agent frameworks and agent runtimes being
different layers, reidmarlow (Reid Marlow) left a comment. Along with agreeing that
splitting resolved from redacted parameters at the port layer is the cleanest part of
the architecture, he added an argument we had not made ourselves:

Pushing the retry clamp down into the policy gateway also solves a second problem —
when a downstream service cannot honor an idempotency key, a timeout retry fires the
external side effect twice.

Good argument. It is also why this post exists: before replying, I wanted to check
whether our code actually works the way he read it.

It does not. The direction is inverted.

Our gateway does not say "the downstream has no idempotency key, so retry less." It
says "this call carries an idempotency key, so try it exactly once." The path
without a key is the one that keeps the max_retries default.

His conclusion still holds here — side effects do not fire twice — but it holds for a
reason one layer further down. Explaining that requires counting all three layers.

2. Layer one: the ternary that decides everything

Before the gateway reaches an adapter it resolves secrets, claims a ledger row, checks
egress policy, and checks rate limits and quota. Then it reaches this:

response = await run_with_timeout_retry(
    _invoke,
    timeout_seconds=timeout_seconds,
    # Durable Agent calls are at-most-once at this boundary.
    # Not every downstream adapter can honor an idempotency key.
    max_retries=1 if kwargs.get("idempotency_key") else self.max_retries,
    timeout_factory=lambda: TimeoutError(
        f"Tool invocation timed out after {timeout_seconds} seconds",
        {"timeout_seconds": timeout_seconds, "tool_ref": tool_ref},
    ),
    wait_min=1,
    wait_max=5,
)

(server/app/kernel/ports/tools/policy.py:349–361)

The two comment lines state the intent plainly: durable agent calls are at-most-once
at this boundary, because not every downstream adapter honors an idempotency key.

Which means the idempotency_key in that ternary is not a capability signal — it is
a path marker. It marks "this call is the kind that has a ledger behind it." That is
a different thing from what Reid read it as, and the difference is exactly what decides
which way the clamp points.

3. `max_retries` counts attempts, not retries

Five ports — tools, storage, vector, secrets, plugins — share one helper:

async def run_with_timeout_retry(
    operation, *, timeout_seconds, max_retries, timeout_factory,
    wait_multiplier=1, wait_min=1, wait_max=10,
):
    @retry(
        stop=stop_after_attempt(max_retries),
        wait=wait_exponential(multiplier=wait_multiplier, min=wait_min, max=wait_max),
    )
    async def _with_retry():
        return await operation()

    try:
        return await asyncio.wait_for(_with_retry(), timeout=timeout_seconds)
    except TimeoutError:
        raise timeout_factory()

(server/app/kernel/ports/common/policy.py:52–74)

stop_after_attempt counts attempts. stop_after_attempt(2) means two calls total,
i.e. one retry. The parameter is named max_retries, its docstring says "Maximum
retries", and the number it actually carries is one less than its name suggests.

I am not inferring this. Our own unit test pins it:

result = await run_with_timeout_retry(
    operation, timeout_seconds=5, max_retries=2, ...
)
assert result == "ok"
assert attempts == 2

(server/tests/unit/test_port_policy_common.py:32–52)

The direct consequence: max_retries=1 means "do not retry." And the repository
passes a literal 1 in six places:

Location	Operation	Actual meaning
`ports/secrets/policy.py:35`	fetch one secret	no retry
`ports/plugins/policy.py:112`	invoke a plugin tool	no retry
`ports/vector/policy.py:93`	ensure a collection exists	no retry
`ports/storage/policy.py:338`	check whether an object exists	no retry
`ports/storage/policy.py:403`	mint a download URL	no retry
`ports/storage/policy.py:441`	mint an upload URL	no retry

Every one of those is read-only or naturally idempotent — precisely the category that
should be retried. I do not think this was intentional. I think someone read 1 as
"retry once."

For reference: storage defaults to max_retries: int = 3 (three attempts), vector to
2, tools to 2. All three docstrings say "Maximum retries."

4. Which makes the shipped answer 1

Back to the ternary. When does kwargs contain an idempotency_key?

The gateway puts one there itself, as long as it was constructed with a trace_writer:

tool_call_id = str(kwargs.get("tool_call_id") or f"call_{generate_ulid()}")
idempotency_key = str(
    kwargs.get("idempotency_key") or f"tool:{run_id}:{tool_call_id}"
)

(policy.py:261–264, written back into kwargs at :298–303)

So: can a shipped tool port ever be built without a trace_writer?

There is exactly one place that constructs this gateway
(app/wiring/container.py:487), and all four of its call sites pass a TraceWriter
constructed on the spot:

app/wiring/services.py:414–416 (agent application service)
app/api/v1/agent/dependencies.py:47–49 (HTTP dependency)
app/modules/workflow/runtime/engine.py:558–561, :606, :693–696 (workflow engine)

No production path leaves trace_writer as None. The self.max_retries branch
is unreachable in the shipped wiring, and that default of 2 is dead code today.

It is also deliberate. The test says so in its name:

async def test_tool_policy_does_not_retry_a_durable_agent_invocation(request_ctx):
    ...
    policy = ToolPolicyGateway(gateway=SideEffectingTool(), ctx=request_ctx,
                               max_retries=2, enable_egress_check=False)
    with pytest.raises(RetryError):
        await policy.invoke(..., idempotency_key="agent-tool:run_side_effect_once:call_once")
    assert attempts == 1

(server/tests/unit/test_port_policy_enforcement.py:375–399)

It passes max_retries=2 explicitly and asserts attempts == 1. This is a tested
at-most-once boundary, not an oversight.

One more thing worth saying out loud: this gateway's timeout_seconds and
max_retries have no configuration entry at all. The wiring does not pass them and
settings.py contains no setting starting with tool_. Changing either means changing
code. (Rate limit and quota do have an entry — they come from the request context, as
ctx.tool_rate_limit_per_minute and ctx.tool_daily_quota.)

5. And even that `2` would rarely fire

tenacity's @retry only retries on exceptions that get raised. None of our three
tool adapters raise:

# adapters/tools/http.py:97–101
except httpx.HTTPError as e:
    return ToolResponse(result=None, success=False, error=str(e))

# adapters/tools/function.py:41–42
except Exception as exc:
    return ToolResponse(result=None, success=False, error=str(exc))

# adapters/tools/mcp.py:323–325
except Exception as exc:
    logger.warning("MCP tool invocation failed for %s: %s", tool_ref, type(exc).__name__)
    return ToolResponse(result=None, success=False, error=str(exc))

httpx.HTTPError is the base class for that library's errors: connection failures,
read timeouts, and everything raise_for_status() raises for 4xx and 5xx.
All of it becomes a return value rather than an exception.

So an HTTP 503, a connection timeout, or an MCP server falling over — the textbook
"worth one retry" transients — are never retried on this path, not even on the
branch that allows retries.

What does get retried? Whatever the tool router raises:

raise ValidationError(f"Tool not registered for tenant/workspace: {tool_ref}")   # router.py:364
raise ValidationError(f"HTTP tool '{tool_ref}' missing url (tool_spec.http.url)") # router.py:454
raise ValidationError(f"Function tool '{tool_ref}' missing entrypoint")          # router.py:512
raise ForbiddenError(...)                                                        # router.py:301

Configuration errors and permission errors — the ones no amount of retrying will
fix. With a wait_exponential(min=1, max=5) sleep in between.

This is not inevitable. The LLM port next door has an _is_retryable predicate and a
test that asserts validation errors are not retried
(test_validation_errors_are_not_retried, test_port_policy_enforcement.py:236–250,
asserting await_count == 1). The tool port has no such predicate.

6. The timeout is a budget for the whole sequence

Look at the last line of that helper again:

return await asyncio.wait_for(_with_retry(), timeout=timeout_seconds)

wait_for wraps the entire retry sequence, not a single attempt. So the tool
port's default 30 seconds is the budget for every attempt plus the backoff between
them.

Meaning: if the first attempt burns the full 30 seconds, a second attempt never
happens — the timeout lands first and gets converted into our own TimeoutError by
timeout_factory(). The already-hard-to-reach retry path only has room when the first
attempt fails fast.

The parameter is called timeout_seconds, and the tool gateway's docstring calls it
"Request timeout" (policy.py:75). It is not a request timeout. It is a total
timeout.

For contrast, the LLM port's streaming path wraps each attempt in its own wait_for
(ports/llm/policy.py:932–937). Same repository, two shapes.

7. Failures leave the gateway wearing a different type

tenacity's @retry defaults to reraise=False: once the stop condition is met and the
last attempt still failed, it raises RetryError with the original exception inside.

This is independent of the retry count. Even with stop_after_attempt(1), a failed
first attempt comes out wrapped. So in the shipped wiring — which always takes the
at-most-once branch — every failure leaving the tool gateway is a RetryError, not
a ValidationError, not a ForbiddenError.

There is an unwrap_retry_error helper for exactly this
(ports/common/policy.py:38–49), but it is only used by error_details to write trace
detail. The copy that gets unwrapped is the one for the books; the exception actually
propagating upward is not swapped back.

Two unit tests have already frozen this into assertions:
test_port_policy_enforcement.py:391 and test_tool_secret_injection.py:215, both
with pytest.raises(RetryError).

The impact is concrete: any caller upstream writing except ValidationError: to turn
"tool not registered" into a 400 will not catch it on this path.

8. Layer two: `tool_call_id` is the identity, `idempotency_key` is an assertion

Before touching an adapter, the gateway claims a row in run_step_tool_calls.

That table carries three unique constraints
(kernel/runtime/db/models/runs.py:185–203); two of them matter here:

Constraint	Columns
`uq_run_step_tool_calls_scope_run_call`	tenant + workspace + run_id + tool_call_id
`uq_run_step_tool_calls_scope_idempotency`	tenant + workspace + idempotency_key

Only the first is used for lookup. _call_statement finds an existing row by
tenant + workspace + run_id + tool_call_id (runtime/runs/tool_calls.py:172–180).

The idempotency_key's role here is an assertion, not a key: once a row is found,
a mismatching tool_ref or idempotency_key raises
ConflictError("Tool call identity was reused with different input")
(tool_calls.py:215–219). The arguments have to match too — compared through
canonical_request_hash.

The default key looks like tool:{run_id}:{tool_call_id}, so on the default path the
two constraints never fight.

One more detail worth noting: what lands in the ledger is the redacted payload
(policy.py:283 passes redacted_parameters); a secret reference keeps only its
secret_id, never the plaintext.

9. How at-most-once actually reaches the disk

The real guarantee lives in three columns: status, lease_expires_at, and
outbound_started_at.

The order is:

claim() writes a row with status="claimed", attempt_count=1, and a lease owned by this worker for max(60, ceil(timeout) + 10) seconds (policy.py:274).
mark_running() stamps outbound_started_at before crossing the boundary (tool_calls.py:473–495). That single write is the pivot of the whole mechanism.
The adapter is called. Each attempt starts by renewing the lease (policy.py:328–329).
complete() writes the terminal state: status = "succeeded" if response.success else "failed" (tool_calls.py:543).

If the process dies during step 3, the next worker to claim finds a row with an expired
lease, and splits on one column:

outbound_started_at is null — the request never left. Safe. Re-claim it and bump attempt_count (tool_calls.py:322–332).
outbound_started_at is set — we went out and do not know what happened. Do not re-run. Mark the row in_doubt, mark the step paused, raise ConflictError("Tool call outcome is in doubt") (tool_calls.py:297–320).

This is where "the side effect does not fire twice" actually comes from. It does
not depend on downstream idempotency support, and it does not depend on how many times
the gateway retries. It depends on the fact that going out the door is itself written
down.

Reid's conclusion holds here because of this layer, not because of that ternary.

10. Replay and re-run are one boolean apart

When a row that already reached a terminal state is claimed again, the default is
replay:

if existing.status in {"succeeded", "failed"}:
    ...
    return ToolExecutionClaim(
        record=existing, run_step=step, replayed=True,
        cached_response=ToolResponse(
            result=payload.get("result"),
            success=existing.status == "succeeded",
            error=existing.error_message,
            metadata={..., "idempotent_replay": True},
        ),
    )

(tool_calls.py:270–289)

Note that {"succeeded", "failed"} contains both: a failure is replayed verbatim.
The gateway sees replayed=True and returns the cached response without ever reaching
an adapter (policy.py:291–296).

Making it actually re-run requires passing retry_failed=True, which clears the
result, bumps attempt_count, marks the step retrying, and starts over
(tool_calls.py:249–269).

And that boolean differs between the two execution paths:

Path	Passes `retry_failed`?	Consequence
Workflow node	`True` (`executors/tool.py:374`, `executors/http.py:55`, `executors/node.py:100`)	A node retry really re-sends the tool call
Agent loop	not passed (`modules/agent/runtime/executor.py:29–42`)	A failed call is replayed as a failure, never re-sent

Both sides have decent comments (the workflow one says the identity is attempt-stable,
so a node retry lands on the previous attempt's record: replay it when it succeeded,
re-execute it when it failed). The divergence itself is written down nowhere.

11. Layer three: workflow nodes, and one name with three meanings

The workflow engine carries a retry loop of its own:

retry_policy = node.get("retry_policy") or policy.get("default_retry_policy") or {}
max_retries = int(retry_policy.get("max_retries", 0) or 0)
...
attempts += 1
if attempts > max_retries:
    final_error_recorded = True
    raise

(modules/workflow/runtime/executor.py:427–428 and :483–485)

Here max_retries really does count retries: the default 0 means no retry, and 2
means at most three executions. Each retry also creates a new step, suffixed
_retry{attempt} (executor.py:380).

So the same name means three different things:

Where	How it is written	Total executions at `max_retries=2`
Tool port (via the shared helper)	`stop_after_attempt(max_retries)`	2
LLM port	`for attempt in range(route.max_retries + 1)`	3
Workflow node	`if attempts > max_retries: raise`	3

The LLM line is at ports/llm/policy.py:921, and its count is pinned by a test as
well: test_max_retries_counts_additional_attempts asserts
port.chat.await_count == 3 for max_retries=2
(test_port_policy_enforcement.py:219–232).

In other words: the tool port is the only one of the three that is off by one, and
it happens to be the only one that produces external side effects. On the bright side,
that is the safe direction to be wrong in.

12. How many times the downstream saw it, the ledger counted it, and the bill counted it

These three numbers are not necessarily equal.

The idempotency key does reach the downstream — on four methods:

idempotency_key = kwargs.get("idempotency_key")
if idempotency_key and method.upper() in {"POST", "PUT", "PATCH", "DELETE"}:
    headers.setdefault("Idempotency-Key", str(idempotency_key))

(adapters/tools/http.py:34–36)

GET does not carry it. Reasonable — but "we pass the idempotency key downstream" needs
an "except on GET" attached to it.

The MCP adapter retries once on its own. When an MCP server answers 401 or 403 with
a WWW-Authenticate challenge, the adapter re-derives a token from the scopes in that
challenge and calls again (adapters/tools/mcp.py:77–94). A test asserts it:
assert len(factory.calls) == 2 (tests/unit/test_mcp_official_sdk_adapter.py:272–312).

That retry is invisible upward: the gateway counts one invocation, the ledger's
attempt_count stays at 1, and the MCP server saw two sessions.

Rate limit and quota are checked once, before any retry. Both check_rate_limit
calls sit at policy.py:311–325; the retry happens at :349. So on that (currently
unreachable) non-durable branch, two downstream requests would consume one unit of
quota.

Billing counts once too: record_cost(..., billed_quantity=1, ..., request_count=1)
(policy.py:419–430), one entry per invoke.

The platform has already thought carefully about this exact class of problem in one
other place, and that docstring deserves quoting in full:

llm_image_max_retries: int = 0
"""
Retry ceiling for image generation, capping the per-route retry budget.

Image generation is billed per generated image and is not idempotent: a
platform-side timeout does not cancel the provider-side generation, so a
retry can bill the workspace again while the platform records a single
usage fact. Zero keeps recorded cost aligned with provider charges.
"""

(settings.py:240–247)

Every word of that transfers to tool calls. Tool calls have no equivalent note and no
equivalent knob.

13. The other boundary defaults the opposite way

Outbound events in the same repository chose the mirror image:

max_dispatch_attempts: int = 64
...
next_attempt = int(row_fresh.attempt_count or 0) + 1
if next_attempt >= self.max_dispatch_attempts:
    await self.repo.mark_failed(row_id, msg, consumer_name=consumer_name)
else:
    await self.repo.mark_retry(row_id, msg)

(kernel/events/dispatcher.py:50 and :76–88)

At-least-once, up to 64 attempts, with backoff handled by the outbox repository.
Deduplication is the consumer's job: a (consumer_name, event_id) checkpoint table
where the unique constraint absorbs the duplicate
(kernel/events/checkpoint.py; an insert conflict returns False, meaning "already
handled").

The contrast is defensible. Events are ours, replayable, and consumers can be made
idempotent; tool calls belong to someone else and cannot be taken back. Two
boundaries defaulting opposite ways is evidence that somebody thought about it.

One thing does not line up, though. The event side is documented — "provides
at-least-once delivery; consumers remain responsible for idempotent side effects"
(README.md:77). The tool side's at-most-once contract appears nowhere in the
docs. It lives in two comment lines at policy.py:352–353 and in the name of one
test.

14. So was the comment right?

Three sentences:

The direction is inverted. We do not clamp because the downstream lacks an idempotency key; we clamp because this call has one, which is our marker for "this one has a ledger."
The conclusion holds. Side effects do not fire twice.
It holds one layer down. It comes from the table with outbound_started_at, and from that table choosing in_doubt over a re-run when the outcome is unknown.

If you are building something similar, two takeaways from this exercise:

Decouple "how many attempts" from "is there an idempotency key." The key is a signal about the downstream's capability; the retry budget is your own risk appetite. Binding them together with a ternary means nine readers out of ten read it backwards — and one reader did, in a direction more intuitive than what our code actually does.
Real duplicate protection lives in a table, not in a counter. A counter only decides how many times you try. Only a row can tell you whether the last attempt made it out the door.

15. Twelve things that do not line up

House rule: whatever I found, I list. Each one gets its impact and a workaround.

① max_retries counts attempts while its name and docstring say retries. Five
ports share the helper; six call sites pass a literal 1 (see the table in section 3),
which means "no retry", and all six are read-only or idempotent operations. Impact:
readers overestimate the system's retry behavior. Workaround: read it as
max_attempts. I plan to open an issue.

② The tool boundary's at-most-once contract only exists in two comment lines and a
test name. The README documents at-least-once for events and says nothing about
tools. Impact: a self-hoster cannot tell from the docs whether their tool call can be
re-sent. I plan to open an issue.

③ Three adapters swallow exceptions into success=False, so the gateway's retry is
dead for all of them — while configuration errors from the router do get retried.
Impact: the transients most worth retrying are not retried, and the config errors least
worth retrying burn 1–5 seconds of backoff. Workaround: everything takes the
at-most-once branch today, so the practical damage is limited to the retry semantics
being nominal.

④ Failures leave the gateway as RetryError with the original type buried inside.
unwrap_retry_error is only applied to trace details, not to the propagating
exception; two unit tests have frozen this as an assertion. Impact: upstream code that
maps exception types to HTTP status codes cannot catch it. I plan to open an issue.

⑤ timeout_seconds wraps the whole retry sequence while its docstring calls it
"Request timeout." Impact: when the first attempt eats the budget, the second never
happens, and the parameter name does not say so. The LLM streaming path is per-attempt
— two shapes in one repository.

⑥ The tool gateway's timeout and attempt count have no configuration entry at all.
The wiring does not pass them and no setting starts with tool_. Impact: self-hosters
have to edit code. I plan to open an issue.

⑦ 🔍 Inference, not observation: the HTTP adapter passes timeout=timeout_s
explicitly to httpx, and that timeout_s is only populated when the tool spec declares
policy.timeout_ms or http.timeout_ms (adapters/tools/router.py:489–502);
otherwise it is None. Per httpx's API, omitting the argument uses the
USE_CLIENT_DEFAULT sentinel, while passing None explicitly means Timeout(None),
i.e. "No timeouts" (httpx/_client.py:352, :370–374; httpx/_config.py:72–84,
read from the library source in our own virtualenv). So a registered tool with no
declared timeout bypasses the client's 30-second timeout and is bounded only by the
gateway's 30-second wait_for. I did not construct this call to prove it, hence
"inference".

⑧ Idempotency-Key is only set on POST/PUT/PATCH/DELETE. Probably correct, but the
sentence "we forward the idempotency key" needs that qualifier.

⑨ 🔍 Inference, not observation: the ledger has two relevant unique constraints, and
claim() only re-queries by run_id + tool_call_id after catching an IntegrityError
(tool_calls.py:384–389). If two different runs supplied the same explicit
idempotency key, the collision would hit the second constraint, the re-query would come
back empty, and a raw IntegrityError would propagate instead of a ConflictError.
The default key embeds the run id, so it cannot collide — which is why I could not
construct this one either.

⑩ The agent path does not pass retry_failed; the workflow path passes True. The
same failed tool call means different things on the two paths: one replays the failure
forever, the other really re-sends. This is plausibly deliberate (an agent loop can
decide to pick a different tool), but it is documented nowhere. I plan to open an
issue.

⑪ The MCP adapter's 401 retry is invisible upward. The downstream sees two
requests while attempt_count stays at 1. Impact: reconciling call volume against the
ledger will not add up. Workaround: that retry only happens on an auth challenge, and
business side effects usually happen after auth, so the exposure is small.

⑫ Quota is decremented once and cost is recorded once, while (on the currently
unreachable branch) the downstream could have been hit twice. The platform already
recognized and handled the same shape for image generation
(llm_image_max_retries: int = 0); the tool path has no equivalent note. Impact: zero
today, because that branch is unreachable — but it returns the moment someone
constructs the gateway without a trace_writer.

Full disclosure

Nothing was executed for this post. Not one command. Every claim comes from reading source and tests in the soit/ repository at commit abf3dc3, plus the httpx library source in our own virtualenv. I did not stand up an environment to watch a crashed row turn into in_doubt, even though the code path is clear. Items ⑦ and ⑨ above are explicitly marked as inference.
I only read the community edition. If the Enterprise or Cloud editions wire retries differently, that is out of scope here.
"Unreachable in the shipped wiring" is a claim about the current wiring, not about every possible caller. The evidence is that the gateway is constructed in one place and all four call sites pass a trace_writer. Construct one yourself without it and that branch comes straight back to life.
None of the twelve items is a security incident. They are naming, documentation, and layering mismatches — not "somebody got in."
This post started as someone else's comment. The argument was Reid's, not mine. All I did was go back and check, and find that the direction was reversed.
Disclosure: I maintain SOIT.

One-sentence takeaway

"How many attempts" and "is there an idempotency key" are two different questions,
and binding them together in a ternary guarantees that readers get it backwards; what
actually keeps a side effect from happening twice is never the counter, it is a row
that remembers whether the request made it out the door.
Our answer happens to be 1 — but it is a 1 stacked out of three independent reasons,
and only one of them was on purpose.

Come and find the holes

The repository is github.com/soit-ai/soit. Every
claim here is checkable:

The ternary is at server/app/kernel/ports/tools/policy.py:349–361; read it together with the two comment lines above it.
For the counting semantics, run our own test: server/tests/unit/test_port_policy_common.py asserts attempts == 2 for max_retries=2.
The lease state machine from section 9 is in server/app/kernel/runtime/runs/tool_calls.py — grep for outbound_started_at and read downward; the in_doubt branch is the pivot of the whole piece.

If I got something wrong — especially if one of the twelve items in section 15 is me
misreading an implementation — please open an issue and say so. I would much rather
hear where it does not line up than be told the design looks clean.

We changed one timeout from 30 to 480 and back to 30. In between, it stopped meaning the same thing

Jude — Wed, 16 Sep 2026 16:14:22 +0000

The short version

In a self-hosted deployment, "how long does a login last" is never one number. In ours it is
two knobs you can turn, six hard-coded deadlines you cannot, and one revocation path:

Thing	Lifetime	Configurable	What happens at the end
Access token	30 minutes	✅ `ACCESS_TOKEN_EXPIRE_MINUTES`	The browser silently swaps it; nobody notices
Session (refresh token)	14 days	✅ `REFRESH_TOKEN_EXPIRE_DAYS`	Password required again
Second-factor challenge	5 minutes	❌ hard-coded	Start the sign-in over
Password reset link	30 minutes	❌ hard-coded	Request another mail
Email verification link	24 hours	❌ hard-coded	Request another mail
Workspace invitation	14 days	❌ hard-coded	The inviter resends
API key	1–365 days, mandatory at creation	per key	The call gets a 401
Ending one device	—	—	That device's next request is a 401

Only the second of the two knobs is the session length. The first one is not — even though
our own documentation still explains it as if it were. That is item ③ in the last section.

1. The commit that set it to eight hours

2026-08-06, commit 69be399, titled fix(deploy): default the access token lifetime to one workday. The message says exactly why:

Access tokens expired after a fixed 30 minutes and there is no refresh
flow, so self-hosted users were silently logged out mid-session.

With no refresh flow, the access token is the session. When it expires the person is out,
and not politely — there is no "your session ended" dialog, just a 401 on the next click. So
the default became 480 minutes, one working day, threaded through both compose profiles.

That was the right call at the time, and the message also names the cost: every sign-in
became an eight-hour grant that nobody could take back. A JWT that has been signed is not
something the server knows about. There is no row to flip. Your options are to wait it out or
to rotate the signing key and drop everybody at once.

For a self-hosted internal platform that cost is not academic. Someone leaves the company. A
laptop goes missing. An engineer signs in on a customer's machine to demo something. In each
of those the thing you want is "end that one login", and the system's only answer was: eight
hours.

2. The commit that set it back to thirty minutes

2026-08-30, commit 09dbf7e, titled feat(identity): make a sign-out mean something, and stop logging people out. 38 files, +1439/-56. Four things, reordered by how much they matter:

A sign-in opens a session row, and the access token carries a sid claim naming it.
Every authenticated request checks that the session is still live, so revocation lands on the next request.
Refresh tokens rotate on every use, and only the hash is stored.
The console renews silently, one renewal serving every request that failed together.

The moment 1 and 2 landed, ACCESS_TOKEN_EXPIRE_MINUTES changed meaning. It no longer decides
how often a person gets kicked out — REFRESH_TOKEN_EXPIRE_DAYS plus the refresh flow decides
that. It now decides exactly one thing: how long a revoked login keeps limping. So it went
back to 30, and the commit message says so:

Access tokens drop to 30 minutes because they no longer bound the session --
they bound how long a revoked session keeps working.

One setting, two commits, two meanings. That is the reason for this post. If you see
ACCESS_TOKEN_EXPIRE_MINUTES in a deployment, work out which of the two systems you are
looking at before you decide whether to raise or lower it.

(That commit message now needs a small asterisk of its own — see item ③. For a token that
carries a sid, revocation is immediate; the sentence describes the tokens that do not.)

3. Why checking the session on every request is nearly free

"JWTs cannot be revoked" is shorthand for "signature-only JWTs cannot be revoked". The
moment your request path contains any server-side lookup, revocation is back on the table. The
lookup is the price.

We did not add a lookup. We put the check on one that was already happening
(server/app/modules/identity/infra/workspace_access.py:30–37):

When the caller's token names a session, that session must still be
live. It is checked here rather than in a separate lookup because this
is already the one database read every authenticated request makes, and
checking it at refresh time alone would leave a signed-out token working
until it expired.

The precondition — true for more systems than people assume — is that authentication here was
never "decode the JWT and wave it through". Every authenticated request has to answer is this
person still a member of this tenant and this workspace, and what are their quotas. That
means reading the database. Given that the connection is open and the transaction has started,
one more primary-key lookup costs approximately nothing.

Counted in the order DatabaseWorkspaceAccessResolver.resolve performs them
(workspace_access.py:23–97):

#	Read	For
1	`user_sessions` by primary key	Is the session still live (only when the token has a `sid`)
2	Tenant membership	Still in this tenant
3	Workspace membership	Still in this workspace
4	Tenant	Tenant-level quotas
5	Workspace	Workspace-level quotas, overriding the tenant's
6	MFA enrolment (conditional)	Only when the workspace requires a second factor

The session check is the first of those, and it is an equality read on a primary key. What
it buys: end a device in the console and that device's next request is a 401. No 30-minute
wait, no denylist to keep warm.

One detail from row 6 worth stealing: when a workspace requires a second factor and the caller
has not enrolled, the resolver does not raise "you are not a member". It raises a 403 carrying
reason: mfa_required (workspace_access.py:63–72). The two look alike from the server and
are completely different from the browser — the first can only render "no access", the second
can send the person to the enrolment page.

4. Why shipping this did not sign everybody out

The scariest part of changing authentication is the instant it deploys. The handling here is
simple and worth copying: sid is an optional claim, and a token without one is honoured
until it expires.

It is written into the docstring of create_access_token
(server/app/kernel/identity/auth.py:53–58):

session_id: Session this token belongs to. Naming it lets a
    sign-out end the access it granted; a token without one is
    accepted until it expires, so upgrading does not sign
    everybody out.

And _require_live_session is only reached when session_id is present
(workspace_access.py:45–46, :100–107). The migration window closes by itself: an old token
lives at most ACCESS_TOKEN_EXPIRE_MINUTES longer, then the client trades its refresh token
for a new one — and the new one carries a sid. Nobody is kicked, and nobody is stuck under
the old rules forever.

There is a precondition: at the moment of the upgrade, clients need a refresh token to trade.
Before 08-30 there was no refresh flow at all, so in practice everyone signed in again within
eight hours and picked up a session naturally. Acceptable.

5. Fourteen days is a hard cap, not a sliding window

This is the one people assume wrong. A session's expires_at is computed once, at issue time
(service.py:405–429):

expires_at=utc_now() + timedelta(days=self._refresh_token_days()),

I went looking for every other write to that column on the session path. There is none.
_issue_session is the only writer; everything else reads it. Refreshing does not extend
it. refresh_session rotates the token and updates last_seen_at, workspace_id,
user_agent and ip_address (service.py:490–498) — and deliberately not expires_at.

So REFRESH_TOKEN_EXPIRE_DAYS=14 means: fourteen days from the last time you typed your
password, no matter how active you were in between. It does not mean "expires after fourteen
idle days".

That is a trade-off, and I think it was chosen correctly. A sliding window means a machine
that is never turned off never has to prove again that it is still you. A hard cap gives the
whole system an upper bound: no login lives longer than fourteen days. The cost is that
active users get interrupted too, at a moment nobody can predict — it depends on what time of
day they signed in two weeks ago.

Want a tighter bound? Lower it. Want "everybody signs in again on Monday morning"? There is no
way to express that today (item ⑥).

6. Refresh tokens: hash only, rotated every time

The session row does not store the refresh token. It stores its SHA-256
(service.py:400–402, :424), the same rule API keys follow. The plaintext is handed to the
client once and the server can never produce it again — the model comment states the purpose
outright: a stolen database cannot be replayed as a sign-in (models.py:365–367).

Every refresh mints a new one (service.py:490–491). The old hash is overwritten, so the old
token stops matching immediately. Which gives a genuinely useful property: a refresh token
is good for exactly one use.

What happens on the second use is where the docstring, the deployment guide, the commit
message and even the test name all say one thing and the code does another. That is item ④,
and it is the most valuable thing I found this time.

7. How the browser hides the expiry

The other half lives in web/app/utils/request.ts. Three pieces add up to silent renewal.

(a) Retry once after a 401 (:206–230):

async function retryWithRefreshedToken(error: any): Promise<any | null> {
  const config = error?.config as (...)
  if (!config || error?.response?.status !== 401) return null
  if (config.skipAuthRefresh || config._retriedAfterRefresh) return null
  if (!storedRefreshToken()) return null
  const token = await refreshAccessToken()
  if (!token) return null
  config._retriedAfterRefresh = true
  config.headers = { ...(config.headers || {}), Authorization: `Bearer ${token}` }
  return request.request(config)
}

Two loop guards: the refresh call carries skipAuthRefresh, and a replayed request carries
_retriedAfterRefresh.

(b) One refresh at a time (:145–183). The comment explains why it is mandatory rather
than nice:

A page loads a dozen requests at once, and an expired token fails all of
them together. Without this, each failure would spend the same refresh
token, and every attempt after the first would look like a replay

A page fires a dozen requests; an expired token fails all of them at once. If each failure
refreshed on its own, they would spend the same refresh token a dozen times, and rotation
guarantees only the first can win. A module-level refreshInFlight promise collapses them
into one.

(c) Give up and go to the sign-in page (:186–203). AUTO_REDIRECT_UNAUTHORIZED is a
hard-coded true (:20); after a 401 it waits a second, clears local storage, and navigates
to /sign-in with the current route in a redirect parameter.

Put that together with the server-side check from section 3 and you can trace exactly what
happens on a device you just ended from the console: next request 401 (session revoked) →
interceptor tries to refresh → the refresh is refused too (_session_is_live fails) → null →
the 401 surfaces → local storage cleared, sign-in page. One request cycle. No token has to
expire for any of it.

While we are here, the honest version of the storage trade-off: both tokens live in
localStorage (auth-session.ts:1–22). That survives a page reload and is simple to
implement; it also means one XSS is worth a session of up to fourteen days. HttpOnly
cookies would close that and open CSRF and cross-site deployment questions instead. We picked
the first, and that is a choice that deserves to be written down rather than defaulted into.

8. One path does not get any of this: SSE

This one matters specifically for an agent runtime, so it gets its own section.

Running an agent or following a workflow in the console is a Server-Sent Events stream
(@microsoft/fetch-event-source), and that path does not go through the axios interceptor
above. In request.ts:430–470, buildAuthHeaders() is called once while opening the
connection, and that header is then fixed for the life of the stream. onerror records the
error and rethrows (:479–487) — and in fetch-event-source, throwing from onerror means
stop retrying.

Three consequences, pointing in different directions:

An open stream does not die when the token expires. Authentication happens when the request is accepted; the connection is not re-checked afterwards. A 40-minute agent run does not drop halfway because of a 30-minute token. That is good.
But ending that device does not cut the stream either. Revocation only affects new requests; a stream already pushing events keeps pushing until it finishes on its own.
And if the token is already expired when the stream opens, the stream just fails — no refresh, no retry. A tab that sat open for half an hour hits exactly this on the first click of "Run". An ordinary request would have been rescued silently. This one is not.

The third is a real gap. The workaround is blunt and works: fire one cheap ordinary request to
renew the token before opening the stream.

9. If you are self-hosting, which number should you touch

A table by threat model beats a recommended value:

What worries you	Which knob	Set it to	Cost
A lost device must be cut off now	none	End that device in the console	None; its next request fails
Nobody will click that button	`ACCESS_TOKEN_EXPIRE_MINUTES`	5–15 minutes	More refreshes, each one a write
"Re-authenticate every N days" compliance	`REFRESH_TOKEN_EXPIRE_DAYS`	N	Interrupts at an unpredictable moment
Shared or kiosk machines	`REFRESH_TOKEN_EXPIRE_DAYS`	1	Password every day
Busy server, fewer database reads	raise `ACCESS_TOKEN_EXPIRE_MINUTES`	—	⚠ Does not work: the session check runs per request, not per refresh

That last row is the one people get wrong. Lengthening the access token does not reduce
database reads, because every request reads membership anyway and the session check rides on
that read. The only thing lengthening it does is keep pre-upgrade tokens without a sid alive
longer, which is the opposite of what you want.

Two operational notes:

SECRET_KEY must be changed, and production will stop you. Shorter than 32 characters, or equal to either known placeholder, and validate_runtime_requirements() fails at startup (settings.py:550–555). Fail-closed, not a warning in a log.
Mail is off by default (SYSTEM_MAIL_ENABLED=false), and that interacts directly with session length. Fourteen days in, everyone must type their password again — and the only route back from a forgotten password is a reset mail. With mail off, bootstrap_admin.py prints User already exists. Skipping bootstrap. and exits (:30–33). There is no second path inside the product. Either turn mail on, or accept that "forgot password" means "write SQL".

10. The six deadlines you never see in config

All of them are constants (service.py:91–103), nothing to do with your deployment:

MFA_CHALLENGE_PURPOSE = "mfa_challenge"
MFA_CHALLENGE_MINUTES = 5
PASSWORD_RESET_MINUTES = 30
EMAIL_VERIFICATION_MINUTES = 60 * 24
INVITATION_DAYS = 14

The second-factor challenge is the interesting one. The state between "password accepted" and
"second factor proved" is carried by a stateless JWT with a purpose: mfa_challenge claim
(service.py:1131–1143). The point is that this ticket cannot be used as an access token —
the authentication entry point rejects any token carrying a purpose
(context_resolver.py:88–92):

# A token minted for one step of sign-in authorizes nothing. Without
# this, presenting the challenge token as a bearer would make the
# second factor optional for anyone who noticed.
if payload.get("purpose"):
    raise UnauthorizedError("Token cannot be used to authorize a request")

"A credential for a step of sign-in is not a credential for being signed in" is the box most
hand-rolled MFA implementations forget to tick. The price is that this ticket is not
revocable — it is stateless, with nowhere to mark it spent, and it is good for the full five
minutes. That is item ⑨.

API keys are a separate path entirely: no session, their own row lookup, their own expiry
check, their own scopes (context_resolver.py:145–175). And the lifetime is mandatory at
creation, between 1 and 365 days (schemas.py:408–412). There is no "never expires" option.
Mildly annoying for automation; "long-lived credentials get reissued" is a default I will
defend.

11. Rotating `SECRET_KEY` no longer signs everybody out

An operational note that flipped when the session table landed.

Before: every access token was signed with SECRET_KEY. Rotate it and every token fails
verification — that was the global sign-out button.

Now: rotate it and every access token still fails at once. But the client takes the 401
and refreshes, and refresh_session never looks at the access token — it takes the
refresh token and queries the database (service.py:460–468). Refresh tokens are random
strings stored as hashes; they have nothing to do with the signing key. The client gets back a
token signed with the new key and the user notices nothing at all.

The good news is that key rotation is no longer a mass interruption. The bad news is that you
can no longer use it as an emergency eject button. And a real "sign everybody out" does not
exist: revoke_all_sessions only touches the caller's own sessions (service.py:536–551),
and the only routes are the three under /me/sessions (router.py:257–277). To cut off
someone who left the company today you close their account (execute_account_deletion ends
all their sessions, service.py:640–660), remove them from the tenant (membership is read on
every request, so the next one is a 403), or go into the database.

12. What you can see: the security pane

With sessions in place, the console lists signed-in devices: user agent, IP, last activity,
each with an End button — except the current one, which deliberately has no button (the commit
message: signing yourself out of the page you are on is a trap). "Sign out everywhere"
keeps the current device by default (settings.tsx:309–317 calls revokeAllSessions(true)).

The "last active" column in the member list comes from the same rows:
last_seen_for_users takes the maximum last_seen_at across a user's sessions
(repository.py:468–483). Mind the resolution — last_seen_at is written only on refresh,
so it is accurate to roughly one access token lifetime (30 minutes), not to the last click.

What you cannot see, so nobody misreads the pane: ended sessions are not listed
(list_by_user defaults to include_ended=False, and include_ended=True has no call site
anywhere in the repository). This is a list of currently signed-in devices, not a sign-in
history. And signing in is not audited at all — the audit table has
identity.session.revoked and no event type for a login.

13. Ten things that do not line up

By convention, this section is about our own problems.

① The "Session timeout" dropdown in the console is decorative.
web/app/console/routes/settings.tsx:1105–1116 renders a 12 hours / 24 hours / 7 days select
with defaultValue="12 hours" and no onChange, no save, no read from the backend. The
comment beside it is honest:

BACKEND-PENDING: session lifetime is an instance setting
(ACCESS_TOKEN_EXPIRE_MINUTES), not a workspace one. Not built
rather than withheld: it needs a per-workspace override the
token issuer would have to read.

Worse than unwired: none of the three options matches either real knob. The real values
are 30 minutes and 14 days; the dropdown offers 12 hours, 24 hours, 7 days. An administrator
reading that screen will reasonably conclude their sessions last twelve hours. Impact:
misinformation. Workaround: edit .env. I intend to open an issue for this; it was not
filed when this was written, so there is no link in the text.

② "Log out" in the account menu only clears the browser. Both entry points
(web/app/components/common/nav-user.tsx:31–36,
web/app/console/shell/icon-rail.tsx:81–86) do the same four things: clear the query cache,
clear the user store, clear local storage, navigate to /sign-in. Neither calls
revokeSession or /me/sessions/revoke-all.

So: you click log out, and the session row stays active. It keeps appearing in the security
pane's device list. Its refresh token stays valid for the rest of the fourteen days — that
string was merely deleted from this browser's local storage. Impact: "sign out" means two
different things in two places — the End button in the security pane really revokes, the Log
out item does not. Workaround: use the security pane. Also intend to open an issue; the
fix looks like a handful of lines (fire a revoke first, proceed either way).

③ Four places in docs and comments still describe the old revocation semantics. All four
say the same thing: after revocation, an already-issued access token keeps working until it
expires.

server/app/settings/settings.py:73–78
.env.example:15–18
server/.env.example:33–36
docs/deployment/production-profile.md:46–50 — verbatim: an access token already issued keeps working until it expires, even after the session behind it was ended

And the code in section 3 exists precisely to make that untrue. A token carrying a sid
gets a 401 on the next request after its session is revoked. Those four notes were written
before or alongside the session table and never updated. Impact: a reader shortens the access
token lifetime to reduce a "revocation delay" that is already zero. Workaround: trust the
code. Also intend to open an issue.

④ "Presenting a spent refresh token ends the session" is documented and not implemented —
and the test is named after the behaviour it does not have. This is the big one, so here it
is in full.

Three places promise it:

service.py:454–457: presenting a rotated-out token ends the session
docs/deployment/production-profile.md:52–55: presenting a spent one ends the session, so a stolen token is usable at most until the real client next renews
commit 09dbf7e: A spent one is a replay, and the answer to a replay is to end the session

The code (service.py:462–468):

session = self.session_repo.get_by_refresh_hash(
    self._hash_refresh_token(refresh_token)
)
if session is None:
    raise UnauthorizedError("Invalid refresh token")

After rotation the old token's hash has been overwritten. No row in the database knows it
any more. Presenting it produces "no match → 401", the session is untouched, and nothing
anywhere records that a replay happened.

The test file is blunter about it (server/tests/unit/test_user_sessions.py:65–77): the
function is called test_replaying_a_rotated_token_ends_the_session, the assertion is
assert len(service.session_repo.list_by_user(user.id)) == 1, and the comment reads the live
session is untouched by a failed replay of the old one. The name and the body contradict
each other, and the body is the accurate one.

Impact: when a refresh token is stolen, whoever uses it first wins. If the attacker
refreshes first, they hold the rotated token and the session; the real client's next refresh
fails with a 401 and they end up signing in again, believing they were simply "logged out".
Nobody is told a replay occurred. Workaround: real replay detection needs a stored
previous-generation hash, or a monotonic counter per session. Also intend to open an issue —
and I consider it more serious than ① or ②, because a security property written into
deployment documentation is something other people will cite as fact.

⑤ The expired status is never written, and session rows are never cleaned up. The model
documents three values for status — active, revoked, expired (models.py:385–386) — and
nothing in the repository ever assigns "expired". Expiry is evaluated at read time, in two
places (_session_is_live, _require_live_session).

Alongside it sits ix_user_sessions_expiry, a composite index on (status, expires_at)
(models.py:373): an index built for a sweeper that does not exist. No worker or script
deletes or archives expired sessions. Impact: user_sessions grows monotonically with
sign-ins; an instance a year old carries a pile of rows that died a fortnight in. Workaround:
a scheduled DELETE of long-expired rows is safe — nothing reads them.

⑥ There is no idle timeout, and no way to say "everyone signs in on Monday".
last_seen_at is recorded (on refresh) but participates in no decision — it drives
ordering and the "last active" column, nothing else. So the common compliance line "a login
unused for 30 days must expire" can only be approximated by shortening
REFRESH_TOKEN_EXPIRE_DAYS, which is an absolute cap rather than an idle window; the two hit
active users very differently. Impact: deployments with that requirement cannot express it.
Workaround: shorten the cap, or revoke stale rows on a schedule.

⑦ Changing your password does not end other sessions; resetting it does.

complete_password_reset (service.py:808–821) walks every session for that user and ends them, with the reasoning in the docstring: a reset is what you do when you think the account is compromised.
change_password (service.py:1517–1528) verifies the old password, writes the new hash, and returns. It touches no sessions at all.

Impact: someone who changes their password because they think somebody saw it leaves that
somebody signed in — while almost certainly believing the change kicked them out. Workaround:
click "sign out of other devices" right after.

⑧ Signing in is not audited; signing out is. The audit table carries
identity.session.revoked (service.py:553–575), identity.mfa.changed,
identity.account.closed and others. There is no event type for a login or a new session.
For a platform whose pitch is governance that asymmetry stands out: you can find out who ended
whose session, but not who signed in, when, or from where. The session row holds exactly that
information (created_at, ip_address, user_agent) — it just never reaches the audit
stream. Impact: sign-in activity is outside audit search and export. Workaround: query
user_sessions directly.

⑨ /login and /refresh are not rate limited. The application registers four middlewares
(tracing, error handling, response envelope, request id) plus CORS (main.py:274–292) — none
of them a limiter. The production Caddyfile is 28 lines and contains no limit_req. Every
setting in the system named rate_limit is an LLM or tool-call quota and has nothing to do
with sign-in.

Related, same item: the second-factor challenge is a five-minute stateless JWT with no
spent-marker, and TOTP codes themselves are not replay-protected (totp.py:63–86 compares
the candidates within ±drift steps and never records which step was used). Impact: brute-force
and code-replay protection in our default stack is entirely the deployer's reverse proxy,
and the documentation does not say so. Workaround: rate-limit /api/v1/login,
/api/v1/refresh and /api/v1/login/mfa at the gateway.

⑩ A multi-tab refresh race can bounce someone to the sign-in page. This is an inference, not
an observation. The single-flight lock from section 7 is a module-level variable
(request.ts:153), so its scope is one tab. Open two console tabs, let the token expire in
both, then use both: each reads the same refresh token from local storage, the first wins and
writes the new one back, the second gets a 401, refreshAccessToken returns null, the 401
surfaces, and AUTO_REDIRECT_UNAUTHORIZED sends that tab to the sign-in page.

I did not reproduce this, so it is a conclusion from reading code. It is here because it
is exactly the problem item ④ would amplify if the documented behaviour were actually
implemented — at that point the loser of the race would not be one tab, it would be the whole
session. Workaround, if it does happen: share the refresh result between tabs via a storage
event or a BroadcastChannel.

Disclosure

Nothing was run for this piece. Every conclusion comes from reading code, git history and tests at commit fb46f20. I did not stand up an instance to watch a revoked session get a 401 on the next request, even though I believe the code path is unambiguous. Item ⑩ is explicitly an inference.
Community edition only. Whatever the Enterprise and Cloud builds add on top of this is out of scope.
"Nearly free" is an architectural judgement, not a measurement. Section 3 argues the marginal cost is close to zero because the check rides on a read that already happens and is a primary-key lookup. I did not benchmark it.
None of the ten items is a security incident. They are documentation drifting from implementation, and features that are not finished.
Disclosure: I maintain SOIT.

One sentence

Without a server-side session record, ACCESS_TOKEN_EXPIRE_MINUTES is your session length
and every value hurts; with one, it is only the ceiling on revocation delay, so shorter is
better and the session length belongs to a different setting. Which of the two you have is
decided not by the setting but by whether your request path already performs a lookup — and if
it already reads membership, you have paid for that lookup already.

Come and find the holes

The repository is github.com/soit-ai/soit, and every claim above is checkable:

git show 69be399 and git show 09dbf7e are the two commits; both messages are shorter and sharper than this post.
The "riding along" comment is in server/app/modules/identity/infra/workspace_access.py; read it together with resolve underneath.
The self-contradicting test in item ④ is in server/tests/unit/test_user_sessions.py — the function name and the assertion are three lines apart.

If any of this is wrong — especially if one of those ten items is me missing an implementation
that does exist — please open an issue and say so. I would rather learn which parts do not
line up than be told the design reads well.

`gh attestation verify` said yes to an image we never released

Jude — Sun, 13 Sep 2026 22:38:41 +0000

The four questions, and the one nobody asks

Pulling a container image somebody else built asks four separate questions. Most
people answer three of them and skip the fourth:

Step	Question it answers	If you skip it
Fetch by digest	are these the bytes I asked for	you are gambling on a mutable tag
Verify provenance	who built these bytes, from what, where	unknown origin
Read the SBOM	what is inside	a CVE lands and you cannot tell if you are affected
Check the release manifest	is this digest the one that version shipped	you may be verifying a different build

The fourth is the one that has no substitute, and it is the one that is almost
always skipped. What follows is all four, run for real against our own v1.0.0.

1. An image wears more than one sha256

This is where the confusion starts. Get an anonymous pull token — a public package
needs no account:

TOKEN=$(curl -s "https://ghcr.io/token?scope=repository:soit-ai/soit/server:pull&service=ghcr.io" \
  | sed -n 's/.*"token":"\([^"]*\)".*/\1/p')

Ask what the v1.0.0 tag resolves to:

curl -sI -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/vnd.oci.image.index.v1+json" \
  https://ghcr.io/v2/soit-ai/soit/server/manifests/v1.0.0

docker-content-digest: sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929
Content-Type: application/vnd.oci.image.index.v1+json
Content-Length: 856

856 bytes: this is an image index, not the image. Inside it, two entries:

manifests[0]: sha256:84e7f539...d32f  2589 bytes  linux/amd64
manifests[1]: sha256:a6bd430d...e1be   565 bytes  unknown/unknown
              vnd.docker.reference.type:   attestation-manifest
              vnd.docker.reference.digest: sha256:84e7f539...d32f

That unknown/unknown entry is not corruption — it is BuildKit's own attestation
for the amd64 manifest, and it is a different mechanism from the Sigstore
signatures we are about to read. First thing people conflate.

One level down, the amd64 manifest says:

config: sha256:021814982fcc214a02a2b322cbc3729b3ce43bd6e191aa0fa39da540a3e3cd1f  8769 bytes
layers: 12, 684216450 bytes compressed (about 652 MiB)

Three hashes so far, three different meanings:

Name	Value for server v1.0.0	Refers to
index digest	`96b80ae1…5929`	the multi-platform index itself
platform manifest digest	`84e7f539…d32f`	the linux/amd64 image
config digest	`02181498…cd1f`	the image config JSON

The signature covers the first one. A fourth hash shows up in section 8, and it
is none of these.

2. Pulling the signatures without Docker and without logging in

The OCI spec defines a referrers API for "things attached to this thing":

curl -s -H "Authorization: Bearer $TOKEN" \
  https://ghcr.io/v2/soit-ai/soit/server/referrers/sha256:96b80ae1...5929

{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}

Same error for the platform manifest digest. Do not conclude "unsigned" from
this. The spec has a fallback: replace the colon with a dash and treat the digest
as a tag. List the tags and there it is:

{"name":"soit-ai/soit/server","tags":[
  "v1.0.0",
  "sha256-3a5b3b1a2d14e0646298826602124ba22e63f8c078f653e506a0a042bfd18246",
  "sha256-236201a5a3ee861ffdb5fdd7ec134454619eb1ab0e777439c4a22a65002f874f",
  "sha256-96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"]}

The last one is ours. Remember the first two — section 9 is entirely about them.
Fetching sha256-96b80ae1… returns a small index holding two Sigstore bundles:

sha256:658b3428...80b9   812 bytes
  artifactType : application/vnd.dev.sigstore.bundle.v0.3+json
  predicateType: https://slsa.dev/provenance/v1
  created      : 2026-08-05T16:14:12.946Z

sha256:59bac8c5...d9cf   814 bytes
  predicateType: https://spdx.dev/Document/v2.3
  created      : 2026-08-05T16:14:22.004Z

Build provenance and an SBOM attestation, signed nine seconds apart. Both manifests
carry subject pointing back at sha256:96b80ae1…5929 — the single place where
"this signature belongs to that image" is actually written down.

Pull the bundle blob and hash it:

curl -sL -H "Authorization: Bearer $TOKEN" \
  https://ghcr.io/v2/soit-ai/soit/server/blobs/sha256:eae5d902...dd86 -o bundle.json
sha256sum bundle.json
# eae5d902186b490570aba6f702f7ce03c46f935e403bf881ec27d99480d7dd86 *bundle.json

The hash equals the digest I asked for. A content-addressed registry means this step
verifies itself; no trust required yet.

3. What the signature actually covers

The bundle has three top-level fields — mediaType, verificationMaterial,
dsseEnvelope. The envelope's payloadType is application/vnd.in-toto+json, and
the base64 payload is the only thing the signature covers:

{
  "_type": "https://in-toto.io/Statement/v1",
  "subject": [{
    "name": "ghcr.io/soit-ai/soit/server",
    "digest": {"sha256": "96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929"}
  }],
  "predicateType": "https://slsa.dev/provenance/v1",
  "predicate": {
    "buildDefinition": {
      "buildType": "https://actions.github.io/buildtypes/workflow/v1",
      "externalParameters": {
        "workflow": {
          "ref": "refs/tags/v1.0.0",
          "repository": "https://github.com/soit-ai/soit",
          "path": ".github/workflows/release.yml"
        }
      },
      "internalParameters": {
        "github": {
          "event_name": "push",
          "repository_id": "910429753",
          "repository_owner_id": "193298865",
          "runner_environment": "github-hosted"
        }
      },
      "resolvedDependencies": [{
        "uri": "git+https://github.com/soit-ai/soit@refs/tags/v1.0.0",
        "digest": {"gitCommit": "8105cae074f1f27d7916acfe02f9d4eabb63169f"}
      }]
    },
    "runDetails": {
      "builder": {"id": "https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0"},
      "metadata": {"invocationId": "https://github.com/soit-ai/soit/actions/runs/31023642816/attempts/1"}
    }
  }
}

One complete sentence: commit 8105cae… on refs/tags/v1.0.0, built by
.github/workflows/release.yml on a GitHub-hosted runner in run 31023642816,
produced the image with digest 96b80ae1….

I had the repository on disk, so I closed the loop immediately:

git rev-parse v1.0.0^{commit}
# 8105cae074f1f27d7916acfe02f9d4eabb63169f

Match. Note repository_id and repository_owner_id: numeric IDs are worth more
than the repository name, because names can be renamed and transferred, IDs cannot.

4. The certificate that lived for ten minutes

verificationMaterial.certificate.rawBytes is a 1735-byte DER certificate:

issuer=O = sigstore.dev, CN = sigstore-intermediate
subject=
notBefore=Aug  5 16:14:11 2026 GMT
notAfter =Aug  5 16:24:11 2026 GMT
X509v3 Subject Alternative Name: critical
    URI:https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0

Two things worth stopping on.

The subject is empty. The field that traditionally names the holder is blank; the
identity moved entirely into that SAN URI, and it is not a person or an organisation —
it is a workflow at a ref.

The validity window is exactly ten minutes. That is keyless signing: the pipeline
trades an OIDC token for a short-lived certificate, signs, and throws the key away.
Nothing to store, nothing to rotate, nothing to steal later. Ten minutes on, the
certificate is expired — and the signature still verifies, for the reason in section 5.

The certificate also carries a block of Sigstore extensions describing the build. The
values below are what I read; I am deliberately not putting names on the OIDs —
those live in Fulcio's OID documentation, and the values speak for themselves:

OID suffix	Value
`.1.1` / `.1.8`	`https://token.actions.githubusercontent.com`
`.1.2` / `.1.20`	`push`
`.1.3` / `.1.10` / `.1.13` / `.1.19`	`8105cae074f1f27d7916acfe02f9d4eabb63169f`
`.1.4`	`release`
`.1.5`	`soit-ai/soit`
`.1.6` / `.1.14`	`refs/tags/v1.0.0`
`.1.9` / `.1.18`	`…/release.yml@refs/tags/v1.0.0`
`.1.11`	`github-hosted`
`.1.12`	`https://github.com/soit-ai/soit`
`.1.15` / `.1.17`	`910429753` / `193298865`
`.1.16`	`https://github.com/soit-ai`
`.1.21`	`…/actions/runs/31023642816/attempts/1`
`.1.22`	`public`
`.1.24`	`repo:soit-ai/soit:ref:refs/tags/v1.0.0`

Extended Key Usage is Code Signing, Key Usage is Digital Signature only, and
there is a CT precertificate SCT timestamped Aug 5 16:14:11.958 2026 GMT — the act
of issuing this certificate was itself logged publicly.

5. Why an expired certificate still counts

verificationMaterial.tlogEntries holds exactly one record:

logIndex        : 2346649359
integratedTime  : 1785946452   ->  2026-08-05T16:14:12Z
kindVersion     : {kind: dsse, version: 0.0.1}

That record lives in Rekor, the public append-only transparency log, and its job is
to witness when. The question a verifier asks is not "is this certificate valid
now" — it expired long ago — but "was it valid at the moment of signing". The log
pins that moment at 16:14:12Z, inside the 16:14:11–16:24:11 window.

A detail from the actual bundle: timestampVerificationData is an empty object.
No RFC 3161 timestamp; the Rekor entry carries the time on its own. Not wrong, just
worth knowing what your trust actually rests on.

6. Where the SBOM lives and what is in it

The second bundle's predicate is a complete SPDX document. The bundle blob is
3437645 bytes:

spdxVersion   : SPDX-2.3
dataLicense   : CC0-1.0
name          : ghcr.io/soit-ai/soit/server
creationInfo  : creators ["Organization: Anchore, Inc", "Tool: syft-1.42.3"],
                created "2026-08-05T16:14:08Z"
packages      : 710
relationships : 2840

Those 710 packages by purl type: 469 pkg:deb, 217 pkg:pypi, one
pkg:generic (python 3.11.15) and one pkg:oci (the image itself).

The 469 Debian packages come from the base image — server/Dockerfile starts at
python:3.11 — and only 217 are our own Python dependencies. That ratio is itself a
finding: two thirds of what you just signed, you did not write.

All three images side by side show the split clearly:

Image	Packages	Composition	SBOM generated
`server`	710	469 deb + 217 pypi	16:14:08Z
`knowledge-worker`	775	469 deb + 281 pypi	16:22:23Z
`web`	1331	1311 npm + 18 apk	16:15:01Z

The 64 extra pypi packages in knowledge-worker include torch, transformers and
nvidia-cublas-cu12. The two images are two targets of one Dockerfile; the entire
difference is uv sync --extra knowledge-worker (server/Dockerfile:19 vs :27).
web is node:24-alpine, so the Debian packages become 18 apk ones.

7. The 16 MiB ceiling, and what it cost us

An SPDX document normally carries a files section recording which files each
package was identified from. All three of ours have none — and it is deliberate,
right there in the workflow (.github/workflows/release.yml:132–149):

jq 'del(.files)
    | .packages |= map(del(.hasFiles))
    | .relationships |= map(select(
        ((.spdxElementId // "") | startswith("SPDXRef-File") | not)
        and ((.relatedSpdxElement // "") | startswith("SPDXRef-File") | not)))' \
  "$f" > "$f.tmp"
mv "$f.tmp" "$f"
size="$(stat -c%s "$f")"
test "$size" -le 16000000

The comment above it gives the reason: per-file SPDX entries push an ML-heavy image's
SBOM past the 16 MiB subject limit of actions/attest. In plain terms, the SBOM had
to be shrunk until it could be signed.

The cost is real. You keep the package inventory and the package-to-package
relationships; you lose "which file did this package come from", which is exactly what
you want when a CVE lands and you need to locate it. The test -le 16000000 line is a
hard gate: if a future image blows past the ceiling the release fails there rather
than shipping an SBOM that cannot be signed. That part I think is right — better a
broken build than a half-signed artifact.

8. An SBOM is a scan result, not the truth

Two things I found in ours.

Five Windows launchers in a Linux image. Among the 710 packages,
Simple Launcher 1.1.0.14 appears five times, each with a CPE and no purl:

SPDXRef-Package-binary-Simple-Launcher-d60858f6579e7bb1
  versionInfo : 1.1.0.14
  externalRefs: cpe:2.3:a:Simple_Launcher:Simple_Launcher:1.1.0.14:*:*:*:*:*:*:*

Syft's binary classifier found them: Python packaging wheels ship small Windows
launcher executables, which will never execute inside this Linux image but are
inventoried all the same. And because the files section was stripped (section 7),
you cannot even find out where they are.

A fourth hash that matches nothing. The pkg:oci entry representing the image
itself reads:

pkg:oci/ghcr.io%2Fsoit-ai%2Fsoit%2Fserver@sha256%3Ae7c993b9ac5d7322058e0169c678b4ce21fe42fac72c5d3374404bf4642939ea?arch=amd64

e7c993b9… is not the index digest, not the platform manifest digest and
not the config digest. Asked for directly, the registry returns 404. I could not
determine what it is — the reasonable guess is an identifier the scanner computes
locally — so I am reporting the observation and not a conclusion. The practical rule
is clear enough: do not reconcile an SBOM's internal digest against the signature's
subject. The binding lives in exactly one place, subject.digest, and that value is
96b80ae1…, matching the registry exactly.

9. Three `v1.0.0` images, all of them verifiable

Back to those two extra tags. Fetched the same way:

Image digest behind the fallback tag	Attestations attached	Signed at
`3a5b3b1a…8246`	provenance only	2026-08-05T15:41:15Z
`236201a5…f874f`	provenance + SBOM	15:52:32Z / 15:52:44Z
`96b80ae1…5929`	provenance + SBOM	16:14:12Z / 16:14:22Z

All three provenance statements name refs/tags/v1.0.0 and the same signing identity.
The commits differ:

Image digest	Commit in provenance	Actions run	Commit subject
`3a5b3b1a…`	`0dacfc52…`	31020921135	`ci(release): create the artifacts directory before image SBOM generation`
`236201a5…`	`ec822c63…`	31021873557	`ci(release): catalog packages only in image SBOMs`
`96b80ae1…`	`8105cae0…`	31023642816	`ci(release): trim image SBOMs to package level before attestation`

All three commits are on main locally, 23:33 / 23:44 / 00:05 in my timezone. The
story reads itself off the commit subjects: the tag was re-pointed twice that
night while the SBOM step was being fixed — and each failed attempt had already
pushed and signed its images before failing.

So this happens:

gh attestation verify \
  oci://ghcr.io/soit-ai/soit/server@sha256:3a5b3b1a...8246 \
  --repo soit-ai/soit --format json

exit code : 0
subject   : 3a5b3b1a...8246
predicate : https://slsa.dev/provenance/v1
ref       : refs/tags/v1.0.0
commit    : 0dacfc5219c4ae57d024f9cb4208e1efad5f0d17
san       : https://github.com/soit-ai/soit/.github/workflows/release.yml@refs/tags/v1.0.0
tlog      : rekor.sigstore.dev @ 2026-08-05T23:41:11+08:00

Exit 0. Nothing is wrong with the signature. It states honestly what it is: an
image built by our release workflow, on refs/tags/v1.0.0, from commit 0dacfc52….
The one thing it does not say is that it is not the image we shipped.

No attack here — we built all three. But swap in a nastier scenario and it holds
immediately: a pipeline compromised once, fixed, re-tagged, with the poisoned
intermediate still sitting in the registry. Attestation will not catch that.
gh attestation verify answers "did this come out of that pipeline". It never
promised to answer "is this the release".

10. The missing link: a manifest that pins tag, commit and digest

That answer is not in the signature. It is an asset on the GitHub Release, which
carries seven files:

soit-1.0.0.tar.gz          3148831
SHA256SUMS                     433
release-artifacts.json        2877
server.spdx.json           3313874
knowledge-worker.spdx.json 4004516
web.spdx.json              3862614
source.spdx.json           3694033

release-artifacts.json is the manifest:

{
  "featureKey": "release.artifacts",
  "schemaVersion": 1,
  "version": "1.0.0",
  "release_tag": "v1.0.0",
  "commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f",
  "clean_worktree_at_tag": true,
  "images": [{
    "component": "server",
    "name": "ghcr.io/soit-ai/soit/server",
    "release_tag": "v1.0.0",
    "digest": "sha256:96b80ae141000adde27cf3dedb27935b5f5d0085d69025cffb4bbb63276d5929",
    "reference": "ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929",
    "sbom": {"format": "spdx-json", "path": "server.spdx.json", "sha256": "b83c3bc7...ef68", "attestation": "..."},
    "provenance_attestation": "..."
  }]
}

It says the thing the signature would not: v1.0.0 is 96b80ae1…, not the other
two. It is assembled by jq inside the publish-release job
(release.yml:300–347) and validated on the spot. I fed the copy downloaded from the
public internet straight into the same script that lives in the repository:

{
  "commit": "8105cae074f1f27d7916acfe02f9d4eabb63169f",
  "images": ["knowledge-worker", "server", "web"],
  "passed": true,
  "release_tag": "v1.0.0"
}
exit=0

178 lines, standard library only, no network
(server/scripts/verify_release_artifacts.py). It insists that:

release_tag equals v + version (:42);
commit is 40 lowercase hex characters and not all zeroes (:45);
clean_worktree_at_tag is true (:49);
every image reference equals name@digest (:91) — a name:tag reference is rejected outright;
digest matches ^sha256:[0-9a-f]{64}$ (:88);
exactly the three expected components are present (:15, :114);
no attestation URL is reused across entries (:107–112).

A test guards that last-but-two rule: it rewrites the example's reference into tag
form and asserts the script fails with digest-pinned
(server/tests/unit/test_release_operations_contract.py:85–88).

11. I reproduced the source tarball, byte for byte, on Windows

The release also ships a git archive tarball (release.yml:253–263) described as
deterministic. Worth testing rather than believing. First, the checksum:

grep "soit-1.0.0.tar.gz" SHA256SUMS | sha256sum -c -
# ./soit-1.0.0.tar.gz: OK

Then rebuild it locally:

git archive --format=tar.gz --prefix="soit-1.0.0/" \
  --output=local.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f

c11179c379ba7390c215133f342c92a7517a548f1536959cb43e187b16a7bab3 *local.tar.gz
5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *soit-1.0.0.tar.gz

Different — and not just at the gzip layer; the inner tars differ too. Comparing
members explains it:

members   : 1551 local / 1551 released   -- same
only-one-side: 0 / 0                     -- same
differing : 1280 members
mtime     : 1785945959 everywhere        -- same
mode      : 0664 everywhere              -- same
example: soit-1.0.0/.github/workflows/quality.yml   local 15270  released 14803

Sizes only, and each delta equals that file's line count. CRLF. My machine has
core.autocrlf=true, so git archive helpfully rewrote the line endings. Turn it
off:

git -c core.autocrlf=false archive --format=tar.gz --prefix="soit-1.0.0/" \
  --output=local2.tar.gz 8105cae074f1f27d7916acfe02f9d4eabb63169f
# 5d3dd50f491a897ba9a184ad6dc474e2ef55508224404266d91b3e1e1d32ae9d *local2.tar.gz

Identical. A Windows box, git 2.55.0, a month later, reproducing the tarball
GitHub Actions produced.

Both halves matter. Determinism is real — git archive writes no build timestamp and
takes mtimes from the commit — but it is sensitive to local git configuration, and
the first suspect when reproduction fails is core.autocrlf, not the publisher.

12. The checklist

Everything above, compressed into something you can paste:

# 1. Resolve the digest (never keep books against a tag)
docker buildx imagetools inspect ghcr.io/soit-ai/soit/server:v1.0.0 | head -3

# 2. Verify build provenance
gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit

# 3. Verify the SBOM attestation (NOT checked by default)
gh attestation verify oci://ghcr.io/soit-ai/soit/server:v1.0.0 --repo soit-ai/soit \
  --predicate-type https://spdx.dev/Document/v2.3

# 4. Check the release manifest: is this digest the one that shipped
curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/release-artifacts.json
curl -sLO https://github.com/soit-ai/soit/releases/download/v1.0.0/SHA256SUMS
sha256sum -c SHA256SUMS

# 5. Run it by digest, not by tag
docker pull ghcr.io/soit-ai/soit/server@sha256:96b80ae1...5929

Step 3 deserves emphasis: gh attestation verify checks provenance only by
default. Without --predicate-type it returned exactly one result here, the SLSA
provenance. With it, the SPDX attestation comes back — 6022876 bytes of JSON holding
the same 710 packages counted in section 6. A lot of people assume "verified" includes
the SBOM. It does not.

Step 4 is the point of this whole piece. Without it, the green checkmark from step 2
shines just as brightly on the 3a5b3b1a… image from section 9.

13. Eight things that still do not line up

House rule: the last section is about our own problems.

1. Our own compose pins tags, not digests. All six services in
docker/docker-compose.images.yml use :${SOIT_IMAGE_TAG:-v1.0.0} (:17–:32), and
the README says the same (README.md:146). We demand digest-pinned references inside
release-artifacts.json and then hand users a tag. Impact: a tag can be re-pointed.
Workaround: step 5 of the checklist. I plan to open an issue; none filed at the
time of writing, hence no link.

2. SHA256SUMS does not cover release-artifacts.json. The 433-byte file lists
five entries — four SPDX documents and the tarball. Neither the manifest nor
SHA256SUMS itself is in there (the latter cannot be, by construction). The manifest
does get its own attestation (release.yml:350–353), so it is not unprotected, but
the instinctive "download everything and run sha256sum -c" does not reach it.
Also going to open an issue.

3. docs/release-process.md lags the pipeline. It tells you to copy the real
tag, commit, digests, SBOM checksums and attestation URLs into an evidence document
and then run the verifier (:24–:32). The pipeline has been doing that
automatically for a while, and uploads the result as a release asset. Impact: a reader
concludes the manifest is hand-written after the fact, which makes it look weaker than
it is. Also going to open an issue.

4. Superseded builds are never cleaned up. The two images from section 9 are still
pullable, still verifiable, still runnable. We have no "delete same-tag intermediates
after a successful release" step. Impact: as described. Workaround: pull the digest
from release-artifacts.json.

5. The verifier checks shape, not truth. provenance_attestation only has to be a
non-empty string (:58, :64, :106, all via _require_text). The script never
fetches the URL and never compares the attested subject digest to the one in the
manifest. Impact: a well-formed manifest full of fabricated URLs passes. Workaround:
it is a structural check, not a trust check — the trust comes from
gh attestation verify, and you need both. Defensible division of labour, but the
docs never say so.

6. REQUIRED_IMAGES is hard-coded. verify_release_artifacts.py:15 pins
{"server", "knowledge-worker", "web"} and line 114 demands exact equality. Add a
fourth image and the release breaks at the final step. Impact: a bad surprise on
release day. Workaround: know it is there — and the case it blocks (silently shipping
with an image missing) is worth more than the nuisance.

7. No process for SBOM false positives. Sections 7 and 8. Nobody owns annotating
Simple Launcher as noise, so the next reader has to work it out again. Impact: the
signal-to-noise ratio degrades as images grow.

8. I could not identify e7c993b9…. That is where section 8 stops. This entry
exists so I do not pretend otherwise: I can confirm it is absent from the registry
and differs from the other three digests; I cannot tell you how it is derived.

Disclosures

This is a verification walkthrough, not a runtime one. I did not pull the images and stand up a stack; every conclusion stops at the bytes-and-signatures layer.
Section 9 is not a security incident. We built all three images ourselves on the same night. It illustrates the semantics of attestation, not a breach.
Section 8's mismatched digest is an observation, not a conclusion. The guess is labelled as a guess.
All numbers belong to the v1.0.0 release (tag commit 8105cae…). Later releases will have different values; the method does not change.
I only exercised the success path of gh attestation verify (exit 0). I did not construct a tampered image to see how it fails.
Disclosure: I maintain SOIT.

One-line takeaway

A passing attestation says "these bytes came out of that pipeline"; it does not say
"this is the release." What joins the two is a manifest binding tag, commit and
digest — ours is release-artifacts.json, watched by a 178-line script whose hardest
rule is a single line: the reference must be name@digest, and a tag reference fails.

Try it, and please poke holes

The repository is github.com/soit-ai/soit, and every step above is reproducible:

the curl calls in section 2 need no account — an anonymous token is enough;
the 3a5b3b1a… image from section 9 is still there; verify it yourself and watch it exit 0;
for the byte-identical rebuild in section 11, remember -c core.autocrlf=false.

If any claim here is wrong — especially the digest in section 8 that I failed to
identify — open an issue and say so. I would rather learn where this is wrong than be
told the pipeline looks thorough.

Four different things are called "replay" in our agent runtime. I read the ledger.

Jude — Thu, 10 Sep 2026 01:06:34 +0000

The short version

replay means four different things in our codebase. Here they are up front:

#	What it is	Entry point	Re-executes?	Cost
1	Evidence replay	`GET /api/v1/observe/runs/{run_id}/replay`	No	One database read
2	Catch-up after a dropped connection	`GET /api/v1/runs/{run_id}/stream?last_event_id=...`	No	One database read, then resubscribe
3	Idempotent replay of a tool call	Inside the tool gateway, same idempotency key arriving twice	No — returns last time's result	One database read
4	Actual re-execution	`POST /api/v1/workflows/{id}/runs/{run_id}/replay` and two other paths	Yes	Runs again, spends money again

All four rest on one thing: the ledger that hits the database first is the authority.

Not the logs. Not the event stream. Not the SSE feed scrolling in your console.
The rows in the tables. That sounds unremarkable until you notice it is what lets three
of those four survive a process restart.

1. Why the word needs splitting

In a demo, these three sentences look like one feature:

"Here's every step of that run."
"Lost your connection? Refresh — the missing steps come back."
"Bad answer? Hit replay."

Engineering-wise they are nothing alike. The first is a read. The second is a
read plus a subscription. The third is a write — it calls the model again,
sends the HTTP request again, spends the money again.

The cost of collapsing them into one word is a user who assumes "replay" is safe
and sends two emails.

So the order below goes from cheapest to most expensive.

2. The tables, and one number that shows up three times

Five tables, all in one file (server/app/kernel/runtime/db/models/runs.py, 396 lines):

runs                    one execution                        Run           :25
run_steps               one step inside it                   RunStep       :120
run_step_tool_calls     execution control for one tool call                :180
run_artifacts           files this execution produced        RunArtifact   :242
run_cost_entries        usage and cost for one metered call                :284

Three more cover long-running work (models/tasks.py, 98 lines): tasks,
task_checkpoints, task_events. Section 7 uses them.

Now the detail worth stopping on: 8192 appears three times in this ledger,
and it means something different each time.

Twice on the run and the step, where summaries are truncated:

input_summary=input_summary[:8192] if input_summary else None,

Once on a tool call result, where anything larger is offloaded to object storage
and the ledger keeps a pointer, a byte count and a sha256:

if len(encoded_result) > 8192:
    ...
    artifact = self.trace_writer.create_artifact(
        run_id=record.run_id,
        step_id=record.run_step_id,
        artifact_type="json",
        storage_key=storage_key,
        mime="application/json",
        size_bytes=len(encoded_result),
        sha256=hashlib.sha256(encoded_result).hexdigest(),
        meta={"kind": "tool_result", "tool_call_id": record.tool_call_id},
    )

The asymmetry is deliberate. Summaries are for humans; losing the tail is fine.
Tool results get reconciled and replayed; losing a byte is not fine.

Section 12 covers a consequence of that asymmetry we have not handled well yet.

3. Replay #1: reassembling the evidence

The cheap one. A GET:

GET /api/v1/observe/runs/{run_id}/replay

One sentence of behaviour: query the five record types by run id, add approvals and
feedback, return the bundle. The implementation
(server/app/modules/observe/application/service.py:212) returns seven keys:

return {
    "run": run,
    "steps": steps,
    "artifacts": artifacts,
    "costs": costs,
    "approvals": approvals,
    "feedback": feedback,
    "trace_spec": to_runtrace_spec(run, steps, artifacts, costs),
}

Six raw record sets, plus trace_spec — the same data flattened into something you can
hand to a tracing backend (kernel/runtime/runs/exporter.py:88). That spec carries two
rollups alongside the timeline: usage_summary (prompt tokens, completion tokens,
embeddings, reranks, milliseconds, storage bytes, requests, vectors) and charge_summary
(amounts grouped by currency).

Nothing here executes. No model call, no tool call, no cost. It is a database read,
so you can call it at any point after the run ended, and the ten-thousandth call costs
what the first one did.

Every query carries tenant_id and workspace_id in its where clause — reading another
workspace's ledger is closed off at the SQL level, not at a middleware you can misconfigure.

4. Replay #2: catching up after the connection drops

The second-cheapest, for the "tab is open, wifi died" case:

GET /api/v1/runs/{run_id}/stream?last_event_id=st_xxxx

Handled at server/app/api/v1/workflow/streaming.py:401. The part that matters:

if last_event_id:
    step_query = select(RunStep).where(
        and_(
            RunStep.id == last_event_id,
            RunStep.run_id == run_id,
            ...
        )
    )
    last_step = db.exec(step_query).first()
    if last_step:
        last_step_time = last_step.created_at
        known_step_ids.add(last_step.id)

steps_query = select(RunStep).where(
    and_(
        RunStep.run_id == run_id,
        ...
        RunStep.created_at > last_step_time if last_step_time else True,
    )
).order_by(RunStep.created_at)

Look at where it reads from: select(RunStep). The database. Not an in-memory ring
buffer, not a broker offset.

That choice buys a specific property: you can reconnect an hour after the run finished,
hand over your last_event_id, and still get the steps you missed. An in-memory buffer
cannot do that — a restart empties it. A broker can, but then you need a broker.

The SSE id: field is the step's primary key (streaming.py:432), so the Last-Event-ID
that browsers resend automatically is already a row id in the ledger. No second cursor
scheme to keep in sync.

One more detail worth borrowing: that query sets populate_existing=True, with a comment
explaining why — the execution side writes from its own session, so this tailer has to
bypass anything its own session cached earlier. That is the kind of line nobody can
reconstruct three months later without the comment.

5. Replay #3: the same idempotency key, twice

This one happens below the surface, inside the tool gateway.

Every tool call gets a run_step_tool_calls row. The table carries three unique
constraints (models/runs.py:182):

UniqueConstraint("tenant_id", "workspace_id", "run_step_id", ...)
UniqueConstraint("tenant_id", "workspace_id", "run_id", "tool_call_id", ...)
UniqueConstraint("tenant_id", "workspace_id", "idempotency_key", ...)

The third is the interesting one. When the same key arrives again and the row is already
terminal:

if existing.status in {"succeeded", "failed"}:
    payload = existing.result_json or {}
    ...
    return ToolExecutionClaim(
        record=existing,
        run_step=step,
        replayed=True,
        cached_response=ToolResponse(
            result=payload.get("result"),
            success=existing.status == "succeeded",
            error=existing.error_message,
            metadata={..., "idempotent_replay": True},
        ),
    )

Last time's result comes back; nothing leaves the process. The metadata carries
idempotent_replay: True so callers can tell this apart from a fresh execution.

If the earlier result was large enough to live in object storage,
load_cached_response (tool_calls.py:636) fetches the artifact — after checking tenant,
workspace, run and step all match, and raising Tool result artifact scope mismatch
if any of them does not.

The point of this layer: replay #4 is only safe to offer because this one exists.
When you re-run, the tool calls whose idempotency keys did not change are not actually
executed a second time.

6. A status that admits we don't know

This is the design I would point at first if someone asked what is unusual about this
ledger.

Claiming a tool call takes a lease (60 seconds by default, widened by the gateway to the
tool's timeout). An expired lease means the executor may be dead. Retry or not?

The code answers by asking whether the request actually left (tool_calls.py:309):

lease_expired = (
    existing.lease_expires_at is not None
    and _aware_utc(existing.lease_expires_at) <= now
)
if lease_expired and existing.outbound_started_at is not None:
    existing.status = "in_doubt"
    ...
    raise ConflictError("Tool call outcome is in doubt")
if lease_expired and existing.outbound_started_at is None:
    existing.status = "claimed"
    existing.attempt_count += 1
    ...

Two branches, split on one field, outbound_started_at:

Died before going out — safe. Re-claim, bump the attempt count.
Died after going out — mark it in_doubt, do not retry, park the step at paused, raise a conflict.

The second branch is the honest one. On the other end is a real system: an order endpoint,
an email, a transfer. The request left and no response came back.
We don't know whether it happened, so we don't guess. The ledger records "in doubt"
and a human decides.

Auto-retrying here is wrong in the specific way that only surfaces when someone gets two
copies of the same email.

7. Replay #4: actually running it again

The expensive one. Three separate paths, three different mechanisms.

(a) Workflows: replay and retry

POST /api/v1/workflows/{workflow_id}/runs/{run_id}/retry
POST /api/v1/workflows/{workflow_id}/runs/{run_id}/replay

The two implementations differ by one check
(modules/workflow/application/service.py:804 and :821): retry requires the source run
to be failed or canceled; replay does not. Both load the original inputs, execute
again, and put source_run_id and control_action in the response.

(b) Agent tasks: replaying a persisted snapshot

More interesting (server/app/wiring/task_drivers.py:82). Rather than "take the inputs and
run", it loads the previous ResponseInteraction snapshot and deliberately strips the
identifiers that belonged to the failed attempt before queueing a new one:

execution_json["assistant_message_id"] = generate_thread_message_id()
payload = dict(execution_json.get("payload") or {})
if payload:
    # Drop identifiers that belong to the attempt being replaced so the
    # replay creates its own response, run and task.
    payload.pop("task_id", None)
    payload.pop("run_id", None)

The old task is then moved to CANCELED with a forward pointer,
retried_as_interaction_id, in its progress payload. The comment is blunt about why:
leaving it queued would report work this task will never perform.

If there is no snapshot, it does not improvise — it fails explicitly with a dedicated
error code, SNAPSHOT_MISSING_ERROR_CODE. No evidence, no replay. I like that one.

(c) Knowledge ingestion: lineage that actually lands in the ledger

The only one of the three that writes the lineage into runs
(modules/knowledge/application/runtime_service.py:848):

run = self.trace_writer.create_run(
    ...
    source_run_id=previous_run.id,
    attempt_no=max(previous_run.attempt_no + 1, task.retry_count + 1),
    request_id=f"knowledge-ingest:{task.id}:{task.retry_count + 1}",
)

runs has both source_run_id and attempt_no, plus a dedicated index,
ix_runs_scope_source_created. This path uses them.

The other two do not. That is item ① in section 12.

8. Why the ledger is trustworthy

Three reasons, all in the code.

Status changes are conditional UPDATEs, not read-modify-write.
The where clause at writer.py:375 carries the old value:

result = self.db.execute(
    update(Run)
    .where(
        Run.id == run_id,
        Run.tenant_id == self.ctx.tenant_id,
        Run.workspace_id == self.ctx.workspace_id,
        Run.status == old_status,
    )
    .values(**values)
    ...
)
if result.rowcount != 1:
    ...
    raise RuntimeTransitionError(f"Concurrent run transition rejected: {old_status} -> {target_status}")

Two executors racing to change the same run: one wins, the other sees rowcount != 1
and is rejected. Not last-write-wins — someone jumped the queue, so error out.

Success is an irreversible terminal state.
From the transition table in kernel/runtime/status.py:

ExecutionStatus.SUCCEEDED: frozenset(),
ExecutionStatus.FAILED: frozenset({ExecutionStatus.RETRYING}),
ExecutionStatus.CANCELED: frozenset({ExecutionStatus.RETRYING}),
ExecutionStatus.EXPIRED: frozenset({ExecutionStatus.RETRYING}),

SUCCEEDED reaches nothing. A success written into the ledger cannot be walked back,
not even to failed. Failures can move to retrying; successes go nowhere.

Outbound notification goes through a transactional outbox, not a live broadcast.
Creating a run and every status change write an outbox row (writer.py:282 and four other
sites) inside the same database transaction as the business data.

The live event bus, by contrast, is best-effort — the last line of _emit_event is:

except Exception:
    return

Swallowed. That is the right call: a failed notification must never block the ledger
write. It also means one thing for anyone verifying behaviour —
reconcile against the ledger, not against what you saw on the event stream.

9. The ledger belongs to the ports, not to the loop

The previous piece argued that governance is a property
of the port rather than of the agent loop. This one adds a parallel claim.

Count who writes to TraceWriter:

File	Mentions of `trace_writer`
`kernel/ports/llm/policy.py`	57
`kernel/ports/storage/policy.py`	47
`kernel/ports/vector/policy.py`	44
`kernel/ports/tools/policy.py`	15
`kernel/ports/plugins/policy.py`	12

Five kernel ports, five policy gateways, one ledger.

Which means: you do not instrument the agent loop, and you do not instrument the
workflow engine. If an operation left through a port, it left a row. The agent loop
calling tool_port.invoke leaves one; a DAG workflow's tool node calling the same
tool_port.invoke leaves one — in the same table, with the same schema.

The converse holds too, and it is the real boundary of this design:
a call that bypasses the ports leaves nothing in the ledger. That is not a bug, it is
what layering means. The ledger records governed operations, not everything the process
happened to do.

10. The boolean the platform computes for you

Mechanism aside, the question a user actually has is simpler: is there enough evidence
for this run?

GET /api/v1/runs/{run_id} returns thirteen governance evidence items
(kernel/runtime/runs/service.py:530 onward):

actor_scope        subject_version     capability_binding   permission_scope
secret_boundary    egress_policy       audit_record         cost_attribution
trace_timeline     tool_call           knowledge_citation   child_workflow
replay_ready

The last one is the boolean. Its criteria are at service.py:516:

replay_missing: list[str] = []
if not steps:
    replay_missing.append("steps")
if response_timeline_applicable and not response_events:
    replay_missing.append("response_events")
if cost_attribution_applicable and not cost_entries:
    replay_missing.append("costs")
if knowledge_citation_applicable and not citations:
    replay_missing.append("citations")
if tool_governance_applicable and not tool_calls:
    replay_missing.append("tool_calls")
if tool_governance_applicable and not audits:
    replay_missing.append("audits")

Note the _applicable guards: it judges against what this run actually did. A pure
chat run has no tool calls and is not marked deficient for lacking them. A run that did
call a tool, but has no matching audit rows, comes back fail — and names the missing
category in missing.

I find that more useful than a docs page promising "full replay". It is a field you can
query, not an adjective. And it can fail — a check that always returns pass is not a
check.

11. Fingerprints in the ledger, not payloads

A ledger you keep for a long time is at risk of becoming a disclosure surface.

The handling starts at tool_calls.py:59. Arguments are redacted before they are
persisted; a key matching one of sixteen sensitive names becomes [REDACTED]:

api_key       apikey          access_token   authorization
client_secret cookie          credential     password
private_key   refresh_token   secret         secret_access_key
session_token token           x_api_key

Keys are normalized before comparison — camel case split, non-alphanumerics folded to
underscores — so apiKey, API-KEY and api_key are treated alike.

One exception is worth knowing: if the value is a dict carrying a secret_id, it is
not redacted. It is already a reference rather than a plaintext, and blanking it would
destroy the one thing you want later: which secret this call used.

Then size. Arguments over 8192 bytes are not stored; three things are kept instead:

return {
    "truncated": True,
    "size_bytes": summary["size_bytes"],
    "request_hash": summary["payload_hash"],
    "argument_names": sorted(str(key) for key in value),
}

request_hash is computed over the pre-redaction original — canonical_request_hash
does a sorted, compact JSON dump and sha256s it.

The effect: the ledger holds no payload, but idempotency still works. The same key
arriving twice is compared on request_hash; a mismatch raises
Tool call identity was reused with different input.

Reconcilable, but not leaky. Second-nicest thing in this ledger, after in_doubt.

12. Seven things that don't line up yet

This section is entirely about our own problems, ordered by impact.

① Workflow replay/retry never writes lineage into the ledger.

runs has source_run_id and attempt_no, plus an index built for them. Across the
whole repo there are 19 create_run( call sites and exactly one passes
source_run_id — the knowledge ingestion path from section 7.

Workflow replay goes through execute_workflow into engine.execute, and the engine
creates the run like this (modules/workflow/runtime/engine.py:145):

run = self.trace_writer.create_run(
    mode=plan.mode,
    subject_kind=plan.subject_kind,
    subject_id=plan.subject_id,
    subject_version_id=plan.subject_version_id,
    input_summary=input_summary,
    run_id=plan.run_id,
)

No source_run_id. No attempt_no.

Impact: source_run_id exists only in the HTTP response body. If the caller does
not store it, the "B is a replay of A" relationship is gone — unqueryable in the ledger,
and ix_runs_scope_source_created indexes nothing useful.

Workaround today: have the caller keep the source_run_id it got back.
Intended fix: thread source_run_id and attempt_no through those two paths into
create_run. I intend to open an issue for this; I had not filed it when this was
written, so there is no link here.

② Re-execution reads a truncated copy of the inputs.

Section 2 noted that input_summary is cut at 8192 bytes. Workflow replay loads inputs
like this (modules/workflow/application/service.py:151):

def _load_run_inputs(self, run: Run) -> dict[str, Any] | None:
    if not run.input_summary:
        return None
    import json
    try:
        parsed = json.loads(run.input_summary)
        ...

It json.loads the summary.

So a run whose inputs exceeded 8KB will fail to parse on replay (truncated JSON generally
is not valid) and return Replay requires inputs or a parseable run input_summary.

Workaround today: pass inputs explicitly instead of letting it read from the ledger —
both endpoints accept an override.
Intended fix: send large inputs to an artifact and keep a pointer, exactly like tool
results in section 2. Same status: issue intended, not yet filed.

③ generate_ulid() does not generate a ULID.

The source says so itself (kernel/commons/ids.py:9):

def generate_ulid() -> str:
    """Generate a ULID-like sortable ID.

    For now, we use UUID4 with prefix. In production, consider using
    python-ulid or similar library for true ULID generation.
    """
    return f"id_{uuid.uuid4().hex}"

UUID4 is random. Not sortable at all — neither the name nor the "sortable" in the
docstring holds.

Bounded but real impact: everything that needs chronological order has to use
created_at rather than the id. The catch-up in section 4 does exactly that. Arguably
forced into the correct implementation.

④ Catch-up uses a strict greater-than.

Following from ③: created_at comes from Python's datetime.now(UTC), and the filter is
RunStep.created_at > last_step_time.

Theoretical consequence: if two steps land on an identical timestamp and the client's
last received event was one of them, the other is skipped by the strict comparison.

I did not reproduce this. datetime.now() resolves to microseconds on modern Linux,
and two steps in one run colliding on the same microsecond takes unusual conditions.
It is listed as a design fragility, not an observed bug — please don't repeat it as
one. The fix is easy once ③ is done: order by (created_at, id).

⑤ Ids in the ledger come in three shapes.

Because generate_ulid() already returns an id_-prefixed string, anything that adds its
own prefix ends up double-prefixed:

Table	How it is generated	What you see
`run_step_tool_calls`	`f"rstc_{generate_ulid()}"`	`rstc_id_xxxx`
`tasks`	`f"task_{generate_ulid()}"`	`task_id_xxxx`
`run_cost_entries`	`default_factory=generate_ulid`	`id_xxxx`

All three work. The third just gives no hint which table the row belongs to.
Cosmetic, no functional impact — but you notice it the moment you start reading rows.

⑥ The Run.status docstring lists 6 statuses; there are 11.

On the model (models/runs.py:79):

"""Status: queued, running, paused, succeeded, failed, canceled."""

ExecutionStatus has eleven: those six plus preparing, waiting_input,
waiting_approval, retrying, expired. Steps add skipped on top.

Impact: anyone writing a client from that comment misses five states. Documentation
drift; a one-line fix.

⑦ The soit runs replay line in the console is display copy — that CLI does not exist.

From the run detail adapter (web/app/console/adapters/run-detail.ts:227):

ledger_code: {
  command: `soit runs replay ${run.id} --dry-run`,
  output: `replaying ${detail.steps.length} steps · verdict on record: ${run.status}`,
},

It renders as a code sample explaining what that panel shows.
But there is no soit CLI in the open-source repo — server/pyproject.toml has no
[project.scripts], and server/scripts/ has no matching entry point.

The thing that does work is the HTTP endpoint from section 3. There is a replay script in
the repo, but it is for the outbox (server/scripts/replay_outbox_event.py, 41 lines —
it returns one terminally failed domain event to the pending queue), which is a different
thing entirely.

I went back and forth on including this. Including it says we haven't kept our own
console copy honest. Leaving it out means a reader types the command from a screenshot and
gets nothing. Included, in the end — the gap between demo copy and real capability is
exactly the kind of thing a reader is entitled to know.

Coming clean

No fresh live run behind this piece. Every conclusion comes from reading soit/ at commit fb46f20, plus the tests already in the repo. I did not stand up an environment, execute a run and then call the replay endpoint. Every claim carries a file and a line number; go check them.
Item ④ in section 12 is an inference, not an observation. I did not construct the colliding-timestamp case. It is listed because a design should not depend on timestamp uniqueness, not because we have seen it break.
Replay does not promise identical output. Replay #4 genuinely runs again — models have temperature, tools talk to real systems, external data moves. What is promised is the same inputs, the same governance policy and complete evidence. There is no record-and-stub harness for tools in the repo.
This is all the community edition. Every path above is in github.com/soit-ai/soit and readable right now.
Disclosure: I maintain SOIT.

One-line version

"Replayable" here is not an adjective. It is a field the platform computes, that you can
query, and that can come back fail — backed by five database tables, four replay paths
with very different costs, and one status willing to admit we don't know whether the other
side did the thing.

Try it, and come argue

The repo is github.com/soit-ai/soit. To check the claims above:

Start it, send any message, take the run_id.
GET /api/v1/runs/{run_id} and look at replay_ready among the thirteen evidence items — if it is fail, missing names what is absent.
GET /api/v1/observe/runs/{run_id}/replay and see what the seven keys hold.

If any of the seven items in section 12 is wrong, open an issue and say so. I would rather
learn where this doesn't line up than be told the design is nice.

I counted our agent loop: 239 lines. Frameworks and runtimes are not the same layer.

Jude — Sun, 06 Sep 2026 23:39:03 +0000

The short version

	Agent framework	Agent runtime
Operates during	the days you write code	every execution after that
Core question	how should the agent think, compose, call	is this call allowed, is it written down, who pays for it
Typical surface	prompt composition, chain/graph authoring, model and tool wrappers, developer experience	permission checks, secret boundaries, egress policy, ledger and replay, cost attribution, approval interrupts
Judged by	expressiveness, time to first agent, ecosystem breadth	can you reconstruct what happened, can you stop it, can you replay it
Where it lives in our repo	`modules/agent/runtime/` (planner + executor + verifier)	`kernel/ports/`, `kernel/runtime/`, `kernel/security/`
Size	239 lines	the policy gateway on one port alone is 475

The last two rows are what this post argues, and they are not a diagram I drew. They are wc -l.

Why this needs a whole post

"Agent platform" currently swallows at least three separate things: the library you write
an agent with, the engine that runs it, and the control plane that governs it. Once
those three share a word, every discussion becomes two people answering different questions —
one is saying "I built an agent in three lines", the other is saying "I need to know who
approved that call at 3pm yesterday". Both are right. Neither is talking to the other.

I am not going to characterize anyone else's internals. That would require me to have read
their code closely enough to be accountable for the claim, and every claim in this post has to
come with a line number. So the post does two things:

describes what the framework layer generally owns — consensus, no assertions about anyone;
uses our own repo as the specimen to locate what the runtime layer owns.

The specimen is github.com/soit-ai/soit, Apache 2.0. Every
line number below refers to commit 3a57ae1.

1. Counting our own agent loop: 239 lines

An agent loop is three things: decide the next step, do it, check whether it is done. In our
repo those are three files:

server/app/modules/agent/runtime/planner.py    98 lines
server/app/modules/agent/runtime/executor.py   42 lines
server/app/modules/agent/runtime/verifier.py   99 lines

The executor is 42 lines and nearly all of it fits here:

class AgentExecutor:
    """Execute tool actions for agent."""

    def __init__(self, tool_port: ToolPort):
        self.tool_port = tool_port

    async def execute_tool(
        self, tool_ref, parameters, ctx, run_id, tool_call_id,
        idempotency_key, run_step_id=None, resume_approval=False, lease_owner=None,
    ) -> ToolResponse:
        """Execute tool call."""
        return await self.tool_port.invoke(
            tool_ref=tool_ref, parameters=parameters, run_id=run_id,
            tool_call_id=tool_call_id, idempotency_key=idempotency_key,
            run_step_id=run_step_id, resume_approval=resume_approval,
            lease_owner=lease_owner, ctx=ctx, strict_registry=True,
        )

This class has no logic. It hands the call to a port. That is not laziness; it is the
thesis of this post: the rules of execution are not written in the loop, they are written on
the port.

The planner is equally plain. It hands messages to the model using native function calling and
gets back either tool calls or text — there is no third case:

response = await self.llm_port.chat(
    messages=planning_messages, model=model, temperature=temperature,
    tools=tool_definitions if tool_definitions else None,
    tool_choice="auto" if tool_definitions else None,
    run_id=run_id, reasoning_effort=reasoning_effort,
)
if response.tool_calls:
    return PlanResult(action="tool", tool_calls=response.tool_calls, ...)
return PlanResult(action="respond", response=response.text or "", ...)

No prompt template DSL. No chain or graph authoring language. No regex pulling Action: out of
model output. The verifier is the same shape: one structured-output tool definition
(verify_response, fields ok and reason) asking the model whether the answer is adequate.

This layer is the framework's home turf, and we deliberately built almost nothing here.
Thin is neither a virtue nor a flaw. It just means we are not competing on expressiveness.

2. So what are the 1,617 lines wrapped around it doing?

server/app/modules/agent/application/service.py is 1,617 lines. Its import block answers the
question on its own:

from app.kernel.identity.guard import workspace_guard
from app.kernel.ports.approvals import ApprovalLedgerPort, ApprovalRecord
from app.kernel.ports.common.rate_limiter import RateLimiter
from app.kernel.runtime.runs.tool_calls import RuntimeToolExecutionService, ToolExecutionCommand
from app.kernel.runtime.runs.writer import TraceWriter
from app.kernel.runtime.tools.approval import tool_approval_rule
from app.kernel.runtime.tools.resolver import ToolResolver

Workspace guard, approval ledger, rate limiter, tool-execution ledger (with leases and
idempotency), trace writer, approval rules, tool resolver. Plus a dedicated control signal for
human approval:

class _AgentApprovalInterrupt(Exception):
    """Internal control signal for a durable human approval checkpoint."""

The loop itself is one while (service.py:653):

while pending_tool_calls or iterations < data.max_iterations:

One line of loop, sixteen hundred lines around it. Almost none of those lines are about how
the agent thinks. They are about whether this call is allowed, whether it is written down, who
it is billed to, and how to resume when a human interrupts it.

If nine tenths of a codebase answers the second set of questions, calling it "another agent
framework" describes the least important tenth of it.

3. Where the line actually is: ten gates on one tool call

server/app/kernel/ports/tools/interface.py is 54 lines: an abstract ToolPort with a single
abstract invoke(). That is the seam.

The thing that does the work is ToolPolicyGateway in
server/app/kernel/ports/tools/policy.py, 475 lines, implementing that same ToolPort. Which
means calling a tool raw and calling a tool through governance look identical to the caller —
the only difference is which implementation got injected.

ToolPolicyGateway.invoke() starts at policy.py:231. One tool call passes, in order:

#	Gate	What it does	Where
1	Secret resolution and redaction	Secret references in the parameters are resolved through `secrets_port`, and a redacted copy is produced for everything downstream that records	`policy.py:253–256`
2	Ledger claim	The call is claimed in the run ledger; `tool_call_id` and an idempotency key are minted or reused	`policy.py:264–290`
3	Idempotent replay	If the claim comes back `replayed`, the cached response is returned and the call is not made again	`policy.py:291–296`
4	Lease	The execution takes a lease of `max(60, ceil(timeout) + 10)` seconds, renewed while it runs	`policy.py:281`, `policy.py:326`
5	Egress policy	Every http URL found anywhere in the parameters goes through `check_egress_policy`	`policy.py:307–309`
6	Rate limit	Per-minute limit keyed on `tool_ref` + tenant + workspace + user	`policy.py:311–318`
7	Daily quota	86,400-second window keyed on `tool_ref` + tenant + workspace	`policy.py:319–324`
8	Tracing	A `soit.tool.invoke` OTel span carrying tenant, workspace, run and step attributes	`policy.py:336–348`
9	Timeout and retry	Shared timeout/retry; with an idempotency key, `max_retries=1`, and the comment says why	`policy.py:349–360`
10	Audit and settlement	Audit log (written with the redacted parameters), step status and metrics, cost	`policy.py:367–383`, `410–425`

Watch how gate 1 and gate 10 cooperate: real values only ever reach the call; the redacted
copy is the only thing that reaches the record. The code carries resolved_parameters and
redacted_parameters side by side precisely so that audit and metrics can never accidentally
take the wrong one. That is not a discipline you can maintain by being careful in application
code. It only works if it sits on the path everything must take.

Gate 9's comment is the tell:

# Durable Agent calls are at-most-once at this boundary.
# Not every downstream adapter can honor an idempotency key.

That is a runtime-layer concern in one sentence. It does not care whether your agent logic is
elegant. It cares whether a retry files the same ticket twice.

4. The fact that proves governance does not live in the loop

At this point you could reasonably say: you just pushed the governance code downstream of your
agent loop, that proves nothing.

So look at a second execution model. Besides the agent loop, the repo has a workflow engine —
modules/workflow/, 6,269 lines, including a 969-line engine.py and an 855-line
executor.py. It is a DAG. It has nothing structurally in common with an agent loop.

How does it call a tool?

# server/app/modules/workflow/runtime/executors/tool.py:428
response = await context.tool_port.invoke(...)

# server/app/modules/workflow/runtime/executors/llm.py:105
response: ChatResponse = await context.llm_port.chat(...)

The same tool_port.invoke. The same llm_port.chat.

Two unrelated execution models, one set of gates. That is the operational definition of a
layer: governance is a property of the port, not of any particular loop. Swap the execution
model above and not one of the ten gates below goes away.

Which is also the technical reason the two layers do not conflict: anything that calls
through these ports gets governed — our loop, a DAG engine, or something else entirely. The
layer underneath cannot tell the difference and does not need to.

5. The dependency list is the most honest positioning statement a project has

What a project says it is, you read in the README. What it actually is, you read in its
dependencies.

In server/pyproject.toml, exactly three core dependencies relate to agents at all:

"mcp>=1.28.1,<2",          # tool protocol
"ag-ui-protocol==0.1.19",  # front-end interaction event protocol
"litellm==1.91.1",         # model invocation

All three are protocols or call layers. There is no agent framework in the core
dependencies. That is not a manifesto, it is pyproject.toml lines 68 to 71.

And LangChain? It is there. Here:

[project.optional-dependencies]
local-embedding = [
    "sentence-transformers>=4.1.0",
    "langchain-huggingface>=0.0.6",
    ...
]

pyproject.toml:75–83 — an optional extra named local-embedding, for running embedding
models locally. Nothing to do with agent orchestration.

While I am here, a correction about us. Our README's tech-stack table lists the LLM row as
OpenAI · Anthropic · DeepSeek · Qwen · LangChain (adapter layer) (README.md:240). That does
not match the dependencies. LangChain is not an LLM adapter layer here; it is an optional local
embedding dependency. That is our documentation misleading readers, and I intend to open an
issue to fix it — I had not filed it when this was written, so there is no link to give you.

6. The seams are other people's protocols, not shapes we invented

A layer boundary is only real if the seam is public. Three seams:

Requests in. POST /api/v1/responses accepts AG-UI's RunAgentInput directly:

# server/app/api/v1/responses/router.py:222
async def create_response(payload: RunAgentInput | ResponseCreateRequest, ...):

A front end does not have to learn a SOIT-specific message shape.

Tools in. Tool references are namespaced strings; adapters/tools/router.py shows three
prefixes — tool:http:*, tool:function:*, and mcp_tool:*. Any MCP server resolves into the
tool registry without a code change.

Events out. The run is persisted and streamed to the front end as AG-UI interaction events
(adapters/agui/agent.py and responses.py, 909 lines together).

Three protocols, no dialect of our own. That is the precondition for two layers being able to
snap together at all.

7. We did not write a framework at the model layer either

adapters/llm/ is 2,619 lines, of which router.py is 534. Those 534 lines do routing,
credential resolution and egress guarding — not prompt composition:

# server/app/adapters/llm/router.py:192 (inside _authorize_provider_target)
await self.egress_guard.authorize(ctx, f"model-provider:{provider_slug}", url)

Both provider-resolution paths (router.py:368 and router.py:422) go through it first. In
other words, calling a model is itself subject to egress policy. If the target host is not
in policy, the call does not leave the box.

One more that is easy to miss: in production, a provider with no configured credential is
rejected outright (router.py:316–324, MODEL_PROVIDER_CREDENTIAL_REQUIRED). "Production mode
refuses to let you cut corners" is a runtime-layer job description. It does nothing for your
developer experience. It exists to stop development-time convenience from reaching production.

8. The boundary is welded shut by CI, not asserted in a doc

Layering usually dies by being true in the documentation and false in the code. So we handed
this one to a tool. The first contract in server/importlinter.ini:

[importlinter:contract:kernel_isolation]
name = Kernel is isolated
type = forbidden
source_modules =
    app.kernel
forbidden_modules =
    app.api
    app.modules
    app.adapters
    app.infra
    app.wiring

The kernel may not import anything above it. The moment the governance kernel depends
backwards on a product module, "swap the execution model and every gate survives" stops being
true. That is not something to leave to good intentions; let CI hit the wall instead.

server/app/kernel/README.md states the rule in prose, and one line of it is worth quoting:

Kernel extension points that need product or infrastructure data must use
provider interfaces registered from wiring/.

When the kernel needs outside data it does not reach for it; it takes a provider interface
registered in wiring/. That is the standing cost of keeping a layer boundary alive.

9. The honest part: those 6,269 workflow lines do overlap

So far I have made us sound very tidy: we only do runtime, not framework.

That is not quite true.

modules/workflow/ is 6,269 lines, with a compiler (compiler.py, 259), variable resolution
(variable_resolver.py, 212), node executors (the tool node alone is 610), resume and reaper
paths. It is an orchestration engine, and it overlaps in function with orchestration
frameworks people already use.

I am not going to argue that ours is somehow a different species. It overlaps. The only
distinction is the one from section 4: it calls tools through the same tool_port.invoke.

So the accurate statement is not "we don't build framework things". It is: we built the
minimum of it we needed, and we made it obey the same rules as everybody else.

10. What "not competitors" concretely means — including what you cannot do today

Three things you can do, and one you cannot.

You can:

Bring framework-side tools in over MCP and have them governed. Any MCP server resolves into the tool registry, and from then on every call it makes goes through the ten gates.
Bring framework-side services in as HTTP plugins. adapters/plugins/http_runtime.py and skill_runtime.py are the two plugin runtime implementations (129 and 85 lines).
Keep your front end. Interaction is an AG-UI event stream, and the request shape coming in is AG-UI's RunAgentInput.

You cannot (and this matters more than the three above):

There is no entry point today for handing us a framework-written agent to host wholesale.
Your loop still runs in your process. What SOIT governs is the hand it reaches out with —
tool calls, model calls, egress — not the reasoning inside it.

That is the real boundary right now. Do not size it up as a universal container. If what you
want is "host my existing agent code as-is and get the whole governance stack for free", this
repo does not do that today. What it does is make every hand that code reaches out with sign
its name.

Confession

This is a read-the-code and read-the-config piece with no new end-to-end run. Every claim can be opened and checked at commit 3a57ae1; whether it behaves this way at runtime is outside what this post verified.
All line counts are wc -l, including blanks, comments and docstrings. Fine for orders of magnitude, wrong for estimating effort.
The README inaccuracy in section 5 is real. Our own documentation misled readers about this. I intend to file an issue; it was not filed when this was written.
I made no claims about any specific framework's internals. "What the framework layer owns" here is the industry-consensus description. Everything with a line number is our own repo.
SandboxToolPort is not a security sandbox. The 61 lines in kernel/ports/tools/sandbox.py are a dry run for pre-release rehearsal: the run exercises the full decision path while the side effect is stopped at the boundary, so rehearsing a release does not actually file a pile of tickets. It stops side effects, not hostile code.
The section 4 argument has a precondition. "Swap the execution model and every gate survives" holds only for execution models that call through the ports. Code that opens its own socket is not governed by any of this — the import contract in section 8 keeps the kernel clean; it does not stop application code from going around.

One sentence

A framework decides how the agent thinks. A runtime decides whether the hand it reaches out
with counts.

These do not compete, because they do not even operate on the same timescale — one acts on the
few days you spend writing code, the other on every execution afterwards.

If they compete for anything, it is attention. Somewhere on the path from demo to
production, a team's attention has to move from the first to the second, and it usually moves
too late — as in, after the first incident.

Try it, or take it apart

Code at github.com/soit-ai/soit, Apache 2.0. Every line
number in this post refers to commit 3a57ae1.

Three files are worth opening, because they carry the entire argument:

server/app/modules/agent/runtime/executor.py — 42 lines, see how empty it is
server/app/kernel/ports/tools/policy.py — 475 lines, the ten gates
server/importlinter.ini — how the boundary is welded shut

If you think the section 4 argument has a hole in it, or you have seen a better way to draw this
line in another project, say so in an issue. The failure mode for a post like this is talking
to myself.

Disclosure: I maintain SOIT.

Your agent request touches 12 containers. Only one of them runs a model.

Jude — Wed, 02 Sep 2026 16:13:18 +0000

In the last post I cut this stack down to four long-running containers. The most common follow-up was not "how did you cut it" but the inverse:

So what are the rest of them actually for?

That deserves a straight answer. When a self-hosted project's quickstart starts a dozen containers, the default reading is either "the architecture never converged" or "someone is cosplaying enterprise." Neither reading is unfair — plenty of projects earn it. So this post does exactly one thing: it walks a single agent run from arrival to completion, and every time the run touches a service, says what that service did at that moment and which guarantee disappears without it.

Here is the least intuitive part of the answer up front: of the twelve containers, exactly one runs a model. Everything else buys the same thing — turning "run an agent" into "run an agent such that afterwards you can audit it, reconcile it, and replay it, and a crashed process does not leave half a state behind."

The short answer

Service	Where it sits in a run	What it carries	What you lose without it
`postgres`	throughout	the ledger: run / step / tool_call / artifact / cost	the physical basis for observability and replay; hard readiness gate
`redis`	at authz, and on cross-instance broadcast	permission cache (5-minute TTL), rate limiter, cross-instance event bus	replicas stop seeing each other's events; every authz check hits the DB
`minio`	when a run produces something large	artifact bytes; the DB keeps only `storage_key` + `sha256`	hard readiness gate; nowhere to put outputs
`milvus`	the retrieval step	vector store	retrieval calls fail (but readiness stays green)
`etcd`	never directly	Milvus's own metadata store, not the platform's dependency	goes wherever Milvus goes
`vault`	when a secret is needed	KV v2 store; credentials stay out of the process and out of `.env`	falls back to an in-process store, lost on restart
`migrate` / `bootstrap`	before the run	one-shot: schema, first admin, first tenant	— (they exit)
`minio-init`	same	one-shot: create the bucket, disable anonymous access	— (it exits)
`api`	throughout	the one process that actually runs a model	it is the thing being demoed
`web`	throughout	the front end	same
`outbox-dispatcher`	after the response is sent	delivers events committed inside the run's transaction, with per-consumer checkpoints	events sit at `pending` forever; nothing downstream ever fires
`knowledge-ingest-worker`	unrelated to a run	document parsing and indexing	uploads never reach the knowledge base
`scheduler`	unrelated to a run	fires due schedules	⚠ the quickstart never starts it — see below

1. First, a correction: it is not 12, it is 14

The quickstart command names twelve services:

docker compose --env-file .env -f docker/docker-compose.yml up -d \
  postgres redis minio etcd milvus vault migrate bootstrap api web \
  knowledge-ingest-worker outbox-dispatcher

docker/docker-compose.yml defines fourteen. The two extras behave very differently:

minio-init is not in the command, but api declares depends_on: minio-init: service_completed_successfully, so Compose starts it anyway, it creates the bucket and runs mc anonymous set none, and exits. Thirteen containers actually start.
scheduler is not in the command and nothing depends on it — it never starts in the quickstart topology at all. That one gets its own section.

The discrepancy itself is trivial. What matters is that "how many services are defined" and "how many you actually run" are two different numbers, and arguments about whether a stack is heavy tend to conflate them.

2. Before the run: the two containers that exit

migrate and bootstrap are one-shot jobs — restart: "no", and a healthy result is Exited (0).

migrate runs sh scripts/migrate.sh once postgres is healthy. bootstrap runs scripts/bootstrap_admin.py once migrate has exited successfully, creating the first admin and tenant (admin@example.com / changeme123 / tenant default by default).

Note the dependency condition: service_completed_successfully, not service_healthy. "Finished, and succeeded" is a different claim from "came up," and getting it wrong gives you a stack where every container is present and the schema is half-applied. api waits on both of these completing before it boots.

If docker compose ps shows those two as Exited, that is the correct state, not a failure.

3. Authz: Postgres is the authority, Redis is a cache

The first thing a request hits is authorization. Redis appears here, but strictly as a cache — the authority is always Postgres.

From server/app/kernel/identity/permissions.py:

class PermissionCache:
    """Permission cache using Redis."""

    def __init__(self, redis_client: redis_async.Redis | None = None):
        self._redis: redis_async.Redis | None = redis_client
        self._redis_pool: redis_async.ConnectionPool | None = None
        self._cache_ttl = 300  # 5 minutes

Three things worth knowing:

The TTL is hard-coded at 300 seconds, not configurable. A permission change can therefore take up to five minutes to propagate everywhere unless something calls invalidate explicitly (it exists — pattern-matched scan_iter + delete).
Losing Redis degrades rather than fails. _get_redis() returns None when settings.redis_url is empty or contains "None"; the caller treats that as a cache miss and falls through to the database. No Redis means slower, not broken.
The same Redis backs the rate limiter (server/app/kernel/ports/common/rate_limiter.py), implemented as a Lua script: ZREMRANGEBYSCORE to drop the expired window, ZCARD to count, then ZADD + EXPIRE if under the limit. Sliding-window counting in a single eval, so there is no read-modify-write race.

So "can I drop Redis?" resolves to: in a demo yes, in production no — and the reason is not the cache, it is the event bus in section 8.

4. The run gets written down: five ledger tables

Authorization passes, the run starts. This is the bulk of what Postgres carries, and it is the heaviest single piece of design in the stack.

One execution writes to five tables:

Table	One row is	Notable columns
`runs`	one execution	`status` / `trace_id` / `request_id` / `parent_run_id` / `source_run_id` / `attempt_no` / `sandbox`
`run_steps`	one step inside it	`step_type` (llm / retrieval / rerank / tool / workflow_node / agent_plan / memory_write / io) / `metrics_json`
`run_step_tool_calls`	one tool call	`idempotency_key` / `request_hash` / `lease_owner` / `attempt_count`
`run_artifacts`	one produced artifact	`storage_key` / `sha256` / `size_bytes` / `mime`
`run_cost_entries`	one metered invocation	`billed_quantity` / `amount` / `currency`

A few columns explain why this is not just two lines in a log file:

parent_run_id / source_run_id / attempt_no (server/app/kernel/runtime/db/models/runs.py). The first is parent-child; the other two are a retry and replay lineage — which run this one was derived from, and which attempt it is. That is what makes "replayable" a mechanism rather than a slogan: a replay is a new run pointing back at its source, not a re-read of a log.
sandbox. Marks a run as a rehearsal rather than real work. The field's own comment is blunt about why: pre-release regression executes real agents, and without the flag their cost and evidence inflate real activity.
input_summary / output_summary capped at 8KB, with metrics_json as a JSON column. The ledger stores summaries; the full payload lives in object storage behind run_artifacts. That split is deliberate — the relational store holds queryable structure, object storage holds bulk.

5. Why tool calls get their own table

run_step_tool_calls is the most heavily constrained of the five. It carries three unique constraints:

UniqueConstraint("tenant_id", "workspace_id", "run_step_id", ...)
UniqueConstraint("tenant_id", "workspace_id", "run_id", "tool_call_id", ...)
UniqueConstraint("tenant_id", "workspace_id", "idempotency_key", ...)

plus an index on ("status", "lease_expires_at"). Together they say one thing: tool calls have side effects, so they have to be at-most-once. One step maps to one tool call; a tool_call_id cannot land twice within a run; the idempotency key is globally unique — that key is the tool:{run_id}:{tool_call_id} from the earlier governed-MCP post.

The lease_owner / lease_expires_at pair is crash recovery: a worker that dies stops renewing, and the row becomes claimable again once the lease expires. This semantic is factored into a shared module (server/app/kernel/runtime/common/lease.py) whose docstring is explicit that every runtime domain executing work outside a request must use the same claim / renew / orphan-recovery primitives. Two constants and one implementation detail are worth remembering: MIN_LEASE_SECONDS = 30 (a smaller configured value is clamped up), LEASE_RENEWALS_PER_LEASE = 3 (the heartbeat interval is a third of the lease), and claims use SKIP LOCKED so concurrent workers do not contend — the comment openly notes SQLite ignores the clause, which is fine for single-worker tests.

This section is the whole article in miniature: these containers exist not because AI is complicated, but because "side-effecting operations must happen exactly once" is expensive in a distributed system, and always has been.

6. Retrieval: what Milvus does, and why etcd tags along

If the run includes a retrieval step, api queries Milvus. The adapter is server/app/adapters/vector/milvus.py, and the index parameters are fixed:

index_params={"index_type": "IVF_FLAT", "metric_type": metric_type, "params": {"nlist": 1024}}

etcd deserves an explicit correction, because it is the container most often misread as padding:

milvus:
  environment:
    ETCD_ENDPOINTS: etcd:2379
    MINIO_ADDRESS: minio:9000
  depends_on:
    etcd: { condition: service_healthy }
    minio: { condition: service_healthy }

etcd is not the platform's dependency; it is Milvus's. Milvus standalone keeps its metadata in etcd and its data files in MinIO. No application code in the repo talks to etcd at all. The honest way to read the topology is therefore: "vector retrieval" is one capability that costs two and a half containers (etcd + Milvus, sharing MinIO). Whether a demo should pay that is a clear trade-off, not a mystery.

One fact carried over from the previous post, because it matters here: the vector store does not gate readiness. In server/app/api/v1/health/router.py, the database and object storage raise 503 when probing fails; the vector store is probed and reported only:

try:
    await vector.check_ready()
    vector_status = "connected"
except Exception:
    vector_status = "unavailable"

The docstring gives the reasoning: non-vector endpoints keep serving during a vector outage, so pulling the instance out of rotation would be an overreaction.

7. Where the big objects go: MinIO

run_artifacts stores storage_key, sha256, size_bytes, mime — not the content. The content is in MinIO.

The one-shot minio-init container does two things: mc mb -p local/soit-artifacts to create the bucket, then mc anonymous set none to close anonymous access. The second is a small correct default: the artifact bucket is not anonymously readable out of the box.

Object storage is a hard readiness gate (probe fails → 503). This is exactly where the previous post's live run broke: on paper you can swap in the local-filesystem adapter, and in practice it does not work inside the official image because the root path is strip("/")-ed into a relative path (filed as issue #43). So the practical verdict stands: MinIO cannot be dropped.

8. The part that starts after the response is sent

By now the run is finished and the response has gone back to the caller. One container is only now getting to work.

api writes domain events into the event_outbox table inside the same transaction as the business data (server/app/kernel/runtime/db/models/events.py). That is the transactional outbox: if the business change committed, the event exists; if it rolled back, the event does not. There is no window where the database changed but the message never went out.

outbox-dispatcher then polls that table as its own process. The row's columns are effectively its state machine: status, available_at, locked_at, lock_owner, lock_expires_at, attempt_count, last_error, processed_at.

The module docstring of server/app/kernel/events/dispatcher.py states the whole flow in one line:

claim rows, run registered handlers with checkpoint idempotency

"Checkpoint idempotency" is a second table, event_consumer_checkpoint, unique on (consumer_name, event_id). Before dispatching, the service asks checkpoints.is_processed(consumer_name, event_id); on success it calls try_record_success. So idempotency is per consumer per event, not per event — if the second of three handlers fails and the row is retried, the first is not re-executed.

That container also exposes its own Prometheus endpoint (expose: 9201, started via start_http_server), and its healthcheck is a scrape of /metrics.

9. Four of these containers are the same code as `api`

This is the part most easily mistaken for microservice sprawl, and it is the opposite. migrate, bootstrap, api, outbox-dispatcher and scheduler all share one build context (build: context: ../server). The released-image path makes it starker: docker/docker-compose.images.yml points migrate, bootstrap, api and outbox-dispatcher at the same image, ghcr.io/soit-ai/soit/server. Only knowledge-worker and web are separate. Three images cover twelve containers.

The difference is the entrypoint, plus which background loops are switched on. The lifespan in server/app/main.py has six flags, each folding one loop into the API process:

Flag	Code default	Loop it folds in
`workflow_orphan_reaper_enabled`	`False` (compose sets `true` for `api`)	reaping orphaned workflows
`schedule_worker_enabled`	`False`	firing due schedules
`account_deletion_sweeper_enabled`	`False`	account deletion sweep
`knowledge_ingest_worker_enabled`	`False`	knowledge ingestion
`outbox_dispatcher_enabled`	`False` (compose hard-codes `"false"`)	outbox dispatch
`response_interaction_worker_enabled`	`False`	durable chat interactions

Which means "how many containers" is largely a deployment decision, not an architectural one. The same code can run as one process or five. Compose splits them, and server/scripts/schedule_worker.py states the reason more plainly than I could:

Separate from the API for the same reason the outbox dispatcher is: a scheduler that shares a process with request handling competes with it, and an API restart should not be a gap in when jobs fire.

10. Production mode refuses to let you cut corners

Those flags look like a matter of taste. Half of them are not, once ENVIRONMENT=production. validate_runtime_requirements() in server/app/settings/settings.py fails closed on each of:

the database URL must carry host, database name, username and password;
the event bus must be redis (the code default is actually memory; compose supplies redis);
outbox_dispatcher_enabled being true is an error — production forbids folding the dispatcher into the API process, it has to be its own;
inline chat-interaction execution is forbidden, and the durable interaction worker is required;
plugin signature verification is required, and at least one public key must be configured — the comment explains why that is checked separately: requiring signatures with no trusted key rejects every package, which reads as a gate but is really a total block.

This is the part I'd most want a skeptical reader to notice. Which services are optional is not an opinion in this repo; it is code that refuses to boot. You may drop Redis and fold the dispatcher into the API for a demo. You cannot do that and also claim to be running production.

One gap I found while writing this

One thing I turned up is not a trade-off, it is a hole:

docker-compose.yml defines a scheduler service that sets SCHEDULE_WORKER_ENABLED: "true" and runs scripts/schedule_worker.py;
the quickstart command does not include it, and nothing depends_on it;
the api container does not set SCHEDULE_WORKER_ENABLED, and the code default is False;
.env.example does not mention the variable;
docs/ never mentions scheduler at all — across the whole repo only the two compose files do.

Net effect: in a stack started per the quickstart, schedules never fire on their own. POST /schedules creates one, the preview endpoint tells you when it would next run, and POST /schedules/{id}/run triggers it by hand — but nothing is polling to claim it when its time comes.

docker-compose.production.yml does include a scheduler, so this is a quickstart coverage gap rather than a missing feature. There is a second-order problem too: the released-image overlay covers six services and scheduler is not one of them, so adding a scheduler to the images-based path silently falls back to a local build.

I plan to file an issue for both halves of this: add scheduler to the quickstart command, and cover it in the released-image overlay. It was not filed yet when I wrote this, so there is no link here — if you want to confirm it yourself, walking the four bullets above in order is enough.

What this post is not

There is no new end-to-end run behind it. The minimal-topology post was executed end to end; this one is a static read of the code and compose files. So the scheduler finding above rests on a code-and-config chain of evidence (service never started + flag defaults to False + no documentation) — I did not stand up the quickstart, create a schedule, and watch it fail to fire. Falsifying it is easy if you want to: start the quickstart stack, create a schedule one minute out, and see whether it runs.
No resource numbers. This is about responsibilities, not footprint. The image-size figures are in the previous post.
Nothing about wiring it into your existing observability. OTEL_ENABLED (default false) and an OTLP endpoint are in compose, and the production file ships an otel-collector, but that is its own article.
The Redis conclusion is conditional. "Fine to drop in a demo" holds because a demo runs a single api replica. Add replicas and the in-process bus stops crossing processes. That is what the production check is protecting.

So where is the weight?

Regroup the twelve by what they guarantee:

2 are the thing being demoed: api, web.
3 are one-shot jobs: migrate, bootstrap, minio-init — they exit.
2 are where the ledger and the artifacts physically live: postgres, minio — also the only two hard readiness gates.
2½ are one capability, vector retrieval: milvus + etcd (etcd being Milvus's dependency, not ours).
1 is secret isolation: vault.
1 is cross-replica broadcast and caching: redis.
2 are background processes split out of the same codebase: outbox-dispatcher, knowledge-ingest-worker.

Exactly one of them runs a model. The rest of the weight buys one thing: the execution leaves evidence behind, and a crash mid-run does not leave half a state.

Whether that is worth it depends entirely on what you are doing. If you just want to see whether an agent runs at all, this stack is too heavy for you — the previous post shows how to get it to four containers. If you need to put an agent inside a process someone will later have to reconcile, these containers are the things you would end up writing yourself.

Try it

Repo: github.com/soit-ai/soit
Full topology: docker/docker-compose.yml
The four-container version: I tried to cut our 12-container stack down to 4
If you think any one of these trade-offs is wrong, open an issue and say so.

Disclosure: I maintain SOIT.

I tried to cut our 12-container stack down to 4. Two of my three conclusions were wrong.

Jude — Sat, 29 Aug 2026 16:10:58 +0000

Self-hosted projects lose most of their prospective users at the first command in the README. Here is ours:

docker compose --env-file .env -f docker/docker-compose.yml up -d \
  postgres redis minio etcd milvus vault migrate bootstrap api web \
  knowledge-ingest-worker outbox-dispatcher

Twelve service names. Issue #25 asked the obvious question: I just want to look at it — do I really need all of them?

No. This post takes the twelve apart: which ones genuinely cannot go, which group leaves as a unit, which one you fold into the API process instead of deleting, and what breaks with each removal.

More importantly: I actually ran the trimmed stack. That matters, because two of the three conclusions I had drawn from reading the code turned out to be wrong. Had I published the paper version, you would have followed it and ended up with a stack that starts and then refuses to show you a UI. I left both mistakes in, because they carry more information than the correct answers do.

The short answer

Service	Can it go?	What it costs you
`postgres`	No	Hard readiness gate — 503 without it
`minio` + `minio-init`	No (see section 2)	On paper the local filesystem replaces it. In the published image that does not work.
`migrate` / `bootstrap`	No	One-shot jobs: schema and the admin account
`api` / `web`	No	They are the thing being demoed
`milvus` + `etcd`	Yes, as a group — with a side effect	Vector search raises at call time, and readiness gets slow enough to mark the API unhealthy
`vault`	Yes	Secrets move to an in-process store, gone on restart
`redis`	Yes, for a demo	No permission cache, single-process event bus
`knowledge-ingest-worker`	Yes	No document ingestion
`outbox-dispatcher`	Fold it in	One environment variable moves it into the API

Twelve becomes seven, and since minio-init, migrate and bootstrap exit when they finish, four containers stay running: postgres, minio, api, web.

What you save is Milvus (a 2.6GB image), etcd, Vault, Redis, the ingestion worker and the outbox container.

1. The readiness endpoint tells you which dependencies are real

The fastest way to find out whether a dependency is hard is not the deployment guide — it is the health check. A deployment guide documents intent; a health check documents behaviour.

In server/app/api/v1/health/router.py, three backends are treated differently:

try:
    db.execute(text("SELECT 1"))
    db_status = "connected"
except Exception:
    raise HTTPException(status_code=503, detail="Database is unavailable")

try:
    await storage.ensure_ready()
    storage_status = "connected"
except Exception:
    raise HTTPException(status_code=503, detail="Object storage is unavailable")

try:
    await vector.check_ready()
    vector_status = "connected"
except Exception:
    vector_status = "unavailable"

Database or object storage down means 503. Vector store down means the field reads unavailable and the endpoint still returns 200. The docstring gives the reason: the platform degrades gracefully when the vector store is down, so a vector outage should be surfaced rather than pull the instance out of rotation.

That leaves two hard requirements: a reachable Postgres, and a writable storage root.

The second one says storage root, not MinIO — and I assumed that distinction meant MinIO could go. That is wrong conclusion number one.

2. Wrong conclusion #1: local filesystem storage does not work inside the image

The storage adapter is built on fsspec (server/app/adapters/storage/fsspec.py), and the base URL resolves in this order:

self.base_url = base_url or settings.storage_url or self._default_local_base_url()

_default_local_base_url() returns a file:// URI under the repository root. So point STORAGE_URL at a local directory (or leave it unset) and storage should land on disk with no object store at all.

I configured exactly that, and the API came up returning 503:

{"success":false,"code":"SERVICE_UNAVAILABLE","message":"Object storage is unavailable"}

Constructing the adapter directly inside the container gave the real error:

PermissionError: [Errno 13] Permission denied: '/app/home'

/app/home is a strange path, given that I passed /home/appuser/soit-storage. The cause is this function:

@staticmethod
def _normalize_root_path(root_path: str) -> str:
    return root_path.replace("\\", "/").strip("/")

strip("/") removes the leading slash too, so the absolute path /home/appuser/soit-storage becomes the relative path home/appuser/soit-storage, which fsspec's LocalFileSystem then resolves against the process working directory. The image sets WORKDIR /app/, so it lands in /app/home/....

And /app is not writable: server/Dockerfile does COPY ./ /app/ without --chown, leaving it owned by root, while the final instruction is USER appuser (uid 10001).

So inside the published image the local-filesystem path is effectively dead: whatever you pass ends up under /app, where a non-root process cannot create directories. Mounting a volume does not rescue it either — Docker creates the mount point owned by root as well. Making it work would mean running the API as root or pre-chowning a mount, and neither belongs in a guide aimed at people trying the project for the first time.

So MinIO stays. It is cheap, at least: one long-running container plus a minio-init that exits, on a couple hundred megabytes — an order of magnitude smaller than the Milvus group.

(For the record, MinIO wears two hats in the full topology: the platform's artifact store, and Milvus's object backend. It was never separable from Milvus anyway.)

3. The vector group leaves as a unit — and it is not a graceful degradation

milvus depends on etcd (metadata) and minio (data). Does the API still start without milvus and etcd? Yes, and the reason is in the adapter's constructor docstring (server/app/adapters/vector/milvus.py): the connection is established lazily on first use, so building the port during dependency injection does not fail when the vector store is unavailable.

But do not expect it to degrade into empty results, because the vector port has no environment-level fallback. From server/app/wiring/container.py:

def _create_vector_port(self) -> VectorPort:
    import os
    if os.getenv("PYTEST_CURRENT_TEST") or os.getenv("SOIT_TESTING") == "1":
        from app.adapters.vector.memory import InMemoryVectorPort
        return InMemoryVectorPort()
    from app.adapters.vector.milvus import MilvusVectorPort
    return MilvusVectorPort()

The in-memory implementation is reserved for test runs; unlike the secrets port, it never consults ENVIRONMENT. The real effect: the platform boots, non-vector features work, readiness honestly reports vector: "unavailable", and knowledge retrieval raises the moment you use it.

The readiness response from the actual run says exactly that:

{"status":"ready","database":"connected","storage":"connected","vector":"unavailable"}

That conclusion held. But the same run surfaced something the code does not show you, which is the next section.

4. Wrong conclusion #2: without Milvus, the web container never starts

That readiness response took 34 seconds to come back.

The reason is not hard to guess: vector.check_ready() has to resolve the milvus hostname and open a connection, and the container is not there, so every request waits out DNS and connect timeouts. The vector probe is fail-soft, but it is not fail-fast — nothing bounds how long it may take.

Which runs straight into Compose's own health check:

{"Test": ["CMD-SHELL", "python -c \"...urlopen('http://localhost:9200/health/ready', timeout=3)\""],
 "Interval": "10s", "Timeout": "5s", "Retries": 5}

The probe times out after 3 seconds, Compose gives it 5, and the endpoint needs 34. It cannot pass. In the run, the API container sat permanently at:

soit-api-1   Up 3 minutes (unhealthy)

The service itself is fine — I logged into it. Only the health check fails. But web declares depends_on: api: condition: service_healthy, so a normal up -d web means web never starts at all. You get a stack with a perfectly working API and no UI, and very little to tell you why.

The fix is small: pass --no-deps for web as well. It is a static frontend; it only needs the browser to reach the API, not Compose's opinion about the API's health.

docker compose ... up -d --no-deps web

Started that way, web comes up healthy and serves HTTP 200.

This is the one finding in this post that reading the code could never produce. On paper you get a guide that looks right and leaves you staring at a dead URL.

5. Vault genuinely does degrade

The secrets port is wired differently (same container.py):

if not settings.vault_url or not settings.vault_token:
    if self._allows_in_memory_adapters():
        from app.adapters.secrets.memory import InMemorySecretValueStore
        return InMemorySecretValueStore()
    raise RuntimeError("Production requires Vault URL and token for the secrets adapter")

_allows_in_memory_adapters() accepts ENVIRONMENT values dev / development / local / test / testing, and compose defaults to development. So leaving VAULT_URL and VAULT_TOKEN empty swaps in the in-process secret store and the Vault container can stay down. Verified in the run: migrate, bootstrap and api all worked with no Vault anywhere.

The cost is in the name: in-process means not durable. The model API key you configure during the demo is gone the moment the container restarts.

6. Redis is three different questions

Redis is interesting because it is not a binary. The three places that use it disagree about what its absence means.

The event bus can be switched. memory is the default; compose is what changes it to redis:

backend = (settings.event_bus_backend or "memory").lower()
if backend == "redis":
    return RedisEventBus(...)
if backend == "memory" and self._allows_in_memory_adapters():
    return InMemoryEventBus()

The same ENVIRONMENT guard applies — in production that branch raises. The in-memory bus only delivers within a single process, which is exactly why it pairs with folding background work into the API process.

The permission cache degrades gracefully. In server/app/kernel/identity/permissions.py the Redis accessor returns None when it cannot connect, and callers treat None as a cache miss and re-check against the database. One less cache layer, same answers.

Rate limiting is a hard dependency that usually never fires. RateLimiter (server/app/kernel/ports/common/rate_limiter.py) is a Redis sliding window with no in-memory equivalent. But the call sites are conditional (server/app/kernel/ports/tools/policy.py):

rate_limit = kwargs.get("rate_limit_per_minute") or self.rate_limit_per_minute
if rate_limit:
    await self.rate_limiter.check_rate_limit(...)
if self.daily_quota:
    await self.rate_limiter.check_rate_limit(...)

No configured limit, no Redis call. Dropping Redis from a demo is therefore safe as long as you do not configure per-tool rate limits or daily quotas. That is the one item here that depends on what you do during the demo.

7. One service you fold in rather than remove

outbox-dispatcher runs the transactional outbox. Its setting says exactly what the flag means (server/app/settings/settings.py):

outbox_dispatcher_enabled: bool = False
"""Enable background outbox dispatcher in the API process."""

The flag does not control whether dispatching happens — it controls where. Compose sets it to false and runs the same logic in a separate container. For a demo, invert it: set OUTBOX_DISPATCHER_ENABLED=true on the api service and skip the container. server/app/main.py reads the flag at startup and attaches the dispatcher to the API's lifespan.

Why production does the opposite: validate_runtime_requirements() contains if self.outbox_dispatcher_enabled: raise ValueError("Production requires the dedicated outbox dispatcher process"). Dispatching and request handling in one process compete for the same resources, and a restart interrupts both at once. Fine for a demo — and note that this is enforced by code, not advised by documentation.

8. Skip the ingestion worker unless you are demoing RAG

knowledge-ingest-worker builds from its own image target (server/Dockerfile) with one extra dependency group:

FROM base AS knowledge-worker
RUN --mount=type=cache,target=/root/.cache/uv \
    /bin/uv sync --frozen --no-dev --extra knowledge-worker

That extra is docling[rapidocr] — document parsing and OCR. While we are here, a common misconception: torch, torchvision and torchaudio live in the local-embedding extra in pyproject.toml, not in knowledge-worker, and not in the API image either. The worker is lighter than people assume — but if your demo never uploads a document, it has no reason to exist.

9. The commands, as actually run

One Compose trap first: api lists milvus and vault in depends_on, so Compose starts them for you even when you leave them off the command line. Every step needs an explicit --no-deps, and you sequence the one-shot jobs yourself.

The env file (all of these override compose defaults):

printf '%s\n' \
  'ENVIRONMENT=development' \
  'VAULT_URL=' \
  'VAULT_TOKEN=' \
  'EVENT_BUS_BACKEND=memory' \
  'OUTBOX_DISPATCHER_ENABLED=true' \
  > .env.minimal

Bring it up on the published images — overlay docker-compose.images.yml and pass --no-build, or Compose will build from source:

COMPOSE="docker compose --env-file .env.minimal -f docker/docker-compose.yml -f docker/docker-compose.images.yml"

$COMPOSE up -d --no-build postgres minio minio-init
$COMPOSE run --rm --no-deps migrate
$COMPOSE run --rm --no-deps bootstrap
$COMPOSE up -d --no-build --no-deps api web

migrate prints a run of alembic upgrades; bootstrap prints Bootstrap completed. along with the admin ids.

Then verify. Remember that readiness takes more than 30 seconds (section 4), so give curl a generous timeout:

curl -s -m 60 http://localhost:9200/health/ready

From the run:

{"status":"ready","database":"connected","storage":"connected","vector":"unavailable"}

vector: unavailable while the whole thing still reports ready is section 1's code path observed from the outside — the output is its own proof.

Then exercise a real path, not just the health endpoint:

curl -s -X POST http://localhost:9200/api/v1/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@example.com","password":"changeme123"}'

An access_token in the response means the database and auth path are both working. The UI is on http://localhost:5000 with the same credentials.

docker ps will show the API as unhealthy, and that is expected (section 4). The service is fine.

What this is not

As usual, the limits:

This is not a supported deployment shape. It is a demo trim. Set ENVIRONMENT=production and every shortcut above is closed off one by one: validate_runtime_requirements() demands the Redis event bus, the dedicated outbox process, Vault, OpenTelemetry, and plugin signature and digest verification. Missing any of them fails startup. That is deliberate fail-closed behaviour.
The API stays unhealthy, so do not hand this topology to anything that orchestrates on container health — Kubernetes probes, or start-up ordering that waits on a healthcheck, will both break.
In-memory means gone on restart — secrets, and any event in flight on the in-memory bus.
Without Milvus, vector features raise rather than return empty. Demoing knowledge bases means putting milvus and etcd back.
The rate-limit caveat is yours to judge: dropping Redis assumes no configured limits.
The default SECRET_KEY is change-me, and bootstrap will warn you about it (InsecureKeyLengthWarning). Harmless for a demo; do not let that value outlive one.

Two bugs found along the way

Writing this turned up two problems of our own. Both are filed, and it seems fair to say so here rather than quietly fix them:

_normalize_root_path() calls strip("/"), which turns absolute paths into relative ones, making the local filesystem storage backend unusable inside a container (section 2). That function is presumably meant to normalise object-storage key prefixes; backends like file://, where an absolute path means something, should not get the same treatment. (issue #43)
The vector readiness probe has no timeout, so a missing vector store drags /health/ready past 30 seconds and makes the Compose health check fail permanently (section 4). Fail-soft was implemented; fail-fast was not. A seconds-level timeout on check_ready() would give you both. (issue #44)

Worth noting: both are things you only hit by actually running a reduced topology, and our own CI runs the full one. Which is probably an argument for supporting the minimal shape officially.

Why write this down at all

If four containers are enough, why does the default ask for twelve?

Because the default topology targets the production shape, not the demo shape. Every service cut above maps to a requirement that production enforces in code: secrets need a real secret manager, events need to cross process boundaries, dispatching needs to scale independently, vectors need to persist. You get something you can experiment against as if it were production, and the price is a first command that looks frightening.

The point is that the distance between those two shapes is measurable in a handful of environment variables — and measuring it happens to be the fastest way to understand the architecture: the health check tells you the hard dependencies, the wiring code tells you which ports have fallbacks, and validate_runtime_requirements() tells you where production draws its line.

But keep the other lesson too: reading the code gives you hypotheses; running it gives you conclusions. Two of my three were wrong, and the wrong two were exactly the ones that would have stopped you.

Try it

SOIT is Apache-2.0 and the code is on GitHub:

Repository: github.com/soit-ai/soit
Full quickstart (the twelve-service path): docs/quickstart.md in the repo
Governance demo: docs/governance-demo.md — a 20-minute local run that walks through permissions, secrets, call auditing, cost attribution, replay and regression

If you get the minimal topology running, or get stuck on a step, open an issue and say so. Right now this trim only exists as a blog post; if the feedback says it is useful, we will turn it into a Compose profile so --profile minimal does the whole thing — and fix those two bugs on the way.

Disclosure: I maintain SOIT.

We open-sourced our agent platform's governance layer: hexagonal architecture and governed execution

Jude — Thu, 27 Aug 2026 13:57:32 +0000

We open-sourced SOIT a few weeks ago. The one
line version: an agent runtime and governance platform for teams that need AI
agents to touch real enterprise systems without losing control.

This post is about the part that is actually hard — not building agents, but
being willing to let them run in production.

The problem isn't building agents. It's trusting them.

Every team I talk to has the same story. The demo took a week and it was great.
Then security, compliance and ops started asking questions, and the project
parked itself at PoC:

What is this agent allowed to do, and who decided that?
Where do its credentials live, and what leaks if a prompt goes wrong?
Which hosts can it reach when a tool call makes an outbound request?
What exactly did it do last Tuesday, and can we prove it step by step?
What did that run cost, and which team pays for it?

Frameworks answer none of these — they orchestrate calls and leave controls to
you. Hosted platforms answer some, but you inherit their model choices, their
data boundary and their pricing. Cloud-vendor agent services answer more, in
exchange for the deepest lock-in of all.

Our answer is to make governance a kernel concern rather than a patch applied
afterwards.

Governed execution

The idea is simple to state: every agent run is a governed run. Chat turn,
agent loop, or workflow run — everything flows through one runtime ledger
(Run / RunStep / Trace), and the same controls apply on every path:

Permissions — tenant and workspace scoping on every resource, RBAC with resource-level grants. Every record carries tenant_id and workspace_id; there are no escape hatches.
Approved capabilities — agents bind to models, tools, knowledge bases and workflows through per-version allowlists. A tool from a plugin, an MCP server, or a built-in adapter looks identical to the agent and passes the same checks.
Secret boundaries — credentials live in Vault with workspace-scoped visibility and are injected at the gateway. Business code never opens a raw HTTP client or LLM SDK.
Egress policy — outbound HTTP from tools is policy-controlled. An agent cannot quietly call a host you never approved.
Ledger, cost, audit, replay — per-step tokens, latency and cost; a full audit log of privileged operations; a trace timeline you can replay step by step.
Separation of duties — the Dev role that builds and runs agents cannot change egress policy, secrets, or installed plugins. That takes a workspace Owner or Admin.

None of this is a wrapper around someone else's runtime. SOIT is a hexagonal
architecture: a stable kernel, versioned JSON Schema contracts on every
primitive, and replaceable adapters at the edges — so the governance layer holds
no matter which model or vector store you swap in.

That last property is the whole reason for the shape. Governance implemented at
the integration layer has to be rewritten for every new tool and every new
provider. Governance implemented at the port layer is written once, and every
adapter inherits it whether it wants to or not.

What's in the box

Four pillars, all of them in the open-source edition:

Build — visual agent assembly with versioning and release management, a DAG workflow editor, a knowledge pipeline (PDF/DOCX/Markdown/HTML into Milvus-backed retrieval), and MCP support: any Model Context Protocol server resolves into the tool registry without code changes, including OAuth 2.1-protected servers.
Execute — an outbox-based event-driven runtime with checkpoints, retries and fallback chains; multi-model routing across OpenAI, Anthropic, DeepSeek, Qwen and any OpenAI-compatible endpoint — including the one on your own GPU.
Observe — a workspace console built on the runtime ledger: live run volume, cost burn, failure rates, drill-down by agent, workflow and tool, plus OpenTelemetry tracing and Prometheus metrics.
Govern — everything in the section above.

It self-hosts with one command:

git clone https://github.com/soit-ai/soit.git
cd soit
cp .env.example .env
docker compose --env-file .env -f docker/docker-compose.yml up -d

A supply chain you can verify

If your agents run in production, so does your agent platform — which makes it
part of your attack surface.

Every SOIT release is built by a tag-triggered pipeline that publishes
digest-addressable images, SPDX SBOMs, Sigstore-backed build provenance and SBOM
attestations, a deterministic source archive, and SHA256SUMS. v1.0.0 shipped
that way: three images on GHCR that pull anonymously, and artifacts you can
check with gh attestation verify before they enter your environment — we ran
it ourselves and got exit 0.

Secret scanning, dependency audit and container scanning run in CI as gates, not
as dashboards.

What SOIT is not

A post like this should also say what you are not getting:

It is not a lightweight chatbot builder. If you want a prompt box and a share link, plenty of tools do that with far less infrastructure.
Content safety and PII detection are not implemented. SOIT exposes a content-safety port and an HTTP adapter so you can plug in a classifier you operate, and inspection outcomes become part of run evidence — but with no adapter configured, no inspection happens. I would rather say that than ship a checkbox that does nothing.
The project is young. v1.0.0 is released and the CI gates are in place, but there is no large-scale production deployment vouching for it yet. We run it ourselves. Early adopters welcome, with eyes open.

License: Apache 2.0

Commercial use, self-hosting, internal deployments and building products on top
are all unrestricted. The core platform is and will remain open source; SSO,
advanced audit reports and multi-region deployment live in SOIT Enterprise. The
commercial boundary is drawn in the feature set, not in the license.

Come break it

The repo is at github.com/soit-ai/soit.
The quickstart takes about ten minutes on a machine with Docker. There are
good first issues seeded and the roadmap is pinned.

If the architecture trade-offs are the interesting part for you — the outbox
runtime, the spec-first contracts, lease-based worker recovery — say so and I
will write those up separately. I also wrote a companion piece on what happens
when an MCP tool call has to obey RBAC, secrets and egress policy:
Five questions to answer before you put MCP in production.

If your agents graduated from notebooks and hit the trust wall, this was built
for you. Tell me where it falls short.

Disclosure: I maintain SOIT.

Five questions to answer before you put MCP in production

Jude — Thu, 27 Aug 2026 13:46:50 +0000

MCP standardized the boring part of giving an agent tools. One streamable HTTP
endpoint, one list_tools call, and the tools show up in the model's callable
list. The first time we wired one up internally it took an afternoon.

Then we tried to put it in production, and someone from security asked five
questions. I could not answer any of them.

This post is those five questions, and how we ended up answering them in SOIT —
an open-source agent runtime with governance in the middle of it. Everything
below points at a file in the repo, because posts like this are unusually easy
to write as a slide deck instead of as software.

1. Who is allowed to call this tool?

MCP does not have an opinion here. Whatever list_tools returns is what the
model can call. Visibility is capability.

In a multi-tenant, multi-workspace deployment that is not enough. SOIT installs
an MCP server as a plugin artifact rather than as a config entry. Tool
references are namespaced — mcp_tool:{server}:{tool}, parsed by
parse_mcp_tool_ref in server/app/adapters/tools/mcp.py — and every
resolution carries a RequestContext holding tenant_id, workspace_id and
user_id.

Two things follow from that:

From the agent's point of view, a tool from a plugin, a tool from an MCP server, and a built-in adapter all look identical. Bindings are typed and versioned.
Permission checks, secret injection, egress limits, audit, cost attribution, trace and replay apply to MCP tools automatically. Nobody writes the governance path twice.

Each agent version also carries a capability allowlist covering models,
knowledge bases, workflows, tools, plugins and MCP servers. So "which MCP tools
can v3 of this agent call" is something you can diff and roll back, rather than
a runtime toggle somebody flipped.

2. Where do the credentials live?

Most MCP integration examples look like this:

{
  "auth": { "type": "bearer", "token": "sk-xxxxxxxx" }
}

A plaintext token in a config file. It ends up in git. It ends up in logs. It
ends up in the config backup somebody exported to a laptop.

SOIT rejects this outright. _build_auth_headers checks for a token or
value field in the auth config and raises:

MCP credentials must use secret_id

Only secret_id is accepted, resolved through SecretsPort at call time. Same
for API keys — and they are only supported in headers, never in a query string,
because query strings leak through logs and referrers.

The real value exists in memory for the duration of the call and nowhere else.
What gets persisted — to the database, to audit records, to traces — is a
redacted copy: ToolPolicyGateway._resolve_secrets builds it in the same
pass that resolves the secret, keeping only secret_id and the signing policy
reference (server/app/kernel/ports/tools/policy.py).

Three auth types are supported: bearer, api_key, oauth2. OAuth follows
2.1 with authorization-server discovery (RFC 9728, RFC 8414 / OpenID Connect)
and resource-bound tokens (RFC 8707), using the client_credentials grant.

One limitation worth stating plainly: the browser-based authorization_code
flow is not implemented. SOIT calls MCP servers on its own behalf, not on
behalf of a user sitting in front of a browser. If you need "call a protected
MCP server as the end user," this does not cover you.

3. Where can it connect to?

This is the one that should worry you most.

You deployed the MCP server, but it is a thing that makes network requests on
your behalf. Put a URL in the tool arguments and it will fetch it. The classic
shape of this is asking it for http://169.254.169.254/ — the cloud metadata
service, holding temporary credentials.

SOIT's egress policy is deny-by-default, in three layers.

Layer one: domain policy. check_egress_policy matches the target domain
against tenant-scoped and workspace-scoped allowlists and blocklists, with the
blocklist winning. The defaults are enable_egress_policy: bool = True and
egress_allowlist: list[str] = [] — an empty allowlist means nothing is
permitted until you say so. And if the policy lookup itself throws, the answer
is deny, not allow:

except Exception as exc:
    raise ForbiddenError(
        "Egress policy lookup failed; request denied",
        {"resource_ref": resource_ref},
    ) from exc

Fail-closed is not a slogan. It is whatever you actually wrote in each except
branch.

Layer two: per-address validation after resolution. Passing the domain check
is not enough — DNS rebinding lets an allowlisted hostname resolve to
127.0.0.1 or 10.0.0.x. So after the domain is allowed, GovernedEgressGuard
actually resolves the hostname and checks
ipaddress.ip_address(address).is_global for every address returned. One
non-public address and the whole request is refused
(server/app/kernel/security/egress.py).

Closed off in the same pass: non-http/https schemes are denied by default, URLs
carrying userinfo (https://user:pass@host/) are denied, and a DNS failure is a
denial rather than a retry.

Layer three: authorization per hop. A URL that cleared both layers returns a
302 pointing at your internal network. Now what? So the outbound HTTPX client is
built like this (server/app/adapters/http/governed_client.py):

async def authorize_request(request: httpx.Request) -> None:
    await guard.authorize(ctx, resource_ref, str(request.url))

event_hooks["request"] = [authorize_request, *request_hooks]
kwargs.setdefault("follow_redirects", False)

Authorization hangs off the HTTPX request event hook, so every request that
actually goes out is checked, redirect hops included — not just the URL you
handed in at the entry point. And redirects are not followed by default.

The MCP adapter builds its sessions with that client, so the whole MCP path —
initialization, list_tools, every call_tool — sits inside these constraints.

4. Can you find out what happened afterwards?

Tool calls are the only place an agent produces real side effects. A model
saying something wrong can be asked again. A tool that changed a row in the
production database changed it.

SOIT persists each tool call as a step of a run, and writes two pieces of
evidence per call:

Gateway audit. log_gateway_request with gateway_type="tool". The request side records the tool_ref, redacted parameters, and the egress decision (allow / deny plus the target URL). The response side records success, result type, metadata, and error. The failure path writes one too — the first thing the except branch does is emit the audit record. That is the one people forget, and the one you need when something has gone wrong.
Step metrics. Latency, success flag, summarized arguments and result, error code and error details.

The same call writes a cost entry with billing_basis="requests", the provider,
and source_port="tools" — so "what did this agent's MCP tools cost this month"
is a query you can drill into by agent, workflow, tool, and source
(source_kind=plugin | mcp | builtin).

On top of that, an OpenTelemetry span soit.tool.invoke carrying tenant,
workspace, run and step ids, for whatever APM you already run.

5. Can you replay it?

The most frustrating property of agent debugging is that it does not reproduce.
Same input, different reasoning.

At the tool layer you can at least be deterministic. Every tool call in SOIT
carries an idempotency key, defaulting to tool:{run_id}:{tool_call_id}, and
claims a leased execution record. If the claim lands on a record that already
completed, the cached response comes straight back and the external tool is
not called again.

The retry policy changes accordingly. The comment says it better than I can:

max_retries=1 if kwargs.get("idempotency_key") else self.max_retries,

Durable Agent calls are at-most-once at this boundary.
Not every downstream adapter can honor an idempotency key.

At-most-once at this boundary, because you cannot assume the MCP server on the
other end honors your idempotency key. Better to call once too few than once too
many — for writes, that trade is not really a choice.

Rate limits and daily quotas come along with it, keyed by tool_ref plus
tenant, workspace and user, so one runaway agent does not burn a whole tenant's
third-party API budget.

What this does not do

The usual honest list:

MCP transport is streamable HTTP only, targeting the MCP SDK v1 line. The stateless 2026-07-28 protocol revision is not supported yet.
OAuth is client_credentials only, no authorization_code (see question 2).
A marketplace for one-click MCP tool installation is on the roadmap; today you install plugin artifacts by hand.
The default egress allowlist is empty, which means your first MCP server will be refused until you add its domain explicitly. That is deliberate, but it does add a step to the quickstart.

Why none of this belongs in the agent framework

A question that comes up constantly: how does this relate to LangChain and
friends?

They are not the same layer. A framework answers "how do I orchestrate this
call." A runtime answers "under whose identity did this call run, with whose
credentials, what could it reach, what evidence did it leave, and can I replay
it." The first is a concern while you write the code. The second is a concern
after the code ships and someone else asks.

You can certainly put permission checks inside a framework, but then every new
tool integration reimplements the governance logic. Push it down into the
runtime's port layer and MCP tools, plugin tools and built-in tools all travel
the same path — which is the reason question 1 could say "nobody writes it
twice."

Try it

SOIT is Apache-2.0 and the code is all on GitHub:

Repository: https://github.com/soit-ai/soit
Quickstart: docs/quickstart.md in the repo
Governance demo: docs/governance-demo.md — a 20-minute local script that walks through permissions, secrets, call audit, cost attribution, replay and regression, and writes a machine-readable report at the end

If you are pushing MCP toward production right now, I would genuinely like to
hear which of the five questions is blocking you. In our experience the hardest
one is not technical — it is "who gets to decide what goes on the allowlist."

Disclosure: I maintain SOIT.

DEV Community: Jude

Our tool gateway says max_retries=2. It calls your tool exactly once.

The short answer

1. This started as a reply to a comment

2. Layer one: the ternary that decides everything

3. max_retries counts attempts, not retries

4. Which makes the shipped answer 1

5. And even that 2 would rarely fire

6. The timeout is a budget for the whole sequence

7. Failures leave the gateway wearing a different type

8. Layer two: tool_call_id is the identity, idempotency_key is an assertion

9. How at-most-once actually reaches the disk

10. Replay and re-run are one boolean apart

11. Layer three: workflow nodes, and one name with three meanings

12. How many times the downstream saw it, the ledger counted it, and the bill counted it

13. The other boundary defaults the opposite way

14. So was the comment right?

15. Twelve things that do not line up

Full disclosure

One-sentence takeaway

Come and find the holes

We changed one timeout from 30 to 480 and back to 30. In between, it stopped meaning the same thing

The short version

1. The commit that set it to eight hours

2. The commit that set it back to thirty minutes

3. Why checking the session on every request is nearly free

4. Why shipping this did not sign everybody out

5. Fourteen days is a hard cap, not a sliding window

6. Refresh tokens: hash only, rotated every time

7. How the browser hides the expiry

8. One path does not get any of this: SSE

9. If you are self-hosting, which number should you touch

10. The six deadlines you never see in config

11. Rotating SECRET_KEY no longer signs everybody out

12. What you can see: the security pane

13. Ten things that do not line up

Disclosure

One sentence

Come and find the holes

`gh attestation verify` said yes to an image we never released

The four questions, and the one nobody asks

1. An image wears more than one sha256

2. Pulling the signatures without Docker and without logging in

3. What the signature actually covers

4. The certificate that lived for ten minutes

5. Why an expired certificate still counts

6. Where the SBOM lives and what is in it

7. The 16 MiB ceiling, and what it cost us

8. An SBOM is a scan result, not the truth

9. Three v1.0.0 images, all of them verifiable

10. The missing link: a manifest that pins tag, commit and digest

11. I reproduced the source tarball, byte for byte, on Windows

12. The checklist

13. Eight things that still do not line up

Disclosures

One-line takeaway

Try it, and please poke holes

Four different things are called "replay" in our agent runtime. I read the ledger.

The short version

1. Why the word needs splitting

2. The tables, and one number that shows up three times

3. Replay #1: reassembling the evidence

4. Replay #2: catching up after the connection drops

5. Replay #3: the same idempotency key, twice

6. A status that admits we don't know

7. Replay #4: actually running it again

8. Why the ledger is trustworthy

9. The ledger belongs to the ports, not to the loop

10. The boolean the platform computes for you

11. Fingerprints in the ledger, not payloads

12. Seven things that don't line up yet

Coming clean

One-line version

Try it, and come argue

I counted our agent loop: 239 lines. Frameworks and runtimes are not the same layer.

The short version

Why this needs a whole post

1. Counting our own agent loop: 239 lines

2. So what are the 1,617 lines wrapped around it doing?

3. Where the line actually is: ten gates on one tool call

3. `max_retries` counts attempts, not retries

5. And even that `2` would rarely fire

8. Layer two: `tool_call_id` is the identity, `idempotency_key` is an assertion

11. Rotating `SECRET_KEY` no longer signs everybody out

9. Three `v1.0.0` images, all of them verifiable

9. Four of these containers are the same code as `api`