Your agent request touches 12 containers. Only one of them runs a model.

In the last post I cut this stack down to four long-running containers. The most common follow-up was not "how did you cut it" but the inverse:

So what are the rest of them actually for?

That deserves a straight answer. When a self-hosted project's quickstart starts a dozen containers, the default reading is either "the architecture never converged" or "someone is cosplaying enterprise." Neither reading is unfair — plenty of projects earn it. So this post does exactly one thing: it walks a single agent run from arrival to completion, and every time the run touches a service, says what that service did at that moment and which guarantee disappears without it.

Here is the least intuitive part of the answer up front: of the twelve containers, exactly one runs a model. Everything else buys the same thing — turning "run an agent" into "run an agent such that afterwards you can audit it, reconcile it, and replay it, and a crashed process does not leave half a state behind."

The short answer

Service	Where it sits in a run	What it carries	What you lose without it
postgres	throughout	the ledger: run / step / tool_call / artifact / cost	the physical basis for observability and replay; hard readiness gate
redis	at authz, and on cross-instance broadcast	permission cache (5-minute TTL), rate limiter, cross-instance event bus	replicas stop seeing each other's events; every authz check hits the DB
minio	when a run produces something large	artifact bytes; the DB keeps only storage_key + sha256	hard readiness gate; nowhere to put outputs
milvus	the retrieval step	vector store	retrieval calls fail (but readiness stays green)
etcd	never directly	Milvus's own metadata store, not the platform's dependency	goes wherever Milvus goes
vault	when a secret is needed	KV v2 store; credentials stay out of the process and out of .env	falls back to an in-process store, lost on restart
migrate / bootstrap	before the run	one-shot: schema, first admin, first tenant	— (they exit)
minio-init	same	one-shot: create the bucket, disable anonymous access	— (it exits)
api	throughout	the one process that actually runs a model	it is the thing being demoed
web	throughout	the front end	same
outbox-dispatcher	after the response is sent	delivers events committed inside the run's transaction, with per-consumer checkpoints	events sit at pending forever; nothing downstream ever fires
knowledge-ingest-worker	unrelated to a run	document parsing and indexing	uploads never reach the knowledge base
scheduler	unrelated to a run	fires due schedules	⚠ the quickstart never starts it — see below

1. First, a correction: it is not 12, it is 14

The quickstart command names twelve services:

docker compose --env-file .env -f docker/docker-compose.yml up -d \
  postgres redis minio etcd milvus vault migrate bootstrap api web \
  knowledge-ingest-worker outbox-dispatcher

docker/docker-compose.yml defines fourteen. The two extras behave very differently:

• minio-init is not in the command, but api declares depends_on: minio-init: service_completed_successfully, so Compose starts it anyway, it creates the bucket and runs mc anonymous set none, and exits. Thirteen containers actually start.
• scheduler is not in the command and nothing depends on it — it never starts in the quickstart topology at all. That one gets its own section.

The discrepancy itself is trivial. What matters is that "how many services are defined" and "how many you actually run" are two different numbers, and arguments about whether a stack is heavy tend to conflate them.

2. Before the run: the two containers that exit

migrate and bootstrap are one-shot jobs — restart: "no", and a healthy result is Exited (0).

migrate runs sh scripts/migrate.sh once postgres is healthy. bootstrap runs scripts/bootstrap_admin.py once migrate has exited successfully, creating the first admin and tenant ([e-mail] / changeme123 / tenant default by default).

Note the dependency condition: service_completed_successfully, not service_healthy. "Finished, and succeeded" is a different claim from "came up," and getting it wrong gives you a stack where every container is present and the schema is half-applied. api waits on both of these completing before it boots.

If docker compose ps shows those two as Exited, that is the correct state, not a failure.

3. Authz: Postgres is the authority, Redis is a cache

The first thing a request hits is authorization. Redis appears here, but strictly as a cache — the authority is always Postgres.

From server/app/kernel/identity/permissions.py:

class PermissionCache:
    """Permission cache using Redis."""

    def __init__(self, redis_client: redis_async.Redis | None = None):
        self._redis: redis_async.Redis | None = redis_client
        self._redis_pool: redis_async.ConnectionPool | None = None
        self._cache_ttl = 300  # 5 minutes

Three things worth knowing:

1. The TTL is hard-coded at 300 seconds, not configurable. A permission change can therefore take up to five minutes to propagate everywhere unless something calls invalidate explicitly (it exists — pattern-matched scan_iter + delete).
2. Losing Redis degrades rather than fails. _get_redis() returns None when settings.redis_url is empty or contains "None"; the caller treats that as a cache miss and falls through to the database. No Redis means slower, not broken.
3. The same Redis backs the rate limiter (server/app/kernel/ports/common/rate_limiter.py), implemented as a Lua script: ZREMRANGEBYSCORE to drop the expired window, ZCARD to count, then ZADD + EXPIRE if under the limit. Sliding-window counting in a single eval, so there is no read-modify-write race.

So "can I drop Redis?" resolves to: in a demo yes, in production no — and the reason is not the cache, it is the event bus in section 8.

4. The run gets written down: five ledger tables

Authorization passes, the run starts. This is the bulk of what Postgres carries, and it is the heaviest single piece of design in the stack.

One execution writes to five tables:

Table	One row is	Notable columns
runs	one execution	status / trace_id / request_id / parent_run_id / source_run_id / attempt_no / sandbox
run_steps	one step inside it	step_type (llm / retrieval / rerank / tool / workflow_node / agent_plan / memory_write / io) / metrics_json
run_step_tool_calls	one tool call	idempotency_key / request_hash / lease_owner / attempt_count
run_artifacts	one produced artifact	storage_key / sha256 / size_bytes / mime
run_cost_entries	one metered invocation	billed_quantity / amount / currency

A few columns explain why this is not just two lines in a log file:

• parent_run_id / source_run_id / attempt_no (server/app/kernel/runtime/db/models/runs.py). The first is parent-child; the other two are a retry and replay lineage — which run this one was derived from, and which attempt it is. That is what makes "replayable" a mechanism rather than a slogan: a replay is a new run pointing back at its source, not a re-read of a log.
• sandbox. Marks a run as a rehearsal rather than real work. The field's own comment is blunt about why: pre-release regression executes real agents, and without the flag their cost and evidence inflate real activity.
• input_summary / output_summary capped at 8KB, with metrics_json as a JSON column. The ledger stores summaries; the full payload lives in object storage behind run_artifacts. That split is deliberate — the relational store holds queryable structure, object storage holds bulk.

5. Why tool calls get their own table

run_step_tool_calls is the most heavily constrained of the five. It carries three unique constraints:

UniqueConstraint("tenant_id", "workspace_id", "run_step_id", ...)
UniqueConstraint("tenant_id", "workspace_id", "run_id", "tool_call_id", ...)
UniqueConstraint("tenant_id", "workspace_id", "idempotency_key", ...)

plus an index on ("status", "lease_expires_at"). Together they say one thing: tool calls have side effects, so they have to be at-most-once. One step maps to one tool call; a tool_call_id cannot land twice within a run; the idempotency key is globally unique — that key is the tool:{run_id}:{tool_call_id} from the earlier governed-MCP post.

The lease_owner / lease_expires_at pair is crash recovery: a worker that dies stops renewing, and the row becomes claimable again once the lease expires. This semantic is factored into a shared module (server/app/kernel/runtime/common/lease.py) whose docstring is explicit that every runtime domain executing work outside a request must use the same claim / renew / orphan-recovery primitives. Two constants and one implementation detail are worth remembering: MIN_LEASE_SECONDS = 30 (a smaller configured value is clamped up), LEASE_RENEWALS_PER_LEASE = 3 (the heartbeat interval is a third of the lease), and claims use SKIP LOCKED so concurrent workers do not contend — the comment openly notes SQLite ignores the clause, which is fine for single-worker tests.

This section is the whole article in miniature: these containers exist not because AI is complicated, but because "side-effecting operations must happen exactly once" is expensive in a distributed system, and always has been.

6. Retrieval: what Milvus does, and why etcd tags along

If the run includes a retrieval step, api queries Milvus. The adapter is server/app/adapters/vector/milvus.py, and the index parameters are fixed:

index_params={"index_type": "IVF_FLAT", "metric_type": metric_type, "params": {"nlist": 1024}}

etcd deserves an explicit correction, because it is the container most often misread as padding:

milvus:
  environment:
    ETCD_ENDPOINTS: etcd:2379
    MINIO_ADDRESS: minio:9000
  depends_on:
    etcd: { condition: service_healthy }
    minio: { condition: service_healthy }

etcd is not the platform's dependency; it is Milvus's. Milvus standalone keeps its metadata in etcd and its data files in MinIO. No application code in the repo talks to etcd at all. The honest way to read the topology is therefore: "vector retrieval" is one capability that costs two and a half containers (etcd + Milvus, sharing MinIO). Whether a demo should pay that is a clear trade-off, not a mystery.

One fact carried over from the previous post, because it matters here: the vector store does not gate readiness. In server/app/api/v1/health/router.py, the database and object storage raise 503 when probing fails; the vector store is probed and reported only:

try:
    await vector.check_ready()
    vector_status = "connected"
except Exception:
    vector_status = "unavailable"

The docstring gives the reasoning: non-vector endpoints keep serving during a vector outage, so pulling the instance out of rotation would be an overreaction.

7. Where the big objects go: MinIO

run_artifacts stores storage_key, sha256, size_bytes, mime — not the content. The content is in MinIO.

The one-shot minio-init container does two things: mc mb -p local/soit-artifacts to create the bucket, then mc anonymous set none to close anonymous access. The second is a small correct default: the artifact bucket is not anonymously readable out of the box.

Object storage is a hard readiness gate (probe fails → 503). This is exactly where the previous post's live run broke: on paper you can swap in the local-filesystem adapter, and in practice it does not work inside the official image because the root path is strip("/")-ed into a relative path (filed as issue #43). So the practical verdict stands: MinIO cannot be dropped.

8. The part that starts after the response is sent

By now the run is finished and the response has gone back to the caller. One container is only now getting to work.

api writes domain events into the event_outbox table inside the same transaction as the business data (server/app/kernel/runtime/db/models/events.py). That is the transactional outbox: if the business change committed, the event exists; if it rolled back, the event does not. There is no window where the database changed but the message never went out.

outbox-dispatcher then polls that table as its own process. The row's columns are effectively its state machine: status, available_at, locked_at, lock_owner, lock_expires_at, attempt_count, last_error, processed_at.

The module docstring of server/app/kernel/events/dispatcher.py states the whole flow in one line:

claim rows, run registered handlers with checkpoint idempotency

"Checkpoint idempotency" is a second table, event_consumer_checkpoint, unique on (consumer_name, event_id). Before dispatching, the service asks checkpoints.is_processed(consumer_name, event_id); on success it calls try_record_success. So idempotency is per consumer per event, not per event — if the second of three handlers fails and the row is retried, the first is not re-executed.

That container also exposes its own Prometheus endpoint (expose: 9201, started via start_http_server), and its healthcheck is a scrape of /metrics.

9. Four of these containers are the same code as api

This is the part most easily mistaken for microservice sprawl, and it is the opposite. migrate, bootstrap, api, outbox-dispatcher and scheduler all share one build context (build: context: ../server). The released-image path makes it starker: docker/docker-compose.images.yml points migrate, bootstrap, api and outbox-dispatcher at the same image, ghcr.io/soit-ai/soit/server. Only knowledge-worker and web are separate. Three images cover twelve containers.

The difference is the entrypoint, plus which background loops are switched on. The lifespan in server/app/main.py has six flags, each folding one loop into the API process:

Flag	Code default	Loop it folds in
workflow_orphan_reaper_enabled	False (compose sets true for api)	reaping orphaned workflows
schedule_worker_enabled	False	firing due schedules
account_deletion_sweeper_enabled	False	account deletion sweep
knowledge_ingest_worker_enabled	False	knowledge ingestion
outbox_dispatcher_enabled	False (compose hard-codes "false")	outbox dispatch
response_interaction_worker_enabled	False	durable chat interactions

Which means "how many containers" is largely a deployment decision, not an architectural one. The same code can run as one process or five. Compose splits them, and server/scripts/schedule_worker.py states the reason more plainly than I could:

Separate from the API for the same reason the outbox dispatcher is: a scheduler that shares a process with request handling competes with it, and an API restart should not be a gap in when jobs fire.

10. Production mode refuses to let you cut corners

Those flags look like a matter of taste. Half of them are not, once ENVIRONMENT=production. validate_runtime_requirements() in server/app/settings/settings.py fails closed on each of:

• the database URL must carry host, database name, username and password;
• the event bus must be redis (the code default is actually memory; compose supplies redis);
• outbox_dispatcher_enabled being true is an error — production forbids folding the dispatcher into the API process, it has to be its own;
• inline chat-interaction execution is forbidden, and the durable interaction worker is required;
• plugin signature verification is required, and at least one public key must be configured — the comment explains why that is checked separately: requiring signatures with no trusted key rejects every package, which reads as a gate but is really a total block.

This is the part I'd most want a skeptical reader to notice. Which services are optional is not an opinion in this repo; it is code that refuses to boot. You may drop Redis and fold the dispatcher into the API for a demo. You cannot do that and also claim to be running production.

One gap I found while writing this

One thing I turned up is not a trade-off, it is a hole:

• docker-compose.yml defines a scheduler service that sets SCHEDULE_WORKER_ENABLED: "true" and runs scripts/schedule_worker.py;
• the quickstart command does not include it, and nothing depends_on it;
• the api container does not set SCHEDULE_WORKER_ENABLED, and the code default is False;
• .env.example does not mention the variable;
• docs/ never mentions scheduler at all — across the whole repo only the two compose files do.

Net effect: in a stack started per the quickstart, schedules never fire on their own. POST /schedules creates one, the preview endpoint tells you when it would next run, and POST /schedules/{id}/run triggers it by hand — but nothing is polling to claim it when its time comes.

docker-compose.production.yml does include a scheduler, so this is a quickstart coverage gap rather than a missing feature. There is a second-order problem too: the released-image overlay covers six services and scheduler is not one of them, so adding a scheduler to the images-based path silently falls back to a local build.

I plan to file an issue for both halves of this: add scheduler to the quickstart command, and cover it in the released-image overlay. It was not filed yet when I wrote this, so there is no link here — if you want to confirm it yourself, walking the four bullets above in order is enough.

What this post is not

• There is no new end-to-end run behind it. The minimal-topology post was executed end to end; this one is a static read of the code and compose files. So the scheduler finding above rests on a code-and-config chain of evidence (service never started + flag defaults to False + no documentation) — I did not stand up the quickstart, create a schedule, and watch it fail to fire. Falsifying it is easy if you want to: start the quickstart stack, create a schedule one minute out, and see whether it runs.
• No resource numbers. This is about responsibilities, not footprint. The image-size figures are in the previous post.
• Nothing about wiring it into your existing observability. OTEL_ENABLED (default false) and an OTLP endpoint are in compose, and the production file ships an otel-collector, but that is its own article.
• The Redis conclusion is conditional. "Fine to drop in a demo" holds because a demo runs a single api replica. Add replicas and the in-process bus stops crossing processes. That is what the production check is protecting.

So where is the weight?

Regroup the twelve by what they guarantee:

• 2 are the thing being demoed: api, web.
• 3 are one-shot jobs: migrate, bootstrap, minio-init — they exit.
• 2 are where the ledger and the artifacts physically live: postgres, minio — also the only two hard readiness gates.
• 2½ are one capability, vector retrieval: milvus + etcd (etcd being Milvus's dependency, not ours).
• 1 is secret isolation: vault.
• 1 is cross-replica broadcast and caching: redis.
• 2 are background processes split out of the same codebase: outbox-dispatcher, knowledge-ingest-worker.

Exactly one of them runs a model. The rest of the weight buys one thing: the execution leaves evidence behind, and a crash mid-run does not leave half a state.

Whether that is worth it depends entirely on what you are doing. If you just want to see whether an agent runs at all, this stack is too heavy for you — the previous post shows how to get it to four containers. If you need to put an agent inside a process someone will later have to reconcile, these containers are the things you would end up writing yourself.

Try it

• Repo: github.com/soit-ai/soit
• Full topology: docker/docker-compose.yml
• The four-container version: I tried to cut our 12-container stack down to 4
• If you think any one of these trade-offs is wrong, open an issue and say so.

Disclosure: I maintain SOIT.