DEV Community: Kingsley Onoh

Why exit code 0 cannot authorize a freight award

Kingsley Onoh — Tue, 01 Sep 2026 13:03:13 +0000

Exit code 0 answers one narrow question: did the child process report an operating-system failure? It does not say that MiniZinc produced a complete solution, that the output belongs to the input we sent, or that an award is safe to persist.

That distinction shaped the clearing boundary in Freight Capacity Auction Clearing Engine. The worker is allowed to run an optimizer. It is not allowed to treat the optimizer as an authority. Before an award enters PostgreSQL, the system needs a configured backend, a normalized version, canonical input and output hashes, a parsed terminal status, and a decision record for every awarded, rejected, or unassigned load.

A clean process exit is only one piece of that evidence.

Process success and freight success are different facts

The tempting implementation is short. Spawn MiniZinc, wait for exit code 0, decode whatever JSON arrived on stdout, then write the selected bids. That design works while every solver run is polite. It fails at the boundary cases that matter most: truncated output, duplicated final records, a terminal status followed by another parsed solver record, an executable that reports a version but lacks the expected capability, or a timeout that leaves descendants alive.

None of those cases should produce a freight commitment.

I split the responsibility across three modules. Process_runner owns the child process. It receives one executable and literal arguments, never a shell command. It bounds stdin, stdout, and stderr independently, applies a deadline, captures typed exit outcomes, and cleans up the process tree. Solver_backend owns health probing and terminal output grammar. Solver_execution joins those pieces and refuses output that lacks the domain status.

The distinction is visible in the types. Process_runner may return Success because the executable exited normally. Solver_backend.parse_minizinc_stream can still return Malformed_output. The latter decision wins.

I was wrong to treat process success as solver success. The tests made the gap concrete. A fixture can emit plausible JSON, exit 0, and omit the terminal status. Another can emit the terminal status twice. Both look healthy if the adapter checks only the process outcome. Neither is a valid clearing result.

One terminal record, exactly once

The parser is small because its contract is narrow. This is the production code from src/solver/solver_backend.ml, lines 353-371:

let parse_minizinc_stream value =
  let lines =
    String.split_on_char '\n' value
    |> List.filter (fun line -> String.trim line <> "")
  in
  let is_separator line =
    let line = String.trim line in
    line <> "" && String.for_all (function '-' | '=' -> true | _ -> false) line
  in
  let rec loop seen_status = function
    | [] -> (match seen_status with Some status -> Ok status | None -> Error Malformed_output)
    | line :: rest ->
        (match parse_line line with
         | Some (`Terminal status) -> (match seen_status with None -> loop (Some status) rest | Some _ -> Error Malformed_output)
         | Some `Other -> (match seen_status with None -> loop seen_status rest | Some _ -> Error Malformed_output)
         | None when is_separator line -> loop seen_status rest
         | None -> loop seen_status rest)
  in
  loop None lines

The parser accepts ordinary solver records before the terminal record. After terminal status, another parsed solver record is malformed, and a second terminal status fails for the same reason. MiniZinc separators and unrecognized lines are tolerated on either side of the terminal record, but they don't count as evidence. Reaching the end without a status is malformed.

This rule covers SATISFIED, OPTIMAL_SOLUTION, ALL_SOLUTIONS, UNSATISFIABLE, UNBOUNDED, UNSAT_OR_UNBOUNDED, UNKNOWN, and ERROR. The important part is not which words are accepted. It is that the worker receives one explicit end state rather than guessing from silence.

The next layer maps a malformed stream to the stable code SOLVER_OUTPUT_INVALID. In bin/worker.ml, the same failure becomes a job error stating that the solver failed before producing terminal evidence. No award persistence runs after that branch.

The child process cannot own the boundary

Terminal parsing would be weak if the process adapter could leak resources or accept hostile command construction. The shared runner therefore keeps executable and arguments separate. It rejects NUL-bearing input before spawn, including a leading NUL that has special meaning in the Lwt process implementation on Windows. Environment names are allow-listed by the caller. There is no string command API.

Output is bounded too. The solver execution path allows up to 1 MiB on stdout and 64 KiB on stderr. Those are operational caps, not optimization parameters. A solver that floods either stream should fail as a process boundary violation, not consume memory until the worker becomes unstable.

Timeout and cancellation share the same cleanup path. On POSIX, the child runs in a private process group. The runner sends a termination signal, allows a bounded grace period, checks whether the group remains, sends a kill signal when needed, reaps the leader, and verifies group absence. The test fixture covers a child whose leader exits while a descendant resists termination. That is an ugly case, and it is exactly why process control lives below the solver adapter.

Windows cleanup is recorded as unproven by the fixture. The code returns a typed termination-unavailable result when it cannot prove cleanup. I would rather carry that gap in the release record than turn POSIX evidence into a cross-platform claim.

Evidence must survive the worker

Parsing one terminal status stops malformed output, but it does not bind the result to the auction. The worker still has to preserve what was solved.

Before execution, it builds canonical JSON from sorted loads and bids. Stable ordering matters because the same freight facts should produce the same input hash regardless of database retrieval order. The worker probes the selected backend, records its normalized version, and hashes the canonical input. After execution, it hashes the output artifact. The configured backend never changes automatically. If MiniZinc is selected and missing, an available OR-Tools binary does not quietly take over.

That no-fallback rule costs availability. It also keeps the evidence honest. A replay or operator can tell which optimizer produced the decision. Silent fallback would make a job look continuous while changing the model, solver behavior, or explanation surface underneath it.

The persistence boundary then writes the job state, solver identity, hashes, award rows, and clearing decisions in PostgreSQL. The release fixture expects both positive and negative decisions. One bid wins; the other remains a rejected decision with its reason. Saving winners alone would make the optimizer look decisive while deleting the evidence needed to explain competition.

Clearing_service.clear adds the final guard. If solver evidence is absent, it returns an infeasible result with SOLVER_EVIDENCE_REQUIRED. Policy filtering can reject bids above reserve, below service requirements, or beyond carrier-share limits. Scoring can rank eligible bids. None of that logic may bypass the evidence requirement in production clearing.

What surprised me about malformed success

I expected the hardest solver tests to be about objective values and capacity constraints. The sharper tests were about output shape.

The canonical fixture has four loads, four carriers, eight eligible bids, and three excluded bids. It produces eleven decisions and resolves a score tie by UUID. Those checks matter because deterministic explanations depend on them. But the fixture that changed my view is smaller: valid-looking output with no terminal status. It proves that a solver adapter can parse useful lines, receive exit code 0, and still have no right to persist the answer.

That failure is dangerous because it resembles success. The process finished. The logs may show no crash. Some assignments may already be present in stdout. An adapter that accepts partial output can create a complete-looking award from an incomplete run.

The same reasoning applies after solving. An award that requires approval stays non-exportable. The local release lifecycle submitted two bids, persisted one award and one rejection with solver hashes, then received HTTP 409 when it requested export before approval. After approval, the export returned HTTP 200 from a frozen snapshot. Solver success did not grant export authority. Approval did.

These are separate gates on purpose. Process execution says the program ran. Terminal evidence says the solver declared an end state. Persistence says the decision and its proof were written together. Approval says a person accepted the operating consequence. Export says the accepted record can leave the system without changing later.

What the local proof says, and what it does not

On 27 August 2026, the local release gate passed the OCaml unit suite, PostgreSQL 16 and Redis 7 integration suites, Dream lifecycle checks, hostile child-process fixtures, MiniZinc artifact tests, DuckDB and Parquet checks, browser journeys, and the packaged Docker lifecycle.

The solver fixture covered optimal, satisfied, infeasible, unbounded, unknown, error, timeout, nonzero exit, and malformed stream cases.

That is measured local validation. It is not production throughput, live carrier behavior, or proof that the larger PRD targets have been met. The published system supports one production clearing mode, single_round_spot. Other auction modes remain explicit unsupported cases. External notification and workflow delivery also remain outside the local proof.

The transferable point is narrow: an optimization result needs a domain completion record, not just a successful process. If the boundary cannot prove which solver ran, what it saw, how it ended, and what it wrote, exit code 0 is administrative trivia.

The timestamp that broke a fresh evidence chain

Kingsley Onoh — Tue, 01 Sep 2026 12:50:14 +0000

2026-08-29 14:03:12.120000+00 and 2026-08-29 14:03:12.12+00 describe the same instant. SHA-256 does not care. It sees different bytes.

That distinction became a release-safety problem in the Edge Fleet Rollout Safety Control Plane. The service records every significant release action in an append-only evidence chain. Each event includes the hash of the previous event, and verification recomputes every hash from stored fields. The first PostgreSQL contract run appended an event and then immediately declared that same event invalid.

Nothing had been tampered with. PostgreSQL had only rendered the timestamp in its normal form.

Why a formatting detail reached the safety boundary

An edge rollout has several kinds of truth that can drift apart. The server may have issued an install command. A device may have acknowledged it. The device may still be running the old artifact. A health sample may belong to the previous observation window. The control plane handles those gaps by treating observations, gate evaluations, approvals, and control actions as evidence.

The evidence table is immutable. A tenant's first event starts with 64 zeroes as its previous hash. Each later event points to the hash before it. Verification reads events in sequence order, reconstructs the canonical JSON object, calculates its digest, and checks both links. A changed payload breaks the event hash. A removed or reordered event breaks the previous-hash link.

The event hash covers more than its payload. It includes the identifier, sequence number, tenant, aggregate type and identifier, event type, actor, occurrence time, trace identifier, and previous hash. That breadth makes the chain useful, but it also means every covered representation is part of the protocol.

The JSON serializer already sorts object keys recursively. I assumed that was the end of canonicalization. I was wrong about where canonicalization ended. Sorting keys gives stable JSON structure; it does not give a timestamp string a stable database representation.

The failing sequence

The PostgreSQL adapter generated a UTC timestamp with six fractional digits. It assembled the event object, serialized it, calculated the digest, and inserted the timestamp and digest in one row. PostgreSQL accepted the TIMESTAMPTZ value, then returned a shorter fractional component when the row was read for verification.

The append path had hashed this value:

2026-08-29 14:03:12.120000+00

The verification path reconstructed the event with this value:

2026-08-29 14:03:12.12+00

Both strings represented one instant. They produced different canonical JSON bytes, so the newly appended event failed at sequence one. The hash chain was doing its job. The input contract was inconsistent.

The discrepancy also threatened every later event. Once one stored digest differed from its recomputed value, the next event would inherit a previous hash that verification could no longer trust. A lexical mismatch at the head of the chain contaminated the meaning of the tail.

This was not an abstract cross-platform concern. The PostgreSQL storage checkpoint recorded the defect during the first contract run against an isolated cluster. The test exercised migration reruns, tenant isolation, evidence append and verification, and database triggers that reject updates and deletes. The append succeeded. The following verification exposed the mismatch.

The production normalizer

The fix lives in src/infrastructure/postgres_storage.cpp. This is the exact function used by the PostgreSQL adapter:

std::string canonicalPostgresTimestamp(std::string timestamp) {
  const auto fractional = timestamp.find('.');
  auto timezone = timestamp.find('+', fractional == std::string::npos ? 0 : fractional);
  if (fractional == std::string::npos || timezone == std::string::npos) return timestamp;
  while (timezone > fractional + 1 && timestamp[timezone - 1] == '0') {
    timestamp.erase(timezone - 1, 1);
    --timezone;
  }
  if (timezone == fractional + 1) timestamp.erase(fractional, 1);
  return timestamp;
}

The append path first converts the service timestamp from an ISO T and trailing Z to PostgreSQL's UTC spelling with a space and +00. The function then removes trailing zeroes from the fractional part. If every fractional digit disappears, it also removes the decimal point. The adapter hashes that normalized value and inserts that same value.

The important property is not the loop. It is the timing. Normalization happens before the event object is serialized and before the row is inserted. The bytes selected for hashing already match the representation that verification will read.

An alternative would insert the row, query it back, hash the returned fields, and update the digest. That path conflicts with the table's immutability rule and makes the append transaction more elaborate. Another option would exclude occurrence time from the hash, which would weaken the event envelope for the sake of a formatting bug. The implemented boundary keeps time covered and makes the storage adapter responsible for its dialect's spelling.

Canonical JSON was necessary but insufficient

The shared canonical JSON code recursively sorts object keys and emits strict compact JSON. That removes variation from map iteration order. It also makes replay digests stable when the same logical object arrives with keys in a different order.

String values remain strings. The serializer cannot know whether a value contains a timestamp, a device key, an artifact digest, or ordinary text. Teaching a generic JSON layer to reinterpret selected strings would hide a database rule inside a shared primitive. The PostgreSQL adapter knows the column type and the database's output form, so it owns this normalization.

SQLite shows why the boundary belongs there. Its adapter stores a six-digit UTC string ending in Z and reads that text back unchanged. PostgreSQL stores a typed timestamp and may shorten the fractional part. Both adapters satisfy the same evidence operation, but they need different lexical preparation to preserve the operation's invariant.

The shared invariant is precise: the event bytes hashed during append must equal the event bytes reconstructed from persisted fields. It is not “all databases must print time the same way.”

Concurrency is part of the same contract

Canonical bytes would still be useless if two workers could claim the same next sequence. The PostgreSQL append transaction takes an advisory transaction lock derived from the tenant identifier before reading the latest event. It then chooses the next sequence, uses the previous event hash, inserts the new event, writes any local operator notice, and commits.

SQLite reaches the same boundary with BEGIN IMMEDIATE and process-level serialization. The mechanisms differ because the deployment shapes differ. SQLite supports the local, Docker-free mode. PostgreSQL supports the production storage contract. In both cases, sequence allocation and evidence insertion share a transaction.

This pairing matters. A hash chain has two dimensions of determinism: stable bytes inside an event and stable order between events. Timestamp normalization fixes the first. Tenant-scoped serialization protects the second.

The test that became more valuable after it failed

The PostgreSQL component test does not inspect the normalizer directly. It asks the storage contract for an event, then asks the same contract to verify the tenant's chain. It also attempts to update the stored event and expects the database trigger to reject the mutation. The final verification remains valid after that rejected attempt.

That test caught a defect an isolated unit test for SHA-256 would never see. The digest function produced the correct digest for the bytes it received. The JSON serializer produced stable output. PostgreSQL stored the correct instant. The defect existed only where those correct components met.

The broader evidence ladder kept the fix in context. The Docker-free build completed 63 of 63 CTest cases. The production image completed 64 of 64, with two documented environment-dependent skips in that image run. The PostgreSQL contract was also run with its database URL supplied, and the build journal records the timestamp defect and corrected pass. The implementation ledger closed at 323 of 323 items.

Those counts do not prove that SHA-256 is unbreakable or that every database version formats every temporal type identically. They prove that this adapter's append, read, trigger, and verification path was exercised as a connected contract.

What I changed in my design habit

I used to treat canonicalization as a serializer feature. The useful unit is larger: producer, serializer, storage type, database output, and verifier. If any member rewrites a covered value, the hash protocol must account for that rewrite before the digest is committed.

The same question now applies to every field in a signed or hashed record:

Will the database change its lexical form?
Will a driver coerce its type?
Will Unicode normalization differ between producer and reader?
Will numeric precision survive a round trip?
Will an export path preserve the same bytes?

This does not mean converting every value to a string. It means naming the representation boundary and testing a full round trip through the real storage engine.

The surprise was useful because the chain rejected a harmless semantic equivalence. Safety systems need that severity. If the verifier silently accepted alternate spellings, it would need a second, fuzzier definition of equality for every covered field. That would turn evidence verification into interpretation.

The cleaner rule is exact: normalize at the boundary that knows the representation, hash once, store those bytes, and verify the bytes read back. In this project, removing four zeroes restored that rule. The size of the patch said nothing about the size of the invariant it protected.

When a Redis Payload Became a Pointer

Kingsley Onoh — Sat, 29 Aug 2026 11:37:04 +0000

{
  "runId": "run-identifier",
  "pkgKey": "object-key",
  "schemaVersion": "simulation-package.v2",
  "theaterCount": 1,
  "generatedAt": 1756378800000
}

That object is valid JSON. It is not a simulation package.

WorldMonitor originally placed the complete package in Redis. The gateway could read the value, parse it, validate the scenario fields, and create work. The publisher later moved the larger object to R2 and left a pointer in Redis. The key name stayed the same. The transport meaning changed underneath it.

If I had patched the parser to accept pointer fields, I would have mixed storage lookup with domain validation. Direct JSON ingestion would become harder to reason about, manual ingestion would inherit R2 concerns it did not need, and retries could create duplicate scenarios once the fetch finally succeeded.

I put one resolution step between JSON parsing and package parsing. It accepts either representation and returns one package shape. Everything after that boundary remains unaware of Redis pointers and object storage.

Transport is not the scenario contract

The gateway receives WorldMonitor data through two routes. A poller reads the latest value from WorldMonitor's Redis instance. An authenticated API route accepts a package directly. Both routes eventually call parseSimPackage() and persist the same scenario fields.

That shared parser is important because WorldMonitor's current package shape is not identical to the gateway's stable shape. The current feed can express simulationRequirement as an object. The gateway stores it as text. Theater fields can be nested differently. Constraints arrive as maps and become arrays. Ranking and strength values are clamped to the accepted range, HTML tags are stripped from text, and the entity list is capped at 20 before persistence.

Those are domain normalization rules. Fetching an object from R2 is a transport rule. I kept them in separate modules because they fail differently.

A pointer with missing R2 configuration is an operational error. A remote object that returns a non-success HTTP status is also an operational error. The object fetch has a 30-second timeout. A retrieved package with the wrong schema is a validation error. The first category may succeed on the next polling cycle. The second category should not enter the simulation queue at all.

One extra branch at the boundary is the cost. Every caller must pass optional R2 configuration even when direct packages are common. That branch buys a single downstream contract and keeps object-storage credentials away from the parser and manual-ingestion route.

Redis is a notification surface, not the package archive

Once the payload moved to R2, Redis became a statement about the latest available run. The pointer carries enough metadata to identify that run and locate its object. It does not become authoritative scenario data until the object is fetched and passes the package parser.

I resisted storing the pointer itself as rawPackage. Doing that would make every later replay depend on a remote object that can be moved, expired, or replaced. The gateway stores the accepted package content in PostgreSQL with the scenario. A report can then be traced to the input the gateway actually validated, even if the publisher's latest pointer changes.

The cost is duplicated storage. R2 keeps the publisher's object and PostgreSQL keeps the gateway's accepted copy. For these scenario documents, reproducibility matters more than avoiding a modest JSON copy. If packages grow large enough to pressure database backups, I would store a content-addressed archive and retain its hash with the scenario. I would not point historical simulations at a mutable “latest” object.

The poller closes its WorldMonitor Redis connection in a finally block after every cycle. That favors isolation over connection reuse. A fresh connection pays setup cost on each poll, but a dead external Redis client cannot linger inside the service and consume retries between scheduled runs. At the current polling rate, predictable cleanup wins.

One poll cycle, one normalization path

The poller connects to a Redis service owned by WorldMonitor, not the Redis instance used by BullMQ. It disables reconnect loops for a single poll, allows one request retry, and gives the connection five seconds. If that cycle fails, the scheduled job can try again later without leaving a reconnecting client behind.

The core path looks like this:

parsed = await resolveSimulationPackage(
  parsed,
  worldMonitorR2Config(),
);

const pkg = parseSimPackage(parsed);

const existing = await db
  .select()
  .from(scenarios)
  .where(
    and(
      eq(scenarios.tenantId, tenantId),
      eq(scenarios.worldmonitorRunId, pkg.runId),
    ),
  );

if (existing.length > 0) {
  return { ingested: false };
}

const [inserted] = await db
  .insert(scenarios)
  .values({
    tenantId,
    worldmonitorRunId: pkg.runId,
    title: pkg.title,
    theaters: pkg.selectedTheaters,
    entities: pkg.entities,
    eventSeeds: pkg.eventSeeds,
    constraints: pkg.constraints,
    simulationRequirement: pkg.simulationRequirement,
    source: SCENARIO_SOURCE.POLLER,
    rawPackage: pkg,
  })
  .returning({ id: scenarios.id });

The excerpt comes from pollWorldMonitor(). Resolution happens before normalization. Duplicate detection happens after normalization has established a trustworthy runId and before any queue message is created.

I also kept the normalized package in rawPackage. The named scenario columns support ordinary product queries, while the JSON copy preserves the accepted input for later diagnosis. Calling it raw is slightly historical because it contains the normalized package rather than the original pointer. That is a naming debt I would fix before exposing the field outside internal tooling.

What surprised me was how easily valid JSON could be mistaken for valid domain data. JSON.parse() answered only whether bytes formed an object. It said nothing about whether that object was a scenario, a pointer, or stale metadata. Once the publisher introduced R2, treating JSON parsing as ingestion success became an attractive lie.

Idempotency needs the tenant and the run

The poller wakes on a schedule. BullMQ can retry. An operator can submit the same package manually after a poll. A network timeout can happen after PostgreSQL commits but before the caller sees success. Duplicate delivery is normal under those conditions.

WorldMonitor's runId gives the gateway a stable source identity, but it is not globally sufficient. Two tenants may deliberately consume the same WorldMonitor run. The idempotency key is the pair of tenantId and worldmonitorRunId.

The application performs a readable duplicate check and returns without enqueueing another simulation. The database schema backs that check with a unique constraint on the same pair. I wanted both. The early query gives a quiet, expected path for the poller. The constraint closes the race where two workers check before either one inserts.

Manual ingestion uses the same parser and key but reports a conflict instead of silently skipping. That difference is about caller intent. A poller repeatedly seeing the latest run should remain quiet. A person or service explicitly submitting a duplicate deserves a clear response that no new scenario was created.

There is a trade-off in binding idempotency to runId. If WorldMonitor republishes corrected content under the same ID, the gateway will keep the first accepted scenario. That is safer than silently changing an input after a simulation may have started. A correction needs a new run ID or a separate revision contract. Mutable source identities would make reports impossible to reproduce.

Validation decides what never reaches the queue

The parser accepts the legacy package and the current version-two package, then returns the gateway's stable SimPackage. It does not let every downstream consumer carry version branches.

The current feed can hold theaters under a simulation context. The parser maps them into the gateway's selected-theater records. A requirement object is joined into one textual instruction. Constraint maps become flat lists. Missing ranking scores or entity strengths receive a neutral value of 0.5, and supplied values cannot escape the zero-to-one range.

The entity cap of 20 is a conscious loss of input. It limits prompt and graph pressure before MiroFish receives the seed document. Keeping the first 20 entries is simple and deterministic, though it assumes WorldMonitor orders them by relevance. If that ordering contract weakens, I would rank explicitly or reject oversized packages. Quietly taking an arbitrary subset would be indefensible.

Sanitization removes HTML tags from text fields because the same data reaches seed documents and a browser-facing product. This is not the only output safety boundary, but it stops source markup from becoming part of the simulation instruction by accident.

Invalid JSON, a missing key, or a package that fails schema validation produces no queue work. The poller logs the condition and returns { ingested: false }. Connection and fetch failures also avoid the queue, while the per-tenant failure tracker can emit an outage event after three consecutive operational failures. A malformed package and an unavailable source are not reported as the same incident.

That distinction keeps alerting useful. An empty feed may be normal between WorldMonitor runs. A broken Redis connection repeated across cycles is an availability problem. Schema drift is a data-contract problem. One generic “ingestion failed” counter would hide which team needs to act.

The package can move without moving the gateway

The current reader supports complete JSON in Redis and a pointer whose object lives in R2. The rest of the pipeline receives the same scenario either way. It records one tenant-scoped source identity, queues one simulation, and retains the accepted package beside the fields the application queries.

I would add versioned correction semantics before allowing an existing run to change. I would also record a content hash beside the source ID so an operator can distinguish a harmless duplicate from conflicting content published under the same identity. Neither addition is needed to understand the present contract.

The important boundary sits earlier: storage location may change, while package meaning and idempotency stay under gateway control.

Tenant Isolation Did Not Fix the Wrong Theater Card

Kingsley Onoh — Sat, 29 Aug 2026 11:36:21 +0000

Simulation Theater

That placeholder appeared on a completed simulation whose predictions already named the actual region. The records were correct. Tenant isolation was correct. Both API calls returned valid responses. The frontend assembled them in the wrong order.

The bug forced me to separate two properties I had been treating as one. A tenant-safe system prevents one customer from reading another customer's prediction. A coherent system presents related simulation and prediction data as one understandable state. The gateway had solved the first property at the API boundary. Separate polling loops left a timing gap in the second.

I kept the tenant boundary strict, then changed how prediction data crosses caches, events, and frontend transforms. The result is a useful lesson from a narrow failure: security conditions must survive every storage layer, while presentation joins need an ordering contract of their own.

The browser never chooses its tenant

Every protected request carries an API key. Fastify hashes that key with SHA-256, finds one active tenant record, and attaches the tenant to the request. Route handlers call requireTenant() and use the resolved ID in their database predicates.

The authentication function is small enough to inspect in full:

export async function authGuard(
  request: FastifyRequest,
  _reply: FastifyReply,
): Promise<void> {
  const apiKey = request.headers['x-api-key'];

  if (!apiKey || typeof apiKey !== 'string' || apiKey.trim() === '') {
    throw new UnauthorizedError('Missing X-API-Key header');
  }

  const hash = crypto.createHash('sha256').update(apiKey).digest('hex');
  const [tenant] = await db
    .select()
    .from(tenants)
    .where(eq(tenants.apiKeyHash, hash))
    .limit(1);

  if (!tenant) throw new UnauthorizedError('Invalid API key');
  if (!tenant.isActive) throw new ForbiddenError('Tenant is inactive');

  request.tenant = tenant;
}

There is no tenant ID header for the caller to edit. A route does not accept tenantId from a query string and hope it matches the credential. The key resolves identity once, then that identity follows the request.

I chose explicit application predicates over PostgreSQL row-level security for this service. Every scenario, simulation, prediction, report, profile, episode, and graph query includes the tenant ID. The benefit is visibility in code and tests. The cost is repetition. One forgotten predicate can become a cross-tenant disclosure.

That trade-off demands real database tests. The verification suite seeds two tenants with separate scenarios, simulations, and predictions. Tenant A cannot list Tenant B's records, fetch its scenario, read its report, or cancel its simulation. Cross-tenant reads return 404 rather than 403 so the API does not confirm that the resource exists. The tests run against Dockerized PostgreSQL, not an in-memory substitute.

If the route count grows enough that predicate review becomes unreliable, I would add database policies as a second boundary. I would not remove the application predicates. Independent checks fail differently.

A cache key is part of the access-control model

Prediction reads use a Redis read-through cache with a five-minute TTL. Redis failure is nonfatal. On a read error, the route queries PostgreSQL. On a write error, it returns fresh database data and logs the cache problem.

Fail-open caching is appropriate because Redis is an accelerator here, not the authority. It creates a confidentiality condition, though: the key must encode every field that changes the authorized result.

The latest-predictions key contains the tenant ID, minimum confidence, and limit. The list key also contains theater, prediction type, cursor, and limit. PostgreSQL repeats predictions.tenant_id = tenant.id inside the cache miss function. Cursor lookup is tenant-scoped too, so a cursor copied from another tenant cannot become a side channel into its ordering.

Leaving the tenant out of the Redis key would defeat correct SQL. Tenant A could populate a shared cache entry and Tenant B could receive it without touching PostgreSQL. That failure never appeared in production because the tenant was part of the key from the implementation stage, but it is exactly the sort of omission that a database-only isolation test can miss.

The five-minute TTL trades freshness for query load. A newly completed run may coexist briefly with an older cached prediction list unless completion invalidates the relevant pattern or the frontend waits for the next cycle. The cache helper includes nonblocking pattern invalidation through Redis SCAN, deleting batches without locking the keyspace. I would add event-driven invalidation to every prediction commit before increasing the TTL.

Correct endpoints can still compose into a wrong screen

The frontend polls simulations and predictions separately. Simulation cards need both datasets. Status, agent count, and round count come from the simulation response. Theater, confidence, prediction summary, factions, and time horizon come from predictions matched by simulation ID.

transformPredictions() stores the latest prediction array in module state. transformSimulations() reads that cache, picks the highest-confidence prediction for each simulation, and falls back to Simulation Theater with confidence 0.75 when no matching prediction is present.

The initial timing bug was simple. Simulation data arrived before the prediction transform populated its cache. The card rendered a plausible placeholder instead of an error, so the UI looked finished while saying the wrong thing. A blank state would have been easier to notice.

The current DataBridge creates the prediction loop before the simulation loop. Prediction updates also dispatch a predictions-updated event for the globe and faction views. Active simulations fetch their scenario details so the swarm hero can replace a placeholder topic with the scenario's theater or title.

That ordering reduces the startup race, but I do not consider it a strict data dependency. Two network requests started in sequence can finish in either order. The current transform still has a fallback because the product must render during partial availability.

The stronger fix at scale would be a small read model for simulation cards. One endpoint could return the simulation with its top prediction and theater under one database snapshot. That would move the join to the server and remove module cache timing from the card contract. Another query shape and cache entry would need maintenance. I kept separate feeds because the prediction timeline and globe already need the full prediction set, and the current polling volume is small.

I expected asynchronous requests to race. I did not expect a safe fallback to conceal the race. Simulation Theater and 0.75 were meant to keep a demo surface from collapsing. In live mode, they could make absent data look measured. The frontend now labels active work as analysis in progress and fetches real scenario detail. I would go further by carrying an explicit dataState into every card so placeholders cannot masquerade as facts.

Lazy rendering needs a memory of its own

The globe renderer loads through a dynamic import. Predictions can arrive before WebGL initialization finishes. Without a handoff, the first prediction event would be lost and the map would remain empty until the next poll.

wireGlobeUpdates() holds the latest prediction array in pendingPredictions. Once the renderer initializes, it applies that pending data before registering the continuing event listener. This is a small client-side mailbox. It gives a lazy component one remembered state rather than asking every producer to know whether the globe exists yet.

Several prediction updates during initialization collapse into the newest set. That last-value behavior is correct for a dashboard that displays current predictions. It would be wrong for an audit view that must render every transition. Event semantics depend on the consumer's job, not on the event bus alone.

The same distinction appears when a simulation becomes idle. The frontend fetches the real agent stance summary, displays it for ten seconds, then returns the hero to demo mode. If that request fails, it logs the error and falls back. Live data gets a bounded presentation window without leaving the landing surface frozen on an old run.

Events carry tenant context without delegating authority

When orchestration completes, the gateway stores the report and predictions before publishing simulation.completed. The envelope includes event type, source, tenant ID, timestamp, and payload. Failure events carry the same tenant context.

Notification delivery is optional and feature-flagged. The publisher uses its configured service credential, not a browser-provided tenant ID. An unavailable notification service logs a warning and leaves the completed simulation intact. After three consecutive WorldMonitor failures for one tenant, a separate tracker emits an outage event once for that streak and resets after a successful poll.

This event path is tenant-aware, but it is not transactional with PostgreSQL. A process can commit completion and die before publication. I accepted that for notification and monitoring consumers because the REST API remains the source of truth. If another service begins triggering money movement or mandatory operations from these events, the gateway needs an outbox table and replayable delivery.

That is the recurring trade-off across the system. API keys, SQL predicates, cache keys, events, and frontend joins each carry a different part of the boundary. Treating “tenant-safe” as an authentication feature would leave four other places capable of undoing it.

Privacy and coherence need separate evidence

The real PostgreSQL tests show that one tenant cannot read another tenant's records. Tenant-scoped cache keys keep Redis from bypassing that rule. Event envelopes preserve ownership when results leave the service. The frontend timing fix addresses a different failure: related data can be private and still be assembled incorrectly.

I would replace placeholder facts with explicit loading states before exposing the dashboard to decision makers. I would also add a simulation-card read model if separate poll completion keeps affecting display. Neither change weakens the current tenant contract.

Security answers who may see a record. Coherence answers whether the record is being understood in the right context. A prediction product needs proof of both.

Why Put a Queue in Front of a One-Job Worker?

Kingsley Onoh — Sat, 29 Aug 2026 11:36:07 +0000

Why use BullMQ if the simulation worker is allowed to process only one job at a time?

The usual queue story is parallelism. Add workers, raise concurrency, increase throughput. That story did not fit this gateway. A MiroFish run starts graph construction, profile preparation, an agent simulation, action-log import, and report generation. The upstream container image occupies about 14 GB after extraction, and the live process shares a local host with PostgreSQL, Redis, the gateway, the frontend, and development tooling.

During the first full English simulation, the process ran for 28 minutes and died with exit code -9. There was no application exception to catch. The operating system killed it under memory pressure. I stopped 21 nonessential containers and recovered roughly 7 GB before rerunning.

After that failure, concurrency: 1 stopped looking conservative. It became the capacity contract.

The queue separates admission from execution

An API request should not hold an HTTP connection open while a swarm runs. The simulation route creates a database record, places a self-contained job on Redis, and returns the simulation ID with a pending status. A worker later receives the tenant ID, scenario ID, simulation ID, agent count, round count, and model provider needed to run the job.

That job shape matters. It contains identifiers and configuration, not open database connections or request objects. BullMQ can persist it across a gateway restart. The worker reloads the scenario and validates tenant ownership when execution begins.

With one active slot, the queue performs three jobs that concurrency alone cannot describe. It absorbs bursts, preserves ordering under limited memory, and gives every accepted simulation an observable state before heavy work starts. A burst can become pending records without becoming many MiroFish processes fighting over the same host.

The trade-off is queueing delay. A long simulation blocks every job behind it. I accepted that because an honest pending state is better than concurrent runs that the machine cannot finish. If demand grows, the next unit of scale is another isolated worker host, not a larger concurrency number on the same host.

Backpressure also has to be visible to the caller. The API stores a simulation before queue admission and returns its ID immediately. A progress route maps pending, queued, graph-building, simulating, reporting, and terminal states into plain phase labels. It reports elapsed time, agent count, round count, and whether the run is active.

The current response does not expose queue position or an estimated start time. With one worker, that omission becomes noticeable as soon as more than a few simulations wait. I would add job age and queue position before adding parallel hosts so operators can distinguish slow execution from slow admission. A queue that protects the machine but leaves users guessing has moved the failure rather than solved it.

The number is in code, not operator folklore

The worker factory carries the limit and the failure instrumentation together:

export function createSimulationWorker(): Worker {
  const worker = new Worker(
    QUEUE_NAMES.RUN_SIMULATION,
    async (job) => {
      const {
        simulationId,
        scenarioId,
        tenantId,
        agentCount,
        roundCount,
        llmProvider,
      } = job.data;

      await runSimulation({
        simulationId,
        scenarioId,
        tenantId,
        agentCount,
        roundCount,
        llmProvider,
      });
    },
    {
      connection: parseRedisUrl(env.REDIS_URL),
      concurrency: 1,
    },
  );

  return worker;
}

createSimulationWorker() is created once during service startup. The API and quick-launch routes both add run-simulation jobs with three attempts and exponential backoff starting at 60 seconds. The worker emits a warning for an intermediate failure and an error with permanentFailure: true when all configured attempts are spent.

I kept the concurrency value beside the worker rather than making it an unchecked environment variable. On this deployment, raising it changes the memory safety model. That deserves a code review and a load test, not a late-night configuration edit.

There is a cost to that choice. Different hosts cannot tune independently without a release, and a future production cluster may have worker classes with different capacities. I would introduce a bounded configuration value when there are at least two proven host profiles. Until then, a configurable footgun does not count as operational flexibility.

Exit code -9 changed what I measured

Before the killed run, I was watching application logs and MiroFish status. Neither source showed a TypeScript error because neither process decided to fail. The host reclaimed memory.

That surprised me because the run had already survived graph generation and entered the expensive simulation phase. A health endpoint could still answer while the machine moved toward exhaustion. Service health and workload capacity were different signals.

The verified rerun used one agent and one round. It moved from orchestration start to completion in about seven minutes and 39 seconds, then produced 12 graph nodes, 12 edges, 37 stored episodes, two profiles, an 8,366-character report, and four predictions. Those figures prove the path works at the measured configuration. They do not prove that the same host can safely run the default 4,096-agent, five-round template shown in the frontend.

I want that limitation stated plainly. Product defaults and operator-verified capacity are not yet the same thing. The queue stops simultaneous runs from multiplying pressure, but it cannot make one oversized run fit.

At higher loads, I would capture per-phase resident memory, container memory peaks, and runtime before changing either agent count or concurrency. The scaling decision needs a measured envelope. A second worker on separate hardware may double throughput. Two jobs inside one memory boundary may only double the chance of another kill.

Retries are bounded, not magical

Three attempts sound like recovery. They are useful for temporary Redis, database, or upstream connection failures. They do not make every orchestration step idempotent.

The gateway writes its simulation record before enqueueing, so a retry reuses that local ID. runSimulation() confirms that the record belongs to the same tenant and scenario. It also saves upstream project and simulation IDs as they become available. Those records give an operator evidence about where a run reached.

The current orchestrator still starts its remote sequence from the graph phase on a fresh BullMQ attempt. If a response was lost after MiroFish created remote state, a retry can create another upstream project. BullMQ provides at-least-once delivery. It does not provide exactly-once behavior across PostgreSQL, Redis, and MiroFish.

I considered making every step resumable before the first deployment. That would require remote idempotency keys or a local execution ledger with step-level reconciliation. The codebase had no evidence that MiroFish accepts a client-supplied key for each state-changing action. Building resume logic on remote IDs alone could be worse than a clean retry because the gateway might continue a project whose actual state it cannot verify.

Bounded duplicate risk remains during ambiguous network failures. The service records remote IDs and errors, caps attempts, and raises a permanent-failure signal. A human can inspect the upstream state before replaying a dead job. I would add a step ledger once run frequency makes manual reconciliation a recurring operation.

Cancellation exposes the next orchestration gap

The API has a cancellation route. It verifies tenant ownership, rejects simulations already in a terminal state, and marks an active record as cancelled with a completion timestamp. That protects the state transition visible to the caller.

It does not yet interrupt a running MiroFish operation. The BullMQ processor calls runSimulation() and waits. The orchestrator does not poll the local simulation row for a cancellation flag between remote phases. A user can cancel the record while remote work continues, and the worker may later try to write another status.

I would not hide that gap behind the word cancellation. Today the route cancels the gateway's declared intent. It is not a remote kill switch.

Cooperative cancellation belongs at phase boundaries. The orchestrator could check the tenant-scoped row before graph polling, simulation start, report generation, and final commit. BullMQ job removal could stop work that has not begun. A true mid-simulation stop depends on MiroFish offering a supported termination operation.

Response time is the cost. Checking local state adds database reads to a long workflow, while checking too rarely makes cancellation feel false. Phase-boundary checks are the sensible first step because they prevent new expensive work without pretending an in-flight remote call can be recalled.

A crash-recovery test has a precise claim

The repository includes a BullMQ crash-recovery test that checks the contracts surrounding Redis connection parsing, self-contained job data, retry options, worker event handlers, and shutdown ordering. It does not kill a live process in the middle of a real MiroFish run.

That boundary matters. The test proves the code has the pieces BullMQ needs to recover a persisted job. It cannot prove how the full stack behaves after the operating system terminates the worker between a remote side effect and a local status update. Calling it a process-kill test would overstate the evidence.

The service shutdown path is still useful. On SIGTERM or SIGINT, it stops HTTP intake and scheduled polling, closes cleanup work, asks the BullMQ worker to close, then disconnects the queue, Redis, database, and telemetry. worker.close() lets active work drain during a normal shutdown. A forced kill remains a different event.

I would test that event with a real child process and real Redis before claiming automatic crash recovery. The test would enqueue a deterministic long job, kill the worker after a known checkpoint, start a replacement, and verify both job state and external side effects. MiroFish makes the last part expensive, which is exactly why the claim should wait for the test.

One slot can still be an architectural choice

The queue gives the gateway durable admission, retries, backoff, failure events, and a clean HTTP boundary. Concurrency one gives the host a chance to finish the work it accepts. They solve different problems.

This design will reach a limit. A growing queue can make results arrive too late even when every job succeeds. The answer then is capacity isolation: dedicated worker hosts, resource limits, and scheduling based on measured run size. The API and job contract can remain while execution moves outward.

For the current deployment, the most important throughput metric is not jobs started per minute. It is simulations completed without the operating system killing the process.

A Simulation ID Is Not a Running Simulation

Kingsley Onoh — Sat, 29 Aug 2026 11:35:23 +0000

createSimulation() returned an ID. startSimulation() still failed.

That was the sixth version of the gateway's live MiroFish path. Five earlier runs had already found a wrong Redis instance, a duplicate simulation record, a synchronous ontology response mistaken for an asynchronous task, a project ID nested under data, and graph status nested under the same wrapper. The new ID looked like progress. MiroFish answered that the simulation had not been prepared.

The seventh run added preparation and exposed one final missing operation: report generation had to be triggered explicitly after the swarm stopped. The eighth run completed the whole route.

None of those failures required a clever algorithm. They required the client to admit that the upstream API was a state machine, even though its surface looked like a collection of ordinary HTTP calls.

A typed client can still encode the wrong lifecycle

I started with the reasonable model: upload context, ask MiroFish to build a graph, start a simulation, then fetch its report. Each action had a client method. Each response could be assigned a TypeScript type. The call sequence was still wrong.

The first clue was ontology generation. I expected a task ID and wrote polling logic around it. The live service returned the ontology synchronously and placed the project ID inside a { data, success } envelope. Graph construction did return a task ID and required polling. Two adjacent operations used different completion models.

Then the graph-status response used another nested payload. The initial poller read status at the top level, so it kept waiting even after the service had completed the job. That is a dangerous integration failure because nothing crashes. The worker appears busy until its timeout expires.

Simulation startup had a deeper sequence. The project ID from graph construction did not double as a simulation ID. The gateway had to create a simulation, retain the returned simulation ID, prepare profiles and configuration, wait for preparation, then start the run. A report was another asynchronous job with its own task ID and poll cycle.

TypeScript helped once those facts were known. It could not discover them from an API that returned HTTP success for an operation whose downstream state was not ready. Live execution supplied the missing contract.

The response normalizers now stay close to the operations they describe. Profile payloads, action logs, graph data, task status, and report state do not pass through one catch-all decoder. Each normalizer accepts the wrapper shapes observed for that operation and rejects data that cannot produce its domain type.

That creates more small functions than one generic unwrapData() helper. The duplication is intentional. A universal unwrapping function would tell the caller that a payload exists, while concealing whether it contains project_id, simulation_id, task_id, or runner_status. The client needs those distinctions to decide what can happen next.

The live failures also changed the test boundary. Unit tests cover the client methods and poll behavior, but recorded response shapes are only hypotheses until a real MiroFish checkout answers. The launch run became a contract test for the whole lifecycle. I would keep a small fixture set from verified responses so future upstream updates fail at the decoder rather than halfway through a simulation.

I turned the call list into phases

The orchestrator records graph_building, simulating, reporting, completed, and failed states in PostgreSQL. Those states are more than UI labels. They mark ownership transitions between the gateway and MiroFish.

During graph building, the gateway generates a seed document from the tenant's scenario, asks for an ontology, starts graph construction, polls the graph task when one exists, fetches graph data, and mirrors it locally. During simulation, it creates and prepares the upstream run before asking the swarm to start. During reporting, it triggers report generation, waits for that task, fetches the report, parses predictions, and commits the terminal result.

This is the actual center of the simulation phase:

const createResult = await mirofishClient.createSimulation(mirofishProjectId);
const mirofishSimId = createResult.data.simulation_id;

await updateSimulationStatus(
  simulationId,
  tenantId,
  SIMULATION_STATUS.SIMULATING,
  { mirofishSimId },
);

await mirofishClient.prepareSimulation(mirofishSimId);
await mirofishClient.pollPrepareStatus(mirofishSimId, GRAPH_TIMEOUT_MS);

await mirofishClient.startSimulation(mirofishSimId, config);
await mirofishClient.pollSimulationStatus(
  mirofishSimId,
  SIMULATION_TIMEOUT_MS,
);

Every method in that sample exists in MirofishClient. The order is fixed because each call creates state required by the next one. Saving mirofishSimId before preparation gives the gateway a recovery handle if the process fails after the upstream simulation has been created.

I separated graph and simulation timeouts as well. Graph-related polls allow 600,000 milliseconds. A simulation can run for 1,800,000 milliseconds. Using one generic timeout would either abandon legitimate simulations too early or leave a stuck graph task alive for far too long.

The trade-off is explicit coupling. The orchestration code knows more about MiroFish than a generic HTTP adapter would. I accepted that because hiding the lifecycle behind a vague execute() method would make recovery impossible. The gateway needs to know which remote ID exists, which phase owns it, and whether repeating an operation is safe.

Retry only the failures that have not produced remote state

The HTTP layer retries connection failures such as refusal, timeout, DNS failure, and reset. It makes at most three attempts with exponential delay starting at one second. It does not retry every non-success HTTP status.

That boundary is deliberate. Repeating a connection attempt is different from repeating a state-changing request. If the network drops after MiroFish creates a project but before the gateway receives the response, an automatic retry could create a second project. The client cannot infer idempotency from an error code. Broad retries would trade a visible failure for hidden duplicate state.

Polling has its own rules. It accepts complete and completed, understands that simulation state may live under runner_status, and throws when the upstream task reports failure. Delay and timeout are explicit inputs. A graph task and a swarm run share polling mechanics without sharing assumptions about duration or response shape.

I was wrong about how much could be normalized at the HTTP boundary. The { data, success } wrapper looked like the obvious common shape, but completion data still differed by operation. Normalizing every response into one universal result type would erase distinctions the orchestrator needs. The final client keeps operation-specific return types and puts only transport behavior in the shared request helper.

Not every missing artifact should fail the run

Once preparation completes, the gateway tries to fetch agent profiles. After simulation, it tries to fetch action logs. Both are valuable because they power stance summaries and local episode memory. Neither is the report itself.

Those enrichment calls sit inside narrow try blocks. A profile-fetch failure logs a warning and lets the run continue. The same rule applies to action logs. Report generation is different. If the report cannot be triggered, polled, or fetched, the simulation cannot be marked complete because the product has no deliverable.

This distinction prevented a small remote-data mismatch from destroying an otherwise useful run. It also prevents the opposite lie: a simulation with no report cannot pass merely because agent profiles were saved.

The fallback path for action logs shows why this is necessary. MiroFish may expose the log through its API, but the gateway also knows how to read the upstream file output when the API shape is unavailable. Malformed lines are skipped and counted rather than crashing the whole import. The importer hashes an action's simulation identity, agent, round, type, and content into a source key, so retrying the import does not duplicate episodes.

That fallback is useful, but it is also coupling to an upstream implementation detail. I would remove it when the API provides a stable action-log contract across releases. Until then, the gateway records the gap instead of pretending the integration is cleaner than it is.

Failure status must not replace the original failure

The most subtle recovery bug sat in the catch block. When orchestration fails, the gateway updates the simulation row to failed, stores the error, and emits a failure event. But the database update can fail too.

An earlier shape risked replacing the original MiroFish error with the secondary persistence error. That would leave the operator debugging the cleanup step rather than the operation that broke the run. The current code wraps failSimulation() in its own try, logs both errors when persistence fails, and rethrows the original exception.

That ordering is an operational contract. Diagnostic cleanup should add evidence. It should not rewrite history.

Completion events follow a related rule. The gateway marks the report and completion time in PostgreSQL before it publishes simulation.completed. Notification delivery is feature-flagged and best effort. An unavailable notification service cannot retroactively turn a finished swarm run into a failed one. The database remains the authority for simulation outcome.

An event consumer may miss a completion notification during an outage. The service keeps that failure visible in logs, and the simulation can still be recovered from the API. Making the event transactionally guaranteed would require an outbox and a delivery worker. I would add that when downstream automation depends on every event, not merely when notifications are useful.

Eight attempts produced one explicit contract

The final live path generated an ontology, built and mirrored a graph, prepared profiles, ran the swarm, imported action logs, generated a report, and parsed four predictions. Its importance is not the happy-path result. The eight attempts left a client that names each remote phase and a database record that tells an operator where work stopped.

I would still change one part at higher volume. Remote IDs and phase timestamps belong in a dedicated execution-attempt table once the same simulation can be replayed or resumed more than once. The current simulation row is enough for one active attempt and bounded retries. It would become ambiguous if several upstream runs competed for the same logical simulation.

For now, the gateway refuses that ambiguity. One simulation owns one scenario, one upstream project, one upstream simulation ID, and one terminal report. The client does not disguise MiroFish as a single call.

An external API is easiest to operate when the code records the states its documentation leaves between the lines.

Why I Put Swarm Memory in PostgreSQL Instead of Zep

Kingsley Onoh — Sat, 29 Aug 2026 11:35:10 +0000

A completed simulation left me with a report and an ownership problem. MiroFish had built the graph, generated the agents, and run their interactions, but the useful memory still lived behind somebody else's service boundary. The gateway could display the final answer. It couldn't reliably ask its own questions of the evidence that produced it.

That distinction mattered because running a swarm covered only half the requirement. The gateway also had to retain graph entities, agent episodes, profiles, reports, and predictions under the same tenant contract as the rest of the service. A provider-controlled memory layer made every later query depend on that provider's availability, billing model, and isolation semantics. The project estimate put Zep consumption at 30,000 to 40,000 credits per simulation. Daily runs would turn memory retrieval into a recurring external dependency before the gateway had served its first reader.

I moved gateway-owned memory into PostgreSQL 16 with pgvector. I did not pretend this removed Zep from MiroFish. The upstream checkout still uses its configured graph provider while it constructs a graph. The boundary is narrower and more defensible: once MiroFish returns graph data and action logs, the gateway owns the persisted copy and every read against it.

The wrong boundary would have copied the upstream system

Replacing an external memory service sounds like a database migration. That framing invites a bad design: reproduce the provider's graph product inside the gateway, then maintain a second graph engine beside MiroFish.

I needed much less than that. The gateway has three read patterns. It retrieves entities by type, walks a bounded neighborhood, and searches agent episodes by meaning. PostgreSQL already holds tenant, scenario, simulation, report, and prediction records. Adding graph nodes, graph edges, and episodes kept the ownership boundary in one transactional store. pgvector covered the semantic query without introducing another network call.

This choice also made the tenant rule visible. Every memory query accepts tenantId as an argument. It isn't recovered from a global context and it isn't optional. The graph tables carry tenant_id and simulation_id, while unique constraints stop the same upstream entity or episode from being inserted twice for one simulation. A node can have the same upstream identifier in two tenants without either tenant seeing the other record.

The relevant comparison put one owned store against another service boundary with separate credentials, failure modes, and billing. PostgreSQL won because the required traversals were bounded and the semantic search was local. If the product later needs arbitrary path analysis across millions of graph edges, that decision deserves another review. The launch scope did not need that machinery.

One schema, two kinds of memory

Graph entities and agent episodes look similar because both can carry embeddings. They have different jobs.

graph_nodes stores named entities mirrored from the MiroFish graph. The default embedding text combines the entity type and name, which gives short labels enough context to be searchable. graph_edges stores typed relationships and their source and target identifiers. agent_episodes holds the actions produced during simulation rounds, including the agent, action type, content, metadata, and a vector derived from the content.

I kept the embedding width at 384 dimensions because the local Xenova/all-MiniLM-L6-v2 model emits exactly that shape. The model runs in process through @xenova/transformers, uses mean pooling, and normalizes the output. The database columns are vector(384), so a model swap that changes dimensions is a schema decision, not a package update hidden in an application release.

That coupling is deliberate. Silent dimensional mismatch is worse than an explicit migration. The table definition, embedding constant, and tests all name 384. A future model can be better on paper and still be the wrong choice if it forces an unplanned rewrite of every stored vector.

The embedding extractor is initialized lazily and shared through one promise. That avoids loading the ONNX model for requests that never touch semantic search and prevents concurrent callers from starting duplicate model loads. The trade-off is cold-start latency on the first embedding request. I accepted it because this gateway runs long simulations, not millisecond search ads. If search becomes an interactive first-screen feature, I would pre-warm the model during service startup and expose its readiness separately.

The query carries the security contract

Here is the actual semantic retrieval function. The important part isn't the cosine operator. It is the set of boundaries around it.

export async function searchEpisodes(
  tenantId: string,
  simulationId: string,
  agentId: number | null,
  query: string,
  topK = 10,
): Promise<EpisodeSearchResult[]> {
  const queryEmbedding = await generateEmbedding(query);
  const vectorLiteral = `[${queryEmbedding.join(',')}]`;
  const agentFilter =
    agentId !== null ? sql`AND agent_id = ${agentId}` : sql``;
  const safeTopK = Math.max(1, Math.min(100, Math.floor(topK)));

  const result = await db.execute(sql`
    SELECT id, tenant_id, simulation_id, agent_id, content,
      embedding <=> ${vectorLiteral}::vector AS distance
    FROM agent_episodes
    WHERE tenant_id = ${tenantId}::uuid
      AND simulation_id = ${simulationId}::uuid
      ${agentFilter}
    ORDER BY distance ASC
    LIMIT ${safeTopK}
  `);

  const rows = Array.isArray(result)
    ? result
    : ((result as { rows?: unknown[] }).rows ?? []);
  return (rows as Record<string, unknown>[]).map(rawRowToEpisodeResult);
}

The excerpt shortens the selected columns for readability, but the function and query are the ones the service executes. tenantId and simulationId are both predicates. An optional agent filter narrows the search without weakening either ownership condition. topK is clamped between 1 and 100 before it reaches SQL.

That cap is not search tuning. It is resource control. A caller cannot turn semantic retrieval into an unbounded result set, and the API doesn't need to trust every upstream parameter. The graph-neighbor query follows the same idea by clamping traversal depth between 1 and 20.

Cosine distance is converted into a similarity score by the row mapper. Keeping that conversion outside the query lets the SQL remain about ordering while the application returns a domain value. It also gives tests a single place to catch vector strings, numeric arrays, and database-driver result shapes.

I trusted mocked persistence for too long

The first graph-store tests mocked the database. They verified query construction and method behavior, which was useful while the schema was moving. They could not prove that PostgreSQL had the vector extension, that Drizzle's custom type crossed the driver boundary correctly, or that pgvector returned the expected ranking.

That was the wrong confidence level for code whose main claim was ownership of persistence.

The project rules say not to mock infrastructure we own. The progress record called out the gap directly: real pgvector integration had been deferred. I added integration coverage against Dockerized PostgreSQL and pgvector. Those tests insert real 384-value vectors, execute the store functions, verify tenant-scoped node reads, check edge traversal, and assert semantic ranking against the database operator.

What surprised me was how much of the risk sat below TypeScript. A perfectly typed number[] says nothing about whether a migration enabled the extension or whether a raw result comes back in the shape the mapper expects. The integration test catches a class of failure that a mock is structurally unable to represent.

There is still a smaller unit-test layer around deterministic behavior, including embedding generation and safe limits. I kept it because model loading is slower than testing a clamp or a mapper. The split test strategy creates two kinds of evidence: fast tests give narrow feedback, while the PostgreSQL suite carries the persistence claim. Treating either one as the whole answer would hide risk.

Mirroring is not pretending to be the source

After graph construction, storeMirofishGraph writes valid nodes before edges. Episodes arrive later from action logs. The gateway uses conflict handling to make repeated writes safe for the same simulation. That matters because orchestration can retry after a network failure even when the upstream operation actually completed.

The mirror is repairable rather than atomic. Nodes are written in parallel, then edges are written in parallel. There is no transaction around the complete graph import. A database failure can leave some nodes present and no corresponding edges.

I accepted that because the upstream graph remains available for the duration of orchestration and every local identity has a conflict key. Retrying the import updates existing nodes and fills missing records instead of multiplying them. A transaction across the full graph would give a cleaner all-or-nothing result, but it would also hold locks while embedding generation and many inserts complete. The current graph is small enough for replay to be the simpler recovery tool.

That decision needs a boundary. The gateway does not mark the local memory phase complete until graph fetch and storage return. A partial mirror from a failed run is diagnostic residue, not a successful retrieval store. At larger volumes, I would stage imported rows under an attempt ID and promote the set after validation. Readers could then ignore incomplete attempts without requiring one long transaction.

I deliberately call this a mirror. MiroFish remains the source of graph-construction output for that run. PostgreSQL becomes the source for gateway reads after ingestion. If graph import fails halfway through, the run should not be described as having durable local memory. If profile or action-log enrichment fails after the simulation itself succeeds, the orchestrator records a warning and continues toward the report. Those two failures have different business meaning, so they do not share one blanket retry policy.

The design also leaves one obvious limitation. storeMirofishGraph uses parallel writes for nodes and then edges. That is appropriate for the small verified graph, which contained 12 nodes and 12 edges. At much larger graph sizes, I would replace per-record writes with staged bulk inserts and measure index cost during ingestion. I would not add that machinery before the data proves it is needed.

The result is not a home-grown replacement for every feature in Zep. It is an owned retrieval contract. A tenant's simulation memory sits beside its scenario and predictions. Semantic search stays inside the gateway's database boundary. Upstream graph construction can change without taking gateway reads with it, provided the returned graph and action-log contracts remain valid.

That is the line I wanted: borrow computation where it saves time, own the memory that gives the computation lasting value.

Zero Is Not an Empty Value in Financial Software

Kingsley Onoh — Thu, 27 Aug 2026 10:24:45 +0000

What does zero downside mean in a cloud commitment recommendation?

It should mean that the modeled draws produced no loss after commitment cost, unused capacity, upfront amortization, and liquidity penalty. In one optimizer run, it meant something else: the helper that summarized the frontier had no idea how to represent an empty array.

The candidate carried a p95 downside loss of 6,300 cents. The frontier summary reported zero. The ranked policy relaxation also suggested zero. Nothing crashed. The run completed, the JSON was valid, and the wrong value looked unusually good.

That is the kind of defect I worry about in financial software. A loud exception stops a decision. A plausible zero can approve one.

The Value Had Already Changed Meaning

The Cloud Commitment Portfolio Optimizer compares commitment candidates across AWS, Azure, and GCP. Each candidate has expected savings, a commitment amount, utilization percentiles, a p95 downside loss, and a set of binding policy constraints. The worker sorts feasible candidates by expected savings, then by downside.

The same candidate list also feeds a frontier summary. That summary reports the best expected savings and the lowest p95 downside among all candidates. If no candidate satisfies the active policy, the worker persists an infeasible run and suggests which policy limit would need to move.

Those are three different uses of the same number.

For a candidate, zero downside describes an economic result. In a frontier, zero can be the best observed value. In a reducer, zero was being used as “there was no first value.” The type was bigint in all three places, so TypeScript could not tell them apart.

The first implementation reduced the list from 0n. That is a common instinct because it keeps a helper total. It is also correct for sums. For a minimum over positive values, it is a trap:

min(6300, 0) = 0
min(4100, 0) = 0
min(1, 0) = 0

Every positive downside loses to the sentinel. The larger the actual risk, the more confidently the summary still says zero.

Why This Was Easy to Miss

The economic calculation itself was not wrong. In the Zig kernel, downside is derived from net savings and clamped at zero. A no-action case can legitimately produce zero across every economic field. In the TypeScript worker, each candidate calculates its own downside distribution, selects the 95th percentile, checks it against the policy budget, and assigns feasibility.

The defect lived after all of that, inside an aggregate used for presentation and infeasibility guidance.

That location matters. Most tests were naturally aimed at the main path: claim a queued run, read the frozen forecast, load versioned price items, produce a recommendation, store the frontier, and mark the run complete. A successful candidate fixture had its expected saving and p95 downside asserted directly. It passed.

The bug became visible only when the test forced an infeasible objective and then inspected two secondary outputs: lowest_p95_downside_loss_cents in the frontier and max_downside_loss_cents in the ranked relaxation. Both should have carried 6300. Both carried the reducer's sentinel.

I was wrong about where the risky code was. I expected the difficult defects to sit in percentile selection, amortization, or provider-specific eligibility. The failure came from a utility function small enough to read without stopping.

There was another reason it survived ordinary review. Zero looked defensive. An empty array would not throw. The API could still return a stable shape. That defensive default removed a runtime failure by creating a business statement.

The Constraint Was Bigger Than a Helper

I could not replace the value with JavaScript Infinity. The worker uses bigint because money crosses the system as canonical decimal strings and must not pass through floating-point numbers. bigint has no infinity value, and introducing a number sentinel would break the numeric contract.

Throwing on an empty list was possible, but emptiness is not always exceptional. A run can produce no candidates when price coverage does not match the forecast scope. That run should become infeasible with an explanation, not fail as an internal error.

Returning zero for an empty list kept the existing frontier schema stable, but only if the reducer handled non-empty lists from a real value. The current minBigInt() does that:

function minBigInt(left: bigint, right: bigint): bigint;
function minBigInt(values: readonly bigint[]): bigint;
function minBigInt(leftOrValues: bigint | readonly bigint[], right?: bigint): bigint {
  if (typeof leftOrValues === "bigint") return leftOrValues < right! ? leftOrValues : right!;
  const [first, ...rest] = leftOrValues;
  return rest.reduce((minimum, value) => (value < minimum ? value : minimum), first ?? 0n);
}

For a non-empty list, the first candidate becomes the seed. A list containing 6300n now returns 6300n. Two values compare with each other, not with a value invented by the helper.

The empty fallback remains 0n, which is a tradeoff rather than a perfect model. The caller also records candidate_count, so a consumer can distinguish an empty frontier from a risk-free candidate. If I redesigned the contract now, I would make the aggregate nullable when candidate_count is zero. That would move the distinction into the schema instead of asking readers to infer it from two fields.

I considered splitting the overload into two named functions: one for comparing two values and one for aggregating a collection. That would reduce the chance that a caller accidentally reaches the array branch, but it would not settle the empty-state question. The return type still has to say whether no minimum exists. A cleaner contract would return bigint | null for the collection form and force buildFrontier() to serialize that absence deliberately. The current repair stayed smaller because the published summary already pairs its amount with candidate_count.

I kept that redesign out of the repair because it would have changed the API and report contract during a provider-expansion batch. The focused correction restored truth for every non-empty frontier without widening the release. A contract migration deserves its own tests and version decision.

The Test Had to Cross the Boundary

A unit test for minBigInt([6300n]) would prove the arithmetic. It would not prove that the number reached the places finance reads.

The integration fixture creates an active policy with a downside budget of 10 cents and a minimum expected saving of 500 cents. It feeds three forecast points of 1,000 cents against a monthly effective commitment cost of 7,300 cents. The resulting candidate has no positive expected saving and carries 6,300 cents of downside.

The test then runs the real worker against PostgreSQL and the object store. It checks four consequences:

The run ends as infeasible.
No recommendation row is inserted.
The persisted relaxation suggests a downside budget of 6300, not zero.
The frontier artifact reports its lowest p95 downside as 6300.

That path matters more than direct helper coverage. The same value passes through candidate evaluation, aggregation, artifact serialization, database persistence, and the API-facing summary. A regression in any one of those steps breaks the test.

The fixture also checks that the artifacts do not expose credentials, raw rows, stack traces, or internal candidate IDs. Financial correctness and disclosure boundaries belong in the same proof path. A correct risk number in an unsafe artifact is still a failed design.

Zero Has Several Jobs, So It Needs Context

There are legitimate zeros throughout this optimizer. No commitment means zero committed capacity. A profitable draw has zero downside loss. A policy can permit zero minimum expected savings. An empty collection can have a count of zero.

The mistake was letting one zero stand in for all of them.

This shows up beyond minimum functions. An absent price is not a zero price. A forecast with no history is not zero demand. An approval that has not been requested is not a rejection. A retry count of zero says no attempt has happened; it does not say delivery succeeded.

The safest representation follows the business state. Use a number for an amount. Use null for an unavailable aggregate when the contract permits it. Use a status for a workflow state. Use an empty collection when there are no members. Problems start when a convenient primitive is asked to carry two of those meanings at once.

What surprised me was not that a reducer could be wrong. It was how far a six-line helper reached. The wrong seed affected the frontier summary, the policy relaxation, the report a reviewer could quote, and any later analysis built from that artifact. The economic kernel had done its job. The summary layer changed the claim.

The Result

Before the repair, the infeasible fixture stored lowest_p95_downside_loss_cents: "0" and suggested a zero downside budget. After the repair, both fields preserve the candidate's "6300". The worker still returns a valid infeasible result, writes no purchase recommendation, and keeps the artifact free of internal or secret data.

The same worker dispatches five provider and instrument paths through the shared evaluation contract. The focused infeasible test now guards the aggregate that all five use. That is more valuable than five copies of the same helper test because it protects the common financial statement at its persistence boundary.

In risk software, empty and zero are different financial statements.

JSONB Was Fine. The Side Effects Needed a State Boundary.

Kingsley Onoh — Tue, 16 Jun 2026 10:26:01 +0000

What should happen when a checklist item sends a client message?

In this portal, that question starts with a milestone stored as a JSON object inside projects.milestones_json. The same milestone can also be a task linkage through tasks.milestoneKey. It can produce a client-visible update. It can fire a Notification Hub event. It can change what the client sees in the portal and what the operator sees in the CLI.

That is too much authority for one checklist item.

The early temptation was simple: keep milestones as JSONB because project setup needed to be fast. A client project does not need a full ceremony for every step. Sometimes the operator needs to add three milestones from the CLI, mark the first one done, and move on. A separate milestone table felt heavier than the problem.

I still think that part was right.

The part I got wrong was assuming the storage choice was the design decision. It wasn't. The real decision was what happens when that flexible JSON object creates side effects outside the JSON field.

The Real Problem

A milestone stored as JSONB is easy to edit and hard to govern. Postgres will store the array. Drizzle will read it back. TypeScript can normalize the shape. None of that answers the business question: when a milestone is marked done, who is allowed to know?

The portal had several competing truths:

The admin route knows which project is being changed.
The client record knows whether notifications are enabled.
The project knows whether it is client-visible or internal-only.
The milestone object knows its own notifyClient and notificationMode values.
The task table knows whether linked work is complete.
The notification layer knows whether to emit an event.

If any one of those edges is checked loosely, the result can be wrong while every individual function still returns 200.

That is the kind of failure that bothers me most. Not a crash. A clean success response attached to the wrong business truth.

The build journals had already taught that lesson elsewhere. One batch found a path where the URL project and update project could diverge inside the same tenant. Another found joined project and client rows being hydrated by ID only after the root row had been tenant-scoped. The first query was safe. The side edge was not.

I was wrong to treat tenant isolation as a problem solved at the start of a request. Every side effect has its own identity boundary.

The Constraints

I did not want a new table just to make the design feel pure. The system was still an internal operating tool. The PRD target was under 50 clients and 20 peak requests per second. The operator needed speed more than relational ceremony.

Milestones also had to stay pleasant from the CLI. The code already had a route that marks a milestone done by a 1-based index. That is not glamorous, but it matches how operators think: first milestone, second milestone, third milestone. Forcing every milestone through IDs too early would make the command surface worse.

But the shortcut had limits.

projects.milestones_json can hold flexible milestone data. It cannot decide whether an email should be sent. It cannot prove the project is tenant-scoped. It cannot decide whether a task title should be backfilled into a milestone key. It cannot stop an internal-only project from producing a client-visible message.

So the storage stayed flexible, and the side effects became strict.

The Design

The core helper is normalizeMilestones() in src/lib/milestones.ts. It does the unglamorous work first: discard empty names, preserve known keys, generate missing keys, deduplicate collisions, and coerce done into a real boolean. That gave the JSONB field a predictable shape without changing the database design.

Then syncProjectMilestoneStatus() handles the inverse direction. If tasks are linked to a milestone key, their state can mark a milestone complete only when the linked tasks justify it. That lets the task table and JSONB array communicate without pretending JSONB is relational.

The more interesting function sits in src/routes/admin/projects.ts. It is small enough to look boring, which is why I like it.

function shouldEmitMilestoneNotification(params: {
  notifyClient?: boolean;
  notificationMode?: 'silent' | 'material_updates' | 'all';
  preferenceMode: NotificationPreferenceMode;
  notificationsEnabled: boolean;
  projectVisibility: typeof projects.visibility.enumValues[number];
}) {
  if (!params.notificationsEnabled) return { emitted: false, reason: 'client notifications disabled' };
  if (params.projectVisibility === 'internal_only') return { emitted: false, reason: 'project is internal only' };
  if (params.notifyClient === false) return { emitted: false, reason: 'notifyClient false' };
  if (params.notificationMode === 'silent') return { emitted: false, reason: 'notificationMode silent' };
  if (params.notifyClient === true) return { emitted: true, reason: 'notifyClient true' };
  if (params.notificationMode === 'material_updates' || params.notificationMode === 'all') {
    return { emitted: true, reason: `notificationMode ${params.notificationMode}` };
  }
  if (params.preferenceMode === 'material_updates' || params.preferenceMode === 'all') {
    return { emitted: true, reason: `project notification preference ${params.preferenceMode}` };
  }
  return { emitted: false, reason: `project notification preference ${params.preferenceMode}` };
}

This is not a clever algorithm. It is a contract.

The order matters. Client notifications being disabled beats everything. Internal-only project visibility beats an eager milestone setting. An explicit notifyClient: false beats the project preference. Silent mode beats a general setting. Only after those denials does the function permit an event.

That order reflects the business. The safest choice must win first.

The route around it does the heavier lifting. It loads the project by id and tenantId. It loads the client by id and the same tenantId. It normalizes milestones before touching the selected 1-based index. It treats an already-completed milestone as idempotent. It writes a portal update. Only then does it ask whether to emit the Notification Hub event.

The notification event itself is fire-and-forget. That is a separate design choice from the milestone boundary. The client state change should not roll back because an email layer is unavailable. The update exists. The portal can show it. The notification failure can be logged and retried outside the request path.

Tests make the boundary real. The event tests cover milestone completion, suppression when notifyClient is false, suppression when project preference is portal-only, and the case where a portal update should exist even when email does not fire. Those are not framework tests. They are business truth tests.

The backfill route is another scar. Tasks gained milestoneKey after milestone JSON already existed. The route defaults to dry-run and requires confirm=true before it mutates task titles and milestone keys. That is what a state boundary looks like when the data model changes underneath a live operator workflow: preview first, then write.

What Surprised Me

I expected the risky part to be JSONB. It was not.

The risky part was side-effect drift. A JSON object can be perfectly valid and still produce the wrong portal update. A project can be tenant-scoped and still attach a joined row carelessly later. A notification can have the right event name and the wrong visibility rule.

That changed how I read the rest of the portal code. I stopped asking only, "Is the row scoped?" I started asking, "Which other facts will this action create, and do they share the same boundary?"

That question shows up everywhere in this project: comments, reports, document caches, capacity notices, stale work, handoff summaries, and client asks. The portal is not just storing facts. It is deciding which facts are safe to expose.

The Result

The final design kept the flexibility that made milestones useful from the CLI, but moved the risk into explicit gates. JSONB still holds the editable project steps. Tasks can link to those steps. Completion can produce a portal update. Notifications can fire only when the client, project, milestone, and preference rules agree.

The latest recorded build gate reached 416 total tests, with 398 passing and 18 skipped. More important than the count, the tests caught wrong-but-running states: missing ask classification, stale internal project noise, QA due noise, generic reply drafts, and untrimmed handoff output.

That is the transferable part. Flexible storage is fine when the business fact is local. The moment it creates a side effect, treat it like a state machine.

Why I made OR-Tools prove it was better than the deterministic dispatcher

Kingsley Onoh — Mon, 15 Jun 2026 20:05:19 +0000

Dispatch optimization needs a lower bound before it needs a clever objective.

In the first real OR-Tools integration, the solver selected fewer assignments than the deterministic fallback it needed to improve. That result made the boundary explicit: CP-SAT could optimize cost, priority, and tie-breakers only after it matched or beat the deterministic feasible assignment count.

The constraint changed how I treated OR-Tools inside the dispatch engine. I had treated the solver as the smarter engine in the room. The code reminded me that dispatch combines math with an operating record. A plan has timestamps, frozen work, post-selection capacity checks, replay metrics, and explanations a dispatcher can defend after the board changes.

The tempting version

The tempting version is simple. Build one boolean variable per eligible technician-job decision. Add constraints for job uniqueness, technician capacity, planning windows, and frozen work. Maximize the objective. Return the result.

That version reads well in a design doc. It is also too trusting for dispatch.

A field-service board has commitments. A dispatcher accepts a plan. A technician starts driving. A supervisor freezes a job. A customer is waiting against an SLA clock. If the solver returns an answer that is mathematically feasible but operationally worse, the system still has to notice.

The code that changed the contract

The final adapter runs deterministic solving first, uses that count as a lower bound, and then lets CP-SAT optimize within that boundary.

val vars = linearArgs(decisions.map(_.variable))
val deterministicAssignmentCount = fallback.solve(model, options).assignments.size
if (deterministicAssignmentCount > 0) {
  val _ = cp.addGreaterOrEqual(LinearExpr.sum(vars), deterministicAssignmentCount.toLong)
}
val coeffs = decisions.map { decision =>
  val assignmentReward = 1_000_000L - (decision.job.priority.rank.toLong * 10_000L)
  val cost = (decision.cost.total * BigDecimal(100))
    .setScale(0, BigDecimal.RoundingMode.HALF_UP)
    .toLong
  assignmentReward - cost - jobIndex(decision.job.id).toLong * 100L -
    technicianIndex(decision.technician.id).toLong
}.toArray
cp.maximize(LinearExpr.weightedSum(vars, coeffs))

That sample is from OrToolsSolverAdapter.solveWithCpSat. The assignment reward is intentionally large. Priority affects the reward. Cost is scaled to an integer. Job and technician indexes act as stable tie-breakers.

The line that matters most is not the maximize call. It is cp.addGreaterOrEqual(LinearExpr.sum(vars), deterministicAssignmentCount.toLong). That line says the solver is allowed to optimize, but it is not allowed to schedule less work than the deterministic feasible path already found.

Why deterministic sequencing stayed

Even after CP-SAT selects decisions, the system does not blindly stamp them into the board. It passes selected decisions through deterministic scheduling. That second stage can still reject work for capacity or planning-window overflow.

At first, that felt redundant. If the solver has constraints, why check again?

Because the dispatch plan is not only a set of pairs. It is a sequence of visits with concrete start times, travel, overtime, and explanation codes. Stable timestamps matter for replay. Stable rejection reasons matter for support. The deterministic layer turns selected pairs into an operating plan that looks the same when the same input snapshot is replayed.

That also protects partial plans. A solver timeout or infeasible slice should not fabricate certainty. The domain has reason codes such as missing_capability, frozen_assignment, capacity_exceeded, outside_planning_window, and solver_timeout. A partial plan with honest unscheduled work is safer than a complete-looking plan built on silence.

Frozen work was the real domain invariant

The solver failure was loud because it affected assignment count. Frozen work is quieter and more dangerous.

The constraint builder treats accepted, completed, and frozen assignments as hard facts. A technician who conflicts with frozen work is rejected. A job that would collide with preserved work does not get moved just because the global objective improves.

That choice is easy to miss if you only look at optimization. A solver optimizes variables. Dispatchers manage promises. Once a human has accepted work, the board has a memory. The optimizer has to respect that memory.

What surprised me

The surprise was not that OR-Tools needed constraints. That is normal. The surprise was that the deterministic implementation became a guardrail for the solver rather than dead code waiting to be deleted.

I kept it for three reasons.

First, it gives the CP-SAT model a feasible assignment lower bound. Second, it gives the app a fallback when native solver loading, runtime failure, or timeout happens. Third, it gives replay a baseline that operators can compare against using SLA hit rate, travel minutes, overtime minutes, churn moves, unscheduled jobs, and solve time.

That makes the deterministic path part of the product, not a temporary scaffold.

The tradeoff

The cost is extra machinery. There are two solve paths. There is trace metadata. There are post-selection checks. There are tests that assert OR-Tools was invoked, no fallback happened, and deterministic results still match where they should.

The benefit is that optimization no longer gets special trust. It has to earn its place inside the operating record.

That is the lesson I took from this build: in systems that move real work, a smarter algorithm is not automatically the source of truth. Sometimes the older deterministic code is the witness that keeps the new optimizer honest.

Evidence Beats Certainty: Why My Classifier Refuses to Pretend Every Product Has an Answer

Kingsley Onoh — Sat, 13 Jun 2026 21:43:17 +0000

Batch 010 found a bug that looked like good news.

The classification worker was finishing its work. Runs moved through the database. Product rows had candidate tariff codes. The regression suite was far enough along that a casual glance could have treated the classifier as alive.

Then one test forced three uncomfortable cases through the loop: no candidate, weak confidence, and a near tie. All three came back looking too clean. The worker was persisting the run as classified, even when the evidence said the product needed review or had no supportable recommendation.

That is the kind of bug I worry about in compliance software. Not the loud crash. The green row.

A customs classifier can fail by throwing an exception. That failure is annoying, but honest. The operator sees it. The queue stops. The job gets retried. The audit trail can say, plainly, that classification did not happen.

The worse failure is a result that looks complete while the evidence underneath is missing or contested.

That was the real Batch 010 scar. The engine already carried the domain rule in its intent: classification is evidence, not a label. But the persistence path was still treating classification as if the only final state that mattered was success. The runtime could produce rejected candidates and confidence values. The database could store failure reasons. The tests could express review states. One narrow path still flattened doubt into completion.

I was wrong about where the risk sat. I expected the hard part to be selecting the tariff code. The harder problem sat one layer later: making sure the code was not selected when the evidence did not deserve that much authority.

Customs data makes that tension obvious. A product row is rarely a clean ontology entry. It is a SKU, a commercial name, a description written by someone under time pressure, a country of origin, a jurisdiction, maybe a material list, maybe an intended use. The difference between a good HS or HTS recommendation and a dangerous one can be a phrase that is absent, ambiguous, or buried in the wrong field.

So I made the classifier refuse to pretend. If a product lacks a candidate, it should be blocked. If the best candidate is too weak, it should go to review. If two candidates are close enough that the lower one is still meaningful, the engine should preserve that tie instead of hiding it behind a confident-looking status.

The decision lives in a small Rust function, which is why I like it. The policy is not scattered across a UI badge, a worker branch, and a reporting query. The worker asks one question: given the runtime outcome, what status should the database store?

pub(super) fn outcome_decision(outcome: &RuntimeClassification) -> OutcomeDecision {
    if outcome.selected_code.is_none() {
        return OutcomeDecision {
            status: "blocked",
            failure_reason: Some("no_candidate"),
        };
    }
    if has_tie_candidate(outcome) {
        return OutcomeDecision {
            status: "needs_review",
            failure_reason: Some("tie_candidate"),
        };
    }
    if outcome.confidence < 0.82 {
        return OutcomeDecision {
            status: "needs_review",
            failure_reason: Some("low_confidence"),
        };
    }
    OutcomeDecision {
        status: "classified",
        failure_reason: None,
    }
}

That function from src/classification/outcome.rs is not clever. It is deliberately plain. It says the classifier has four questions to answer before it earns the right to call a run classified.

First, did the runtime select any code at all? If not, the run is blocked/no_candidate. The operator should not see an empty answer wearing the same status as a resolved classification.

Second, did the runtime find a meaningful tie? The rule runtime marks lower-ranked matches as rejected candidates, and a near tie gets the reason tie_score. In that case the selected code still matters, but it is not enough. The run becomes needs_review/tie_candidate.

Third, did the selected code clear the confidence floor? The current worker uses 0.82 as the line below which a product should not pass as clean. That number is a code-backed threshold, not a production claim. It is there because the engine needs a deterministic boundary for review routing.

Only after those checks does the run become classified.

The order matters. No candidate is different from low confidence. Low confidence is different from a tie. A tie with a selected code is different from a rule pack that found nothing. If those cases all share a green status, the UI can only lie or become complicated later. If the status and reason are precise at write time, the rest of the product can stay simpler.

The test that caught this is the kind of test I wish more systems had before they gained users. It does not test the happy path with a cotton shirt and a confident tariff code. It creates three products with names that force the worker to admit uncertainty.

One row has no matching rule. One row matches with a confidence below the floor. One row matches two close candidates, 6205.20 and 6205.30, close enough that the rejected candidate still belongs in the record. The assertion is not only that the worker completes three jobs. It checks the stored status, the failure_reason, the selected code where one exists, the candidate code list, and the tie_score reason inside rejected candidates.

That last part matters. I did not want a review queue filled with vague work items that say, "please check this." I wanted the queue to carry the reason the machine gave up authority. A reviewer should know whether they are handling an empty result, a weak result, or a contested result.

The same logic affects audit exports. An audit pack that says a product was classified is different from an audit pack that says the system found two close candidates and routed the run to review. In both cases, the export has value, but it answers a different question. One says, "here is the evidence behind the recommendation." The other says, "here is the evidence behind the refusal to recommend."

That distinction changes the product shape. The engine stores matched rules, rejected alternatives, confidence, risk band, rule pack version, input snapshot, reviewer decisions, and failure reasons. It also freezes the product and rule pack facts at queue time. If the product description changes after the job enters the queue, the worker still evaluates the snapshot it was handed. If the active rule pack changes later, historical runs still point back to the pack version that produced them.

That is slower to reason about than a direct request that always reads current product state. It is also safer. A compliance review is not asking, "what would the system say today?" It often asks, "what did the system know then, and why did it make that call?"

What surprised me was how much of the architecture flowed from that one sentence.

The classifier uses a PostgreSQL job table instead of pretending a background job is a fire-and-forget detail. A worker leases rows, marks attempts, and exits if a run is already terminal. Product import refuses rows that lack required facts such as SKU, name, description, country, jurisdiction, product type, materials, or intended use. Rule packs have activation gates before they become active. Reviewer overrides append structured corrections instead of mutating the machine result. Audit exports are rendered from frozen snapshots instead of live joins that could drift.

Those choices sound separate until Batch 010 ties them together. If the worker writes the wrong status, every careful snapshot around it becomes less trustworthy. The audit export preserves the wrong conclusion. The review queue misses the item. The dashboard looks cleaner than the evidence. Optional integrations can fire the wrong event. A bad status is not a display bug. It is an evidence bug.

The fix was small because the earlier design had already made room for it. The database had a status field and a failure reason. The runtime returned selected and rejected candidates. The tests could create all three edge cases. Once the regression exposed the lie, the code only had to make the domain decision explicit.

I also changed how I read passing tests after that. A test that proves a worker completed is not enough for a compliance loop. Completion is only a transport fact. The domain fact is whether the stored row still carries the same uncertainty the runtime produced. That is why the regression checks status, failure_reason, selected_code, candidate_codes, and rejected candidate reasons in one place. If any one of those drifts, the row may still look finished, but the evidence contract is broken.

That is the lesson I took from it, and I mean lesson in the practical sense, not as a slogan. If a domain has reviewable uncertainty, model that uncertainty before the happy path spreads through the codebase.

For this project, uncertainty has names: no_candidate, low_confidence, and tie_candidate. Those names are not UI copy. They are durable outcomes.

A classifier that always returns an answer is easy to demo. It is also easy to distrust. In customs work, the more serious promise is narrower: when the evidence is good enough, store the recommendation; when it is not, store the reason it stopped.

That is why the Trade Compliance Classification Engine refuses to treat every product as solved. Certainty is useful only when the record can prove how it was earned.

Why I Made Stale Forecasts Fail Instead of Falling Back to Do Nothing

Kingsley Onoh — Fri, 12 Jun 2026 23:03:41 +0000

The UI showed ready, do_nothing, and a blank reason field.

A facility manager reading that screen would assume the engine had looked at the peak window, checked the assets, and decided there was nothing worth doing. The interface looked calm. The audit trail looked complete.

That was not true.

The forecast behind the plan had expired. The planner should never have scored it. But the fallback path did exactly what I told it to do: when no action was selected, choose do_nothing so the operator gets a safe recommendation instead of an empty response.

Safe fallback became false confidence.

The product has a simple rule: physical feasibility comes before economics. A battery cannot discharge below its minimum state of charge. A building load cannot curtail past its comfort limit. A flat-rate tariff cannot justify peak curtailment because there is no peak signal to respond to. Those are business truths encoded as code.

That rule exists because energy planning has a dangerous temptation: collapse every problem into money. If the demand charge is high enough, the spreadsheet always finds a saving. Real facilities do not work that way. Operators know some processes cannot move, some comfort limits cannot bend, and some battery cycles are not worth spending for a small peak reduction. The planner has to encode that judgment before it calculates expected savings.

The bug came from treating stale forecasts like another physical constraint.

In a real infeasible window, do_nothing is useful. If every battery is depleted, every comfort limit blocks curtailment, and every flexible process is already at its limit, doing nothing is a valid operational recommendation. It tells the operator: the engine understood the window and found no feasible savings-positive action.

A stale forecast is different. It means the engine did not have permission to reason about the window at all. The input is invalid. The correct output is a failed plan with an explicit reason.

I got that boundary wrong in the first implementation.

The code had all the pieces in separate places. createPlan detected stale forecasts. generateCurtailmentPlan recorded a stale-forecast rejection. But the bottom of the planner had a broad fallback: if no selected actions exist, add do_nothing. That line was written for infeasible windows, not invalid input, but it had no way to know the difference.

The fix looks small because the hard part was naming the boundary.

Batch 009 had already moved the planner away from fake input. Forecast creation loads qualified interval readings from the database. Plans check that the selected forecast and tariff belong to the requested site. The selected band travels as forecastBandKw, and the action payload records both confidence_band and forecast_band_driver. Those pieces made the failure more embarrassing, not less. The system had evidence discipline at the edges, then lost it in one central fallback.

const staleForecastFailureReason* = "stale forecast cannot be used for a new plan without explicit override"

proc planStatusForDecision*(decision: PlannerDecision; stale: bool): (string, string) =
  if stale: ("failed", staleForecastFailureReason)
  elif decision.selectedActions.len > 0: ("ready", "")
  else: ("failed", "no feasible planner action")

proc generateCurtailmentPlan*(ctx: TenantContext; input: PlannerInput; tariff: PlannerTariff; assets: seq[PlannerAsset]; staleForecast: bool): PlannerDecision =
  validateInput(ctx, input)
  var selected = newJArray()
  var rejected = newJArray()
  var binding = newJArray()
  let curtailAllowed = addStaleAndTariffRejections(tariff, staleForecast, rejected, binding)
  var selectedSavings = 0.0
  for asset in assets:
    if asset.rejectMissingAssetTelemetry(rejected):
      continue
    if asset.assetType == "battery":
      addBatteryAction(asset, input, tariff, staleForecast, selected, rejected, selectedSavings)
      addChargeBatteryAction(asset, input, tariff, staleForecast, selected, rejected, selectedSavings)
    elif asset.assetType in ["building_load", "flexible_process"]:
      addCurtailAction(asset, input, tariff, curtailAllowed, staleForecast, selected, rejected, selectedSavings)
      addShiftLoadAction(asset, input, tariff, staleForecast, selected, rejected, selectedSavings)
  if selected.len == 0 and not staleForecast:
    selected.add(%*{"action_type": "do_nothing", "reason": "no feasible savings-positive action after physical constraints"})
  addBindingRejections(rejected, binding)
  makeDecision(input, tariff, selected, rejected, binding, selectedSavings)

That and not staleForecast is the visible change. The real design change is above it: planStatusForDecision owns the distinction between invalid input and feasible output.

Before that split, status came from selectedActions.len. If there was at least one selected action, the plan became ready. That is a bad proxy because selected actions can be generated by fallback logic. The status needs to know why the planner had no action.

The stale forecast flag now travels through the planning path as an input validity marker, not just another rejected-action reason. It still appears in rejectedActions so the UI can show the operator what blocked the run. But it also controls persisted status and failure_reason so API consumers and replay logic do not treat the plan as a valid no-op decision.

What surprised me was how much of the surrounding architecture existed because of this one boundary.

The same boundary shows up in the schema. curtailment_plans stores status, confidence_band, input_snapshot, plan_actions, rejected_actions, savings_estimate, risk_summary, and failure_reason as separate fields. That separation matters because a failed plan with rejected actions is not the same thing as a ready plan with rejected actions. The operator sees the human explanation either way, but the status tells the rest of the system whether the plan can be accepted, replayed, or escalated.

Forecasts store p10, p50, and p90 bands. Plans record confidence_band and forecast_band_kw. The service checks that a forecast belongs to the same site as the plan. It checks whether the tariff changed after the forecast. It checks whether the forecast is older than the allowed window. All of that is careful work, but one broad fallback at the bottom of the planner erased the meaning.

That is the part I was wrong about. I assumed a safe fallback is always safer than a hard failure.

In operational software, a false safe state can be worse than an error. An error asks for attention. A ready no-op plan closes the loop. It tells the operator they can move on.

The tests now capture the boundary directly. One unit test calls generateCurtailmentPlan with a stale forecast and asserts that no do_nothing action is selected. Another calls planStatusForDecision with stale input and asserts that the persisted status is failed, not ready. The Playwright journeys cover the other side of the behavior: when the inputs are valid but the constraints block action, the operator still sees recommended and rejected action sections, binding constraints, and decision controls.

That is why I like this failure as a design story. It did not ask for more code. It asked for a better state model. The planner needed two kinds of negative answer: one where the business should not act because no feasible action exists, and one where the software should not answer because its input has expired. Both are negative. Only one is a recommendation.

The same distinction shaped replay. Backtests compare planner, no-action, and threshold policies using the same historical input snapshot. A replay can include a no-action policy because it is an intentional baseline. That is different from a planner run falling into do_nothing because its forecast input had expired. Same words. Different contract.

It also shaped operator feedback. A user can accept, reject, or modify a ready plan, and operator_feedback.original_snapshot preserves the recommendation at the time of the decision. That only works if ready means ready. If stale input can still reach ready status, the audit trail becomes a record of the operator reacting to a recommendation the engine should never have issued. The database can preserve the snapshot perfectly and still preserve the wrong thing.

That is why I prefer status fields that carry domain meaning, even when they feel strict. failed is not a bad product outcome when it protects the operator from bad evidence. A failure reason such as stale forecast cannot be used for a new plan without explicit override gives the next workflow something honest to do: rebuild the forecast, refresh the tariff, or ask the operator for an override. A ready no-op gives downstream code no reason to pause.

I now treat do_nothing as a domain decision, not as an absence handler.

That rule carries across the codebase. Missing asset telemetry becomes ASSET_TELEMETRY_INVALID before scoring. A flat-rate tariff produces a tariff-matrix rejection before curtailment can enter selected actions. Battery state of charge and cycle limits reject discharge before expected savings are calculated. Each one is visible because the planner has to show what it refused to do.

The result is less forgiving code, and that is the point. A planner that fails with a clear reason is safer than a planner that returns a calm answer from bad inputs.

I would carry this further if I rebuilt the planner from scratch. staleForecast is still a boolean moving through function calls. It works, and the tests pin the behavior, but an explicit input-validity type would make the boundary harder to blur later. Something like PlanInputStatus could separate ready, stale forecast, tariff mismatch, and missing history before the planner sees any assets. That is a better shape for the next version because it makes invalid input impossible to confuse with an infeasible action set.

The transferable lesson is narrow: fallback logic needs a domain name. If you cannot name the state it represents, it will eventually hide a state you meant to expose.