DEV Community: Kingsley Onoh

JSONB Was Fine. The Side Effects Needed a State Boundary.

Kingsley Onoh — Tue, 16 Jun 2026 10:26:01 +0000

What should happen when a checklist item sends a client message?

In this portal, that question starts with a milestone stored as a JSON object inside projects.milestones_json. The same milestone can also be a task linkage through tasks.milestoneKey. It can produce a client-visible update. It can fire a Notification Hub event. It can change what the client sees in the portal and what the operator sees in the CLI.

That is too much authority for one checklist item.

The early temptation was simple: keep milestones as JSONB because project setup needed to be fast. A client project does not need a full ceremony for every step. Sometimes the operator needs to add three milestones from the CLI, mark the first one done, and move on. A separate milestone table felt heavier than the problem.

I still think that part was right.

The part I got wrong was assuming the storage choice was the design decision. It wasn't. The real decision was what happens when that flexible JSON object creates side effects outside the JSON field.

The Real Problem

A milestone stored as JSONB is easy to edit and hard to govern. Postgres will store the array. Drizzle will read it back. TypeScript can normalize the shape. None of that answers the business question: when a milestone is marked done, who is allowed to know?

The portal had several competing truths:

The admin route knows which project is being changed.
The client record knows whether notifications are enabled.
The project knows whether it is client-visible or internal-only.
The milestone object knows its own notifyClient and notificationMode values.
The task table knows whether linked work is complete.
The notification layer knows whether to emit an event.

If any one of those edges is checked loosely, the result can be wrong while every individual function still returns 200.

That is the kind of failure that bothers me most. Not a crash. A clean success response attached to the wrong business truth.

The build journals had already taught that lesson elsewhere. One batch found a path where the URL project and update project could diverge inside the same tenant. Another found joined project and client rows being hydrated by ID only after the root row had been tenant-scoped. The first query was safe. The side edge was not.

I was wrong to treat tenant isolation as a problem solved at the start of a request. Every side effect has its own identity boundary.

The Constraints

I did not want a new table just to make the design feel pure. The system was still an internal operating tool. The PRD target was under 50 clients and 20 peak requests per second. The operator needed speed more than relational ceremony.

Milestones also had to stay pleasant from the CLI. The code already had a route that marks a milestone done by a 1-based index. That is not glamorous, but it matches how operators think: first milestone, second milestone, third milestone. Forcing every milestone through IDs too early would make the command surface worse.

But the shortcut had limits.

projects.milestones_json can hold flexible milestone data. It cannot decide whether an email should be sent. It cannot prove the project is tenant-scoped. It cannot decide whether a task title should be backfilled into a milestone key. It cannot stop an internal-only project from producing a client-visible message.

So the storage stayed flexible, and the side effects became strict.

The Design

The core helper is normalizeMilestones() in src/lib/milestones.ts. It does the unglamorous work first: discard empty names, preserve known keys, generate missing keys, deduplicate collisions, and coerce done into a real boolean. That gave the JSONB field a predictable shape without changing the database design.

Then syncProjectMilestoneStatus() handles the inverse direction. If tasks are linked to a milestone key, their state can mark a milestone complete only when the linked tasks justify it. That lets the task table and JSONB array communicate without pretending JSONB is relational.

The more interesting function sits in src/routes/admin/projects.ts. It is small enough to look boring, which is why I like it.

function shouldEmitMilestoneNotification(params: {
  notifyClient?: boolean;
  notificationMode?: 'silent' | 'material_updates' | 'all';
  preferenceMode: NotificationPreferenceMode;
  notificationsEnabled: boolean;
  projectVisibility: typeof projects.visibility.enumValues[number];
}) {
  if (!params.notificationsEnabled) return { emitted: false, reason: 'client notifications disabled' };
  if (params.projectVisibility === 'internal_only') return { emitted: false, reason: 'project is internal only' };
  if (params.notifyClient === false) return { emitted: false, reason: 'notifyClient false' };
  if (params.notificationMode === 'silent') return { emitted: false, reason: 'notificationMode silent' };
  if (params.notifyClient === true) return { emitted: true, reason: 'notifyClient true' };
  if (params.notificationMode === 'material_updates' || params.notificationMode === 'all') {
    return { emitted: true, reason: `notificationMode ${params.notificationMode}` };
  }
  if (params.preferenceMode === 'material_updates' || params.preferenceMode === 'all') {
    return { emitted: true, reason: `project notification preference ${params.preferenceMode}` };
  }
  return { emitted: false, reason: `project notification preference ${params.preferenceMode}` };
}

This is not a clever algorithm. It is a contract.

The order matters. Client notifications being disabled beats everything. Internal-only project visibility beats an eager milestone setting. An explicit notifyClient: false beats the project preference. Silent mode beats a general setting. Only after those denials does the function permit an event.

That order reflects the business. The safest choice must win first.

The route around it does the heavier lifting. It loads the project by id and tenantId. It loads the client by id and the same tenantId. It normalizes milestones before touching the selected 1-based index. It treats an already-completed milestone as idempotent. It writes a portal update. Only then does it ask whether to emit the Notification Hub event.

The notification event itself is fire-and-forget. That is a separate design choice from the milestone boundary. The client state change should not roll back because an email layer is unavailable. The update exists. The portal can show it. The notification failure can be logged and retried outside the request path.

Tests make the boundary real. The event tests cover milestone completion, suppression when notifyClient is false, suppression when project preference is portal-only, and the case where a portal update should exist even when email does not fire. Those are not framework tests. They are business truth tests.

The backfill route is another scar. Tasks gained milestoneKey after milestone JSON already existed. The route defaults to dry-run and requires confirm=true before it mutates task titles and milestone keys. That is what a state boundary looks like when the data model changes underneath a live operator workflow: preview first, then write.

What Surprised Me

I expected the risky part to be JSONB. It was not.

The risky part was side-effect drift. A JSON object can be perfectly valid and still produce the wrong portal update. A project can be tenant-scoped and still attach a joined row carelessly later. A notification can have the right event name and the wrong visibility rule.

That changed how I read the rest of the portal code. I stopped asking only, "Is the row scoped?" I started asking, "Which other facts will this action create, and do they share the same boundary?"

That question shows up everywhere in this project: comments, reports, document caches, capacity notices, stale work, handoff summaries, and client asks. The portal is not just storing facts. It is deciding which facts are safe to expose.

The Result

The final design kept the flexibility that made milestones useful from the CLI, but moved the risk into explicit gates. JSONB still holds the editable project steps. Tasks can link to those steps. Completion can produce a portal update. Notifications can fire only when the client, project, milestone, and preference rules agree.

The latest recorded build gate reached 416 total tests, with 398 passing and 18 skipped. More important than the count, the tests caught wrong-but-running states: missing ask classification, stale internal project noise, QA due noise, generic reply drafts, and untrimmed handoff output.

That is the transferable part. Flexible storage is fine when the business fact is local. The moment it creates a side effect, treat it like a state machine.

Why I made OR-Tools prove it was better than the deterministic dispatcher

Kingsley Onoh — Mon, 15 Jun 2026 20:05:19 +0000

Dispatch optimization needs a lower bound before it needs a clever objective.

In the first real OR-Tools integration, the solver selected fewer assignments than the deterministic fallback it needed to improve. That result made the boundary explicit: CP-SAT could optimize cost, priority, and tie-breakers only after it matched or beat the deterministic feasible assignment count.

The constraint changed how I treated OR-Tools inside the dispatch engine. I had treated the solver as the smarter engine in the room. The code reminded me that dispatch combines math with an operating record. A plan has timestamps, frozen work, post-selection capacity checks, replay metrics, and explanations a dispatcher can defend after the board changes.

The tempting version

The tempting version is simple. Build one boolean variable per eligible technician-job decision. Add constraints for job uniqueness, technician capacity, planning windows, and frozen work. Maximize the objective. Return the result.

That version reads well in a design doc. It is also too trusting for dispatch.

A field-service board has commitments. A dispatcher accepts a plan. A technician starts driving. A supervisor freezes a job. A customer is waiting against an SLA clock. If the solver returns an answer that is mathematically feasible but operationally worse, the system still has to notice.

The code that changed the contract

The final adapter runs deterministic solving first, uses that count as a lower bound, and then lets CP-SAT optimize within that boundary.

val vars = linearArgs(decisions.map(_.variable))
val deterministicAssignmentCount = fallback.solve(model, options).assignments.size
if (deterministicAssignmentCount > 0) {
  val _ = cp.addGreaterOrEqual(LinearExpr.sum(vars), deterministicAssignmentCount.toLong)
}
val coeffs = decisions.map { decision =>
  val assignmentReward = 1_000_000L - (decision.job.priority.rank.toLong * 10_000L)
  val cost = (decision.cost.total * BigDecimal(100))
    .setScale(0, BigDecimal.RoundingMode.HALF_UP)
    .toLong
  assignmentReward - cost - jobIndex(decision.job.id).toLong * 100L -
    technicianIndex(decision.technician.id).toLong
}.toArray
cp.maximize(LinearExpr.weightedSum(vars, coeffs))

That sample is from OrToolsSolverAdapter.solveWithCpSat. The assignment reward is intentionally large. Priority affects the reward. Cost is scaled to an integer. Job and technician indexes act as stable tie-breakers.

The line that matters most is not the maximize call. It is cp.addGreaterOrEqual(LinearExpr.sum(vars), deterministicAssignmentCount.toLong). That line says the solver is allowed to optimize, but it is not allowed to schedule less work than the deterministic feasible path already found.

Why deterministic sequencing stayed

Even after CP-SAT selects decisions, the system does not blindly stamp them into the board. It passes selected decisions through deterministic scheduling. That second stage can still reject work for capacity or planning-window overflow.

At first, that felt redundant. If the solver has constraints, why check again?

Because the dispatch plan is not only a set of pairs. It is a sequence of visits with concrete start times, travel, overtime, and explanation codes. Stable timestamps matter for replay. Stable rejection reasons matter for support. The deterministic layer turns selected pairs into an operating plan that looks the same when the same input snapshot is replayed.

That also protects partial plans. A solver timeout or infeasible slice should not fabricate certainty. The domain has reason codes such as missing_capability, frozen_assignment, capacity_exceeded, outside_planning_window, and solver_timeout. A partial plan with honest unscheduled work is safer than a complete-looking plan built on silence.

Frozen work was the real domain invariant

The solver failure was loud because it affected assignment count. Frozen work is quieter and more dangerous.

The constraint builder treats accepted, completed, and frozen assignments as hard facts. A technician who conflicts with frozen work is rejected. A job that would collide with preserved work does not get moved just because the global objective improves.

That choice is easy to miss if you only look at optimization. A solver optimizes variables. Dispatchers manage promises. Once a human has accepted work, the board has a memory. The optimizer has to respect that memory.

What surprised me

The surprise was not that OR-Tools needed constraints. That is normal. The surprise was that the deterministic implementation became a guardrail for the solver rather than dead code waiting to be deleted.

I kept it for three reasons.

First, it gives the CP-SAT model a feasible assignment lower bound. Second, it gives the app a fallback when native solver loading, runtime failure, or timeout happens. Third, it gives replay a baseline that operators can compare against using SLA hit rate, travel minutes, overtime minutes, churn moves, unscheduled jobs, and solve time.

That makes the deterministic path part of the product, not a temporary scaffold.

The tradeoff

The cost is extra machinery. There are two solve paths. There is trace metadata. There are post-selection checks. There are tests that assert OR-Tools was invoked, no fallback happened, and deterministic results still match where they should.

The benefit is that optimization no longer gets special trust. It has to earn its place inside the operating record.

That is the lesson I took from this build: in systems that move real work, a smarter algorithm is not automatically the source of truth. Sometimes the older deterministic code is the witness that keeps the new optimizer honest.

Evidence Beats Certainty: Why My Classifier Refuses to Pretend Every Product Has an Answer

Kingsley Onoh — Sat, 13 Jun 2026 21:43:17 +0000

Batch 010 found a bug that looked like good news.

The classification worker was finishing its work. Runs moved through the database. Product rows had candidate tariff codes. The regression suite was far enough along that a casual glance could have treated the classifier as alive.

Then one test forced three uncomfortable cases through the loop: no candidate, weak confidence, and a near tie. All three came back looking too clean. The worker was persisting the run as classified, even when the evidence said the product needed review or had no supportable recommendation.

That is the kind of bug I worry about in compliance software. Not the loud crash. The green row.

A customs classifier can fail by throwing an exception. That failure is annoying, but honest. The operator sees it. The queue stops. The job gets retried. The audit trail can say, plainly, that classification did not happen.

The worse failure is a result that looks complete while the evidence underneath is missing or contested.

That was the real Batch 010 scar. The engine already carried the domain rule in its intent: classification is evidence, not a label. But the persistence path was still treating classification as if the only final state that mattered was success. The runtime could produce rejected candidates and confidence values. The database could store failure reasons. The tests could express review states. One narrow path still flattened doubt into completion.

I was wrong about where the risk sat. I expected the hard part to be selecting the tariff code. The harder problem sat one layer later: making sure the code was not selected when the evidence did not deserve that much authority.

Customs data makes that tension obvious. A product row is rarely a clean ontology entry. It is a SKU, a commercial name, a description written by someone under time pressure, a country of origin, a jurisdiction, maybe a material list, maybe an intended use. The difference between a good HS or HTS recommendation and a dangerous one can be a phrase that is absent, ambiguous, or buried in the wrong field.

So I made the classifier refuse to pretend. If a product lacks a candidate, it should be blocked. If the best candidate is too weak, it should go to review. If two candidates are close enough that the lower one is still meaningful, the engine should preserve that tie instead of hiding it behind a confident-looking status.

The decision lives in a small Rust function, which is why I like it. The policy is not scattered across a UI badge, a worker branch, and a reporting query. The worker asks one question: given the runtime outcome, what status should the database store?

pub(super) fn outcome_decision(outcome: &RuntimeClassification) -> OutcomeDecision {
    if outcome.selected_code.is_none() {
        return OutcomeDecision {
            status: "blocked",
            failure_reason: Some("no_candidate"),
        };
    }
    if has_tie_candidate(outcome) {
        return OutcomeDecision {
            status: "needs_review",
            failure_reason: Some("tie_candidate"),
        };
    }
    if outcome.confidence < 0.82 {
        return OutcomeDecision {
            status: "needs_review",
            failure_reason: Some("low_confidence"),
        };
    }
    OutcomeDecision {
        status: "classified",
        failure_reason: None,
    }
}

That function from src/classification/outcome.rs is not clever. It is deliberately plain. It says the classifier has four questions to answer before it earns the right to call a run classified.

First, did the runtime select any code at all? If not, the run is blocked/no_candidate. The operator should not see an empty answer wearing the same status as a resolved classification.

Second, did the runtime find a meaningful tie? The rule runtime marks lower-ranked matches as rejected candidates, and a near tie gets the reason tie_score. In that case the selected code still matters, but it is not enough. The run becomes needs_review/tie_candidate.

Third, did the selected code clear the confidence floor? The current worker uses 0.82 as the line below which a product should not pass as clean. That number is a code-backed threshold, not a production claim. It is there because the engine needs a deterministic boundary for review routing.

Only after those checks does the run become classified.

The order matters. No candidate is different from low confidence. Low confidence is different from a tie. A tie with a selected code is different from a rule pack that found nothing. If those cases all share a green status, the UI can only lie or become complicated later. If the status and reason are precise at write time, the rest of the product can stay simpler.

The test that caught this is the kind of test I wish more systems had before they gained users. It does not test the happy path with a cotton shirt and a confident tariff code. It creates three products with names that force the worker to admit uncertainty.

One row has no matching rule. One row matches with a confidence below the floor. One row matches two close candidates, 6205.20 and 6205.30, close enough that the rejected candidate still belongs in the record. The assertion is not only that the worker completes three jobs. It checks the stored status, the failure_reason, the selected code where one exists, the candidate code list, and the tie_score reason inside rejected candidates.

That last part matters. I did not want a review queue filled with vague work items that say, "please check this." I wanted the queue to carry the reason the machine gave up authority. A reviewer should know whether they are handling an empty result, a weak result, or a contested result.

The same logic affects audit exports. An audit pack that says a product was classified is different from an audit pack that says the system found two close candidates and routed the run to review. In both cases, the export has value, but it answers a different question. One says, "here is the evidence behind the recommendation." The other says, "here is the evidence behind the refusal to recommend."

That distinction changes the product shape. The engine stores matched rules, rejected alternatives, confidence, risk band, rule pack version, input snapshot, reviewer decisions, and failure reasons. It also freezes the product and rule pack facts at queue time. If the product description changes after the job enters the queue, the worker still evaluates the snapshot it was handed. If the active rule pack changes later, historical runs still point back to the pack version that produced them.

That is slower to reason about than a direct request that always reads current product state. It is also safer. A compliance review is not asking, "what would the system say today?" It often asks, "what did the system know then, and why did it make that call?"

What surprised me was how much of the architecture flowed from that one sentence.

The classifier uses a PostgreSQL job table instead of pretending a background job is a fire-and-forget detail. A worker leases rows, marks attempts, and exits if a run is already terminal. Product import refuses rows that lack required facts such as SKU, name, description, country, jurisdiction, product type, materials, or intended use. Rule packs have activation gates before they become active. Reviewer overrides append structured corrections instead of mutating the machine result. Audit exports are rendered from frozen snapshots instead of live joins that could drift.

Those choices sound separate until Batch 010 ties them together. If the worker writes the wrong status, every careful snapshot around it becomes less trustworthy. The audit export preserves the wrong conclusion. The review queue misses the item. The dashboard looks cleaner than the evidence. Optional integrations can fire the wrong event. A bad status is not a display bug. It is an evidence bug.

The fix was small because the earlier design had already made room for it. The database had a status field and a failure reason. The runtime returned selected and rejected candidates. The tests could create all three edge cases. Once the regression exposed the lie, the code only had to make the domain decision explicit.

I also changed how I read passing tests after that. A test that proves a worker completed is not enough for a compliance loop. Completion is only a transport fact. The domain fact is whether the stored row still carries the same uncertainty the runtime produced. That is why the regression checks status, failure_reason, selected_code, candidate_codes, and rejected candidate reasons in one place. If any one of those drifts, the row may still look finished, but the evidence contract is broken.

That is the lesson I took from it, and I mean lesson in the practical sense, not as a slogan. If a domain has reviewable uncertainty, model that uncertainty before the happy path spreads through the codebase.

For this project, uncertainty has names: no_candidate, low_confidence, and tie_candidate. Those names are not UI copy. They are durable outcomes.

A classifier that always returns an answer is easy to demo. It is also easy to distrust. In customs work, the more serious promise is narrower: when the evidence is good enough, store the recommendation; when it is not, store the reason it stopped.

That is why the Trade Compliance Classification Engine refuses to treat every product as solved. Certainty is useful only when the record can prove how it was earned.

Why I Made Stale Forecasts Fail Instead of Falling Back to Do Nothing

Kingsley Onoh — Fri, 12 Jun 2026 23:03:41 +0000

The UI showed ready, do_nothing, and a blank reason field.

A facility manager reading that screen would assume the engine had looked at the peak window, checked the assets, and decided there was nothing worth doing. The interface looked calm. The audit trail looked complete.

That was not true.

The forecast behind the plan had expired. The planner should never have scored it. But the fallback path did exactly what I told it to do: when no action was selected, choose do_nothing so the operator gets a safe recommendation instead of an empty response.

Safe fallback became false confidence.

The product has a simple rule: physical feasibility comes before economics. A battery cannot discharge below its minimum state of charge. A building load cannot curtail past its comfort limit. A flat-rate tariff cannot justify peak curtailment because there is no peak signal to respond to. Those are business truths encoded as code.

That rule exists because energy planning has a dangerous temptation: collapse every problem into money. If the demand charge is high enough, the spreadsheet always finds a saving. Real facilities do not work that way. Operators know some processes cannot move, some comfort limits cannot bend, and some battery cycles are not worth spending for a small peak reduction. The planner has to encode that judgment before it calculates expected savings.

The bug came from treating stale forecasts like another physical constraint.

In a real infeasible window, do_nothing is useful. If every battery is depleted, every comfort limit blocks curtailment, and every flexible process is already at its limit, doing nothing is a valid operational recommendation. It tells the operator: the engine understood the window and found no feasible savings-positive action.

A stale forecast is different. It means the engine did not have permission to reason about the window at all. The input is invalid. The correct output is a failed plan with an explicit reason.

I got that boundary wrong in the first implementation.

The code had all the pieces in separate places. createPlan detected stale forecasts. generateCurtailmentPlan recorded a stale-forecast rejection. But the bottom of the planner had a broad fallback: if no selected actions exist, add do_nothing. That line was written for infeasible windows, not invalid input, but it had no way to know the difference.

The fix looks small because the hard part was naming the boundary.

Batch 009 had already moved the planner away from fake input. Forecast creation loads qualified interval readings from the database. Plans check that the selected forecast and tariff belong to the requested site. The selected band travels as forecastBandKw, and the action payload records both confidence_band and forecast_band_driver. Those pieces made the failure more embarrassing, not less. The system had evidence discipline at the edges, then lost it in one central fallback.

const staleForecastFailureReason* = "stale forecast cannot be used for a new plan without explicit override"

proc planStatusForDecision*(decision: PlannerDecision; stale: bool): (string, string) =
  if stale: ("failed", staleForecastFailureReason)
  elif decision.selectedActions.len > 0: ("ready", "")
  else: ("failed", "no feasible planner action")

proc generateCurtailmentPlan*(ctx: TenantContext; input: PlannerInput; tariff: PlannerTariff; assets: seq[PlannerAsset]; staleForecast: bool): PlannerDecision =
  validateInput(ctx, input)
  var selected = newJArray()
  var rejected = newJArray()
  var binding = newJArray()
  let curtailAllowed = addStaleAndTariffRejections(tariff, staleForecast, rejected, binding)
  var selectedSavings = 0.0
  for asset in assets:
    if asset.rejectMissingAssetTelemetry(rejected):
      continue
    if asset.assetType == "battery":
      addBatteryAction(asset, input, tariff, staleForecast, selected, rejected, selectedSavings)
      addChargeBatteryAction(asset, input, tariff, staleForecast, selected, rejected, selectedSavings)
    elif asset.assetType in ["building_load", "flexible_process"]:
      addCurtailAction(asset, input, tariff, curtailAllowed, staleForecast, selected, rejected, selectedSavings)
      addShiftLoadAction(asset, input, tariff, staleForecast, selected, rejected, selectedSavings)
  if selected.len == 0 and not staleForecast:
    selected.add(%*{"action_type": "do_nothing", "reason": "no feasible savings-positive action after physical constraints"})
  addBindingRejections(rejected, binding)
  makeDecision(input, tariff, selected, rejected, binding, selectedSavings)

That and not staleForecast is the visible change. The real design change is above it: planStatusForDecision owns the distinction between invalid input and feasible output.

Before that split, status came from selectedActions.len. If there was at least one selected action, the plan became ready. That is a bad proxy because selected actions can be generated by fallback logic. The status needs to know why the planner had no action.

The stale forecast flag now travels through the planning path as an input validity marker, not just another rejected-action reason. It still appears in rejectedActions so the UI can show the operator what blocked the run. But it also controls persisted status and failure_reason so API consumers and replay logic do not treat the plan as a valid no-op decision.

What surprised me was how much of the surrounding architecture existed because of this one boundary.

The same boundary shows up in the schema. curtailment_plans stores status, confidence_band, input_snapshot, plan_actions, rejected_actions, savings_estimate, risk_summary, and failure_reason as separate fields. That separation matters because a failed plan with rejected actions is not the same thing as a ready plan with rejected actions. The operator sees the human explanation either way, but the status tells the rest of the system whether the plan can be accepted, replayed, or escalated.

Forecasts store p10, p50, and p90 bands. Plans record confidence_band and forecast_band_kw. The service checks that a forecast belongs to the same site as the plan. It checks whether the tariff changed after the forecast. It checks whether the forecast is older than the allowed window. All of that is careful work, but one broad fallback at the bottom of the planner erased the meaning.

That is the part I was wrong about. I assumed a safe fallback is always safer than a hard failure.

In operational software, a false safe state can be worse than an error. An error asks for attention. A ready no-op plan closes the loop. It tells the operator they can move on.

The tests now capture the boundary directly. One unit test calls generateCurtailmentPlan with a stale forecast and asserts that no do_nothing action is selected. Another calls planStatusForDecision with stale input and asserts that the persisted status is failed, not ready. The Playwright journeys cover the other side of the behavior: when the inputs are valid but the constraints block action, the operator still sees recommended and rejected action sections, binding constraints, and decision controls.

That is why I like this failure as a design story. It did not ask for more code. It asked for a better state model. The planner needed two kinds of negative answer: one where the business should not act because no feasible action exists, and one where the software should not answer because its input has expired. Both are negative. Only one is a recommendation.

The same distinction shaped replay. Backtests compare planner, no-action, and threshold policies using the same historical input snapshot. A replay can include a no-action policy because it is an intentional baseline. That is different from a planner run falling into do_nothing because its forecast input had expired. Same words. Different contract.

It also shaped operator feedback. A user can accept, reject, or modify a ready plan, and operator_feedback.original_snapshot preserves the recommendation at the time of the decision. That only works if ready means ready. If stale input can still reach ready status, the audit trail becomes a record of the operator reacting to a recommendation the engine should never have issued. The database can preserve the snapshot perfectly and still preserve the wrong thing.

That is why I prefer status fields that carry domain meaning, even when they feel strict. failed is not a bad product outcome when it protects the operator from bad evidence. A failure reason such as stale forecast cannot be used for a new plan without explicit override gives the next workflow something honest to do: rebuild the forecast, refresh the tariff, or ask the operator for an override. A ready no-op gives downstream code no reason to pause.

I now treat do_nothing as a domain decision, not as an absence handler.

That rule carries across the codebase. Missing asset telemetry becomes ASSET_TELEMETRY_INVALID before scoring. A flat-rate tariff produces a tariff-matrix rejection before curtailment can enter selected actions. Battery state of charge and cycle limits reject discharge before expected savings are calculated. Each one is visible because the planner has to show what it refused to do.

The result is less forgiving code, and that is the point. A planner that fails with a clear reason is safer than a planner that returns a calm answer from bad inputs.

I would carry this further if I rebuilt the planner from scratch. staleForecast is still a boolean moving through function calls. It works, and the tests pin the behavior, but an explicit input-validity type would make the boundary harder to blur later. Something like PlanInputStatus could separate ready, stale forecast, tariff mismatch, and missing history before the planner sees any assets. That is a better shape for the next version because it makes invalid input impossible to confuse with an infeasible action set.

The transferable lesson is narrow: fallback logic needs a domain name. If you cannot name the state it represents, it will eventually hide a state you meant to expose.

Why I Froze Simulation Inputs Before the Solver Ran

Kingsley Onoh — Thu, 28 May 2026 13:24:24 +0000

A planner can approve the right transfer for the wrong reason.

That was the failure mode I kept coming back to while building the Inventory Allocation Simulator. The solver could be mathematically correct, the recommendation could have positive net value, and the UI could show a clean explanation. But if the explanation reread today's warehouse and SKU tables after yesterday's simulation finished, the audit trail would be fiction.

Inventory data changes constantly. A lane gets disabled. A SKU margin changes. Inbound units arrive. Demand history is corrected because a stockout period was recorded as zero sales. If a completed simulation depends on the current state of those tables, its story changes every time the business updates its planning data.

I was wrong to treat that as a reporting problem at first. It was a data contract problem.

The failure hiding in a normal design

The first design looked harmless: store a simulation run, run the worker, persist recommendations, and render the detail page by joining back to warehouses, SKUs, inventory, lanes, and policies. Most CRUD systems are written that way because joins are cheap and normalized tables keep data clean.

That design breaks the moment a simulation becomes evidence.

The recommendation is not just a row saying transfer 30 units. It needs to explain the constraint that bound the decision, the demand scenario that created the shortage, the service-level tail left unmet, and the tradeoffs accepted. In this project those fields live in explanation: binding_constraints, scenario_sensitivity, accepted_tradeoffs, net_value, and solver diagnostics.

If the explanation uses live tables, a planner can open the same completed run on Tuesday and see different supporting facts from Monday. That is worse than no explanation. It creates confidence in a record that no longer matches the decision.

The constraint I chose

I made simulation creation the boundary.

create_simulation_run! authorizes the planner, validates the scenario count, captures every planning surface needed by the solver, and stores it inside simulation_runs.input_snapshot. The worker consumes that snapshot. The detail page reads that snapshot. Demand scenarios are stored with the run. Completed runs do not ask the mutable catalog what the world looks like now.

The core function is small, which is the point:

function capture_simulation_input_snapshot(
    store::AbstractTenantAdminStore,
    ctx::TenantContext,
    policy_id,
)::NamedTuple
    authorize!(ctx, "run_cancel", "simulation")
    parsed_policy_id = _uuid_value(policy_id)
    policy = _snapshot_policy(store, ctx, parsed_policy_id)
    return (
        tenant_id = string(ctx.tenant_id),
        policy = policy,
        warehouses = [_warehouse_response(row) for row in fetch_warehouses(store, ctx.tenant_id, _snapshot_page())],
        skus = [_sku_response(row) for row in fetch_skus(store, ctx.tenant_id, _snapshot_page())],
        inventory_positions = [_inventory_response(row) for row in fetch_inventory_positions(store, ctx.tenant_id, _snapshot_page())],
        demand_history = [_demand_response(row) for row in fetch_demand_history(store, ctx.tenant_id, _snapshot_page())],
        transfer_lanes = [_lane_response(row) for row in fetch_transfer_lanes(store, ctx.tenant_id, _snapshot_page())],
    )
end

That snapshot is not elegant. It is intentionally blunt. The run carries the policy, warehouses, SKUs, inventory positions, demand history, and transfer lanes it used. SNAPSHOT_MAX_ROWS is set to 1_000_000, which tells you the tradeoff plainly: this is a batch planning system, not a real-time transfer executor.

What surprised me

The snapshot decision also fixed a forecasting bug before it could become a solver bug.

Stockout periods are dangerous because they make demand look low. In clean_demand_history, the system stores both observed units and adjusted units. If demand was zero and lost sales were 82, the cleaned demand is 82. That value feeds the scenario generator. The stockout row also inflates uncertainty, because a period with unavailable inventory is less trustworthy than normal sales.

The Batch 019 test mutates live inventory and demand after a run is created. Then it runs the worker and checks that the scenario baseline still comes from the frozen snapshot, not the updated live row. That was the test that made the architecture feel real. It did not just prove a function. It proved the system can remember what it believed when the recommendation was created.

The alternative was versioning every table. That would give better diff history, but it would also make the MVP harder to operate. I chose the snapshot because the project needed completed-run honesty more than a general temporal database. A future version could move to event-sourced planning records. This one needed a concrete audit boundary.

Where the solver fits

The solver reads the frozen snapshot and stored scenarios, then builds a JuMP model over lanes, SKUs, inventory, service level, warehouse capacity, transfer cost, and safety stock. The model is allowed to fail. In fact, readable failure is part of the contract.

A region rule can block every feasible transfer. A max transfer cost can make the plan infeasible. A timeout can return no acceptable incumbent. Those cases return diagnostics instead of pretending every scenario has a transfer.

That mattered for the Journal topic because failure diagnostics have to describe the same world the solver saw. The _constraint_report function can name max_transfer_cost_cents, region blocking, sender safety stock, and receiver service-level constraints. If those facts came from live tables, the failure message could drift too.

The result

The deterministic solver fixture produces a 30-unit transfer from WH-SURPLUS to WH-NEED, with lane_capacity and receiver_service_level as binding constraints and a 30,900-cent net value. The large benchmark ran 50 warehouses, two thousand SKUs, and 100 scenarios in 17,928.4753 ms and generated two thousand recommendations.

Those numbers matter less than the contract behind them. A completed run is not a report over current data. It is a preserved decision record. Once I made that boundary explicit, the rest of the system had somewhere honest to stand.

Why I Moved Redis Acknowledgement Outside the Database Transaction

Kingsley Onoh — Fri, 22 May 2026 10:22:55 +0000

One good parcel event, one bad parcel event, one batch. That was enough to lose the good one.

The consumer read both from Redis Streams. The first event was valid, so the app wrote a shipment event and a claim case, then acknowledged the Redis message. The second event had no tenant id, threw an exception, and rolled back the PostgreSQL transaction. Redis had already been told the first message was done.

The database said nothing happened. Redis said the message was gone.

That is the kind of bug that looks like an operations mystery later. A carrier says it sent the event. The stream no longer shows it as pending. The claim queue has no case. Everyone starts looking at logs, but the data has already contradicted itself.

The part I got wrong

I treated Redis acknowledgement as part of processing. That was too early.

The first version of DeliveryEventConsumerJob.pollOnce() processed each stream record inside a TransactionTemplate. It wrote through ShipmentEventIngestionService.upsertFromEvent(...), then acknowledged the message in the same loop. It felt clean because both actions sat inside the same method and both happened after the event handler returned.

But Redis is not inside the PostgreSQL transaction. XACK does not care whether the database later commits. Once Redis removes the message from the pending list, the recovery path changes. If the transaction rolls back after that ack, PostgreSQL loses the row and Redis loses the retry handle.

That is not a duplicate problem. It is a vanished work problem.

The codebase already had idempotency in the right place for duplicates. ShipmentEventIngestionService builds a tenant-scoped dedup key from the event source and takes a PostgreSQL advisory transaction lock before inserting into shipment_events. The table also has a (tenant_id, dedup_key) uniqueness constraint. A replay can create at most one shipment event row and one claim case.

I was wrong to optimize for avoiding replay. Replay was safe. Premature ack was not.

The shape of the fix

The fix was small, but the boundary it moved mattered.

public int pollOnce() {
    if (!Boolean.TRUE.equals(properties.enabled())) {
        return 0;
    }
    List<String> processedMessageIds = transactionTemplate.execute(status -> {
        if (!lockManager.tryAcquireTransactionLock(LOCK_NAME)) {
            return List.of();
        }
        streamClient.ensureConsumerGroup();
        String consumer = consumerName();
        List<DeliveryGatewayTrackingEvent> events = streamClient.readPendingEvents(
                consumer, BATCH_SIZE, Duration.ZERO);
        if (events.isEmpty()) {
            events = streamClient.readNewEvents(consumer, BATCH_SIZE, Duration.ofMillis(250));
        }
        List<String> messageIds = new ArrayList<>();
        Map<UUID, TenantContext> tenantsById = new HashMap<>();
        for (DeliveryGatewayTrackingEvent event : events) {
            process(event, tenantsById);
            messageIds.add(event.messageId());
        }
        return messageIds;
    });
    if (processedMessageIds == null || processedMessageIds.isEmpty()) {
        return 0;
    }
    streamClient.acknowledgeAll(processedMessageIds);
    return processedMessageIds.size();
}

The transaction now returns the message ids only after the database work succeeds. Only then does the job call acknowledgeAll.

That creates two possible failure states, and both are tolerable.

If the database fails, no ack happens. Redis still has the pending messages. The next poll retries them.

If the database commits and the ack fails, Redis may replay messages whose database rows already exist. That is fine. The dedup key, advisory lock, and unique index collapse the replay back to the existing shipment event. The system may do extra work, but it will not create a second claim.

This is the tradeoff I should have chosen from the start: duplicate work over lost work.

Why the batch cache came later

The ack fix uncovered a different problem. The 50 events/sec acceptance test passed in isolation, then failed during full regression. The batch took 1,805 ms against a 1,000 ms target.

At first glance, that looked like Redis overhead. It wasn't. The consumer was reading 50 messages and doing one batched ack, but process(...) still resolved the same tenant and integration setting once per event. One tenant, 50 events, 50 database lookups before the hot path even reached shipment ingestion.

The fix was not a global cache. A global tenant cache would create stale security behavior and make feature flags harder to trust. The right cache lived inside one poll batch:

Map<UUID, TenantContext> tenantsById = new HashMap<>();
for (DeliveryGatewayTrackingEvent event : events) {
    process(event, tenantsById);
    messageIds.add(event.messageId());
}

process(...) now uses computeIfAbsent to authorize each tenant once per poll. Every poll still checks whether the tenant has delivery-gateway enabled. Nothing survives across polls. The latency win comes from removing repeated reads, not weakening the gate.

What surprised me was how easy this bug was to miss. The system could pass duplicate-message tests, tenant-scope tests, and manual exception flow tests while still failing a real throughput target. It needed the acceptance test with 50 actual Redis messages and PostgreSQL writes to reveal the hidden cost.

The database boundary still does the hard work

The consumer job is only the outer shell. The correctness boundary lives in ShipmentEventIngestionService.upsertFromEvent(...).

That service takes the event, normalizes carrier, tracking number, status, timestamps, snapshots, and metadata. It builds an idempotency key under the tenant id. It locks that key with pg_advisory_xact_lock. It upserts the shipment, inserts the event with on conflict do nothing, updates the visible shipment status only when the event timestamp is not older, writes audit records, then asks ClaimAutoCaseService whether the status deserves a case.

Only three statuses create default cases: failed_attempt, returned, and exception. A delivered scan does not open a claim. An unknown scan does not open a claim. The tracking timeline records context, but operations work begins only when the event status represents an operational exception.

That separation matters because not every carrier signal should become work. A tracking system records facts. A claims system creates ownership.

The result

The regression test that mattered is blunt: publish a valid event and then an invalid event in the same Redis batch. Run pollOnce(). Assert that no shipment event and no claim case exist, and that both Redis messages remain pending.

That test now passes.

The throughput proof also passes: 50 delivered events process in under 1 second locally, and because they are delivered events, they create zero claim cases. The duplicate stream test publishes two messages with the same source event and gets one shipment event, one claim case, and zero pending Redis messages after successful processing.

The lesson here is not "ack after commit" as a slogan. The real rule is narrower: if one system owns retry visibility and another system owns business durability, the retry signal must not be cleared until the business fact is committed.

The Document Number Is Reserved Before the PDF Exists

Kingsley Onoh — Mon, 11 May 2026 08:21:48 +0000

The hard part of document numbering is not incrementing an integer.

It is deciding what happens when the integer is reserved, rendering starts, and the render fails.

PostgreSQL sequences are built for speed. They are not built for legal numbering. A sequence advances even if the surrounding transaction rolls back. For most applications that is fine. For invoices and board records, a gap is not invisible. If number 41 exists and number 43 exists, someone will ask what happened to 42.

I needed a numbering system that could do three things at once: serialize per entity, type, and year; let different sequences run in parallel; and preserve a reason when a number is skipped.

That became the two-phase allocator in src/documents/numbering.ts.

Phase one reserves the next number. The allocator locks the tuple (entity_id, document_type, year) with pg_advisory_xact_lock, checks for the oldest unclaimed gap, and only then advances the high-water mark. It writes the reservation to pending_allocations with a TTL.

Phase two either finalizes the reservation against a document row or releases it into sequence_gaps.

The core reservation shape is small:

await acquireAllocatorLock(c, entityId, documentType, year);

let reservedNumber = await tryReclaimOldestGap(c, entityId, documentType, year);
if (reservedNumber === null) {
  reservedNumber = await computeNextHighWater(c, entityId, documentType, year);
}

await c.query(
  `INSERT INTO pending_allocations
     (id, entity_id, document_type, year, reserved_number,
      reserved_by_api_key_id, reserved_at, expires_at, metadata)
   VALUES ($1, $2, $3, $4, $5, $6, now(),
           now() + ($7::text)::interval,
           $8::jsonb)`,
  [allocationId, entityId, documentType, year, reservedNumber, apiKeyId, ttl, metadata],
);

The advisory lock boundary matters. FZE invoices for 2026 serialize against other FZE invoices for 2026. LLC board resolutions do not wait on them. A document type has its own counter space because letterhead #1 and compliance letter #1 can both be real first documents in their own family.

The part I got wrong early was the SQL shape for reclaiming a gap.

The first version used an update with a subquery and a limit against the same table. PostgreSQL flattened the plan in a way that ignored the limit and updated every eligible row. That is the kind of bug that looks impossible until you inspect the affected rows. The fixed version uses a CTE target so the candidate set is materialized before the update runs.

const res = await c.query(
  `WITH target AS (
     SELECT id FROM sequence_gaps
      WHERE entity_id = $2 AND document_type = $3 AND year = $4
        AND reclaimed_at IS NULL
      ORDER BY reaped_at ASC
      LIMIT 1
      FOR UPDATE SKIP LOCKED
   )
   UPDATE sequence_gaps
      SET reclaim_allocation_id = $1, reclaimed_at = now()
    WHERE id IN (SELECT id FROM target)
    RETURNING id, gap_number`,
  [null, entityId, documentType, year],
);

That one CTE is the difference between reclaiming one documented gap and mutating the whole backlog.

The reaper is the other half of the design. A reservation can expire before it is attached to a document. Maybe Puppeteer failed. Maybe the sidecar was down. Maybe storage rejected the upload. The service cannot just delete the reservation and pretend the number never happened. It moves the number to sequence_gaps with reason = 'reaper_swept'.

There are also explicit reasons: abandoned reservation, admin documented gap, manual void. That vocabulary is important because auditors do not need a philosophical explanation. They need a row that says why the number did not produce a document.

What surprised me was how much of the design was about not being too clever. I could have hidden this behind one nextDocumentNumber() helper and let failures be retried. That would make the happy path smaller and the audit story weaker. The split between pending allocation and sequence gap is more verbose, but the data tells the truth.

The same pattern shows up in the tests. The reaper-race gate is not a decorative concurrency test. It exists because the allocator is only correct under pressure: concurrent reservations, expired rows, reclaimed gaps, and failed finalization must all preserve one property. No two documents get the same number in the same sequence, and no missing number lacks a reason.

The document number is reserved before the PDF exists because rendering is fallible. The audit trail has to survive that fallibility.

The Audit Trail Is a Data Structure, Not a Log Message

Kingsley Onoh — Mon, 11 May 2026 08:21:05 +0000

Logs can explain what a service thought happened.

They do not prove what happened.

Klevar Docs needed an audit trail for rendered documents, invoice events, credit note applications, signatures, voids, and attachments. The usual answer is an events table. Insert a row whenever something happens. Add timestamps. Keep it forever. That is useful, but it is still just a table unless the table can detect tampering.

The hash chain is the difference.

Each entity gets its own ordered chain. A row stores the event payload, a SHA-256 hash of the canonical payload, the previous row hash, the chain index, and a link hash that binds those values together. If someone changes a payload, deletes a prior row, swaps entity rows, or reorders entries, verification fails.

The append path is transactional with the document change. That detail matters more than the hashing. If the document row commits and the chain row rolls back, the proof is incomplete. If the chain row commits and the document row rolls back, the proof references a thing that does not exist. insertChainEntry() is designed to run inside the caller's transaction.

The core logic is direct:

const allocRes = await tx.execute(
  sql`SELECT fn_allocate_chain_index(${entityId}::uuid)::text AS idx`,
);
const chainIndex = BigInt(allocRes.rows[0]!.idx);

let previousHash: string | null = null;
if (chainIndex > 1n) {
  const priorRes = await tx.execute(
    sql`SELECT payload_hash FROM document_hash_chain
         WHERE entity_id = ${entityId}::uuid
           AND chain_index = ${(chainIndex - 1n).toString()}::bigint`,
  );
  previousHash = priorRes.rows[0]!.payload_hash;
}

const chainLinkHash = computeChainLinkHash({
  content_hash: contentHash,
  previous_hash: previousHash,
  chain_index: chainIndex,
  entity_id: entityId,
});

There are two design choices hidden in that snippet.

The index comes from fn_allocate_chain_index(entity_id), not a PostgreSQL sequence. The same rollback problem that makes sequences wrong for legal document numbers also applies to chain indices. A verifier expects the chain to be contiguous. If index 19 is missing because a transaction rolled back after a sequence increment, the verifier cannot know whether that is harmless or tampering.

The link hash includes entity_id. That prevents a row from one entity being copied into another entity's chain without detection. Klevar has one group boundary, but the legal proof is per entity. FZE, LLC, and Ltd cannot share a chain just because the service is single-tenant.

The verifier is a walker, not a database query. It receives rows sorted by chain_index, checks ordering, checks the genesis row, checks each previous_hash, recomputes each link hash, and returns the first mismatch. It also reports intentionally broken indices. That last category is important because some retention or force-purge action may be documented rather than hidden. A broken chain can be honest if the break is recorded and visible.

What surprised me was the dependency on canonical JSON. Hashing JavaScript objects directly is a trap because key order and serialization details can drift. The service pins canonicalize@2.0.0 and runs an RFC 8785 boot assertion before Fastify binds a port. If canonicalization changes, the server refuses to start. That is not paranoia. It is the cost of using hashes as legal proof.

The hash chain also changed how I think about events. events_emitted is the integration outbox for Hub and Webhook Engine fanout. It is operational. document_hash_chain is proof. Those two surfaces overlap, but they are not the same thing. A notification can be retried, delayed, or dropped without changing the legal document. A chain append cannot be treated that way.

The biggest tradeoff is operational weight. A chain gives you another invariant to maintain, another verifier to run, another repair story to document, and another failure mode to alert on. The alternative is worse: a document archive that can produce files but cannot prove nobody altered the history.

For this system, the archive without the chain would be storage. The chain turns it into evidence.

Why a Rendered Invoice Can Still Fail Send

Kingsley Onoh — Mon, 11 May 2026 08:20:52 +0000

An invoice PDF can exist and still not be an invoice package.

That sentence shaped the send path. The easy implementation would render the invoice, store the PDF, try to build the XML, and log a warning if the compliance layer failed. The client still gets a document. The API still returns success. The business can "fix it later."

That is exactly the failure I did not want.

Klevar Docs treats e-invoicing as part of finalization, not as decoration after rendering. For an invoice that routes to Factur-X, the send path has to build CII XML, validate it, embed it into the PDF, convert the container to PDF/A-3b, validate that with veraPDF, upload the XML, and replace the PDF with the conformant output. If any hard requirement fails, the send fails.

The orchestrator is intentionally thin. buildComplianceArtifacts() does profile resolution and dispatches to a branch. The Factur-X branch owns the hard path. The XRechnung branch owns the public-sector German XML path.

The Factur-X branch is where the philosophy is visible:

const valid = validationRaw.valid === true;
if (!valid) {
  await markDocumentFacturXFailed(db, document.id);
  throw new FacturXValidationRejectError({
    reason: 'schematron_failed',
    errors: Array.isArray(validationRaw.errors) ? validationRaw.errors : [],
    invoice_id: invoice.id,
    factur_x_validation_status: 'fail',
  });
}

const fallbackResult = await embedAndValidateWithFallback(
  plainPdfBytes,
  build.xml,
  fallbackContext,
  fallbackDeps,
);

if (!fallbackResult.conformant) {
  throw new PdfARequiredRejectError({
    reason: 'pdf_a_non_conformant',
    stage_failed: fallbackResult.stage_failed,
    invoice_id: invoice.id,
  });
}

There are two details in that snippet that matter.

First, validation failure is a typed 422, not an internal server error. The invoice shape is wrong. The operator needs to fix the invoice, not restart the service. Missing BT fields, sidecar 4xx failures, and schematron failures all become the same class of business-visible rejection.

Second, the document row is marked as failed before throwing. That was a later correction. Without it, a failed send could leave the row looking cleaner than reality because the broader transaction never committed. The code now best-effort updates documents.factur_x_validation_status = 'fail' so a polling operator sees the truth after a rejection.

What surprised me was how much logic had to live before the sidecar call. I expected mustangproject to be the hard part. The harder part was building the DTO without lying. Seller name, seller VAT ID, registration ID, address, client identity, tax category, reverse charge handling, unit code, document type, original invoice reference, payment terms, issue date, due date. Every one of those fields has a business rule.

The builder is pure TypeScript for that reason. It reads frozen invoice, client, and entity snapshot data and returns a transport DTO for the Java sidecar. It does not query the database. It does not default from live entity state. It does not mutate the invoice.

That purity paid off when credit notes entered the path. A credit note is not just a negative invoice. It carries a different document type code and can need a preceding invoice reference. The builder got an explicit discriminator for invoice versus credit_note, and the sidecar maps that into the XML. The alternative was to infer from amounts, which would have been clever and wrong.

XRechnung is intentionally different. German public-sector invoices route to standalone XML. The human-readable PDF remains a companion, not the legal container. That means the branch skips PDF/A-3b embedding work and persists XML as the authoritative artifact. Today the validator status can be pass, fail, or skipped because the KoSIT validation lane had an acknowledged gap. I kept that explicit instead of pretending both branches had identical maturity.

The system now has an uncomfortable but correct behavior: it can render a beautiful PDF, then refuse to send it.

That is the point. A valid-looking artifact is not enough. The service needs to know whether the document is acceptable for the legal path it is taking. For a plain letterhead, PDF bytes may be enough. For an invoice that routes to Factur-X, the XML and PDF/A container are part of the document. If they fail, the document failed.

I would rather make the operator fix a rejection than let a client receive a file that only looks complete.

The PDF Looked Correct Because the Template Was Wrong

Kingsley Onoh — Mon, 11 May 2026 08:20:09 +0000

The first FZE letterhead looked fine.

That was the problem.

The rendered PDF had the right legal name, the right registration label, the right address, the right contact line, and the right visual structure. It passed the visual check because every value in the template matched the entity I was testing with. Then I looked at what would happen if the same bundle rendered an LLC document.

It would still say FZE.

That failure is more dangerous than a crash. A crash stops the send path. A wrong legal identity in a PDF can leave the system looking healthy while the document is unusable. The template authoring lane had hardcoded FZE identity strings because the snapshot did not expose the fields the template needed. The implementation had chosen the fastest path to a green render instead of stopping at the missing data contract.

I was wrong to treat the template as the place where legal identity could be finished. The template is presentation. The entity row is state. The document snapshot is the contract between them.

The fix started by widening the entity surface. captureEntitySnapshot() now freezes address, registration, contact, legal names, country code, VAT identifier, officer data, brand, banking config, retention posture, and document policy into the document row. It also redacts fields that should not land on PDFs, like private tax identifiers and operational banking rollout notes.

The load-bearing part is that the snapshot always returns stable shapes. Handlebars runs in strict mode. If a template asks for entity.address.single_line, the address path must exist even when the row is old. Empty object is better than undefined because an old document can still render through the same code path.

The function is blunt about that boundary:

export function captureEntitySnapshot(row: EntityRowForSnapshot): EntitySnapshot {
  if (typeof row.id !== 'string' || row.id.length === 0) {
    throw new ValidationError('entity_snapshot_invalid', { reason: 'id missing' });
  }

  return {
    id: row.id,
    entity_id: row.entity_id,
    name: row.name,
    type: row.type,
    brand: cloneJsonLike(row.brand),
    banking_config: redactBankingConfigForDocuments(row.banking_config),
    address: cloneJsonObject(row.address),
    registration: redactRegistrationForDocuments(row.registration),
    contact: cloneJsonObject(row.contact),
    legal_name: toNullableString(row.legal_name),
    country_code: toNullableString(row.country_code),
    vat_identifier: toNullableString(row.vat_identifier),
    officers: cloneJsonObjectArray(pickOfficers(row)),
  };
}

That code does not look dramatic. It is the reason a document can be re-rendered later without asking the live entities row what the company looks like today.

The surprise was CSS. The HTML literals were obvious once I started searching. The comments in the stylesheet were not. The composer injects CSS directly into a <style> block and hashes the full rendered HTML. A comment like Klevar FZE primary is still rendered output. It becomes part of the document hash. It can leak into the PDF byte stream. It can fail an entity-neutral regression even if the visible page looks correct.

That changed the template rule: anything inside the bundle is document output. Body HTML, shared partials, stylesheet comments, labels, helper inputs. None of it gets to carry entity identity unless it comes from the snapshot or the body payload.

The renderer also became stricter in a different direction. composeHtml() uses a private Handlebars instance for each render, not the global singleton. Helpers like formatCurrency, formatDate, markdown, eq, and officerRoleLabel live inside that private instance. That prevents a test, module, or future template family from registering a helper globally and changing another document type by accident.

The snapshot fix did not end at source code. The regression had to prove the failure could not return. The gate renders every authored bundle against multiple seeded entities and searches for identity strings from the wrong entity. If an LLC render contains an FZE registration value, the test fails. If a stylesheet comment leaks a company name, the test fails. If someone adds a new bundle and hardcodes a legal name because it is faster, the gate catches it.

The deeper lesson is that PDF generation is not the domain. Legal attribution is the domain.

A renderer that accepts a body and returns bytes is easy to build. A renderer that can explain where every legal identity field came from is the system. Once I saw that, the architecture became clearer: templates do not own identity, live rows do not own old documents, and snapshots do not carry private operational fields just because the database has them.

That distinction now runs through the rest of Klevar Docs. Factur-X builder data comes from the snapshot. Board resolution officer data can default from the snapshot. Payment details freeze at issue time. Hash-chain verification depends on content hashes that include the rendered output. If a lower layer cheats, every upper layer can look correct while proving the wrong thing.

The PDF looking correct was the warning. The system only became correct when the source of correctness moved out of the template.

Reachable Is Not the Same as Correct

Kingsley Onoh — Mon, 11 May 2026 08:19:56 +0000

The CLI could create a credit note.

That was the bad news.

The umbrella documents compose command was built to make all document types reachable through one stable surface. Pass a type, an entity, and a JSON body. The server looks up the per-type Zod schema, validates the body, renders the document, and returns the same envelope shape every time.

For generic documents, that is exactly right. A letterhead, proposal, quote, statement, receipt, minutes document, reference letter, or compliance letter can be a schema-gated render through the generic pipeline.

Credit notes were different. So were invoices, pro-forma invoices, and board resolutions.

Those document types do not just render. They create specialized rows, allocate numbers, enforce state machines, inherit fields from source invoices, attach payment behavior, and feed later compliance paths. When documents compose --type credit_note went through the generic renderer, it produced a document row but bypassed createCreditNote(). That meant the credit note skipped the inheritance logic that copies terms from the original invoice.

The symptom appeared later as a Factur-X validation problem. The XML layer complained about missing payment terms. The real bug was earlier: the API made a specialized document reachable through the wrong path.

I had treated coverage as correctness. It was not.

The fix was not to delete the umbrella. The umbrella is still the right operator surface. The fix was to split the server path into two routes inside the same endpoint: specialized dispatch first, generic render second.

The registry is explicit:

export const COMPOSE_SPECIALIZED_DISPATCH: Partial<Record<TemplateType, ComposeDispatcher>> = {
  invoice: invoiceDispatcher,
  pro_forma_invoice: proFormaDispatcher,
  credit_note: creditNoteDispatcher,
  board_resolution: boardResolutionDispatcher,
};

That tiny map carries a big rule. If a type has specialized business logic, compose must call the specialized service. If it does not, compose falls through to renderDocument(). Exactly one path fires.

The dispatcher contract is intentionally boring. It maps the umbrella body into the specialized service input, calls the service, and wraps the returned row in the same envelope shape as a rendered document. It does not recompute totals. It does not duplicate inheritance. It does not allocate its own number. If the specialized service owns the rule, the dispatcher is only an adapter.

Here is the credit note path:

const row = await createCreditNote({
  db: args.db,
  pool: args.pool,
  entityId: args.entityId,
  input,
  apiKeyId: args.apiKeyId ?? null,
  requestId: args.requestId ?? null,
});

return wrapRowAsOutput({
  documentId: row.document_id,
  entityId: args.entityId,
  type: 'credit_note',
  documentNumber: row.document_number ?? '',
  status: row.status ?? 'draft',
  locale: args.locale ?? 'en',
});

That looks like extra ceremony until you compare it to the failure. The old path returned a rendered artifact while skipping the domain rule. The new path returns a draft row because specialized documents often require a later send or execute step to produce their final artifact. The response shape stays stable, but the lifecycle is honest.

What surprised me was that the bug survived because both layers were internally correct. The CLI sent a valid body. The compose endpoint validated it. The generic renderer produced a document. The Factur-X validator was right to reject the later XML. No single module was obviously broken in isolation.

The boundary was wrong.

That forced a different kind of test. Unit tests on the dispatcher are not enough. The integration test has to assert the side effect that only the specialized service can create: invoice numbering, credit-note inheritance, board-resolution numbering, generic letterhead finalization. The test is not "does compose return 201?" It is "did the right domain path fire?"

This is one of the more useful patterns in the project because it applies outside documents. An umbrella command is good when operators need one muscle memory. It becomes dangerous when the umbrella erases domain-specific behavior. Reachability is a UX property. Correctness is a domain property.

The compose endpoint now carries both.

Confidence Is Not Ownership

Kingsley Onoh — Wed, 06 May 2026 16:00:28 +0000

What should a finance queue do when two credible records point at the same case?

An invoice discrepancy and a contract breach can describe the same dispute. They can also describe two different disputes with the same counterparty, the same currency, and nearly the same amount. That is the trap in a finance operations queue. The data looks related before anyone has proved ownership.

The Workbench ingests exceptions from invoice reconciliation, transaction reconciliation, contract lifecycle events, webhook dead letters, manual operator entry, and signed Hub fanout. Every source carries its own identifiers. Some are reliable. Some are only reliable inside the upstream tool that produced them.

I had to decide what the system should do when a new exception looks like it belongs to an existing dispute.

The tempting version is simple: compute a score, pick the highest dispute, attach the exception. That makes demos feel clean. Exceptions flow in, disputes become richer, and the queue stays small.

It is also how a finance system quietly corrupts its own audit trail.

Once an exception is attached to a dispute, every later action inherits that fact. SLA timers, resolution playbooks, Notification Hub events, audit PDF exports, and operator comments all treat the relationship as true. If the relationship was only probable, the system has converted probability into evidence.

That conversion is the real design problem.

The Scoring Shape

The correlator in src/drw/domain/correlator.clj uses seven signals. Source reference and entity id each carry 0.15. Counterparty carries 0.25. Currency carries 0.10. Amount carries 0.15. Date carries 0.10. Category carries 0.10.

Those weights are not magic in the sense of being secret. They are visible because this is a spec project. But the structure matters more than the numbers.

Counterparty is the gate. A candidate dispute is not eligible unless it belongs to the same tenant, is not terminal, has the same counterparty, and falls within the correlation window. Only then does scoring begin.

That means the correlator is not a general similarity search. It is a tenant-scoped dispute ownership test.

The amount signal has a 10 percent tolerance and only scores when the currency also matches. The date signal checks whether the exception was observed within 72 hours of the dispute creation time. The source reference and entity id signals compare against exceptions already attached to the candidate dispute, not just fields on the dispute itself.

That last part matters. A dispute becomes easier to recognize as it accumulates evidence. The first invoice mismatch may create the dispute. A later webhook dead letter with the same upstream reference can now match the attached evidence, even if the dispute record itself does not carry that reference.

The core function is plain Clojure:

(defn score-candidates
  ([tenant-id exception disputes attached]
   (score-candidates tenant-id exception disputes attached {}))
  ([tenant-id exception disputes attached opts]
   (let [cfg (merge default-config opts)
         review (get-in cfg [:thresholds :review])]
     (->> disputes
          (map-indexed vector)
          (filter (fn [[_ dispute]]
                    (candidate-eligible? tenant-id exception dispute cfg)))
          (map (fn [[idx dispute]]
                 (assoc (score-candidate tenant-id exception dispute attached cfg)
                        :sort-index idx)))
          (filter #(>= (:score %) review))
          (sort-by (juxt (comp - :score) :sort-index))
          (mapv #(dissoc % :sort-index))))))

There are two details in that function I care about.

First, eligibility happens before scoring. A cross-tenant dispute receives no score. A terminal dispute receives no score. A different counterparty receives no score. The function does not let a high amount or date match compensate for a broken boundary.

Second, ties preserve input order through :sort-index. That is not glamorous. It prevents unstable review queues where two equal candidates swap positions between renders and make operators think the system changed its mind.

What I Was Wrong About

I initially treated the correlation score as the hard part.

It was not. The harder question was what the system does with the score.

There are three possible outcomes in src/drw/domain/exceptions.clj. If no candidate passes review, the exception creates a new dispute and attaches immediately. If the best candidate hits the auto-merge band and auto-merge is explicitly enabled, the exception attaches and records an auto-merged correlation. Otherwise, the system creates pending correlation records and emits a dispute.correlation_pending event.

That middle branch is intentionally hard to reach. The .env.example values set AUTO_MERGE_THRESHOLD=0.92 and REVIEW_THRESHOLD=0.70, while the source correlator defaults are lower for unit-level behavior. The runtime config is stricter because this is finance operations. False attachment costs more than a larger review queue.

What surprised me is that pending correlation became a domain object, not a UI convenience.

The queue needed an id, a tenant id, an exception id, a target dispute id, a score, a rationale, a status, a decided-by user, and decision timestamps. That is a lot of structure for something that could have been a modal row.

But the moment an operator accepts or rejects a candidate, that decision becomes part of the case history. A rejected match is useful evidence. It says someone looked at the overlap and decided the exception did not belong there. If the same upstream source sends a related item later, the prior rejection explains why the system did not combine the cases earlier.

That is why correlation records live next to exceptions and disputes instead of inside a transient UI response.

The Failure Mode Hidden In Good Matches

The most dangerous false match is not a ridiculous one.

It is the match that looks reasonable.

Same counterparty. Same currency. Amount within 10 percent. Observed inside three days. Category is billing. If those signals point to the wrong open dispute, the system does not look broken. It looks efficient.

The damage appears later. A Workflow Engine playbook starts against the wrong case. A Notification Hub event tells an operator that the dispute is ready for resolution. The audit PDF now contains an exception that belongs somewhere else. Nobody sees the root mistake because every downstream artifact is internally consistent.

That is the kind of bug that worries me more than a 500 response.

A 500 stops the flow. A wrong attachment keeps moving.

The design answer was to make confidence create a decision, not mutate the dispute. A review-band candidate becomes work for an operator. The UI exposes accept and reject actions. The API carries the same boundary. The audit log records correlation creation and later decisions.

The Workbench still supports auto-merge, but it is a policy choice. It has to be enabled. The score has to clear the higher band. The code does not pretend the existence of a scoring function means the business has accepted the risk.

Why Tenant Scope Belongs Inside The Algorithm

Tenant isolation is usually discussed at the HTTP layer. API key comes in, tenant id gets attached to the request, handlers filter queries.

That is necessary, but it is not enough here.

Correlation is a cross-entity operation by nature. It compares a new exception against many existing disputes and attached exceptions. If the algorithm accepts a list that accidentally contains another tenant's disputes, the HTTP layer is already too far away to save it.

So score-candidate checks tenant equality itself. score-candidates filters candidates through candidate-eligible?, which repeats tenant, status, counterparty, and time-window checks before scoring.

This is defensive duplication with a purpose. The route should pass tenant-scoped collections. The domain should still reject anything outside the tenant boundary. In a single-tenant fixture, this looks redundant. In a two-tenant test, it is the difference between "the route behaved" and "the invariant held."

The same philosophy appears in reports. The audit PDF renderer captures a tenant snapshot, renders with strict token lookup, and the setup check renders two tenants to make sure one tenant's identity literals never appear in the other tenant's output.

The theme is the same: do not trust a boundary because a previous layer probably handled it.

The Result

The finished local build passed 160 tests with 869 assertions. The full-flow E2E drives invoice adapter polling into exception creation, assignment, investigation, Workflow Engine resolution polling, Notification Hub event capture, and audit PDF generation.

The number I care about most is smaller: the dashboard guard. It caught a practical operations failure. The first dashboard shape rendered every dispute link in the tenant fixture. The fix capped the overview at 50 open disputes and kept totals intact.

That is the Workbench in miniature. Preserve the facts. Limit the surface. Make the operator decide when the machine only has a probability.

Confidence is useful. Ownership is a human or policy decision.