The encrypted secret was bytes. It round-tripped perfectly in the in-memory store. The unit tests passed. In Postgres, the column was JSON, and bytes is not JSON-serializable. The secret store blew up at the one boundary the in-memory test never crossed.
That was the pattern of the entire v0.2.0 field test on HivePlane. The hard part wasn't the logic — it was the data model and the boundaries. Every defect was a value that was fine in Python and broke one step later, at a persistence seam.
This is the article about tracing each value across every persistence boundary — because the bug lives one boundary further than your unit test.
One defect per seam-type
| Defect | The value that crossed the seam | Where it broke |
|---|---|---|
| D-5 | Secret ciphertext as bytes
|
Postgres JSON persistence rejected binary |
| D-7 | A desired_specs row keyed by id |
Second source hit a PK collision — id vs uniqueness |
| D-8 | The kill-switch record existed |
GET /tools/disabled listed enabled tools — state vs record |
| D-9 | A workload with runs, deleted | Raw FK violation 500 instead of a 409 with a reason |
| #573 | A backup manifest pinning alembic_head=0034
|
Schema head was 0035 — restore against the wrong schema |
Read that table as the taxonomy. The seam, not the logic, is where each one failed.
The bytes that broke JSON
D-5 was the cleanest example. The encrypted secret vault stored ciphertext as bytes. In the in-memory store, it round-tripped perfectly. The unit tests passed. In Postgres, the column was JSON, and bytes is not JSON-serializable. The secret store blew up at the persistence boundary — the one place the in-memory test never exercised.
The fix wasn't in the model; it was at the boundary: base64-encode the ciphertext on the way to Postgres, decode on the way back. The model stays bytes. The seam learns to translate. The unit test that uses the in-memory store still passes — and still proves nothing about the seam.
tip: Your in-memory store and your Postgres store share an interface, not a serialization. A value that round-trips in memory can fail at the JSON column. Trace the value across the boundary the in-memory test never crosses.
The second source that broke the first
D-7 was an id-versus-uniqueness confusion. The reconcile loader keyed desired_specs by id. One source worked. A second source, applying the same desired state, hit a primary-key violation — because the row already existed, and the key assumed one source. The uniqueness constraint and the id were the same field, and they meant different things.
The fix is composite keys and idempotent upserts: (source_id, workload_id). A second source is a second row, not a collision. The bug only appears when you have two sources, which is why the single-source field test never caught it.
The enabled tool that was "disabled"
D-8 was a state-versus-record confusion that I still find funny. GET /tools/disabled returned a list that included enabled tools. The kill-switch record existed in the table; the tool was not actually disabled. The endpoint reported the record, not the state. A "disabled tools" list that includes enabled tools is worse than no list — it trains the operator to ignore it.
The fix: the endpoint reports the state, derived from the record and the current tool status, not the record alone. A record of a disable that was later reversed is history, not a current fact.
The FK that returned 500
D-9 was the database teaching me API design. Deleting a workload that had runs returned a 500 — a raw foreign-key violation surfacing as an internal error. The caller got a server error; the operator got no reason. The fix is a 409 with "workload has N runs; delete the runs first or cascade." A constraint violation is a user-facing condition, not a server fault. The seam between the database and the API is where a 500 becomes a 409.
The schema head the manifest lied about
573 was the one that could have been catastrophic. A backup manifest pinned alembic_head to 0034. The live schema head was 0035. A restore would have succeeded — the manifest said the schemas matched — against a database that was one migration ahead. The restore verified the manifest's claim, not the world's truth.
The fix: the restore verifies the actual schema head against the manifest, and refuses a mismatch. A manifest that lies about its own head is a backup that restores you to the wrong version of your own system.
What I learned
Trace each value across every persistence boundary: binary-vs-JSON, global-key-vs-tenant-scope, id-vs-uniqueness, state-vs-record, FK-vs-409, manifest-head-vs-schema-head. A unit test that stops at the seam passes; the bug lives one boundary further. The Postgres-gated tests — the ones CI runs with a database — are the ones that catch it. And a constraint violation is a user condition, not a server fault: the seam between the database and the API is where 500 becomes 409 with a reason.
The v0.2.0 field test runs zero-skip against the real stack with real PostgreSQL — 36/36 scenarios, 67/67 container tests — and the defect ledger (D-1…D-11) is the evidence that the seams, not the logic, were the work. Full receipts in the field-test report and the Docker test report.
References
-
HivePlane —
pip install hiveplane==0.2.0 - Field test report (36/36) · Docker report (67/67)
- Migration guide (v0.1.0 → v0.2.0)
- State store design
Pick one value in your system and trace it from the model to the Postgres column to the API response. Where does it change shape? That's where your next bug is — and where your unit test stops.
Top comments (1)
D-5 is why running SQLite or in-memory dicts for test harnesses always bites you later. Python dictionaries happily store raw bytes, custom classes, or unpickled objects without complaint. You only discover Postgres jsonb or Pydantic validation rejects the wire format once psycopg actually tries to serialize the query.
D-8 also hits anyone doing audit tables or event logs. If you query the history table instead of folding the events or checking an active state flag, the mere existence of a disable event gets treated as the current truth. I had a runner report workers as stopped for three days because the healthcheck was reading the historical incident log instead of the live lease table.