DEV Community

Daniel Ioni
Daniel Ioni

Posted on

#16 Tests Passed. Production Still Failed Closed Incorrectly: What We Learned Building Evidence Governance in MyZubster

#evidence #governance #myzubster #passed

16 Tests Passed. Production Still Failed Closed Incorrectly: What We Learned Building Evidence Governance in MyZubster

We have been building a small real-world pilot inside MyZubster around a surprisingly simple event:

one person gives kefir culture to another person by hand.

There is no payment.

There is no automatic blockchain transaction.

There is no assumption that participation is research consent.

And yet this small interaction forced us to confront a much larger engineering question:

How should a system decide which evidence may persist, which evidence may become a cryptographic commitment, and which evidence must never become public?

This week we moved that question from documentation into executable policy.

And then we discovered something more valuable than another green test suite:

our policy passed every test and still behaved incorrectly in the real runtime.

Here is what happened.

The pilot

The current MyZubster kefir flow models a physical handover between a donor and a recipient.

The lifecycle is deliberately explicit:

ACCEPTED
   ↓
HANDED_OVER
   ↓
RECEIVED
   ↓
RECORDED
Enter fullscreen mode Exit fullscreen mode

The donor confirms the physical handover.

The recipient independently confirms receipt.

Only after those confirmations can the interaction become a recorded event.

The flow is free by design:

paymentRequired = false
Enter fullscreen mode Exit fullscreen mode

Recording the handover also does not automatically mean recording something on a blockchain.

That distinction became increasingly important as we started designing a broader research and evidence-governance layer.

From "data" to evidence classes

We introduced an evidence classification model.

The initial classes are:

PUBLIC
RESTRICTED
PARTICIPANT_ONLY
EPHEMERAL
Enter fullscreen mode Exit fullscreen mode

They represent different disclosure and persistence expectations.

For example, a participant-only handover may legitimately exist inside the system while still being inappropriate for public anchoring.

Ephemeral information should not silently become permanent evidence at all.

This led to an important architectural separation:

EVENT
  ↓
EVIDENCE CLASSIFICATION
  ↓
POLICY
  ↓
COMMITMENT ELIGIBILITY
  ↓
PUBLICATION / ANCHORING POLICY
Enter fullscreen mode Exit fullscreen mode

A cryptographic hash does not magically make sensitive information safe.

And the ability to create a commitment does not imply permission to publish it.

Privacy should be a policy, not a comment

We implemented an evidence policy service and started enforcing it in the kefir handover path.

Among the rules:

missing classification → deny

PUBLIC → commitment may be prepared

PUBLIC + explicit permission → public anchoring may be allowed

RESTRICTED → private commitment possible, public anchoring denied

PARTICIPANT_ONLY → private commitment possible, public anchoring denied

EPHEMERAL → persistent commitment denied
Enter fullscreen mode Exit fullscreen mode

The critical default is simple:

Missing classification must fail closed.

We also made new kefir handovers default to participant-only, non-research evidence.

Conceptually:

evidenceClass: PARTICIPANT_ONLY
purpose: kefir_hand_delivery_evidence
researchEligible: false
publicAnchoringAllowed: false
Enter fullscreen mode Exit fullscreen mode

This is important because operational participation is not the same thing as research participation.

Using a marketplace does not equal research consent.

Giving or receiving kefir does not equal research consent.

Contributing code does not equal research consent.

Research participation requires a separate explicit process.

The tests were green

We added Node tests covering the policy.

The suite included cases such as:

✔ missing classification fails closed
✔ PUBLIC evidence can prepare a commitment
✔ PUBLIC evidence requires explicit permission for public anchoring
✔ RESTRICTED evidence may have a private commitment but cannot be publicly anchored
✔ PARTICIPANT_ONLY evidence may have a private commitment but cannot be publicly anchored
✔ EPHEMERAL evidence cannot become persistent evidence or commitment
Enter fullscreen mode Exit fullscreen mode

The existing handover tests also passed:

✔ free kefir handover is mounted in the pilot API
✔ handover states advance only through explicit confirmations
✔ donor and recipient confirmations are separated
✔ free handover never implies payment or blockchain evidence
✔ kefir handover defaults to participant-only non-research evidence
✔ commitment preparation is governed by evidence classification
Enter fullscreen mode Exit fullscreen mode

Final result:

tests 16
pass 16
fail 0
Enter fullscreen mode Exit fullscreen mode

Everything looked correct.

It wasn't.

The legacy-data problem

Before considering the feature complete, we inspected existing handovers directly in MongoDB.

There were two legacy records without the new evidence classification.

One of them had already reached:

state = RECORDED
Enter fullscreen mode Exit fullscreen mode

and had an existing blockchain commitment from an earlier version of the system.

This was the perfect test case.

According to the new policy, asking the system to prepare another commitment for this legacy record should fail:

classification = MISSING
        ↓
fail closed
        ↓
EVIDENCE_CLASS_REQUIRED
Enter fullscreen mode Exit fullscreen mode

So we tested it against the running API.

Instead we got:

HTTP=200
Enter fullscreen mode Exit fullscreen mode

The commitment preparation succeeded.

That was the first important failure.

MongoDB said MISSING. Mongoose said PARTICIPANT_ONLY.

We compared three ways of reading exactly the same document.

Raw MongoDB:

MISSING
Enter fullscreen mode Exit fullscreen mode

Hydrated Mongoose document:

{
  "evidenceClass": "PARTICIPANT_ONLY",
  "purpose": "kefir_hand_delivery_evidence",
  "disclosureAudience": [
    "donor",
    "recipient"
  ],
  "researchEligible": false,
  "publicAnchoringAllowed": false
}
Enter fullscreen mode Exit fullscreen mode

Mongoose with .lean():

MISSING
Enter fullscreen mode Exit fullscreen mode

That exposed the bug.

The legacy MongoDB document did not contain an evidence classification.

But the Mongoose schema had defaults for new handovers.

When the legacy record became a hydrated Mongoose document, those defaults made the record appear classified even though the classification had never been persisted.

Our policy wasn't failing.

We were giving the policy the wrong representation of reality.

The route effectively saw:

MongoDB:
classification = MISSING

        ↓ Mongoose hydration

Application:
classification = PARTICIPANT_ONLY

        ↓

canPrepareCommitment()

        ↓

ALLOWED
Enter fullscreen mode Exit fullscreen mode

This distinction matters enormously in systems dealing with consent, privacy, authorization, provenance or evidence.

A schema default is not historical evidence.

A value synthesized by an ODM is not necessarily a value a participant actually supplied, approved or previously persisted.

Fixing the trust boundary

We changed the commitment gate so that the security decision uses the classification actually persisted in MongoDB.

Conceptually:

const rawEvidence =
  await MarketplaceHandover.collection.findOne(
    { _id: handover._id },
    { projection: { evidenceClassification: 1 } }
  );

const commitmentPolicy = canPrepareCommitment(
  rawEvidence?.evidenceClassification
);
Enter fullscreen mode Exit fullscreen mode

The hydrated document can still be useful for normal application behavior.

But for this specific policy decision, the important question is:

Was this classification actually stored?

If the answer is no, the system fails closed.

We also strengthened the regression test to ensure the route does not silently return to:

canPrepareCommitment(
  handover.evidenceClassification
);
Enter fullscreen mode Exit fullscreen mode

for this decision.

Again, the suite passed:

16/16
Enter fullscreen mode Exit fullscreen mode

We restarted the service.

We tested the legacy handover again.

And it still returned:

HTTP=200
Enter fullscreen mode Exit fullscreen mode

That led to the second bug.

We were restarting the wrong process

Our environment contained a Sepolia frontend/backend setup managed partly through PM2.

We had been restarting the apparent Sepolia process after changing the backend.

But then we inspected the process actually listening on port 5003.

It wasn't the PM2 process.

The live gateway was:

/usr/bin/node /root/myzubster/scripts/start-gateway-systemd.js
Enter fullscreen mode Exit fullscreen mode

and its cgroup belonged to:

myzubster.service
Enter fullscreen mode Exit fullscreen mode

So we had:

source code patched       ✓
tests against source      ✓
PM2 restarted             ✓

actual gateway restarted  ✗
Enter fullscreen mode Exit fullscreen mode

The real API was being served by systemd.

This explained why our code and tests said one thing while the runtime continued doing another.

We restarted the actual service:

myzubster.service
Enter fullscreen mode Exit fullscreen mode

The new process came online, connected to MongoDB and listened again on:

127.0.0.1:5003
Enter fullscreen mode Exit fullscreen mode

Then we repeated the same runtime test.

The result we wanted

This time:

{
  "success": false,
  "code": "EVIDENCE_CLASS_REQUIRED",
  "message": "La classificazione dell’evidenza non consente la preparazione del commitment."
}
Enter fullscreen mode Exit fullscreen mode

And:

HTTP=409
Enter fullscreen mode Exit fullscreen mode

Now the complete chain behaved as intended:

LEGACY MONGODB DOCUMENT
classification = MISSING
        ↓
read persisted classification
        ↓
canPrepareCommitment(undefined)
        ↓
EVIDENCE_CLASS_REQUIRED
        ↓
HTTP 409
Enter fullscreen mode Exit fullscreen mode

That was our actual proof.

Not merely:

unit tests = green
Enter fullscreen mode Exit fullscreen mode

but:

policy tests
    +
legacy database state
    +
real authentication
    +
real participant authorization
    +
real service process
    +
real API request
    =
runtime fail-closed proof
Enter fullscreen mode Exit fullscreen mode

Evidence governance is becoming infrastructure

This work is part of a larger separation we are introducing in MyZubster.

We now distinguish between operational participation, contribution, research candidacy and research participation.

In simplified form:

PILOT PARTICIPANT
      ≠
CONTRIBUTOR
      ≠
RESEARCH CANDIDATE
      ≠
RESEARCH PARTICIPANT
Enter fullscreen mode Exit fullscreen mode

Research consent must be explicit.

Candidate inclusion is not research consent.

Marketplace activity is not research consent.

A GitHub contribution is not research consent.

We have also started documenting a separate private-verifiability research direction around the question:

What is the minimum information required to verify an event without unnecessarily exposing participants, relationships, locations or activities?

That work currently remains a research/design track.

We explicitly document things that are not implemented, including:

Monero settlement integration: NOT IMPLEMENTED

Zero-knowledge prototype: NOT IMPLEMENTED

Private reputation system: NOT IMPLEMENTED

Formal university validation: NOT CLAIMED
Enter fullscreen mode Exit fullscreen mode

That boundary is intentional.

Documentation should describe the system we actually have, not the system we hope people assume we have.

What we learned

The biggest lesson was not about MongoDB.

It was about evidence.

In an ordinary CRUD application, a schema default may simply be convenient.

In an evidence system, the difference between:

"value exists"
Enter fullscreen mode Exit fullscreen mode

and:

"the application supplied a default because the value was missing"
Enter fullscreen mode Exit fullscreen mode

can be a trust-boundary issue.

The same applies to consent.

The same applies to provenance.

The same applies to privacy classifications.

And the deployment lesson was equally useful:

A test suite proves properties of the code it executed. It does not prove that the process receiving real requests is executing that code.

We needed both.

What is still missing

There is plenty left to build.

The current work establishes policy and governance foundations, not a completed research infrastructure.

We still need stronger integration tests around persisted legacy records, explicit migration strategies for old evidence, clearer lifecycle management for evidence classifications, and further separation between private commitments and genuinely public evidence.

The private-verifiability work is even earlier.

Zero-knowledge proofs are not implemented.

Monero integration is not implemented.

Private reputation is not implemented.

Formal university validation is not claimed.

And our current research-participant registry remains deliberately empty until an appropriate explicit research-consent process exists.

Why the kefir pilot matters

A hand-delivered jar of kefir may sound like an unusually small thing to build infrastructure around.

That is precisely why it is useful.

The interaction is understandable:

one person offers something
another accepts
the donor hands it over
the recipient confirms receipt
the system records what happened
Enter fullscreen mode Exit fullscreen mode

From there, difficult questions appear naturally.

Who is allowed to confirm each step?

What counts as evidence?

Who may see it?

Should it persist?

Can it be hashed?

Can that hash be made public?

Does participation imply research consent?

What happens to records created before those rules existed?

Those questions apply far beyond kefir.

The principle we're converging on

The direction can be summarized in one sentence:

Verifiable does not have to mean public.

And we can make it slightly stronger:

verification ≠ publication
participation ≠ research consent
schema default ≠ persisted evidence
commitment ≠ permission to anchor publicly
green tests ≠ verified runtime behavior
Enter fullscreen mode Exit fullscreen mode

The next phase is to keep turning those distinctions into enforceable system properties rather than leaving them as documentation.

Because if privacy, consent and provenance matter, they should not depend on everyone remembering what a comment in the code once said.

Top comments (0)