DEV Community: Lee HeeJun

The day my database refused to return a row I had just written

Lee HeeJun — Sun, 14 Jun 2026 08:01:22 +0000

For a while, my multi-tenant SaaS isolated tenants the way most apps do: every query carried a WHERE organization_id = :current_org clause, enforced in the application layer. It works. Until it doesn't — one missing filter, one new endpoint that forgets the convention, one ORM relationship that loads more than you expected, and one tenant can see another tenant's data.
For most products that's a bug. For a product whose entire value proposition is being the trustworthy custodian of someone else's records, it's existential. "We filter by organization in the code, trust us" is not a sentence I wanted to say to a security reviewer.
So I moved isolation down a layer — into the database itself, with Postgres Row-Level Security (RLS). This is a short write-up of how that rollout went, and specifically about the moment it appeared to break, which turned out to be the moment it proved it was working.
The shape of the design
The idea behind RLS is simple: instead of trusting every query to filter correctly, you attach a policy to the table, and Postgres refuses to return rows that don't match — no matter what the query says.
The policy needs to know who is asking. The common pattern is to push the current tenant into a session-scoped runtime parameter (a GUC), and have the policy read it:
sql-- set per request, transaction-scoped
SELECT set_config('app.current_org_id', :org_id, true);

-- policy on each tenant table
CREATE POLICY tenant_isolation ON some_table
FOR ALL
USING (organization_id = current_setting('app.current_org_id', true))
WITH CHECK (organization_id = current_setting('app.current_org_id', true));
Two design choices matter here.
First, fail closed. I wrapped the GUC lookup in a helper that returns NULL when the parameter is missing, rather than throwing or defaulting to something permissive. Because organization_id = NULL is never true in SQL, a request that forgets to set its tenant context sees zero rows — not everything. The absence of identity is treated as "you are no one," not "you are everyone."
Second, the app connects as a non-superuser, non-BYPASSRLS role. RLS politely does nothing for superusers and table owners. If your app's database role can bypass the very policies you wrote, you've built a very expensive no-op. The runtime role has exactly the grants it needs and nothing more; privileged maintenance happens through a separate role.
The chicken-and-egg table
One table fought back: the membership table that maps users to organizations.
Every request resolves its org_id by querying that table ("which org does this user belong to?"). But the org GUC isn't set yet at that point — resolving it is the whole reason we're reading the table. If the membership table's read policy depends on the org GUC, you deadlock yourself out of every request.
The fix is to give that one table a policy keyed on the user identity (set earlier in the request) rather than the org identity (set later). The user GUC is available before org resolution; the org GUC is not. Writes to the membership table are blocked entirely on the app path and go through the privileged role, because membership changes are not something a normal tenant request should ever do directly.
Small thing, but if you get the ordering wrong, nothing works, and the failure looks terrifyingly total.
The moment it "broke"
Here's the part I actually wanted to write about.
I enabled the policies table by table, smoke-testing each one. Most went green. Then I enabled it on a table behind a create endpoint — make a thing, read it back, return it to the user — and it threw:
sqlalchemy.exc.InvalidRequestError: Could not refresh instance ''
The handler had just inserted a row, committed, and then tried to read it back. And the read came back empty. The database refused to return a row the same request had created seconds earlier.
For about thirty seconds this looks like a catastrophic regression. It is the opposite. It is the system working exactly as designed.
The cause: transaction-scoped GUCs (set_config(..., true), equivalent to SET LOCAL) are cleared on COMMIT. The pattern was:
pythondb.add(row)
db.commit() # <- SET LOCAL tenant context is wiped here
db.refresh(row) # <- new transaction, no GUC, policy sees NULL -> 0 rows
After the commit, the next statement runs in a fresh transaction with no tenant context. The fail-closed helper returns NULL. The policy matches nothing. The refresh() finds no row — including the one we just wrote — and errors.
This is the fail-closed design proving itself in production. A session that hadn't re-established who it was got shown nothing. If the isolation had been sloppy — defaulting to "see everything" when context is missing — this would have silently succeeded, and I'd have shipped a system that returns data to sessions with no identity. The error was the receipt.
The fix is to re-inject the tenant context immediately after committing, so legitimate work continues to carry its identity across the commit boundary:
pythondef commit_and_restore(db, ctx):
db.commit()
if ctx: # tenant request: re-apply GUCs for the next transaction
set_config(db, "app.current_user_id", ctx.user_id)
set_config(db, "app.current_org_id", ctx.org_id)
# privileged/admin sessions have no ctx and pass straight through
I routed every commit-then-keep-querying handler through that helper, and watched the same endpoints go green — this time because isolation was working, not because it was absent.
What I took away

Fail-closed is only useful if you can tell it apart from a bug. The "Could not refresh" error and a genuine isolation hole would look completely different in production, but in the moment they feel identical. Knowing your design well enough to read the failure correctly is the skill.
Test the negative, eventually. Watching your own tenant's data appear is necessary but not sufficient. The real property is that another tenant's data does not appear. Single-tenant smoke tests can't prove that; the fail-closed refresh error was, ironically, the strongest live evidence I got that the empty case behaves correctly.
Defense in depth means not trusting your own code. RLS doesn't replace careful query-writing. It's the layer that holds when the careful query-writing eventually slips — and over a long enough timeline, it slips.

The most reassuring thing a security layer can do is refuse you when you haven't proven who you are. Even when "you" is the request that wrote the row.

Building a verifier that doesn't ask you to trust the issuer

Lee HeeJun — Sat, 13 Jun 2026 17:56:54 +0000

Building an Offline Verifier for AI Decision Evidence

Most audit-log systems have a quiet dependency baked into them:

to believe the log, you have to believe the system that wrote it.

A cloud provider sealing its own logs is still the cloud provider vouching for itself. A SaaS platform exporting its own audit trail is still asking the reviewer to trust the platform that produced the evidence.

That is not always wrong. But it is not independent.

If the only way to check the evidence is to trust the party that generated it, then the evidence has a structural ceiling.

I have been building AURORA, a system that seals AI decisions into signed, timestamped evidence records.

This week I shipped the part that makes the word independent more real:

an offline verifier.

A reviewer can now take an AURORA evidence bundle, drop the .zip into a browser, and verify the bundle against the cryptographic material inside it.

No OpenSSL.

No shell.

No account.

No trust in me required.

The verifier checks the manifest, file-hash inventory, RSA signature, record hash chain, and RFC 3161 timestamp evidence. Each result is shown with its own scope boundary.

Here is what I learned building it.

1. The verifier has to be provider-neutral

The first important design decision was not about UI.

It was about what the verifier should actually verify.

A naive verifier checks a vendor:

Is this a valid GlobalSign timestamp?

or:

Is this a valid FreeTSA token?

That sounds reasonable at first. But it couples your verification logic to a specific provider.

The moment you change timestamp providers, add a second provider, or let enterprise customers bring their own TSA, your verifier becomes a provider adapter instead of an evidence verifier.

That is not the shape I wanted.

So the verifier checks the bundle contract, not the vendor.

It checks:

the canonical record bytes hash to the recorded data_hash
the RSA signature verifies over the canonical JSON with the bundled public key
the public key fingerprint matches the published fingerprint
the record hash chain is self-consistent where the required material exists
the RFC 3161 timestamp token imprint binds to the timestamped message
the bundle file inventory matches the recorded SHA-256 hashes

None of those checks require the verifier to care which TSA vendor issued the timestamp.

A bundle timestamped by a free TSA and a bundle timestamped by a qualified TSA should travel through the same verification model.

The trust layer can change.

The verification contract should not.

That separation matters.

If the verifier is tied to a single provider, then the evidence format inherits the lifecycle of that provider relationship. If the verifier is tied to the bundle contract, the evidence format can outlive provider changes.

That is the property I wanted first.

2. One core, two consumers, no hidden app dependency

The verifier has two consumers.

The first is a standalone CLI. That is for auditors, engineers, or reviewers who want to run verification outside AURORA entirely.

The second is the backend /verify/bundle endpoint. That is for the browser-based upload flow.

The obvious move would be to put the verifier core in one shared package and import it everywhere.

I did not do that.

The standalone verifier needs to stay dependency-minimal. An external reviewer should not need to install a FastAPI app, database client, auth layer, storage SDK, or internal AURORA package just to check a signature.

So the core is a pure library:

bytes in.

structured result out.

No database.

No service imports.

No app.crypto.

No TSA client imports.

No HTTP assumptions.

The CLI reads a path, loads the bundle, calls the core, and renders text or JSON.

The backend receives an uploaded .zip, calls the same core, and returns the structured result.

To keep the two copies aligned, the verifier core is vendored into the backend by a sync script. The script stamps a SHA-256 of the source copy, and CI can run a --check mode to fail the build if the backend copy drifts from the standalone verifier.

This feels redundant until you remember the product goal.

The whole point is not to ask people to trust AURORA more than necessary.

A verifier that only works inside AURORA’s app tree is not an independent verifier. It is just another internal endpoint.

The standalone path has to remain real.

3. The bug that taught me the most: normalize, do not overfit

The first regression caught a very small mistake with very large implications.

I added a manifest check for the bundle format.

The verifier had a whitelist of accepted format strings.

My test fixture passed.

A real production bundle failed.

The reason was not cryptographic. It was not a broken signature. It was not a malformed bundle.

It was casing.

Production used:

AURORA_Verification_Bundle

The verifier expected a slightly different casing and separator convention.

The bundle was valid.

My equality check was too brittle.

The fix was to normalize before comparing:

trim
lowercase
normalize separators
then compare against the canonical format identity

The lesson is simple:

when validating identifiers you control on both sides, normalize first.

Exact string whitelists are useful at protocol boundaries. But when your own system already has historical or production variants, a strict literal check can turn valid evidence into a false failure.

False negatives in a verifier are dangerous.

Imagine an auditor uploads a legitimate evidence bundle and the tool says FAILED because one internal label used a different underscore pattern.

That is not integrity.

That is your validator mistaking formatting for evidence.

4. Failure modes need to be honest

This is the part I care about most.

A verifier has power because it produces a verdict.

That also makes it dangerous.

If it hides ambiguity, rounds warnings into passes, or makes trust claims it has not actually proven, it becomes another system the reviewer has to trust blindly.

That defeats the purpose.

So the verifier separates required checks, warnings, unavailable checks, and non-applicable checks.

For example, AURORA currently has a dual-path timestamp condition in some bundles.

The timestamp hash bound at sealing time and the direct hash of the bundled timestamp token can differ because the bundled token may come from a separate TSA call.

That looks suspicious if you only compare bytes mechanically.

But it is not necessarily a tampering signal.

So the verifier reports it as a warning, with a plain-language explanation. The integrity verdict can still pass if the required signature, hash, manifest, and record-chain checks pass.

Same with timestamp trust status.

If a TSA is non-qualified, the result says non_qualified.

If a qualified trust-list validation has not been performed, the verifier does not pretend it has.

That boundary is part of the result.

Not hidden.

Not dressed up.

Not silently promoted.

The principle is:

say exactly what was checked, exactly what passed, exactly what failed, and exactly what was not proven.

That may sound obvious.

It is not how a lot of software behaves.

A verifier that hides its own limits is just another trust demand.

5. HTTP semantics matter more than they look

The browser verifier exposed another small but important distinction:

a malformed upload and a failed verification are not the same thing.

If someone uploads a random file, or a .zip without a manifest, the server could not perform bundle verification.

That is a structural input problem.

It should return something like:

422 Unprocessable Entity

But if someone uploads a well-formed AURORA bundle and the RSA signature fails, that is different.

The verifier successfully ran.

The answer is just no.

That should return:

200 OK
{
  "ok": false,
  "result": "FAILED"
}

A failed cryptographic check is not a server error.

It is a valid verification result.

Collapsing both cases into a generic 400 loses the distinction between:

I could not read this evidence.

and:

I read this evidence, and it does not verify.

For auditors, that distinction matters.

For debugging, it matters.

For trust, it matters.

6. The browser is convenience. The CLI is independence.

The new browser verifier is useful because it removes friction.

A reviewer does not need OpenSSL installed. They do not need to know which command verifies a signature or how to parse an RFC 3161 token.

They can upload the bundle and see the axes:

manifest
file inventory
canonical record hash
signature
record hash chain
timestamp evidence

That is the accessible path.

But it is not the only path.

The CLI remains important because it is the fully independent path.

The browser flow still asks AURORA’s server to run the verifier core. That is useful, but not maximally independent.

The CLI lets a reviewer run the same verification logic outside AURORA’s infrastructure.

That is why the verification guide now has both:

browser-based bundle verification for no-CLI review
OpenSSL / CLI verification for fully independent offline review

The browser lowers the barrier.

The CLI preserves the trust boundary.

Both matter.

7. What the verifier does not do

The verifier does not tell you the AI decision was correct.

It does not tell you the decision was lawful.

It does not tell you the decision was fair.

It does not tell you a regulator approved it.

It does not tell you a court will admit it.

That is not its job.

Its job is narrower:

is this evidence package intact?
do the hashes match?
does the signature verify?
does the timestamp evidence bind to the expected message?
can an external reviewer reproduce the integrity checks?
does the result state its own limits clearly?

That narrowness is intentional.

AURORA is not trying to be the judge.

It is trying to be the custody layer.

Where it lands

The result is now live:

aurora-audit.com/verify-bundle

An external reviewer with no account, no CLI, and no reason to trust me can drag an AURORA evidence bundle into the browser and watch the verifier check the package.

The core is still beta.

The rough edges still show.

Qualified timestamp trust-list validation is a separate trust layer and is still being hardened. The verifier does not pretend otherwise.

But the important part is real now:

the evidence can leave the original workflow and still be checked.

That is the line I wanted to cross first.

If you have built independently verifiable evidence systems, I would be interested in how you handled the trust-boundary question:

where did you draw the line between “verify” and “trust me”?

Building RFC 3161 Layer 2 Verification for AI Decision Evidence

Lee HeeJun — Thu, 11 Jun 2026 16:06:04 +0000

Most developers think timestamping means sending a request to a Time Stamping Authority and storing the response.

Something like this:

POST /tsa
Content-Type: application/timestamp-query

<binary timestamp request>

But that is not verification.

That is only asking for a timestamp.

If you want to use a timestamp as evidence, you need to verify what came back.

And in RFC 3161, that means dealing with CMS SignedData.

The problem

An RFC 3161 timestamp response is not just a date.

It is a cryptographic structure.

Inside the response, there is a timestamp token. That token is usually a CMS SignedData object containing TSTInfo, signer information, signed attributes, certificates, and a signature.

If you want to treat the timestamp as evidence, you need to answer several questions:

Does the timestamp token refer to the payload I actually submitted?
Does the message imprint match my payload digest?
Which certificate signed the timestamp token?
Is the CMS signature valid?
Was the signer certificate valid for timestamping?
Can this verification be repeated later by another party?

Most basic integrations stop too early.

They parse the response, extract the generated time, store the token, and assume that is enough.

For a normal application timestamp, that may be acceptable.

For an evidence system, it is not.

What I am building

I am building AURORA, a cryptographic evidence layer for AI decisions.

The goal is not to decide whether an AI decision was fair, legal, or correct.

The goal is narrower:

preserve verifiable evidence of what was recorded.

A sealed AI decision record can include:

SHA-256 record hash
RSA digital signature
RFC 3161 timestamp token
public verification URL
downloadable evidence bundle

The timestamp layer forced me to go deeper into RFC 3161 and CMS verification than I originally expected.

The code snippets below are simplified for explanation, but the verification boundaries are the same.

Layer 1: Parsing the timestamp response

The first layer is relatively straightforward.

You parse the timestamp response, extract the timestamp token, and read the embedded TSTInfo structure.

Conceptually:

tsr = tsp.TimeStampResp.load(token_der)

timestamp_token = tsr["time_stamp_token"]
signed_data = timestamp_token["content"]

tst_info = tsp.TSTInfo.load(
    signed_data["encap_content_info"]["content"].parsed.dump()
)

Then you check the message imprint.

message_imprint = tst_info["message_imprint"]
hashed_message = message_imprint["hashed_message"].native

if hashed_message != expected_digest:
    raise ValueError("Timestamp imprint does not match payload digest")

This confirms that the timestamp token refers to the digest you expected.

That matters because the TSA does not timestamp your original JSON, PDF, or database record directly.

It timestamps a digest.

If the digest in TSTInfo does not match your payload digest, the token is not evidence for your record.

But Layer 1 is still not enough.

It proves that the token contains the expected imprint.

It does not fully prove that the CMS signature is valid.

Layer 2: Verifying CMS SignedData

The harder part is verifying the CMS SignedData structure itself.

This is where the integration becomes less like an API call and more like applied cryptography.

A timestamp token is a signed object.

To verify it properly, you need to inspect the signer, the signed attributes, the encapsulated content, and the certificate usage.

1. Do not assume `certs[0]` is the signer

One easy mistake is assuming that the first certificate inside the CMS certificate set is the signer certificate.

That is unsafe.

CMS does not guarantee that certs[0] is the signer.

The signer must be selected using the SignerInfo identifier.

Conceptually:

def select_signer_certificate(certs, signer_info):
    signer_id = signer_info["sid"]

    for cert in certs:
        if cert_matches_signer_id(cert, signer_id):
            return cert

    raise ValueError("Signer certificate not found")

The signer can be identified by issuer and serial number, or by subject key identifier.

If you choose the wrong certificate, verification fails.

Worse, you may build an implementation that appears to work only because one provider happens to order certificates in a convenient way.

That is not a verification strategy.

That is an accident.

2. Verify the `message-digest` signed attribute

CMS signed attributes usually include a message-digest attribute.

This digest must match the digest of the encapsulated TSTInfo content.

Conceptually:

expected_digest = extract_message_digest_attr(signed_attrs)
actual_digest = sha256(tst_info_der).digest()

if expected_digest != actual_digest:
    raise ValueError("CMS message-digest does not match TSTInfo")

This step is easy to misunderstand.

The signature is not simply over your original payload.

In this structure, the signature is over the signed attributes.

Those signed attributes include a digest of the encapsulated content.

So the chain is:

original payload
→ payload digest
→ TSTInfo message imprint
→ TSTInfo DER
→ CMS signedAttrs message-digest
→ CMS signature

Each boundary matters.

If one layer is skipped, the evidence chain becomes weaker.

3. Handle the signed attributes encoding correctly

This was one of the most annoying parts.

CMS signedAttrs are encoded with an IMPLICIT context-specific tag on the wire.

But for signature verification, the signed attributes need to be verified as a SET OF attributes.

In simplified form:

signed_attrs_for_verification = b"\x31" + signed_attrs_wire[1:]

That small encoding detail can break verification for hours.

The data may look logically correct.

The parsed object may look correct.

The certificate may be correct.

The digest may be correct.

But if the exact bytes passed into signature verification do not match the expected DER encoding, the signature fails.

This is one of the places where “I parsed the object” and “I verified the evidence” become very different things.

4. Verify the CMS signature

Once the signer certificate is selected and the signed attributes are encoded correctly, the signature can be verified using the signer certificate’s public key.

Conceptually:

public_key.verify(
    signature_bytes,
    signed_attrs_for_verification,
    padding.PKCS1v15(),
    hash_algorithm,
)

If this succeeds, the CMS signature over the signed attributes is valid.

If it fails, the timestamp token should not be treated as verified evidence.

At this stage, you are no longer merely trusting that the TSA returned something that looks valid.

You are checking the cryptographic structure yourself.

5. Check certificate usage

A valid signature is still not the entire story.

The certificate should also be valid for timestamping.

For RFC 3161 timestamping, that usually means checking for the id-kp-timeStamping extended key usage.

Conceptually:

eku = cert.extensions.get_extension_for_class(
    x509.ExtendedKeyUsage
).value

if ExtendedKeyUsageOID.TIME_STAMPING not in eku:
    raise ValueError("Certificate is not valid for timestamping")

Without this check, you may verify a signature from a certificate that was not intended to issue timestamp tokens.

For an evidence layer, that distinction matters.

The question is not only:

“Was this signed?”

The better question is:

“Was this signed by the right kind of certificate, for the right kind of purpose, over the right content?”

Why this matters for AI decision evidence

AURORA is built around AI decision records.

A record might describe a loan decision, an insurance claim decision, a hiring review, a moderation outcome, a risk score, or another system-generated decision event.

For internal analytics, a normal database log may be enough.

For audit evidence, it is weaker.

A database timestamp says:

“This is what the database currently says.”

A cryptographic timestamp says:

“This digest existed at this time, according to an external timestamp authority.”

A verified cryptographic timestamp says:

“We checked that the timestamp token, imprint, signature, signer certificate, and evidence chain are internally consistent.”

That is a stronger evidence boundary.

What AURORA does not claim

AURORA does not determine whether an AI decision was fair.

It does not determine whether a decision was ethical.

It does not replace legal review.

It does not guarantee regulatory compliance.

Those are separate questions.

AURORA focuses on evidence preservation.

It tries to answer a narrower but important question:

“What exactly was recorded, and can that record be verified later?”

Key takeaways

The main lessons from implementing this layer:

A timestamp response is not the same as a verified timestamp.
RFC 3161 timestamp tokens contain CMS SignedData.
The message imprint must match the payload digest.
The CMS message-digest signed attribute must match TSTInfo.
The signer certificate should be selected by SignerInfo, not by array position.
signedAttrs encoding details matter.
The signer certificate should be valid for timestamping.
Evidence systems should expose verification metadata, not hide it.

The result

AURORA’s verification layer is designed to verify RFC 3161 timestamp evidence instead of blindly trusting that a TSA response is valid.

That timestamp layer sits alongside:

SHA-256 record hashing
RSA signatures
public verification pages
downloadable evidence bundles

The broader goal is simple:

AI decision records should be verifiable later.

Not just logged.

Verified.

Project: https://aurora-audit.com

Sample PDF: https://aurora-audit.com/sample-audit.pdf

Have you implemented RFC 3161, CMS SignedData, or timestamp verification before?

I would be interested in hearing what edge cases you ran into.

DEV Community: Lee HeeJun

The day my database refused to return a row I had just written

Building a verifier that doesn't ask you to trust the issuer

Building an Offline Verifier for AI Decision Evidence

1. The verifier has to be provider-neutral

2. One core, two consumers, no hidden app dependency

3. The bug that taught me the most: normalize, do not overfit

4. Failure modes need to be honest

5. HTTP semantics matter more than they look

6. The browser is convenience. The CLI is independence.

7. What the verifier does not do

Where it lands

Building RFC 3161 Layer 2 Verification for AI Decision Evidence

The problem

What I am building

Layer 1: Parsing the timestamp response

Layer 2: Verifying CMS SignedData

1. Do not assume certs[0] is the signer

2. Verify the message-digest signed attribute

3. Handle the signed attributes encoding correctly

4. Verify the CMS signature

5. Check certificate usage

Why this matters for AI decision evidence

What AURORA does not claim

Key takeaways

The result

1. Do not assume `certs[0]` is the signer

2. Verify the `message-digest` signed attribute