DEV Community: gentlyding

Read Replicas Promised Scale. They Also Gave You Stale Data.

gentlyding — Mon, 21 Sep 2026 14:57:02 +0000

You added a read replica to take pressure off the primary. Reads got faster, the primary stopped sweating, everyone was happy — until a user updated their profile, hit refresh, and saw the old value stare back at them. Or an order flipped to "paid" in the admin panel while the customer-facing page still said "pending" for ten seconds. None of that is a bug in your code. It's the shape of asynchronous replication, and almost nobody designs for it up front.

This isn't a post arguing against replicas. It's a post about the one cost nobody puts on the whiteboard: a read replica gives you throughput, not freshness. Those are different promises.

What "asynchronous replication" actually means

With a typical primary-replica setup (Postgres streaming replica, MySQL read-only replica, a cloud reader endpoint, etc.), the replica applies changes after they've committed on the primary. There's a lag — usually milliseconds, sometimes seconds, occasionally minutes when something backs up.

During that window, the replica is serving data that is simply not current. Not wrong, exactly. Just older than the primary by the length of the lag.

primary:  UPDATE users SET name='Ada' WHERE id=7;  -- commits at T0
replica:  still has name='Bob' until the WAL/redo reaches it (~T0 + lag)

If a client reads from the replica in that gap, it gets 'Bob'. The write succeeded; the read just didn't see it yet. That's a read-after-write (read-your-own-write) violation, and it's the single most common surprise people hit after adding a replica.

Where the lag actually comes from

It's easy to blame "the network," but the sources are more varied:

Apply-queue backlog. The replica can receive WAL faster than it can apply it, especially under heavy write load or large transactions.
Long-running queries on the replica. A reporting query saturating I/O delays replication apply.
One huge transaction. A multi-GB bulk update can stall the apply thread for its full duration.
Backups / VACUUM / maintenance windows. These compete for the same resources.
Network partition or replica restart. Now you're behind by however long it was down.

The point: lag is not a constant you can subtract away. It's a distribution, and the tail is what bites you.

The instinct that makes it worse: route everything to the replica

The tempting setup is "writes to primary, reads to replica" — send all SELECTs to the read endpoint and call it scaling. That's also the setup that produces the profile-refresh bug above, because some reads must be strongly consistent:

The row you just wrote (read-your-own-write).
Balances, inventory counts, payment/order status — anything where showing a stale value causes a real-world mistake.
Anything another user just changed that the current user is actively acting on.

Pushing those through a replica "because it's a read" is how you get a customer who paid twice because the available-balance read lagged, or an admin who approves something they shouldn't because the status hadn't caught up.

What actually works (without throwing away the replica)

You don't need synchronous replication (which trades throughput for freshness — often the opposite of why you added a replica). You need to be selective about which reads need to be current.

1. Read-after-write: pin the session to the primary right after a write.
The simplest version: after a mutating request, keep that user's subsequent reads on the primary for a short window (a few seconds). The user sees their own change immediately; everyone else tolerates the lag. Many connection pools support a "primary for this session" flag; in a Spring app you can route through a replication-aware datasource that flips to primary for the remainder of the request or a short TTL.

2. Version / token handoff.
When you write, return a version or timestamp to the client. On the next read, the client sends it back; if the replica's applied position is behind that version, read from the primary instead. More precise than a fixed time window, and it survives the "user walked away and came back 20 minutes later" case.

# after a write, tell the client how far the data has progressed
response.headers["x-write-version"] = str(user.version)
# next read: only escalate to primary if the replica hasn't caught up
if replica_applied_version < int(request.headers["x-write-version"]):
    read_from(primary)

3. Route by data criticality, not by SQL verb.
Decide per query: is this read allowed to be slightly stale? Analytics dashboards, search indexes, public profiles, "last updated" lists — those are replica-friendly. Account balances, order state, anything transactional — primary. Encode that decision in your data-access layer, not in ad-hoc routing scattered across the codebase.

4. Don't forget the cache has the same disease.
A cache is just a faster, more aggressive replica. "Set a TTL" is not an invalidation strategy; it's a confession that you'll serve stale data for the length of the TTL. For read-after-write correctness, prefer write-through (update the cache on write) or explicit invalidation on the write path over relying on expiry. The same read-your-own-write logic applies: after you write, make sure the next read doesn't hit a stale cache entry.

When the replica is exactly the right tool

To be clear, replicas are great — for the right reads:

Analytical and reporting queries that would otherwise hammer the primary.
Full-text search indexes.
Public, non-personalized content.
Any read where "a few seconds old" is fine, which is most reads.

The mistake isn't using a replica. It's treating "replica" and "consistent" as the same word.

The metric you should be watching

If you run a replica and you're not tracking replication lag, you're flying blind. Every engine exposes it: Postgres via pg_stat_replication (or comparing pg_last_wal_receive_lsn / pg_last_wal_replay_lsn), MySQL via Seconds_Behind_* / replica-lag metrics. Plot it, alert on the tail, and — crucially — keep it visible where a human decides whether a stale read just caused a problem. A p99 lag of eight seconds tells you exactly how far behind a confused user might be. Concretely, on Postgres you can see the gap directly:

-- how far behind is each replica, in bytes of WAL not yet replayed
SELECT client_addr,
       pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS bytes_behind
FROM pg_stat_replication;

The part nobody warns you about

Adding a read replica feels like free scale. It's not free — it's a trade where you give up "every read sees the latest write" in exchange for "reads don't load the primary." That trade is almost always worth it. But you have to make the trade consciously: enumerate the reads that must be current, route only those to the primary (or pin the session, or hand off a version token), and stop assuming SELECT means consistent. The replica solved your throughput problem. It quietly created a freshness problem you now own.

Merkle Trees vs Hash Chains for Audit Logs: A Practical Decision Guide

gentlyding — Thu, 17 Sep 2026 11:21:55 +0000

If you're building an audit log and want it tamper-evident, you'll hit the same fork I did: do I link records in a sequential hash chain, or build a Merkle tree? Both let a third party detect tampering, but they make different tradeoffs — and one of them will quietly fail your audit if you pick wrong.

This isn't a "which is better" post. It's a decision guide based on how the log is actually written and who has to verify it.

The two structures, in one paragraph each

Hash chain (sequential). Each record carries the hash of the previous record's contents. To verify, you walk from the oldest to the newest, recomputing each link. If any record's payload changed, its link breaks — and every link after it breaks too.

record[0].hash = H(payload[0])
record[n].hash = H(payload[n] || record[n-1].hash)

Merkle tree. Records are leaves; you hash pairs up to a single root. To verify one record, you don't replay the whole log — you present the root plus the small set of sibling hashes on the path from that leaf to the root (a "membership proof"). The root is the only thing you need to trust or anchor externally.

root = H( H(leaf0 || leaf1) || H(leaf2 || leaf3) )

How to decide

Ask three questions.

1. Are records appended strictly in order, by one writer?

Audit logs almost always are. Events happen in time; you append as they arrive. A sequential hash chain fits this shape like a glove — the "previous hash" pointer is just "the thing you wrote a moment ago," and there's no bookkeeping.

A Merkle tree can do ordered appends too, but you pay a cost: every append can change the root, and if you want a stable root to anchor (say, hourly), you're really building a series of Merkle trees — one per batch. That's a perfectly good design; it's just more machinery than a chain for the common case.

2. Does a verifier ever need to prove one record without replaying everything?

This is where Merkle wins. Suppose a regulator wants to confirm a single transaction from 2023 without downloading your entire 40 GB log. With a hash chain, they must replay from the beginning (or from the last anchored checkpoint) to reach that record. With a Merkle tree, you hand them the root plus a ~log2(n) proof and they're done.

If your verifiers are third parties who sample records rather than replay the whole trail, Merkle membership proofs are the difference between "here's a 2 KB proof" and "here's a 40 GB file."

3. Do you need to prove a record was absent?

A Merkle tree gives you a clean answer to "this record is not in the log" (a proof of non-membership, with the right tree design). A sequential hash chain cannot prove absence at all — it can only confirm what is there. If your compliance requirement includes "demonstrate we never logged X," that's a Merkle property, not a chain property.

The tradeoff table

	Hash chain	Merkle tree
Append cost	O(1) per record	O(1) per leaf, but root changes
Verify one record	Replay from anchor (O(n))	Membership proof (O(log n))
Prove absence	No	Yes (with design)
External anchor	Anchor the head hash	Anchor the root
Mental model	Trivial	Needs care (ordering, balancing)
Best when	Single ordered writer, full-trail verification	Third-party sampling, large logs, absence proofs

The hybrid I'd actually ship

For most audit-log systems, you don't pick one — you use both, at different layers.

The hash chain is the write path: every appended record links to the previous, giving you append-only, tamper-evident history with zero extra machinery.
Periodically (hourly, daily), take the current chain head and build a Merkle tree over the batch, or simply anchor the head hash to an external trusted timestamp (RFC 3161). The anchored root is what a third party trusts; the chain is what they replay to drill into any record.

This gives you the chain's simplicity on the hot path and the Merkle tree's efficient sampling when a verifier shows up with a specific question.

Where I was wrong

In an earlier post I waved this away with "a simple sequential chain beats a full Merkle tree." That's true for the write path of a single ordered writer. It's wrong if your verifiers sample records, need absence proofs, or can't replay a 40 GB trail. The honest answer: the chain is the default, the tree is the upgrade when verification scale demands it.

The part neither structure solves

A hash chain and a Merkle root both prove integrity — that bytes weren't changed. Neither proves when the bytes were written, and neither survives someone who controls the database rewriting both the record and its links. The fix for both is the same: anchor the head (or root) to an external trusted timestamp (RFC 3161) so "this existed by then" becomes independently provable, and keep the verification logic runnable by a party who isn't you.

Build the chain. Anchor it. Let someone else replay it. That's the whole game.

Event Sourcing Gives You an Event Store, Not an Audit Log

gentlyding — Tue, 15 Sep 2026 11:27:48 +0000

A lot of teams reach for Event Sourcing and then treat the event store as their audit log. It's an easy mistake: both are append-only-ish streams of "what happened," so they look like the same thing. They aren't, and the gap shows up exactly when an auditor asks you to prove it.

This post is about where the two diverge, why an event store fails the audit test on its own, and the minimum you have to add so an event-sourced system can actually produce evidence.

They solve different problems

Event Sourcing is an application pattern. Its job is to let you rebuild domain state by replaying events, and to drive read models (projections) from those events. The primary reader of an event store is your own code.

An audit log is an evidence pattern. Its job is to let a third party — an auditor, a regulator, a forensic investigator, sometimes a hostile one — prove what happened, when, and that it hasn't been rewritten since. The primary reader of an audit log is someone who does not trust you.

Those two readers pull the design in opposite directions.

The gap, side by side

	Event store (ES)	Audit log
Purpose	Rebuild state; drive projections	Prove "who did what, when" to a third party
Reader	Your own services	An auditor who doesn't trust you
Third-party verifiable?	Usually no	Must be yes
Tamper-evident?	Not by default	Required
Time source	App-generated wall-clock	External, anchored
Survival of deletion	Events get scavenged/compacted	Records must persist for the retention period

If you've shipped Event Sourcing and someone asks "can you prove these events weren't edited?", the honest answer from a vanilla event store is usually "no."

Three things a raw event store is missing

1. Third-party verifiable integrity. An event store typically just stores events. Nothing binds event n to event n-1 in a way an outsider can check. "Our system says the history is intact" is not the same as "anyone can recompute a hash chain and see it matches." You need the chain, and you need to expose it — the auditor has to be able to replay it themselves.

2. Immutability that survives operations. Event stores are not actually append-forever. EventStoreDB scavenges deleted streams; Kafka drops segments past the retention window; projections are regenerated from scratch whenever you change a handler. All of that is correct for Event Sourcing and fatal for an audit trail. If "compacting old history" is a routine ops task, you don't have an audit log — you have a cache of recent state.

3. An external time anchor. Event timestamps in an event store are written by your application, from its own clock. Whoever controls that server can backdate an event or rewrite one and the timestamp will happily agree. A real audit log anchors each batch of records to a trusted timestamp authority (RFC 3161) so "when this was written" stops being something you can quietly edit.

Bridging the gap without throwing ES away

You don't have to abandon Event Sourcing. You add a verifiable audit trail alongside it, fed by the same events:

When an event is appended, compute a content hash chain over the event — each link hashes its own canonical payload plus the previous link's hash.
Periodically anchor the chain head to an RFC 3161 timestamp authority, so the chain has an external "this existed by then" proof.
Expose a verification report — anyone can replay the chain, recompute hashes, and check the timestamps. Proof, not assertion.

The crucial part: the audit source is the immutable event stream + chain + anchors, not the projection. Read models are derived; they can lag, rebuild, or disagree, and none of that should be able to silently rewrite history.

A minimal append-side snippet (hash chain over a canonical event payload, not a database row):

import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;

class AuditAppender {
    private final MessageDigest sha256 = MessageDigest.getInstance("SHA-256");
    private byte[] headHash = new byte[0]; // genesis link is empty

    byte[] append(String canonicalEventJson) {
        byte[] content = canonicalEventJson.getBytes(StandardCharsets.UTF_8);
        sha256.reset();
        sha256.update(content);
        sha256.update(headHash);        // bind this link to the previous one
        byte[] link = sha256.digest();
        headHash = link;               // advance the chain head
        return link;
    }
}

canonicalEventJson is the point: it must be a stable serialization of the event — fixed key order, no insignificant whitespace, numbers in a fixed format. Hash the raw object's toString() and two semantically identical events will produce two different hashes, which defeats the whole chain.

Two traps to avoid

Don't treat a projection as evidence. A read model is regenerated whenever a handler changes. "The audit says the balance was X" cannot come from a thing you rebuild on a whim. The evidence is the chained event stream; the projection is just a view.
Canonicalize before you hash. JSON key ordering, trailing whitespace, and 1.0 vs 1.00 all change a hash. Define one canonical form for your event payload and hash that, or your verification step will flag false tampering on perfectly honest events.

Bottom line

Event Sourcing gives you a great way to rebuild state and drive projections. It does not, by itself, give you an audit log — because an audit log's whole job is to be provable to someone who doesn't trust you, and a vanilla event store can't do that. Add a content hash chain, anchor it to external time, and expose verification. Then your event store and your audit trail coexist, and the audit trail actually holds up.

Stop Trusting the App: Enforcing Append-Only at the Database Layer

gentlyding — Fri, 11 Sep 2026 11:52:50 +0000

Most "audit logs" are append-only by convention, not by enforcement. The application inserts a row and politely promises never to UPDATE or DELETE it. That promise holds right up until the moment someone does one of these:

A bug in the app calls repo.delete(id) on the wrong entity.
An operator connects with psql and runs a cleanup script against the production table.
A compromised dependency exfiltrates the credentials and rewrites history to cover its tracks.

If a third party is expected to trust your audit trail, "we don't update it" is not a guarantee — it's a hope. The constraint has to live below the application, in the database itself. Here are the patterns that actually work, and where each one still leaks.

Pattern 1 — An INSERT-only role

The cheapest real guarantee: the application connects with a database role that physically cannot modify what it already wrote.

-- a role the app uses at runtime
CREATE ROLE app_audit LOGIN PASSWORD '...';

-- the audit table is owned by a separate, higher-privileged role
CREATE TABLE audit_events (
  id        BIGSERIAL PRIMARY KEY,
  payload   JSONB NOT NULL,
  prev_hash BYTEA,
  cur_hash  BYTEA NOT NULL,
  written_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- grant ONLY insert (and select, if the app reads its own log)
GRANT INSERT, SELECT ON audit_events TO app_audit;
-- explicitly: no UPDATE, no DELETE, no TRUNCATE

Now even a code bug that calls delete resolves to a permission error at the driver, not a silent row removal. This is the single highest-leverage change and almost nobody does it.

The leak: a superuser or the table owner can still mutate rows. So this stops accidents and low-privilege compromises, not a DBA with full access — which is exactly why you still want a hash chain underneath (see the "why this isn't enough" note at the end).

Pattern 2 — Triggers as a backstop

Triggers catch mutations made through any connection, including the owner, unless the session disables them.

CREATE OR REPLACE FUNCTION reject_audit_mutate()
RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
  RAISE EXCEPTION 'audit_events is append-only; mutations are forbidden';
END;
$$;

CREATE TRIGGER audit_no_update
  BEFORE UPDATE OR DELETE ON audit_events
  FOR EACH STATEMENT
  EXECUTE FUNCTION reject_audit_mutate();

The leak: ALTER TABLE ... DISABLE TRIGGER is available to the table owner and superusers, and session_replication_role = replica skips triggers entirely in Postgres. Treat triggers as defense-in-depth, not the perimeter.

Pattern 3 — A foreign key that points backward

Make each row reference the previous row's hash, and put a FOREIGN KEY on it. Deleting any row except the newest now fails the constraint on the row that follows it.

ALTER TABLE audit_events
  ADD CONSTRAINT fk_prev
  FOREIGN KEY (prev_hash) REFERENCES audit_events (cur_hash);

The leak: the newest row has no follower, so deleting the tail still works. And healing the chain after a legitimate bulk operation is painful. Useful as one layer, not sufficient alone.

Pattern 4 — Tombstones instead of DELETE

The cleanest way to honor "right to erasure" laws without making the log mutable: you never delete. You append a redaction event.

INSERT INTO audit_events (payload, prev_hash, cur_hash)
VALUES (
  '{"type":"redaction","target_id": 42,"reason":"GDPR Art.17 request"}',
  (SELECT cur_hash FROM audit_events ORDER BY id DESC LIMIT 1),
  ... -- hash of (payload || prev_hash)
);

The original row stays intact and verifiable; the log now records that it was asked to forget. An auditor sees both the event and the request to erase it. This is the only pattern that satisfies "we must be able to delete" and "the log must stay tamper-evident" at the same time.

Pattern 5 — WORM at the storage layer

Push the table (or a time partition of it) onto write-once storage: a read-only tablespace, an object-lock bucket, or a partitioned scheme where last month's partition is ALTER TABLE ... SET (read_only = true) / moved to immutable media.

-- freeze a partition
ALTER TABLE audit_events_2026_08 SET (read_only = true);

The leak: it's coarse-grained (you freeze whole periods, not individual rows) and it's enforced by the platform, so it shares the same trust boundary as the platform admin. Good for the "we can prove we didn't touch last quarter" claim, weaker for fine-grained guarantees.

The combination that holds up

In practice you want layers, not a single trick:

App connects as an INSERT-only role (Pattern 1) — stops the common bug and the low-priv compromise.
A trigger (Pattern 2) catches the rest at the SQL level.
Tombstones (Pattern 4) replace every DELETE so erasure requests are logged, not destructive.
Period partitions go read-only (Pattern 5) for the long tail.
And underneath all of it, a hash chain (each row hashes its payload plus the previous row's hash) plus an external trusted timestamp, so that even a database superuser who can mutate a row cannot do it without breaking verification — and cannot convincingly fake when it happened.

That last point is the actual perimeter. Database constraints protect you from mistakes and from attackers who lack DB privileges. They do not protect you from someone who controls the database. For that threat, the chain + timestamp has to be verifiable by a party who doesn't trust your infrastructure at all — which is a separate post, but it's the reason none of the patterns above are sufficient on their own.

The takeaway: append-only is a property you enforce in three places — the role, the table, and the math. Skip any one and "append-only" is just a comment in your codebase that nobody obeys.

Trusted Timestamps (RFC 3161): How to Anchor a Hash Chain So No One Can Rewrite the Past

gentlyding — Thu, 10 Sep 2026 16:08:13 +0000

A hash chain proves a record was not modified after it was written. It does not prove when it was written, and it does nothing against someone who controls the server and can re-sign the whole chain. RFC 3161 trusted timestamps close that gap. This is a practical look at what they are, how to wire them in, and where the footguns are.

The gap a hash chain leaves open

If you hash-chain your records — hash[n] = H(payload[n] || hash[n-1]) — anyone can later walk the chain and prove no entry was altered. Tamper with payload[2] and every subsequent link breaks. That property is real and useful.

But it assumes one thing the chain itself cannot guarantee: trust in when the signing happened.

It can't prove when a record was created. The whole chain could have been generated yesterday and backfilled.
It can't stop a server operator with the signing key from regenerating the entire chain from scratch, picking whatever payload they like, and presenting it as the original.
It can't defend against "we lost the logs for March, so we rebuilt them" — a story auditors hear more often than they should.

A hash chain is integrity. It is not time and it is not non-repudiation. Those need an anchor outside your control.

What RFC 3161 actually gives you

RFC 3161 defines a protocol for a Time Stamp Authority (TSA): a third party you trust to vouch for "this exact hash existed at this exact time."

The flow is small:

You compute the hash of whatever you want to anchor (one record, or the head hash of your whole chain).
You send that hash to the TSA — not the data, just the digest. The TSA never sees your content.
The TSA returns a Time Stamp Token: a CMS signature (PKCS#7) that binds your hash to a timestamp, signed with the TSA's private key.
You store the token next to the record (or the chain head).

The token says, in effect: "A hash equal to this one was presented to me at 2026-09-10T14:03:11Z, signed, TSA: Acme-Time." Because it's signed by the TSA, you can't forge it, and because the hash is inside the signed blob, you can't swap the content after the fact.

Two consequences worth stating plainly:

The TSA attests to the hash, not the data. You still store the data yourself; the token only proves the data's digest existed at a time.
The token proves external, third-party time. Even if an attacker owns your server and your signing keys, they can't produce a valid token for a hash dated before they took over — unless they also compromised the TSA, which is the whole point of outsourcing that trust.

Two anchoring strategies

Per-record stamping. Stamp every record as it arrives. Maximum granularity, maximum TSA calls. Fine for low-volume, high-value events (a compliance audit log, a signing certificate log). You store token[n] alongside record[n].

Periodic chain-head stamping. At a fixed interval (every minute, every flush), take the current head hash of the chain and stamp that. One token anchors everything written since the previous token. This is usually the right trade-off: a single TSA call amortizes over thousands of records, and it still proves the entire chain up to that head existed at that time.

For most systems I'd reach for periodic head-stamping, with per-record stamping reserved for the few events where "exactly when" is itself the evidence (a key rotation, a privilege grant).

What a request looks like

Using Bouncy Castle (Java), a minimal stamp request:

import org.bouncycastle.tsp.TimeStampRequestGenerator;
import org.bouncycastle.tsp.TimeStampRequest;
import org.bouncycastle.tsp.TimeStampResponse;
import org.bouncycastle.tsp.TimeStampToken;
import org.bouncycastle.cms.CMSSignedData;

import java.io.OutputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import java.security.MessageDigest;
import java.util.Base64;

MessageDigest md = MessageDigest.getInstance("SHA-256");
byte[] digest = md.digest(payload); // the record's hash, or your chain head

TimeStampRequestGenerator gen = new TimeStampRequestGenerator();
gen.setCertReq(true); // ask the TSA to include its cert in the response
TimeStampRequest req = gen.generate(
    org.bouncycastle.tsp.TSPAlgorithms.SHA256, digest);

byte[] der = req.getEncoded(); // DER-encoded ASN.1 request

URL tsu = new URL("https://tsa.example.com/tsa");
HttpURLConnection c = (HttpURLConnection) tsu.openConnection();
c.setRequestMethod("POST");
c.setRequestProperty("Content-Type", "application/timestamp-query");
c.setDoOutput(true);
try (OutputStream os = c.getOutputStream()) { os.write(der); }

byte[] respDer = c.getInputStream().readAllBytes();
TimeStampResponse tsr = new TimeStampResponse(respDer);
tsr.validate(req); // throws if the response doesn't match your request

TimeStampToken token = tsr.getTimeStampToken();
byte[] tokenDer = token.getEncoded();           // store this
String tokenB64 = Base64.getEncoder().encodeToString(tokenDer);

What you persist is tokenB64 (or the raw DER). The TSA URL, the digest algorithm, and the request nonce are configuration, not secrets.

Verification

Verification is the part auditors actually run, so it must be boring and offline-capable:

// tokenDer = the stored DER; digest = the hash you recompute from the data
import java.security.MessageDigest;
import java.security.cert.X509Certificate;
import java.util.Collections;
import java.util.Date;
import org.bouncycastle.cert.X509CertificateHolder;
import org.bouncycastle.cert.jcajce.JcaCertStore;
import org.bouncycastle.cms.CMSSignedData;
import org.bouncycastle.tsp.TimeStampToken;
import org.bouncycastle.util.Store;

// Validate against YOUR pinned TSA root, not the certs bundled in the
// token. Trusting the embedded certs would let a rogue TSA vouch for itself.
X509Certificate tsaRoot = loadTrustedCert("tsa-root.pem"); // pinned, from config
Store<X509CertificateHolder> trust =
    new JcaCertStore(Collections.singletonList(tsaRoot));

TimeStampToken token = new TimeStampToken(new CMSSignedData(tokenDer));

// 1) the token's embedded hash must equal the data's current hash
byte[] signedDigest = token.getTimeStampInfo().getMessageImprintDigest();
if (!MessageDigest.isEqual(signedDigest, digest)) {
    throw new IllegalStateException("hash mismatch - data was altered");
}

// 2) the TSA's signature must validate against your trusted store
token.validate(trust); // throws on bad signature or untrusted issuer

// 3) the time must be within the window you expect
Date t = token.getTimeStampInfo().getGenTime();
// compare t against the record's claimed time / policy window

Three checks, in order: the imprint matches your data, the TSA signature is valid against a cert you trust, and the time falls where policy says it should. If all three pass, you have third-party proof the data existed at t.

The footguns

This is where most implementations quietly break.

TSA availability is now your dependency. If the TSA is down, you can't anchor. Run a periodic head-stamp on a schedule with retries and backoff; don't block your write path on the TSA. Stamp asynchronously, store the pending state, reconcile when the TSA comes back.

Network egress may be restricted. Many compliance environments block outbound traffic by default. The TSA call is outbound HTTPS to a specific host — it needs an explicit allowlist entry, or your anchoring silently stops working.

Trust the cert, not the URL. Validate the token against a TSA certificate you pinned (or a known root), not against the domain you happened to call, and never against the certs returned inside the token. A MITM on the TSA endpoint who returns a self-signed token should fail step 2.

Token lifetime outlives the TSA's certificate. TSA signing certs expire (typically 1–10 years). A token signed in 2026 with a cert valid to 2030 is fine today, but in 2035 the cert is expired and naive signature checks may reject it. For long-term evidence you need either:

a long-lived, qualified TSA whose root you trust indefinitely, or
LTANS / evidence-record renewal: periodically re-anchor an older token inside a newer one, building a verifiable chain of timestamps so the evidence stays provable after any single cert expires. RFC 4998 (ERS) covers this formally.

Skip this and your "proof it existed in 2026" becomes unverifiable in 2035. For audit evidence with a multi-year retention requirement, this is not optional.

Don't stamp the wrong thing. Stamping payload before you've computed the chain hash, or stamping a mutable field, anchors the wrong bytes. Stamp the immutable digest — the chain head, or the record's canonical hash — never a re-serialized object that might differ byte-for-byte on reload.

Free vs paid TSAs. Public free TSAs exist (some CAs offer one) but have rate limits and no SLA; for production evidence use a commercial or qualified TSA. The cost is per-stamp and usually trivial, but it is a real external bill.

Where this leaves you

After anchoring, tampering requires compromising both your server and the TSA — and if you use a qualified TSA, that's a regulated entity with its own audit trail. The chain gives you integrity; the timestamp gives you when and non-repudiation by an outside party. Together they answer the three questions an auditor actually asks: was it changed, when was it written, and can you prove that without trusting the person who wrote it.

The hash chain is the easy part. The timestamp is what makes it hold up in front of someone who doesn't trust you.

How We Made Audit Logs Immutable at the Storage Layer

gentlyding — Mon, 07 Sep 2026 11:22:14 +0000

Everyone talks about the hash chain. It's the sexy part — a tamper-evident linked list of records an auditor can verify independently. But a hash chain only proves the bytes weren't changed. It does nothing if the storage layer underneath lets you rewrite history more quietly: shift a timestamp, change a value and re-anchor the chain, or have the whole table silently drift when a server's timezone changes.

Immutability is won or lost at the storage layer. Here's the concrete design we landed on for a self-hosted audit log system (PostgreSQL + Spring Boot + JPA).

1. Store absolute instants, never wall-clock strings

The single most important rule: a timestamp in an audit record must be an absolute moment, not a wall-clock label.

In Java that means java.time.Instant — a point on the timeline with no zone attached. In PostgreSQL it means TIMESTAMP WITH TIME ZONE (aka timestamptz), which stores the moment as UTC under the hood.

@Entity
@Table(name = "audit_log")
public class NormalizedLog {
    @Column(nullable = false)
    private Instant eventTime;          // absolute moment, zone-free

    // Canonical ISO string kept *purely* for hashing (see §3)
    @Column(name = "event_time_iso", length = 64)
    private String eventTimeIso;
    // ...prevHash / curHash, actor, action, etc.
}

The trap we deliberately avoid: storing LocalDateTime. A LocalDateTime like 2024-03-15 09:30:00 carries no zone. The moment it hits the database, its true meaning depends on whatever timezone the connection or JVM happens to be in. Move the server, change a config, restore from a backup taken elsewhere — and every "when" in your audit trail is now ambiguous or simply wrong. For a compliance log, that's fatal.

2. UTC at rest, local only at read time

We store everything as UTC. The raw database shows UTC values, and that is expected, not a bug. The display/analysis timezone is a runtime setting (kept in a config table, editable from the UI, applied live) — it is never nailed to the JVM.

That last point matters. We explicitly do not do any of these:

-Duser.timezone=... on the JVM
TimeZone.setDefault(...) in code
hibernate.jdbc.time_zone (a no-op for Instant anyway)
a Hikari connection-init-sql that SETs a session timezone

Why? Because those don't just display in a zone — they can mutate what gets written. The storage baseline must stay UTC, unambiguously.

A real bug this prevented: "today's events" and daily-trend buckets. If you bucket by the UTC date, an event at 07:30 Asia/Shanghai (which is 23:30 UTC the previous day) lands on the wrong day. So the day-bucketing query does the conversion in the database, per the user's configured zone:

SELECT to_char(event_time AT TIME ZONE :tz, 'YYYY-MM-DD') AS d, COUNT(*)
FROM audit_log
WHERE event_time >= :from
GROUP BY 1 ORDER BY 1;

And "since when" counts compare the timestamptz column directly (hitting the ix_al_event_time index) instead of string-slicing the ISO column — which would reintroduce the UTC/local mismatch.

![Time storage: UTC at rest, local at read]

3. Keep a canonical string — but only for the hash

One subtlety: Instant has nanosecond precision, and a DB round-trip can truncate or reformat it. If you hash the Instant object directly, the hash can change between writes and reads even when nothing was tampered with. So each record also stores a canonical ISO-8601 string (eventTimeIso) that the hash chain is computed over. The string is stable; the chain stays verifiable.

// TimeUtil: lenient ISO-8601 parse → Instant (UTC)
public static Instant parse(String s) {
    try { return OffsetDateTime.parse(s).toInstant(); } catch (Exception ignored) {}
    try { return Instant.parse(s); } catch (Exception ignored) {}
    try { return LocalDateTime.parse(s).toInstant(ZoneOffset.UTC); } catch (Exception ignored) {}
    return Instant.now();
}

Note the last branch: an input with no offset is treated as UTC. That's an explicit ingest convention — never infer a source's local zone, or you'll bake ambiguity into the chain.

4. Erasure vs. immutability: keep the hash, redact the PII

"Immutable" doesn't mean "keep every byte forever." Under GDPR's right-to-erasure you must be able to purge a person's PII. But if you delete or rewrite the record, the hash chain breaks and you lose tamper-evidence for everyone.

Our compromise: redact the PII fields in place (actor, IP, raw message, geo tags) but leave curHash untouched, and flip an erased flag. The chain (prevHash → curHash) stays mathematically intact and still verifies; verification simply skips erased records.

@Modifying
@Query("update NormalizedLog l set l.erased = true, l.erasedAt = :now, l.erasedBy = :by, "
     + "l.sourceIp = :mask, l.actor = :mask, l.rawMessage = :mask, "
     + "l.beforeValue = :mask, l.afterValue = :mask, "
     + "l.geoCountry = null, l.geoRegion = null, l.geoCity = null, l.geoIsp = null, l.geoScope = null "
     + "where l.erased = false and l.actor = :actor")
int eraseByActor(@Param("actor") String actor, @Param("now") Instant now,
                 @Param("by") String by, @Param("mask") String mask);

So erasure is itself an audited, non-destructive event — which is exactly what a regulator wants to see.

5. The hash chain is the glue

prevHash and curHash make the whole table an append-only, verifiable chain (the subject of my previous post on building a tamper-evident audit log). Storage immutability is what makes that chain trustworthy: fixed UTC instants, stable canonical strings, and erasure that preserves hashes instead of rewriting them.

Takeaways

Audit timestamps = Instant + timestamptz. Never LocalDateTime.
Store UTC. Convert to the user's zone only when reading / displaying / bucketing.
Don't nail the JVM's timezone — it mutates writes, not just reads.
Keep a canonical string for hashing; hash the string, not the Instant.
Erasure and immutability aren't opposites: redact PII, keep the hash.

If you're building a self-hosted audit log and want the verifiable-chain side of this, I wrote about the hash-chain design here: https://dev.to/gentlyding/how-we-built-a-tamper-evident-audit-log-for-soc-2-and-iso-27001-evidence-jl4

How We Built a Tamper-Evident Audit Log for SOC 2 and ISO 27001 Evidence

gentlyding — Sat, 05 Sep 2026 01:21:43 +0000

Most teams treat "audit logging" as an afterthought: pipe everything into a SIEM or a big Elastic cluster, then hope the auditor is satisfied. In practice, that's where the pain starts.

I'm Qin Kang, and I built Log Audit Platform after watching a client burn roughly six engineering-months duct-taping Splunk + spreadsheets together for an ISO 27001 audit. The result was expensive, fragile, and the auditor still asked the one question that sinks most log pipelines:

"How do I know these logs weren't edited after the fact?"

This post walks through how we designed an audit log that an auditor can actually trust — without sending your data to a third party.

1. Why auditors don't trust raw logs

A raw log file is just text. Even if it's shipped to a "secure" bucket, nothing cryptographically ties one line to the next. An attacker (or a well-meaning operator) who gains write access can quietly rewrite history:

Change a DELETE into a READ.
Backdate an event.
Remove the record of a privilege escalation entirely.

When an auditor asks "can you prove this wasn't altered?", a raw log answers with trust me. That's not good enough for SOC 2 (CC7.2 / CC8.1) or ISO 27001 (A.8.15 / A.8.16).

The fix isn't "more storage". It's integrity by construction.

2. What SOC 2 and ISO 27001 actually want from logging

Stripped of jargon, the control families want three things:

Completeness — you captured the events that matter (auth, admin actions, config changes, data access).
Integrity — a record, once written, can't be silently changed.
Availability for review — an auditor (or your own security team) can independently verify both of the above.

Notice the word independently. SOC 2 and ISO 27001 auditors don't just take your word for it; they want evidence they can re-run. That's the design goal we optimized for.

3. Hash-chain design: why a sequential hash works for audit logs

Instead of storing events as isolated rows, every record carries the hash of the previous record's hash chain. Conceptually:

record[0].hash = H(payload[0])
record[n].hash = H(payload[n] || record[n-1].hash)

If an attacker tampers one record, its hash changes and every subsequent link fails verification — the auditor sees exactly where the chain breaks.

Verification walks the chain from the oldest record to the newest. If any payload was altered — even a single character — the recomputed hash at that link no longer matches the stored hash, and every subsequent link breaks too. The verification report marks exactly where the chain was broken.

Why a simple sequential chain rather than a full Merkle tree? For an audit log, records are append-only and verified in order, so a linear chain is simpler to verify, easier to explain to an auditor, and has no reconstruction complexity. (We're looking at optional RFC 3161 timestamp anchoring as a future external-WORM option, but the in-chain integrity is the core.)

The practical takeaway: editing one record is mathematically impossible to hide.

4. GDPR right-to-erasure without breaking the chain

GDPR's right to erasure (Art. 17) collides with an immutable log: you can't just DELETE FROM audit_log a user's rows, because that breaks the chain and the audit trail.

Our approach is cryptographic erasure / anonymization, not physical deletion:

Personal data is stored in a separate, keyed store, referenced by token from the audit record.
On a valid erasure request, we destroy the key material for that subject. The audit record remains (required for the integrity trail), but the linked identity becomes unrecoverable ciphertext.
The hash chain stays intact, because we erase the key, not the log.

This satisfies both sides: the regulator gets erasure; the auditor keeps a verifiable timeline.

5. Evidence pack automation: turning logs into auditor-ready ZIPs

The second thing auditors hate is hunting. They don't want your raw database; they want the control mapped to the evidence.

So we ship evidence packs: pre-built, exportable bundles that map collected events to specific control IDs (e.g. SOC 2 AC-2, ISO 27001 A.8.16). One click produces a ZIP containing:

The relevant filtered records.
A hash-chain verification report (proving the bundle itself is intact).
A control-to-evidence mapping sheet.

The auditor runs the verification themselves. No spreadsheet gymnastics, no "trust me".

6. Self-hosted deployment: one JAR

Compliance data shouldn't leave your perimeter. Log Audit Platform runs self-hosted:

Backend: Spring Boot 3 + Java 17+
Frontend: Vue 3 admin UI, embedded in the build
Storage: PostgreSQL
Shipping: a single runnable JAR + Docker Compose

No vendor backend, no telemetry pipe, no API calls home after activation. It runs in your VPC, on-prem, or air-gapped. For teams that can't use a cloud SIEM for regulatory reasons, that's the whole point.

7. Pricing: one-time license, not a per-seat SaaS tax

SIEM pricing scales with ingest volume and seats — exactly the metrics that go up when you start taking compliance seriously. We priced it as a one-time license:

Single — $199 (one instance)
Business — $699 (up to 5)
Enterprise — $2,499 (full source + white-label + OEM)

Optional annual maintenance for updates. No per-seat tax, no surprise overage bills right before your audit.

Try it

If you're preparing for a SOC 2, ISO 27001, or GDPR audit and want an audit log that auditors can verify independently, take a look:

https://logaudit.toolsder.com

I'd genuinely like feedback from security engineers and compliance folks — what logging controls have been the biggest pain in your audits? What would make you trust a self-hosted log like this one?

Qin Kang — independent developer, building self-hosted compliance tooling.