DEV Community: László Hadházy

The descriptor survived, const did not — full-stack Rust

László Hadházy — Mon, 07 Sep 2026 12:45:35 +0000

One skeleton, many screens argued that admin screens should be declared as typed data rather than coded, and it ended by claiming the idea was independent of the stack:

draw the boundary as a one-way dependency — domains depend inward on a framework that knows nothing about them — and validate it with a zero-diff refactor of a screen you already trust.

That was React and TypeScript. This is the same claim re-run in Rust, where a descriptor can be a compile-time constant and a template is a macro. Because the first result is already published, the second stack is a replication with a control rather than a fresh opinion — which is rare enough to be worth doing properly.

Companion to Topcoat and the shrinking cost of full-stack Rust. That post was written from the announcement and promised a follow-up reporting where the rough edges actually show. This is it, from the pilot that followed: a small admin panel built on Topcoat 0.6.2 and Toasty 0.10.0, and the four questions that post committed to answering. The pilot is open source — a clean clone runs both screens and the test that decides the argument.

That phrase, a compile-time constant, is where the title comes from, so it is worth saying now what it buys and why I wanted it. A TypeScript descriptor is an array of objects assembled when the module loads. A Rust one can be more than that: &'static, Copy, allocated never, fully checked before the program starts. Going in, that looked to me like the same idea in a stricter form — if declaring a screen as data is good, then declaring it as data the compiler can see through and verify must be better still. I treated that property as the thing worth protecting, and the pilot was partly a test of whether it could be.

The stack is deliberately a young one. Topcoat is six weeks old: Tokio's team announced it on 22 July 2026, the pilot pins 0.6.2, and the project still expects breaking changes. It is not the only full-stack Rust framework — Leptos and Dioxus have been at this for years — but it is the only one of them that answers the question without WebAssembly, server-rendering by default and keeping the reactive layer out of a Wasm bundle. That is the part nobody has years of experience with, and the part worth testing.

It came out differently, and not in the direction I expected.

The only data in this post is metadata

There is no warehouse here and no pipeline. There is a table of invoices with a dozen invented rows, and those rows are not the data that matters. The data that matters is the descriptor — the small typed declaration saying that an invoice screen has these columns, read from these fields, rendered these ways. That is metadata in the plainest sense: it describes something else, it is versioned separately from the thing it describes, and it is reviewed on its own terms.

Which is the shape of the rest of Cordata's work. A pipeline is a descriptor, not a program makes the identical argument one domain over — put source, steps, target, contract and policy into one versioned declaration, and onboarding a new source becomes a configuration change rather than a release. A lineage event, a data contract, a retention rule: all the same move, taking something smeared across code and turning it into a document you can diff, review, and refuse to merge.

Metadata is worth the effort for a reason that is not documentation. It is constraint. A descriptor is worth having for what it stops a domain from doing — the pipeline that cannot write without declaring a contract, the screen that cannot render a column it never declared. In this pilot that constraint is a type, which makes it something other than advisory: the descriptor is the entire vocabulary a screen is allowed, and a domain that needs something the vocabulary does not have cannot quietly work around it. It gets a compile error, and somebody has to decide what the vocabulary should say instead. Everything reported below came from pushing on that boundary until it answered.

Every metadata layer meets the same test eventually, whatever it describes. The declaration covers the ordinary cases comfortably. Then a case turns up that it has no vocabulary for, and there are two ways out.

The first is an escape hatch: one field that accepts arbitrary code or text — a raw_sql string on a pipeline step, a render callback on a column, a free-text notes blob on a contract. It is quick, and it solves not only today's case but every future one, which is exactly the problem. Nothing outside the domain can read what ends up in that field, so from the day it exists the declaration stops describing the system and starts describing only the parts nobody needed an exception for. The layer still looks like a constraint in the type signature, but it has stopped acting as one.

The second way out is slower: name the new case and add it to the vocabulary. A declared Link cell type rather than a render callback. A declared join step rather than a SQL string. The cost is real — the framework has to change every time a domain needs something genuinely new, and that change is reviewed by whoever owns the framework rather than whoever wanted the feature. What it buys is that the declaration still says what the system does, and still refuses what the system should not do.

Which of the two a layer reaches for is the only question about it that really matters, and it is normally answered years in, on a system far too large to watch.

Two admin screens are small enough to watch. So this is a post about a Rust web framework on the surface and a post about metadata underneath — and if you never write a line of Rust, the second half is the one that still applies to you, because the same question is waiting in whatever declaration layer you do own.

The experiment

The pilot has two screens, picked to sit on either side of the boundary. Each is its own module — users.rs and invoices.rs — and those two modules are what this post means by the domains: the code that knows what a user is, or what an invoice is. Facing them is the framework: descriptor.rs, which defines what a description may say, and table.rs, which renders one. The framework knows about neither users nor invoices, and the entire experiment is about what can travel from a domain to it.

The first screen, users, lists name, email, seats, joined and status — all plain values, every one of them covered by the cell types the framework already ships. The second, invoices, was written by hand first and made deliberately harder, carrying three things the descriptor could not express:

ambient context — a money amount formatted for a locale that arrives from the request, not from the row: 4.850,00 € in German, $4,850.00 in English
a derived value — a column computed from two fields rather than read from one
structure — a cell that renders as a link rather than as text

Those three names are used for the rest of the post, and the third is the one worth pinning down now, because the result turns on it. A descriptor of this shape answers one question per column: given a row, what string does this cell show? A derived value is still an answer to that question — compute whatever you like, hand back a string. A link is not an answer to it at all, because a link is a different shape of output: an <a> element wrapping the text, carrying a second value, the destination, which has nowhere to live in a description that only knows how to say "string". So structure here means not what a cell contains, but what kind of thing the cell is.

Screen one exists to be trusted so that screen two can be refactored against it — and that word, refactored, is carrying the whole method, so it is worth being exact about what it means here.

invoices was written the ordinary way first: its own markup, its own loop over the rows, its own formatting, answering to nothing but itself. Refactoring it means deleting all of that and rebuilding the screen as a descriptor handed to the same shared table component users already uses — the same generic component, with no invoice-shaped special case added inside it. The screen's own code changes completely, while what the browser receives must not change at all.

The order matters more than it looks. If both screens are written through the abstraction from the start, the abstraction gets to decide what correct looks like, and the test can only agree with itself. Writing invoices by hand first produces a control: a screen built by somebody solving the real problem, free to do whatever that problem needed — including the three things above, which the descriptor had no vocabulary for. Refactoring it afterwards asks a question that can genuinely come back no. Can the abstraction reproduce, byte for byte, a screen that was never designed to fit it?

The diagram, in words: both domains hand descriptions inward to one table component that does not name any domain type, and nothing points back out. Of the three things tested against that boundary, values crossed unchanged, structure required a change on the framework side, and ambient context could not cross at all while the accessor remained a plain function pointer.

Because Topcoat renders on the server, the zero-diff claim stops being a visual judgement and becomes a text comparison. The test script captures the HTML for every route before a refactor and diffs it after, which turns the whole question into a pass or a fail: either diff returns nothing, or the abstraction has changed what the user sees and is therefore wrong.

The test script gave one false positive before I trusted it. Perturbing a column header produced the failure it should. Reverting that change with mv produced the failure again — because mv restores the original modification time, leaving the source older than the compiled binary, so cargo skipped the rebuild and the server kept serving the previous build. Nothing anywhere reported a skipped step. It now forces the sources current on every run, which costs 1.4 seconds and removes the category.

That is the same shape as the five traps in the trace_id post: a check that is confidently wrong because something upstream silently did not happen. Prove your test discriminates before you believe it.

Result one: the boundary held for values and broke at structure

First refactor, changing nothing in the framework:

ZERO DIFF  users
CHANGED    invoices   — one column

Five of the six columns reproduced byte for byte, including both I expected to break it. The computed column survived outright. The locale-formatted money survived too, though the next section explains why that was an accident of the fixture rather than a result.

Exactly one column failed:

- <td class="cell cell-text">
  <a href="/invoices/2026-0041">2026-0041</a></td>
+ <td class="cell cell-text">
  &lt;a href="/invoices/2026-0041"&gt;2026-0041&lt;/a&gt;</td>

(wrapped to fit this surface — each cell is one line in the real output)

The accessor's type was fn(&T) -> String, and view! escapes an interpolated string. That is correct behaviour — it is the cross-site scripting defence — not a bug to route around.

So: an accessor returning a string can express any cell that is a value, and no cell that is structure. It can compute anything at all from the row, and it can nest nothing at all inside the cell it hands back.

Every column in a descriptor carries two things: an accessor saying what to show, and a cell kind saying how to render it. The cell kind is an enum, and that enum is the framework's entire vocabulary of cell shapes — a domain may choose from it and cannot extend it, which is the constraint from earlier in this post expressed as a type:

pub enum CellKind<T: 'static> {
    Text,
    Number,
    Date,
    Badge,
    Link { href: Accessor<T> }, // added by this result
}

So fixing the link column meant adding that Link variant, plus one if let inside the table component's template — the single place that decides whether a cell's text goes into the <td> bare or wrapped in an anchor:

if let CellKind::Link { href } = &col.kind {
    <a href=(href(row))>((col.get)(row))</a>
} else {
    ((col.get)(row))
}

Both changes are framework-side, and the variant carries something worth noticing: a second accessor for the destination, which is exactly the value that had nowhere to live when a column could only produce one string. The domains did not change, and the table component still does not name any domain type — it learned a shape, not a subject.

That inverts the React result, and this is the first thing worth stealing. There, a column definition held a render function, so any domain could return arbitrary markup, which is more flexible and no boundary at all. The Rust version has the stricter boundary, and the price is that a genuinely new presentation costs a deliberate, single, framework-wide decision.

Result two: the locale broke the thing I thought was the point

The money column survived the first refactor only because the currency formatter had the locale hardcoded, which let the closure capture nothing and coerce to a plain function pointer. The moment the locale became a real one, the accessor needed to capture it, and a function pointer cannot.

Where the locale is stored turns out not to matter, which is worth saying because the obvious objection is that locale is data like anything else. On this site it is a route segment, /en/… and /de/…. It could as easily be a cookie, an Accept-Language header, or a column in a user-preferences table read at login — and frequently is. All of those reach the render the same way: resolved once per request, then threaded down to whatever formats a number. None of them puts the locale on the invoice row that the accessor is handed, and that is what makes it ambient rather than data.

The distinction is worth keeping sharp, because part of this genuinely is row data. An invoice may well carry its own currency code, and a plain function pointer reads that perfectly happily — a React codebase I worked on last year threads the locale as a parameter and takes the currency from the record, which is exactly the right split. What cannot live on the row is which human is looking at it.

The smallest thing that compiled was a global: set the locale per request, read it per cell. It kept the descriptor a compile-time constant, and it is worse than the problem it solves. Two descriptors differing only by locale cannot exist at once, and what a cell renders depends on when it is read. I committed it with a note saying exactly that, so the cost sat in the code rather than in prose.

All three routes then reported zero diff — the two screens, with invoices counted twice because a German route had been added alongside the English one for exactly this test. The output was identical and the program was worse, which is precisely the limit of what this test can tell you: it compares what the reader sees, never what it cost to produce. A green run here was not evidence that the global was acceptable. It was evidence that the test script cannot see a global at all.

The fix is to let the accessor capture after all, which means boxing it:

pub type Accessor<T> =
    Box<dyn Fn(&T) -> String + Send + Sync>;

That is the moment the descriptor stops being a compile-time constant. It is no longer Copy, no longer static, and it allocates once per column per request. On day one of the pilot I wrote that down as the thing most likely to break the thesis.

What that actually cost, measured

lines of Rust	function pointers + global	boxed closures	change
`descriptor.rs`	66	88	+22
`table.rs`	58	58	—
`users.rs`	72	54	−18
`invoices.rs`	119	88	−31
total	315	288	−27

wc -l, before and after that one refactor. Later work grew these files, so the numbers come from the change itself rather than from the repository as it stands today.

The expensive version is smaller. The framework grew twenty-two lines — a documented type alias and a constructor function — and the domains shrank forty-nine, because keeping the descriptor constant had required a macro generating two static column arrays per locale, plus the global to smuggle the locale in. All of that went away.

What was lost is real: Copy, const, &'static, zero allocation. What was gained is also real: the locale is captured rather than smuggled, two locales coexist, and a cell's value no longer depends on when it is read.

And the call site still reads as data:

col(
    if de { "Betrag" } else { "Amount" },
    CellKind::Number,
    move |i: &Invoice| money(l, i.amount_cents),
),

That is not meaningfully further from a struct literal than the original was, which is the finding in one line: the declarative shape survived the change, and const did not.

The four questions the first post promised

Developer experience. Mostly unremarkable, which is the compliment. Two things cost real time:

A page function cannot share a name with a module in scope, and the five-error cascade that follows describes the macro expansion rather than the cause.
A generic component needs Send + Sync bounds the React version never had — though the compiler suggested exactly the right fix, which is more than can be said for some.

Error messages. The spans are consistently good and the vocabulary is not:

Every probe pointed at the source line, never into expanded code.
Malformed markup gets a purpose-written diagnostic — closing tag div does not match opening tag p, with the exact column and no trait bounds anywhere in it.
The vocabulary is where it thins out. Feeding $() something that cannot cross to JavaScript produces an unsatisfied bound on a trait called Surrogated, a word that appears nowhere in the documentation, followed by a list of implementors ending "and 25 others".

So the boundary is policed correctly and explained poorly, which means a junior would land in the right file every time and need telling once what those names mean.

The rebuild loop. It holds up, with a warm incremental rebuild at 0.47 seconds. To find out whether macro expansion comes to dominate as an app grows, I generated components carrying template blocks and measured the slope:

`view!` blocks	warm rebuild
6	0.47s
31	0.52s
81	0.57s
156	0.71s

Roughly linear at about 1.6 milliseconds per block, so five hundred components would still land near 1.3 seconds, which means expansion is not the thing you will notice. The browser-reload half of the loop remains untested.

Where the trade-offs land. Swapping the fixtures for a real Toasty model over SQLite changed the field types from string slices to owned strings, widened an integer, and made the row query asynchronous — and not one accessor changed. The query completes before rendering starts, so the descriptor never sees the asynchrony. That held because every column is a scalar already on the row; a lazily-loaded relation would need the accessor to reach the database, and its signature is synchronous. That case is untested and I am not going to pretend otherwise.

The take-home

I went in believing the descriptor should be data, and that Rust's function pointers were a stricter, purer version of the same idea. They are stricter, but past the first cell that needs context from outside the row they are not purer, and they are not cheaper either — keeping them cost a macro, two duplicated arrays and a global, which is more machinery than the boxed closure it was avoiding.

So the finding is that const-ness was never the property that mattered. What made the original abstraction work is the one-way dependency: domains hand the framework descriptions, the framework knows nothing about domains. That held unchanged across all three attempts — through a new cell shape, through a new source of ambient context, and through replacing the entire data layer. What broke was narrower than it felt at the time: the accessor's type, a plain fn pointer, and with it the descriptor's standing as a compile-time constant. Both of those are decisions about how a description is represented, not about what it is allowed to say. The React post took the descriptor's being plain data for the idea, and I took its being a constant for a stricter version of the same idea, when in both cases the idea was the dependency direction underneath.

Which generalises past admin screens, and back to the metadata argument this started from. A declaration is worth having because of what it refuses to let a domain do, not because of where it is stored or when it is evaluated. Ask of any descriptor — pipeline, screen, policy — which of its properties are the boundary and which are just how it happens to be built today. They are easy to confuse, and only one of them is worth defending.

The pilot is at github.com/cordata-tech/topcoat-descriptor-pilot. A clean clone builds and serves both screens, and scripts/zero-diff.sh check runs the test that decides the argument. NOTES.md carries the findings as they were recorded, including the ones that went the other way and one probe I designed wrong.

Built with an AI pair, which is worth stating on a post about developer experience: Topcoat's first release was April 2026, so there is little training data and the README is most of the corpus. The probing of error messages was mine; the scaffold was not. If you have piloted Topcoat and found the rough edges somewhere else — or think a descriptor of closures gives up more than I am admitting — I would like to hear it. The door is open at cordata.tech/contact. Related reading: One skeleton, many screens is the React original this replicates, and A pipeline is a descriptor, not a program makes the same argument about metadata one domain over.

The trace_id was the easy part — Rust, OTel, Loki

László Hadházy — Sun, 30 Aug 2026 10:30:55 +0000

An earlier post listed what the difficult middle of a data platform actually consists of, and one item on that list was "observability that points at the right wire when something breaks". This post is what that one item cost.

The system is a multi-chain ingestion pipeline: Rust services subscribing to Ethereum, Solana and Polkadot, publishing to Kafka, running on Kubernetes with Prometheus, Loki and Tempo behind it. An ordinary enough shape. What makes the observability requirement unusual is not the architecture but what the data does to it.

Companion to One port, three client crates. That post covered how one Rust service reads three blockchains through a single trait, and stopped at the boundary where the design was done. This one picks up after it: the same service, an issue estimated at an afternoon, and the four conditions that had to hold before the result meant anything — plus a fifth that arrived with the follow-up. Reading the first is not required; the traps below are about OpenTelemetry, tracing and Loki rather than about chains.

Blockchain sources are ephemeral in a way most sources are not. A database you are replicating from still has the row tomorrow. A chain's recent blocks are served by an RPC provider for a while and then they are not, and provider retention is a commercial decision rather than a promise to you. If the pipeline silently drops an event, the evidence it existed may be gone before anyone notices — and nothing upstream will complain, because nothing upstream knows you were listening.

So "did we see everything, and did we handle it correctly" is not a comfort question on this platform — it accounts for most of what the product is, which is why an issue that reads like a formatting nicety was not a cosmetic one.

The issue was one sentence: log lines should carry the active OpenTelemetry trace_id so a line in Loki links to its trace in Tempo. It was estimated at a field, a formatter and an afternoon.

The formatter did take about an afternoon. Making the field mean anything took the rest of the work, and three of the four conditions that had to hold failed silently — none of them raised an error, panicked or turned a test red, and each produced output that looked entirely correct.

That silence is the part worth generalising, because observability code has a property that makes it unusually prone to it: it is the instrument you use to check whether everything else works, so nothing else is watching it. A broken pipeline raises an error someone sees. A broken metric reads zero, and zero is a plausible number.

Four things, not one

Before a trace_id in a log line is useful, all of these have to hold:

A span exists around the work.
That span closes, so it gets exported.
The log event is emitted inside it, and inside the right one.
The formatter can see the span's OTel context and writes it out.

The issue named only the fourth, and the service had the first and nothing else.

The diagram, in words: two paths leave the opened span. The first closes it and exports it, producing the trace. The second carries the log event through the formatter into the JSON line, then through the log pipeline to the clickable link. Both have to finish before that link opens anything; the issue described only the last step of the second.

The formatter is the only step of that chain visible from a log line, which is why an afternoon was a fair estimate for it and a bad one for the feature.

Trap 1 — a span that never closes is never exported

The service's ingestion loop carried #[instrument] on its top-level run function, so a span did exist — exactly one, opened at startup and closed at shutdown.

tracing-opentelemetry ends and exports an OTel span in the subscriber's on_close callback. A span that never closes is never sent. The process could run for a week with tracing fully configured, the OTLP exporter connected and healthy, and Tempo would receive nothing at all.

Nothing in the system reports this, because from every component's point of view everything is working: the exporter is up, the sampler says AlwaysOn, and the configuration is correct. There is simply never anything for it to send.

The fix is a span whose lifetime matches a unit of work:

Some(Ok(event)) => {
    let span = info_span!(
        "ingest_event", event_id = %event.event_id()
    );
    span.in_scope(|| {
        // ... one event's worth of work
    });
    // the span drops here, and only here does it export
}

An #[instrument] on a long-running function is a logging construct, not a tracing one. It scopes your log lines correctly and it will never produce a trace.

Trap 2 — `builder.trace_id` is a root-span-only answer

To write the field, the formatter needs the trace id of the span it is inside. tracing-opentelemetry stores its per-span state in the registry's extensions, and the obvious implementation is three lines:

let extensions = span.extensions();
let otel = extensions.get::<OtelData>()?;
otel.builder.trace_id

OtelData is public, the field is right there, and this works. On root spans.

From tracing-opentelemetry-0.28.0/src/layer.rs:887:

// Record new trace id if there is no active parent span
if !parent_cx.has_active_span() {
    builder.trace_id = Some(self.tracer.new_trace_id());
}

On a child span it is None. The trace id lives in the parent context instead, reachable via parent_cx.span().span_context().trace_id().

This is a nasty shape of bug, because the naive version passes the naive test. Open one span, log inside it, assert the field is present — green. Every real log line in the service is inside a child span, so the field would have been silently absent in production and reliably present in CI.

The test that catches it logs from inside a child:

let root = tracing::info_span!("root");
let _root = root.enter();
let child = tracing::info_span!("child");
let _child = child.enter();
tracing::info!("block ingested");

It is worth proving that the test discriminates rather than assuming it. Replacing the parent-context fallback with return None fails that test and only that test — the other three stay green.

The obvious fixture is not a simpler version of the real case. It is a different case — and when the library branches on the difference, a green test is pinning the branch you do not ship.

Trap 3 — `.instrument()` on the wrong future compiles and does nothing

Each chain adapter runs its subscribe-and-reconnect loop in a spawned task. tokio::spawn does not inherit the caller's span, so every reconnect warning was emitted outside any span at all — invisible to the whole scheme.

The obvious fix, applied three times:

tokio::spawn(async move { run_subscription_loop(...).await; })
    .instrument(span);

It reads correctly, and it is completely inert.

tokio::spawn returns a JoinHandle, and JoinHandle is itself a Future. So .instrument() accepts it and produces an instrumented handle, which is dropped on the next line without ever being polled. The task — already handed to the executor — still has no span.

The type system has no objection, because nothing here is type-incorrect. The only signal is a lint:

warning: unused `tracing::instrument::Instrumented` that must be used
    = note: futures do nothing unless you `.await` or poll them
help: use `let _ = ...` to ignore the resulting value

Note that the suggested fix is precisely wrong — let _ = ... silences the one piece of evidence that the change did nothing. The message body is the actual information: you built a future and threw it away.

tokio::spawn(
    async move { run_subscription_loop(...).await; }
        .instrument(span)
);

.instrument() decorates a future you still own. Once a future is with the executor there is nothing left to decorate — same for .timeout(), .boxed(), and every other combinator. If you are calling one on a JoinHandle, ask what you actually meant.

Trap 4 — every span closed, and they all landed in one trace

This one survived implementation, review of the implementation, and a written retrospective. It was caught on a second review pass, by asking a question none of the tests asked: not does the span close, but what is it a child of.

ingest_event was created contextually, which makes it a child of run — the process-lifetime span from Trap 1. run is the root, so run generates the only trace id that ever exists, and every child inherits it.

Two sibling event spans under one root, printed from an actual test run:

trace_ids: [
    "f3da095a59a8db160b9f4605b879b2e0",
    "f3da095a59a8db160b9f4605b879b2e0",
]  same = true

So: spans close, spans export, logs carry a well-formed 32-hex-character trace id, every test passes — and clicking a log line opens a "trace" containing every event the process has handled inside the backend's retention window. On a blockchain ingester pulling three chains, that is several hundred thousand spans a day in one trace. That is not a trace in any useful sense; it is a haystack with a link pointing at it.

The fix is one keyword — make each unit of work its own root:

let span = info_span!(
    parent: None,
    "ingest_event", event_id = %event.event_id()
);

Same probe after the change:

trace_ids: [
    "508d439ce5a118553871703aea27d140",
    "29ec1a9e882c4970d778e47143d50833",
]  same = false

Parenting to run bought nothing anyway — run never closes, so it never exports. The child was inheriting an id from a span that does not exist downstream.

The general shape: "the span closed" and "the span is in a sensible trace" are separate claims, and only the first one is easy to test. A test that asserts a trace id is present passes identically whether you have one trace per event or one trace per process.

The formatter, and the advice I gave myself that was wrong

tracing-subscriber's fmt layer and the OTel layer are independent — the JSON formatter serialises fields and span names, and never looks at the OTel SpanContext. A Layer cannot add fields to an already-constructed Event either; the visitor is read-only. So the injection has to happen during serialisation, which means implementing FormatEvent.

There are two ways to do that, and I picked wrong first. My own note-to-self said: own the serialisation with serde_json, because splicing a field into another formatter's output is fragile string surgery.

Reading tracing-subscriber's fmt/format/json.rs changed my mind. The stock implementation uses WriteAdaptor and SerializableSpan, neither of which is public. Reimplementing means writing both by hand — including the part that pulls formatted fields out of span extensions and re-parses them as JSON, which carries a maintainer comment in that file calling it an ugly fix. Roughly 120 lines of replicated private internals, drifting from upstream as the crate evolves, to add one key.

The fragility argument was also only half right. Suffix surgery is fragile — inserting before the closing brace means reasoning about whether the object is empty. Prefix surgery has no such case: the record always opens with {, and inserting immediately after it preserves the stock key order exactly.

let mut record = String::new();
self.inner
    .format_event(ctx, Writer::new(&mut record), event)?;

match current_trace_id(ctx, event) {
    Some(trace_id) =>
        write_with_trace_id(&mut writer, &record, trace_id),
    None => writer.write_str(&record),
}

The whole edit is one strip_prefix, one guard, and a fallback that emits the record untouched if the assumption ever breaks. A test pins that the delegated JSON shape survives.

"Don't do string surgery" is a good prior, not a rule. It was written for the case where you own the format — and here the entire point was not owning it.

Absent, not zero

An invalid span context yields the all-zero trace id. Writing "trace_id":"00000000000000000000000000000000" would satisfy a careless test — the key is present, it is 32 hex characters — and would then match the dashboard's derived-field regex, producing a clickable link to a trace that cannot exist.

The formatter omits the key entirely instead. Two of its four tests exist only to pin that: no OTel layer installed (a supported deployment — the exporter is optional), and no span entered.

For a correlation id, absent and wrong are not adjacent failure modes — they are opposites. An absent field tells the reader there is nothing to follow, whereas a wrong one sends someone down a dead end and costs them the time it takes to prove it is one.

And then the pipeline was wrong too

Verifying end to end found the part that no amount of Rust would have fixed: two breaks between the service and the dashboard link, neither in the application.

The derived-field matcher was written in logfmt. The Grafana datasource config had been carrying this since the stack was built:

derivedFields:
  - datasourceUid: tempo
    matcherRegex: "trace_id=(\\w+)"

The service emits JSON, so trace_id= never appears in a line that reads "trace_id":"...". It had matched nothing for weeks, and there was no way to notice, because nothing emitted a trace id either. Two halves of a feature, both missing, each hiding the other.

The log aggregator only parsed the outer envelope. Vector's aggregator ran one transform:

. = parse_json!(.message)

but its Kafka source had no decoding block, so the default byte codec made .message the agent's envelope rather than the service's log line. That single parse lifts the envelope and leaves the application's JSON as an escaped string still sitting inside .message. Querying the log store directly, the stored top-level keys were:

['file', 'kubernetes', 'message', 'source_type', 'stream']

None of the service's own fields made it into that list: level, target, span and the event id were all still inside the escaped string in .message. Every structured field the service works to emit was one parse_json short of being queryable — which also explained something I would otherwise have blamed on the log store, namely that the field list for those streams only ever showed labels.

There is a trap in fixing that too. The existing ! means abort on failure, and Vector drops the event and increments an error counter. Defensible for the envelope, which the agent always produces. Not defensible for the inner payload: every other container in the cluster logs plain text, and a ! there silently discards all of it.

With both ends repaired, the two halves finally describe the same string. The matcher, in JSON rather than logfmt:

derivedFields:
  - datasourceUid: tempo
    matcherRegex: '"trace_id":\s*"(\w+)"'

And a line off the ingest path, captured from a test that renders records through the same formatter production uses — emitted as one line, wrapped here so the keys after trace_id stay readable:

{
  "trace_id": "90a79d9c4db18e37d9f45c22338fa3e1",
  "timestamp": "2026-08-25T14:37:59.805729Z",
  "level": "INFO",
  "fields": {
    "message": "chain event ingested",
    "event_id": "01a0395b-4c7a-7112-b4ee-2c154b83f8c0",
    "event_type": "block_ingested",
    "lag_us": 3259,
    "block_number": 1
  },
  "target": "chain_ingestor::application::ingest_blocks",
  "span": {
    "chain": "ethereum",
    "event_id": "01a0395b-4c7a-7112-b4ee-2c154b83f8c0",
    "name": "ingest_event"
  },
  "spans": [
    {
      "chain": "ethereum",
      "event_id": "01a0395b-4c7a-7112-b4ee-2c154b83f8c0",
      "name": "ingest_event"
    }
  ]
}

trace_id is the first key because the formatter splices it after the opening brace rather than appending it, which is what keeps the matcher a plain match instead of something that has to reason about position. Everything after it is queryable now that the inner payload is parsed, so level, event_type and lag_us are filters rather than text a human reads.

That is the entire feature: a line in the log store, a pattern that matches it, and a link that resolves to a trace holding that one event.

Trap 5 — a sampler that decides per span, not per trace

Trap 4's fix is one keyword, which is what it costs to read. What it costs to run is arithmetic.

A root span per event pins span count to event count, and the Kafka publisher already carries its own #[instrument], so each event produces two spans rather than one. Three chains at current block rates works out to roughly 222,000 events a day — call it 444,000 spans. The next issue adds transaction streaming to the Ethereum adapter, which takes Ethereum alone from 7,200 events a day to about 1.3 million. Two and a half million spans, from one chain, against a Tempo ingester with a 384Mi memory limit.

Storage is not where this breaks — retention is 24 hours and the volume stays in single-digit gigabytes. What does not scale is the ingester's memory and the OTLP exporter's batch queue, which drops spans when it fills and reports nothing when it does. Those drops cluster in exactly the busy periods worth looking at, so an accidental sample at an unknown rate is strictly worse than a deliberate one at a known rate.

That makes AlwaysOn — the sampler in place until now, which exports every trace and decides nothing — the wrong default at this volume. Replacing it is the obvious move, and the replacement carries a trap of precisely the same shape as the four above:

// re-decides at every span
Sampler::TraceIdRatioBased(ratio)

// decides once, at the root, and every descendant inherits it
Sampler::ParentBased(
    Box::new(Sampler::TraceIdRatioBased(ratio)))

should_sample is called per span. A bare ratio sampler therefore evaluates each span independently, so a parent can be sampled in while its child is sampled out. The trace arrives in Tempo with spans missing, and nothing in it indicates that anything is missing. That is worse than not sampling at all, for the same reason an all-zero trace_id is worse than an absent one: it is a dead end that costs someone the time it takes to prove it is one.

Writing the test for that was the interesting part. The obvious fixture — pick a ratio, check what the children do — passes against both samplers, because for most ratios they agree. They differ in exactly one case, and the test has to sit on it: a span whose parent arrived already sampled out, at ratio 1.0. At 1.0 the bare sampler keeps every trace id unconditionally, so it clears every other assertion in the module and fails only here, where the right answer is to drop a span the ratio would have kept. Get that fixture wrong and the suite passes against the broken sampler.

The fix then arrived carrying a version of its own trap. The ratio is a fraction, and 100 is what someone thinking in percentages writes — a valid f64, and TraceIdRatioBased reads anything >= 1.0 as always-on. A config edit meant to cut export by ninety-five percent silently produces the maximum instead. It is refused at load now; clamping it would have been one more thing that looks like it worked.

The real span rate is still unmeasured, so the deployed ratio is 1.0 and the number will follow the measurement. The wrapper was not the part to defer, though. A rate that is too high shows up on a dashboard; a half-sampled trace looks exactly like a whole one.

The take-home

Five traps, one property in common: each produced output that looked right.

A span that never closes exports nothing, and nothing reports it.
A trace id read from the wrong field is absent in production and present in CI.
A combinator applied to the wrong future compiles, runs, and does nothing.
Spans that close correctly can still all land in one trace, and every test still passes.
A sampler that decides per span rather than per trace yields traces that are half there and look whole — and only one fixture in the space tells the broken one from the correct one.

The thread through them is that observability code has no independent observer. Everywhere else in a system, being wrong eventually surfaces as an error someone sees. Here, being wrong surfaces as telemetry that looks fine — and you find out during the incident it existed to help with.

Two habits would have caught all of them. Write the chain from work happens to human sees it before estimating any link in it. And for each link, ask what it would look like if it were broken — if the answer is "the same", that link needs a test that would fail.

If you have hit a different flavour of silently-fine telemetry — or think per-event root spans are the wrong call and run should have been the trace — I would genuinely like to hear which one bit you. The door is open at cordata.tech/contact. Related reading: One port, three client crates for the architecture of the service this post instruments.

One port, three client crates — a hexagonal Rust ingester

László Hadházy — Fri, 28 Aug 2026 15:39:51 +0000

Most of what makes a data platform hard is not the modelling. It is the boundaries: the seams where someone else's SDK meets code you have to maintain, and where a vendor's idea of a "block" meets yours. Those seams are where change arrives, and drawing them in the wrong place is expensive in a way that stays invisible until the second integration.

This is a worked example of drawing one — real trait, real adapter code, and the reasoning that produced them. The system ingests on-chain data from three chains through three client crates written by teams with no reason to agree on anything. If you work on ingestion of any other kind — CDC connectors, partner APIs, IoT feeds, market data — the chains are incidental and the problem is identical: heterogeneous sources, a domain that should not know about them, and the question of how much to abstract before it costs more than it saves.

Notes from the workbench. A design walkthrough rather than a tutorial: how one Rust service reads three blockchains through a single trait, why that trait has the signature it does, and — the part most ports-and-adapters write-ups skip — the three places we deliberately stopped abstracting.

Every crypto observability project starts with one chain. Someone writes an Ethereum indexer that subscribes to newHeads, decodes blocks and publishes to Kafka, and it does the job. Two months later a second chain arrives — Solana, Polkadot, an L2 — and the pressure question is whether it lives in the same service or gets its own.

The same service is usually the honest answer. The operational cost of running two ingester binaries — two deployments, two config surfaces, two dashboards, two on-call runbooks — is high, and none of it is fundamental. What is fundamental is that each chain is served by a different Rust crate with a completely different shape. The Ethereum crate (alloy) hands you a WebSocket subscription that yields Header values. The Solana crate (solana-client) makes you run two clients side by side — a PubsubClient for slot notifications and an RpcClient for actually fetching blocks. The Polkadot crate (subxt) yields decoded Block<PolkadotConfig> values, but reaching their extrinsics costs two further awaits.

Those are three genuinely different mental models, and the reflex fix — a big Chain enum branching to per-chain code paths everywhere the ingester runs — grows a new arm each time a chain is added, and eventually leaks each crate's idioms into every layer above it. The better move is a port: one Rust trait that all chain sources implement, with just enough shape to compose the layer above without knowing which chain is downstream.

What actually makes it hexagonal

Ports and adapters gets described as "use interfaces at the boundaries", which is true and useless — every codebase has interfaces at boundaries. The actual rule is narrower, and it is about which direction the dependencies point.

The domain defines the trait. The adapter implements it. The domain never imports the adapter.

That inverts the direction dependencies naturally want to flow. Without inversion you get layering: the application imports the Ethereum client, wraps it in something, and the wrapper's shape is decided by whatever alloy happens to expose. With inversion the trait is written to suit the use case, and each vendor SDK has to contort itself to fit — which is the point, because the contortion is then confined to one file per vendor.

The diagram, in words: chains drive the system from the top, through adapters that wrap each vendor SDK and implement the ChainSource port. The core reacts, then calls outward through EventSink and MetricsSink to Kafka and Prometheus below. The arrows show data; the dependencies all run the other way, because every one of those five adapters is written against a trait the core defines, and the core names none of them. main.rs sits outside the picture and wires the concrete adapters in at construction time.

In Rust this is enforced rather than encouraged. alloy appears in exactly one file; if the application module tried to import it, the build would fail on an unresolved dependency, because the crate graph does not permit it. That is a stronger guarantee than a convention nobody checks at review time.

Why a driving port hands back a channel

Hexagonal classifies ports by who invokes whom. That is the whole test, and it is worth stating as a question about the actor on the other side rather than about the direction data happens to move:

Driving (primary) ports are invoked by the outside world. Something out there acts on its own — a request arrives, a block is produced — and our application is what gets kicked into behaviour.
Driven (secondary) ports are invoked by the application. The actor on the other side does nothing until our use case reaches out and kicks it into behaviour. A database does not write itself; Kafka does not decide to accept a message.

The diagram above stacks them: driving at the top, driven at the bottom. Cockburn's original hexagon and most renderings you will meet elsewhere put driving on the left and driven on the right — the same distinction, turned ninety degrees.

Data-flow direction is the wrong test and gives the wrong answer here. EventSink moves data outward and ChainSource moves it inward, but that is not what separates them. What separates them is whose activity decides that work happens at all. The sink does nothing until our use case tells it to. The chain does not wait for us — blocks are produced whether we are listening or not, and our job is to keep up with an actor we do not control.

That is deliberately not a claim about who makes the function call at the transport layer, because the answer there varies and does not matter. Our Solana adapter polls: it subscribes to slot notifications and then fetches each block over RPC, one call per slot. A backfill adapter would poll a whole block range with no subscription at all. Neither is a driven port.

Hiding that difference is exactly what the adapter is for. Whether the external world pushes at us or has to be pulled from is a property of someone else's SDK, and letting it change the port's classification would mean the application's structure shifts every time a vendor changes their delivery model. The adapter polls on behalf of an actor whose schedule it does not set — the port stays driving either way, and the use case above it never learns which it was.

So ChainSource is a driving port, and the chain adapters sit on the driving side where an HTTP controller would in a request-driven service.

The interesting part is the shape it takes, because the usual driving adapter calls into the application. An HTTP handler receives a request and invokes a use case. A message listener receives a message and invokes a handler. Push, in both cases, with the adapter owning the call.

Ours inverts that, handing back a receiver and letting the application pull from it:

async fn subscribe(&self) -> IngestorResult<(
    mpsc::Receiver<Result<ChainEvent, IngestorError>>,
    SourceHandle,
)>;

That inversion is deliberate, and cancellation is the reason. A push-based driving adapter owns the control loop — it decides when the application runs. This service cannot allow that, because it has to wait on incoming events and a shutdown signal at the same time — tokio::select!, which races several futures and proceeds with whichever finishes first. A handler being called from inside an adapter has nowhere to put that choice. Ownership of the loop has to stay with the application.

So the port keeps the classification (the chain drives) and inverts the delivery (the application pulls). Written out, the trade is: give up the adapter's ability to call in, gain the application's ability to compose event handling with cancellation, backpressure and anything else it needs to select! over.

The cost lands on the adapter, which now has to run something of its own to fill that channel — a task, a client it owns, and a reconnect loop. Which is the next section's problem, and where the three crates stop resembling each other.

Which pattern is doing which job

Three patterns are stacked here, and their names get used interchangeably often enough to be worth pinning down:

	What it is	Where it lives here
Hexagonal / Ports & Adapters	Architectural. Dependency inversion at the system boundary	`ChainSource`, `EventSink`, `MetricsSink` and the module layout that enforces them
Adapter (GoF, object adapter)	Object-level. Wraps one vendor API behind an interface we own	`EthereumAdapter` wrapping `alloy::Provider`; `SolanaAdapter` wrapping `PubsubClient` + `RpcClient`
Bridge (GoF)	Object-level. Stands between us and a family of interchangeable backends	Not the chain adapters — `PrometheusMetricsSink` instruments via the `metrics` crate's `Recorder` trait, so the backend swaps without touching call sites (see the driven side)

Hexagonal is the architecture; the GoF Adapter is one common way to fill its adapter slot. Adapter wraps one vendor API, Bridge abstracts over a family of them — and the two compose, because an adapter at the port boundary can use Bridge internally, which is exactly what the metrics sink does.

A fourth pattern shows up at the layer above: fan-out (task-per-source). The application runs one tokio::spawn per adapter and composes cancellation with a select!. That is only expressible because every adapter returns the same channel-and-handle pair — the pattern above the port is enabled by the shape chosen at it.

The port in full

#[async_trait]
pub trait ChainSource: Send + Sync + 'static {
    fn chain(&self) -> Chain;

    async fn subscribe(
        &self,
    ) -> IngestorResult<(
        mpsc::Receiver<Result<ChainEvent, IngestorError>>,
        SourceHandle,
    )>;

    async fn is_healthy(&self) -> bool;
}

chain() is a tag and is_healthy() backs the readiness probe. Everything above concerned subscribe(), and there is one alternative worth addressing, because it is the shape most Rust reviewers reach for first.

The port's return type — a channel and a cancel handle rather than a stream — is what lets three genuinely different client crates plug in without the application layer knowing which one it has.

Why not impl Stream? It is the idiomatic Rust answer, it composes with the whole futures ecosystem, and it would have been wrong here. A stream carries values; it carries no way to stop producing them. So the caller would have to compose cancellation itself and every adapter would need a cancellation token threaded through its internals to produce a stream that honours it. That token-plumbing spreads through the trait hierarchy, each crate ends up implementing shutdown slightly differently, and the uniformity the port existed to provide is gone — three chains that look alike at the type level and behave differently on SIGTERM.

By returning (mpsc::Receiver<_>, SourceHandle), the port pushes the cancellation problem inside the adapter. Each adapter spawns its own task, owns its own client, and hands the caller two things: a channel to read events from, and an opaque handle whose Drop implementation shuts the task down. The caller never touches a cancellation token; the adapter's spawn/select internals never leak upward.

Three adapters

All three adapter files start the same way — domain imports, client-crate imports, a struct and an impl block — and then diverge almost completely.

Each snippet opens with that file's vendor imports, because those lines are the boundary — every crate-specific name in the whole service appears in one of these three blocks.

Ethereum (ethereum.rs) uses alloy's WebSocket provider. Provider::subscribe_blocks() returns a Subscription<Header>; the adapter drains that stream, converts each header into a ChainEvent::BlockIngested, and pushes it to the mpsc.

use alloy::{
    providers::{Provider, ProviderBuilder},
    rpc::types::Header,
    transports::{ws::WsConnect, TransportError},
};

let ws = WsConnect::new(&ws_url);
let provider = ProviderBuilder::new().connect_ws(ws).await?;
let sub = provider.subscribe_blocks().await?;
let mut stream = sub.into_stream();

while let Some(header) = stream.next().await {
    let event = normalise_block_header(header);
    event_tx.send(Ok(event)).await?;
}

Solana (solana.rs) needs two clients running side by side. There is no combined "subscribe to full blocks" primitive — PubsubClient::slot_subscribe() gives you slot notifications, and for each slot you have to call RpcClient::get_block_with_config() separately to fetch the actual block body.

use solana_client::{
    nonblocking::{pubsub_client::PubsubClient, rpc_client::RpcClient},
    rpc_config::{CommitmentConfig, RpcBlockConfig},
    rpc_response::SlotInfo,
};
use solana_transaction_status_client_types::{
    TransactionDetails, UiConfirmedBlock, UiTransactionEncoding,
};

// slot_subscribe hands back two things: the stream and an
// unsubscribe callback. The callback is dropped — a subscription here
// ends by dropping the client along with it.
let (mut slot_notifications, _unsubscribe) =
    pubsub_client.slot_subscribe().await?;

while let Some(slot_info) = slot_notifications.next().await {
    // second client, second round trip — the slot notification carries
    // no block body
    let block = rpc_client
        .get_block_with_config(slot_info.slot, rpc_block_config)
        .await?;
    let event = normalise_slot(slot_info.slot, block, Utc::now());
    event_tx.send(Ok(event)).await?;
}

Polkadot (polkadot.rs) uses subxt 0.50, whose block API changed shape in the 0.50 release — client.stream_blocks() yields finalized blocks by default, but the block header doesn't carry the timestamp. That has to be pulled from Timestamp.Now storage:

use subxt::{client::Block, OnlineClient, PolkadotConfig};
use subxt::rpcs::{client::RpcClient, methods::LegacyRpcMethods};

let client = OnlineClient::<PolkadotConfig>::from_url(&ws_url).await?;
let mut block_stream = client.stream_blocks().await?;

while let Some(block) = block_stream.next().await {
    let block = block?;
    let at_block = block.at().await?;
    let timestamp_addr = subxt::dynamic::storage::<(), u64>("Timestamp", "Now");
    let ms: u64 = at_block.storage().fetch(timestamp_addr, ()).await?.decode()?;
    // …build ChainEvent::BlockIngested with a real block_time
}

Three crates, three streaming models, three timestamp-extraction strategies. What they share is the shape at the boundary: each returns an IngestorResult<(mpsc::Receiver<_>, SourceHandle)> from subscribe(), and each spawns its own task that owns the client and pushes events into the channel.

The diagram, in words: the application layer talks to ChainSource as a trait object; three concrete adapters — each wrapping a different client crate — implement it. The application never imports a chain-specific crate; every crate's types stay inside its own adapter file.

The dependency arrow only points one way. The application module doesn't import alloy, solana-client, or subxt. It couldn't compile if it tried — those imports live in exactly one place per chain.

The driven side, briefly

Everything so far has been the driving port, because that is where the three crates disagree. The driven ports are worth a look for contrast — they are shaped differently, and for reasons that follow from the classification rather than from taste.

#[async_trait]
pub trait EventSink: Send + Sync + 'static {
    async fn publish_one(&self, event: ChainEvent) -> IngestorResult<()>;
    async fn flush(&self) -> IngestorResult<()>;
}

/// Outbound port: observability counters.
///
/// Infallible by design — recording a metric must never fail an ingestion.
///
/// # Cardinality contract
///
/// The label set used by implementations of this trait is closed: the only
/// permitted label dimension is `chain`. Implementations must not introduce
/// additional dimensions, and new methods must not take parameters carrying
/// unbounded values — block numbers, transaction hashes, wallet addresses,
/// log indices.
///
/// The rule lives at the port, not at the adapter, because future backends
/// (OTLP-metrics, vendor SaaS) inherit it via this trait. Enforcement is
/// code review at the call site: there is no compile-time guard, and a
/// panicking sink would turn a slow cardinality bug into a fast process kill.
pub trait MetricsSink: Send + Sync + 'static {
    fn record_block_ingested(&self, chain: Chain);
    fn record_event_published(&self, chain: Chain);
    fn record_chain_head_lag_ms(&self, chain: Chain, lag_ms: i64);
    // ...
}

EventSink needs neither a channel nor a cancel handle, because it is request-and-response: the application calls, the adapter answers, and the call ends. That is the ordinary shape for a driven port, and it is the shape ChainSource could not use — a subscription has no point at which the call is finished.

MetricsSink is different again — synchronous, returning nothing at all. The port removes the failure case rather than documenting a convention about it: there is no Result for a caller to mishandle, and no way for an observability problem to become an ingestion problem.

Its cardinality contract is worth the space it takes. Prometheus stores one time series per unique label combination, so a single unbounded label — a block number, a transaction hash — turns a per-block counter into an OOM kill on the scrape target, weeks later, with nothing failing in a test. That is invisible at review time unless someone has written down what "closed" means.

A port is a good place to put an invariant that outlives any particular adapter. That is a use for the boundary beyond swapping implementations, and the one most likely to be missed when ports and adapters get described as "just interfaces".

What the port bought us

Once every adapter returns a channel receiver, everything above the port becomes uniform. The fan-out layer that runs all three adapters concurrently is one tokio::spawn and one select! per adapter to compose cancellation with event reception, with no chain-specific branching anywhere in it.

for source in chain_sources {
    let (mut events, _handle) = source.subscribe().await?;
    let chain = source.chain();

    tokio::spawn(async move {
        loop {
            tokio::select! {
                biased;
                _ = &mut cancel_rx => break,
                event = events.recv() => match event {
                    Some(Ok(e))  => publish_to_kafka(chain, e).await,
                    Some(Err(e)) => log_and_reconnect(chain, e).await,
                    None         => break,
                }
            }
        }
    });
}

Reconnect and backoff end up uniform without being shared. Because each adapter owns its own subscribe loop, the reconnect logic — exponential backoff, max-attempts, "send SourceDisconnected on the channel when retries exhaust" — lives inside the adapter, not in the application. All three use the same 2s → 4s → 8s → 64s cadence, not because the port enforces it — there is no shared code between the subscribe loops — but because fixing what "the boundary" meant made the pattern trivial to copy.

Observability follows the same shape. Each adapter calls a shared observe_head_lag(...) helper, which clamps the value and then calls MetricsSink::record_chain_head_lag_ms(chain, lag) — the trait method above. One metric, chain_head_lag_ms, distinguished by its chain label rather than by three per-chain metric names, which is the cardinality contract being honoured rather than merely stated.

What we deliberately did NOT abstract

Every abstraction has a failure mode where it doesn't know when to stop. "Ports and adapters" easily becomes "shared crate that every adapter must import", and then the shared crate becomes a place where you paper over the client crates' differences with common helpers, and then the helpers become the actual interface — the port is decorative. Each of the following is a place we stopped:

One adapter file per chain, no shared adapter helpers. ethereum.rs, solana.rs, polkadot.rs. The rate-limited-warn atomic that keeps a clock-skew warning from firing every block now exists in all three, copied rather than extracted. That is roughly seven duplicated lines per chain, and it buys the certainty that changing one chain's warn cadence cannot silently change another's. At three copies this is still the right trade; it is also the point at which a fourth chain would make us re-examine it, which is worth saying out loud rather than discovering later.
No shared run_subscription_loop generic. Tempting — the shape is nearly identical across the three — but "nearly identical" hides subxt's two-await-hop for extrinsics, Solana's separate RpcClient::get_block_with_config call per slot, and Ethereum's alloy subscription-id reconnect quirk that the outer reconnect loop is written specifically to route around. A generic loop over impl Stream would either need a config trait wider than the port itself, or it would push those idiosyncrasies down into the adapter as flags — worse than duplication.
The domain conversion is per-adapter. normalise_block_header, normalise_slot, normalise_block. Each takes crate-specific types (alloy's Header, solana's UiConfirmedBlock, subxt's Block<PolkadotConfig>) and produces the domain-owned ChainEvent. This function is the actual adapter boundary — the one place per chain where crate-specific types are named. Making it generic would defeat the entire point of the pattern.

Recording what you did not abstract is what keeps the abstraction boundary where you put it. Each of those is a point where further extraction is defensible in isolation, and each would have added a layer justified by symmetry rather than by a requirement — which is how a port stops being a boundary and becomes a shared-code dependency with a trait on top.

Honest limits

The one-spawn-per-chain shape holds below O(dozens) of sources. Each adapter owns a Tokio task, an mpsc buffer and a connection, which stays cheap at forty and stops being cheap at four hundred — a per-rollup adapter for every L2, say. At that point you want a shared event loop with per-source filters, paid for in coarser cancellation.

The port also assumes every source is a live subscription. Historical backfill needs a second port on the same side — a range source rather than a stream — because stretching ChainSource to cover both would put a mode flag in the one place the design works hardest to keep clean.

The take-home

The pattern underneath all of this is ports and adapters, applied narrowly: three traits owned by the application core, five adapters implementing them, and a module layout that makes the dependency direction a compile error rather than a convention. Nothing exotic — the value came from where the boundaries were drawn and from being deliberate about where they stopped.

Two things worth stealing, independent of the stack:

Give a driving port over an unbounded source a channel and a cancel handle, not a stream. That is the shape that lets the application compose cancellation and backpressure without a token threaded through every adapter. Driven ports do not need it — EventSink is a plain async call, because a request ends.
Keep the crate-specific types inside one file per source. The domain conversion function is the real boundary: everything above it is a ChainEvent, everything below it is alloy::Header or subxt::Block<PolkadotConfig> or whatever the client crate hands you.

You pay one spawn and one mpsc per source. In exchange, adding the next one is a new adapter file, a new arm in a config enum, and no change anywhere above the port — a trade worth taking on any ingester whose sources are heterogeneous.

If you have drawn this boundary somewhere different — or think returning a stream would have been the right call and can say why — I would genuinely like to hear where it held or bent. The door is open at cordata.tech/contact. The companion post on the observability side of this same service is now up: The trace_id was the easy part — what it took to make a log line point at the right trace, and the four things that had to be true before it meant anything.

The catalog is the API — a governed mesh over MCP

László Hadházy — Thu, 27 Aug 2026 08:14:00 +0000

A domain engineer should be able to ask an assistant "which sensitivity level applies to a column that stores an EU IBAN?" and get the answer from the live LakeFormation policy rather than from a wiki page somebody last edited in 2023.

That sentence is easy to write and considerably harder to earn. The interesting part is not that a model can answer the question — it is where the answer comes from, and what stops the model from answering when it should not.

Companion to Fabric + Mesh on AWS. I wrote that sentence at the end of July; that post's § 3 ended with a forward-pointer and no landing page: "the next affordance this ontology deserves is an MCP-exposed catalog tool… It is a natural next step; it is not in the reference implementation yet." This post is the design behind that sentence. The tool is still unbuilt; what follows marks which parts are protocol, which are pattern, and which are neither yet.

The Model Context Protocol moved a long way while those posts were being written. Everything below is checked against revision 2026-07-28, and the revision is named on purpose: a post about a protocol that dates itself is more useful than one that pretends to be timeless.

§ 1 — Why the catalog, and not the warehouse

The reflex when someone says "let the agent query our data" is text-to-SQL against the lake. Part 2 § 7 argued why that is the wrong surface: raw SQL against a governed lake is unsafe for an agentic consumer, and a typed semantic API above it is what makes a question answerable without hallucination.

That names one tier — the semantic layer, where monthly_recurring_revenue means one thing to every consumer. This post is about the other, and the two are easy to swap. The semantic layer answers questions about numbers: what was fraud recall at 30 days last quarter? The catalog answers questions about the data itself: which sensitivity level applies to a column that stores an EU IBAN? Who owns the claims domain? May this role read it?

None of it is queryable from the lake. It is metadata about the datasets rather than data in them — LakeFormation's LF-tags, ownership records, grant expressions, descriptors — four stores behind four APIs in the governance account, reached through the console or Terraform.

A determined engineer can assemble the answer by hand, and nothing here is impossible without an assistant. But that is the wrong counterfactual. Nobody writes a four-API join at 16:40 while adding a column to a table — they copy the tag from the column next to it, or ask in Slack and get somebody's memory of a decision made in March. The tool competes against guessing, not against scripting. And "stores an EU IBAN" is not a lookup key in any case: no record is filed under that phrase, so answering means matching an informal description against the columns already classified — which is § 7's problem, not SQL's.

A catalog question is not a query with tighter permissions on it. It is a different question against a different store — and the answer counts only if the model retrieved it rather than knew it.

That constraint is sharper than it looks. A domain engineer tagging a new column asks which sensitivity level applies to an EU IBAN, and the model answers high — fluently, and correctly: that is exactly how policy_curated.customer.iban is tagged in the reference.

Now the same engineer tags an IBAN column in an analytics extract that is replicated across regions. The sensitivity is still high, so the answer still looks right. But residency records where the data sits, not where the account holder banks — the source table is residency: eu and this copy is residency: global, which changes which grants reach it while leaving the sensitivity untouched. A model reasoning from world knowledge will offer eu anyway, because that is what EU IBAN says, and validation accepts it — eu is a permitted value. The tag set is legal, it is wrong, and nothing downstream will say so.

That is the failure mode — right about the obvious key, quietly wrong about the one carrying the policy. The platform team owns this ontology, and they are not standing behind the domain engineer while the column gets tagged. Retrieving how comparable columns are already classified is what puts them there in effect; asking a model what it knows about IBANs cannot. So the assistant's job here is retrieval and routing; the judgement stays in the policy store.

§ 2 — Tools or Resources — the decision everything else hinges on

MCP exposes two server-side primitives that look interchangeable in a diagram and are not. The specification separates them by who decides. Tools are model-controlled — the language model discovers and invokes them from its own reading of the conversation. Resources are application-driven — the host application decides what to put in context, typically through a picker the user drives.

A catalog is both, and splitting it correctly is the most consequential decision in the design.

The LF-tag ontology is a Resource. From § 3 of the reference: five keys, twenty-one values, small enough to fit on a whiteboard. That is a document, and a host should be able to pin it into context wholesale — the same way an IDE pins an open file — so the model reasons against the full vocabulary instead of discovering it one lookup at a time.

A grant check is a Tool. "May this role read this table?" is a question with arguments, asked mid-reasoning, whose answer the model cannot predict. It has to be invoked.

The costs of getting this backwards are asymmetric. Everything-as-Tools burns turns re-fetching a twenty-one-value vocabulary the model could have held from the start. Everything-as-Resources is worse — the model cannot look anything up on its own, and a host forced to anticipate every dataset a conversation might touch will anticipate wrong.

The split shows up in the wire format, and it is worth seeing the two side by side. The ontology is one fixed document, so it is a Resource with a uri. Datasets are parameterised, so they are ResourceTemplates with a uriTemplate — the protocol uses RFC 6570 URI templates:

// ← resources/list — one fixed document, addressed directly
{
  "resultType": "complete",
  "ttlMs": 300000,
  "cacheScope": "private",
  "resources": [
    {
      "uri": "cordata://ontology/lf-tags",
      "name": "lf-tag-ontology",
      "title": "LF-tag ontology",
      "description": "The governance-owned tag vocabulary. Keys, permitted values, and what each value implies.",
      "mimeType": "application/json"
    }
  ]
}

// ← resources/templates/list — a shape, expanded per domain and table
{
  "resultType": "complete",
  "ttlMs": 300000,
  "cacheScope": "private",
  "resourceTemplates": [
    {
      "uriTemplate": "cordata://domain/{domain}/dataset/{table}",
      "name": "dataset",
      "title": "Dataset descriptor",
      "description": "Schema, ownership, LF-tags and contract for one published dataset.",
      "mimeType": "application/json"
    }
  ]
}

A URI with no variables in it is not a template — it is a resource that has been filed in the wrong list. The distinction is mechanical, and it is the same one the section has been arguing all along.

Template arguments can be auto-completed through the protocol's completion API, which means a host can offer a domain picker without the server writing a picker.

Diagram read-out. Three tiers, and the boundary that matters is not the one between accounts. Outside the governed accounts sit the host application, which pins the ontology as a Resource, and the language model, which invokes lookups as Tools — the host may well run inside a consumer account, but the model inference usually does not, which is the whole reason § 6 exists. Both paths enter the central governance account — the same one that holds the LakeFormation admin plane in the reference topology — where the caller's token is validated as this server's own audience before anything is read. That validation is the account boundary doing its job: the server holds its own LakeFormation grants rather than borrowing the caller's, which is what § 5 means by refusing passthrough. Where scope falls short the answer is a challenge, not an empty list — an empty one would tell the agent the dataset does not exist, which is a different sentence and a false one (§ 5).

The producer accounts are where the distinction has to be exact. They hold two different things, and the catalog reaches one of them. Column schema lives in the domain's Glue catalog, and get_dataset reads it — that is metadata, and the dotted edge is real traffic. The S3 objects beside it hold the rows, and they carry no inbound edge at all. So the line the design defends is the metadata plane against the data plane, not one account against another: a catalog server that reads a producer's Glue catalog is behaving correctly, and the same server reading a producer's S3 object is the failure § 4 is built to prevent.

Which leaves the decision the section opened with, now with the consequences attached. Resource or Tool is not a tidy way to organise an API surface — it decides who may pull a thing into context: the host, deliberately and in advance, or the model, mid-reasoning and on its own initiative. That single choice then fixes what varies by the caller's scopes (§ 5), what a cache may hold and for how long (§ 7), and which primitive the safety envelope has to be enforced on rather than annotated (§ 4). Get it backwards and none of those three has a stable answer, because the thing being governed keeps changing hands. Everything downstream is a consequence of this split, which is why it comes first.

§ 3 — The tool set, and where its schema comes from

The catalog server exposes four tools. § 1 put three questions to the catalog — who owns the claims domain? which sensitivity level applies to a column that stores an EU IBAN? may this role read it? — and each already has its answer sitting in a store the reference topology holds: ownership in DataZone, schema in the domain's Glue catalog, the permitted vocabulary in the LF-tag ontology, the reach of a role in the LakeFormation grants. The sensitivity question takes two tools rather than one, because retrieving precedent and validating a proposal are different operations — the exchange below shows why.

Tool	Question it answers	Determinism
`list_domains`	What domains exist, and who owns them?	Exact
`get_dataset`	What is this dataset's schema, owner, and tag set?	Exact
`resolve_lf_tag`	Is this tag set valid against the ontology?	Exact
`check_grant_expression`	Would this role's grant expression reach this dataset?	Exact

Every one is a lookup or an evaluation. None asks a model to decide anything — which is what makes the tool set auditable, and which is the inclusion rule in full: a question earns a tool when its answer already exists as a record in the governance plane. That rule is what generates the exclusions in § 4 as well — rows are data rather than metadata, a per-user grant listing is an access-review artefact with its own approval path, cross-domain composition is a subscription workflow. Those are not gaps in the tool set; they are the same rule, applied.

Most of the schema for these tools does not need inventing. Part 1 § 2 ended on the rule that one Pydantic model is imported by both CI and the executor, because two implementations of the same rules drift. The catalog server becomes its third importer: ownership and the policy contract are the descriptor's own fields, so get_dataset reuses them rather than restating them. Only the column list is declared locally, because the descriptor does not carry one — columns come from the domain's Glue catalog, which is the same plane boundary the § 2 diagram draws.

What follows is a fragment of the catalog server's own source: a module-level declaration of one tool. It is evaluated once, when the server process starts, and the resulting object is what every tools/list response hands back. Nothing here runs per request — the two model_json_schema() calls happen at import, which is why a change to the descriptor model reaches the advertised schema without anyone editing the server:

from pipeline_runtime.descriptor import Contract, Metadata

GET_DATASET = {
    "name": "get_dataset",
    "title": "Dataset lookup",
    "description": "Schema, ownership and policy tags for one published dataset.",
    "inputSchema": {
        "type": "object",
        "properties": {
            "domain": {
                "type": "string",
                "description": "Owning domain, e.g. fraud or claims",
                "x-mcp-header": "Domain",
            },
            "table": {"type": "string"},
        },
        "required": ["domain", "table"],
        "additionalProperties": False,
    },
    # The governance-owned halves are not redefined here. One model, three
    # consumers: CI validation, the executor, and now the catalog server.
    "outputSchema": {
        "type": "object",
        "properties": {
            "metadata": Metadata.model_json_schema(),   # name, domain, owner
            "contract": Contract.model_json_schema(),   # freshness_sla, lf_tags
            # Columns are the one part the descriptor does not carry: they are
            # read from the domain's Glue catalog, so their shape is declared here.
            "columns": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "type": {"type": "string"},
                        "lf_tags": {"type": "object",
                                    "additionalProperties": {"type": "string"}},
                    },
                    "required": ["name", "type", "lf_tags"],
                    "additionalProperties": False,
                },
            },
        },
        "required": ["metadata", "contract", "columns"],
        "additionalProperties": False,
    },
}

One declaration, three consumers. The model that fails a pull request when a descriptor is malformed is the same model that tells an agent what shape to expect. When a field is added, all three move together or none do.

The declaration says what the answer will look like. The handler is what fills it in, and it runs once per tools/call — two reads against the domain's account, joined on the column name:

def get_dataset(domain: str, table: str, caller: Identity) -> dict:
    authorize(caller, domain)          # § 5 — before either read, not after

    # The server runs in the governance account; the catalog it needs lives in
    # the domain's. One assumed role per call, session-named for the caller, so
    # the producer account's CloudTrail attributes the read to a person.
    producer = boto3.Session(**assume_reader_role(domain, caller))
    glue = producer.client("glue")
    lf   = producer.client("lakeformation")

    # `<domain>_curated` is the per-domain Glue database the blueprint creates
    # (reference § 2: name = "${var.domain_name}_curated"), so the tool takes a
    # domain and a table and never asks the caller for a database name.
    cols = glue.get_table(DatabaseName=f"{domain}_curated", Name=table)
    cols = cols["Table"]["StorageDescriptor"]["Columns"]

    tagged = lf.get_resource_lf_tags(
        Resource={"TableWithColumns": {
            "DatabaseName": f"{domain}_curated", "Name": table,
            "ColumnNames": [c["Name"] for c in cols]}},
        ShowAssignedLFTags=True,
    )
    by_column = {
        c["Name"]: {t["TagKey"]: t["TagValues"][0] for t in c["LFTags"]}
        for c in tagged["LFTagsOnColumns"]
    }

    return {                            # validated against outputSchema on the way out
        "metadata": metadata_for(domain, table),
        "contract": contract_for(domain, table),
        "columns": [
            {"name": c["Name"], "type": c["Type"],
             "lf_tags": by_column.get(c["Name"], {})}
            for c in cols
        ],
    }

Nothing in there is clever, which is the point: two AWS calls and a dictionary join. The role assumption is the one line worth pausing on — it is the cross-account hop from § 1 of the reference, and naming the session after the caller is what keeps the producer account's CloudTrail useful. A read attributed to mcp-catalog-server tells an auditor nothing; one attributed to the engineer who asked is the same behaviour record Behaviour-first governance already projects from. Neither call can return a row — get_table returns column definitions and get_resource_lf_tags returns tags, and there is no third call to Athena. The read-only property § 4 insists on is a fact about which APIs the handler is able to reach, visible in the source, rather than a claim made in an annotation.

The protocol is strict about the contract this creates: where an output schema is declared, servers MUST return structured results conforming to it, and clients SHOULD validate against it.

The x-mcp-header annotation on domain is worth a note. A parameter marked this way is mirrored into an Mcp-Param-Domain HTTP header, so a load balancer or WAF can route on it without parsing the request body — which maps directly onto the account-per-domain topology from § 1 of the reference. It carries its own warning, and it is a real one: header values are visible to network intermediaries, so sensitive parameters must never be marked this way. A domain name is fine. A subject identifier is not.

The IBAN question, answered

Back to the sentence that started this. A domain engineer is tagging a new column and asks which sensitivity level applies to one holding an EU IBAN.

The answer does not come from the model's opinion about IBANs. It comes from precedent already in the catalog. § 3 of the reference tags exactly this column, in Terraform:

resource "aws_lakeformation_resource_lf_tags" "iban_column_high" {
  database { name = "policy_curated" }
  table    { name = "customer" }
  column   { name = "iban" }

  lf_tag {
    key    = "sensitivity"
    values = ["high"]
  }
}

So the exchange is a retrieval, then a validation:

// → tools/call — find how comparable columns are already classified
{ "name": "get_dataset",
  "arguments": { "domain": "policy", "table": "customer" } }

// ← result
{ "resultType": "complete",
  "structuredContent": {
    "columns": [
      { "name": "iban", "type": "string",
        "lf_tags": { "sensitivity": "high", "residency": "eu",
                     "subject_type": "customer" } }
    ],
    "owner": "policy-domain"
  } }

// → tools/call — validate the proposed tag set before it is written
{ "name": "resolve_lf_tag",
  "arguments": { "sensitivity": "high", "residency": "eu",
                 "subject_type": "customer" } }

The second call is not a formality. It runs the same policy.resolve() the executor runs, against the same governance-owned vocabulary, and it fails the same way:

def resolve(declared: dict[str, str]) -> dict[str, str]:
    known = ontology()  # governance-owned vocabulary
    for key, value in declared.items():
        if key not in known:
            raise UnknownTagKey(key, allowed=sorted(known))
        if value not in known[key]:
            raise UnknownTagValue(key, value, allowed=known[key])
    return declared

An assistant proposing sensitivity: hgh gets UnknownTagValue with the permitted set attached, at the moment of tagging — rather than a table written with a tag nothing grants against. That is the defence-in-depth the original callout claimed, and it is worth being clear about which half does the work: the model reduces search time and the ontology refuses invalid answers, and neither half is sufficient alone.

§ 4 — The safety envelope is enforced, not annotated

The tempting shortcut is declaring the tool set read-only. MCP has a field that looks exactly like it should do the job:

{ "annotations": { "readOnlyHint": true, "destructiveHint": false } }

The schema's own documentation removes any doubt about what that is worth — this is the doc comment on ToolAnnotations in schema/2026-07-28/schema.ts:

NOTE: all properties in ToolAnnotations are hints. They are not guaranteed to provide a faithful description of tool behavior (including descriptive properties like title). Clients should never make tool use decisions based on ToolAnnotations received from untrusted servers.

So readOnlyHint is display metadata. It tells a host how to render a confirmation dialog. It enforces nothing, and a safety envelope that depends on it is not an envelope. Read-only has to hold by construction. The handler has no write path, the database role has no INSERT, and the grants stop at SELECT and DESCRIBE. A server built that way stays read-only no matter what its annotations say.

What the tool set deliberately does not expose matters as much as what it does:

No raw rows. The catalog answers questions about datasets, never from them. An agent that can read customer.iban values has defeated the entire LF-tag model in one call.
No per-user grant listings. "Would this expression reach this dataset" is answerable. "Show me everyone who can read the claims domain" is an access-review artefact with its own approval path, not a tool call.
No join hints across domains. Cross-domain composition is a subscription workflow in DataZone, and routing around it through a helpful assistant is exactly the failure the workflow exists to prevent.

Every call gets logged into the governance-plane event stream that Behaviour-first governance § 4 already builds its audit projection from. The tool set is not an exception to the governance model — it is another source of behaviour, and it should show up in the same query a supervisor's ninety-day access question runs against.

One further trap, new enough in the specification to be easy to miss. MCP has no protocol-level session, so servers needing state across calls hand back a handle — and the specification's guidance here — non-normative, but unusually direct — is that for authenticated servers "a handle is a name, not a capability", to be re-validated against the caller's authorization on every call. A dataset_id returned by list_domains is not a ticket. If the second call skips that re-check, the first call has quietly become an access-granting operation.

§ 5 — Authorization is the governance plane

This is the part that changed most, and it lands directly on the mesh argument.

Honesty about scope first, because the specification's framing is easy to misread. Protocol Requirements opens by declaring authorization OPTIONAL for MCP implementations, and then says why in the next three lines: HTTP-based transports SHOULD conform, STDIO transports SHOULD NOT — they "retrieve credentials from the environment" instead — and alternative transports MUST follow the best practices of their own protocol.

So OPTIONAL is about which transports the specification covers, not about how much security a server owes its callers. A stdio server is a local subprocess started by the host: there is no browser to redirect, no audience to bind a token to, and the operating system already decided who may run it. An OAuth flow would protect nothing the process boundary does not already protect. Nothing about that concession applies to a catalog server, which is remote, HTTP, and multi-tenant by construction — the SHOULD lands squarely on it.

And once a server is inside that scope, the requirements stop being soft:

MCP servers MUST implement OAuth 2.0 Protected Resource Metadata (RFC 9728); clients MUST use it for authorization-server discovery.
Clients MUST implement Resource Indicators (RFC 8707) and send resource on both authorization and token requests.
Servers MUST validate that access tokens were issued specifically for them as the intended audience, and MUST NOT accept or transit any other tokens.

That last requirement is the confused-deputy protection stated normatively: token passthrough is prohibited. A catalog server cannot take the token it was handed and replay it against LakeFormation on the caller's behalf. It has to be its own audience, with its own grants.

On AWS that prohibition costs nothing, because there is nothing to pass through to. LakeFormation has never heard of a bearer token. Its grants are written against IAM principals, so an OAuth access token is not a weaker credential there — it is not a credential at all. The identity has to be translated, and that translation is where this section stops being about MCP and starts being about your accounts:

_SAFE = re.compile(r"[^\w+=,.@-]")          # STS: alphanumerics and +=,.@- only

def assume_reader_role(domain: str, caller: Identity) -> dict:
    # 1. Scope check, against the token this server already validated as its
    #    own audience. The token does not travel any further than this line.
    if f"catalog:{domain}:read" not in caller.scopes:
        raise InsufficientScope(required=f"catalog:{domain}:read")

    # 2. The server assumes the domain's reader role under its OWN identity.
    #    The caller is carried as a name and a session tag, never as a
    #    credential — an OIDC `sub` may contain characters STS rejects.
    out = boto3.client("sts").assume_role(
        RoleArn=f"arn:aws:iam::{ACCOUNTS[domain]}:role/cordata-catalog-reader",
        RoleSessionName=f"mcp-{_SAFE.sub('-', caller.sub)}"[:64],
        Tags=[{"Key": "cordata:caller", "Value": caller.sub[:256]}],
        DurationSeconds=900,
    )["Credentials"]

    return {"aws_access_key_id": out["AccessKeyId"],
            "aws_secret_access_key": out["SecretAccessKey"],
            "aws_session_token": out["SessionToken"]}

assume_role rather than assume_role_with_web_identity is deliberate. Federating the caller's own JWT into AWS would mean forwarding a token this server accepted as its own audience on to a third party — the passthrough the MUST NOT above prohibits. The token stops at the scope check.

So the governance chain has three links, no two of them enforced by the same system, and only one of them living in the MCP server at all:

Link	Enforced by	Owned by	Decides
Role trust policy	IAM, in the domain's account	Platform team, via the account blueprint	Whether this server may assume the role at all
LF-tag grant expression	LakeFormation, governance account	Governance	Which columns come back
Scope check	The MCP server's own code	The server	Whether the call is attempted in the first place

The server holds none of that policy — a scope check and an AssumeRole call is the whole of it, which is the entire reason a compromised catalog server is a contained problem rather than a catastrophic one.

The role's trust policy decides who may assume it at all, and it is platform-owned — part of the same domain-account blueprint as everything else in § 2 of the reference, not something a domain writes by hand:

data "aws_iam_policy_document" "catalog_reader_trust" {
  statement {
    effect = "Allow"

    # sts:TagSession is not optional here — without it the Tags= argument
    # in the handler fails, and the caller's identity never reaches CloudTrail.
    actions = ["sts:AssumeRole", "sts:TagSession"]

    principals {
      type        = "AWS"
      identifiers = [var.catalog_server_role_arn]  # governance account, one role
    }

    condition {
      test     = "StringEquals"
      variable = "sts:ExternalId"
      values   = [var.catalog_external_id]
    }
  }
}

One named principal, in one account. Nothing else, in any account, can assume this role. A domain that wants to stop answering catalog questions removes the trust statement, and no coordination with the server is required.

The LF-tag grant on that role decides which columns come back, and it is governance-owned — written once against a tag expression, never against a table name.

And the scope check stays in the MCP server, necessarily — AWS never sees the caller's token, so no IAM policy can condition on what is in it. That check is ordinary application code and can therefore have bugs, which is why the role it guards is scoped to a single domain: a bug there costs one domain's metadata, not every domain's.

The sentence that matters most for a federated mesh sits in the tools chapter, and repeats verbatim for resources/list. The tool set MUST NOT vary per-connection, but it MAY vary "by the authorization presented on the request — for example, returning only the tools the caller's granted scopes permit — since credentials are per-request input, not connection state."

That is account-per-domain expressed in a protocol: a fraud engineer's token surfaces fraud datasets, and the identical server shows a different catalog to the governance function. Not a filtered view rendered by a client — a different response, decided server-side, per request.

And when the caller asks for something outside their scope, the correct answer is not an empty list:

HTTP/1.1 403 Forbidden
WWW-Authenticate: Bearer error="insufficient_scope",
                         scope="catalog:claims:read",
                         resource_metadata="https://mcp.example.com/.well-known/oauth-protected-resource",
                         error_description="Read access to the claims domain is required"

An empty result tells an agent "no such dataset", and a competent agent will believe it and tell the user the data does not exist. A scope challenge tells it what to ask for, and the client can run a step-up authorization flow. The specification also asks servers to emit all required scopes in a single challenge rather than one at a time, because incremental challenges force repeated round-trips for one operation.

This maps onto grant expressions better than anything designed for the purpose would have. "You cannot see this" and "this does not exist" are different sentences, and only one of them is true.

§ 6 — DSGVO (the GDPR) — what the design keeps out of scope

Two legal statements, both narrow enough that anyone can check them against the text. Art. 28(3)(a) DSGVO requires a processor to act only on the controller's documented instructions. Art. 5(2) requires the controller to be able to demonstrate compliance. Everything else in this section is architecture rather than law, and I will keep the two apart.

The architectural claim is about scope. § 4's rule — questions about datasets, never from them — means a sensitivity=high tag and a column name cross the boundary and an IBAN never does. That is worth stating plainly, because the reflex assumption about an assistant near a governed lake is the opposite one. An assistant that reaches rows raises the whole processor question; this design's contribution is to keep one surface on the other side of that line rather than to answer it.

What the protocol contributes is evidence. Audience-bound tokens make a call attributable to a named client acting for a named principal rather than to an ambient credential, and prohibited token passthrough stops the chain widening silently to a fourth party. Whether the resulting record is legally required is not mine to say. Where it goes is checkable, and it is worth being precise rather than waving at the earlier post.

Every call in this design leaves an AWS trail before it leaves an MCP one: sts:AssumeRole into the producer account, then glue:GetTable and lakeformation:GetResourceLFTags inside it. Those are CloudTrail events, and the cross-account stitching in Behaviour-first governance § 4 already reads exactly this shape — its own diagram has the AssumeRole into producer hop in it. The session name from the handler is what makes the trail resolve to a person rather than a service. One caveat that post is explicit about and this one should repeat: CloudTrail data events are not on by default, so "no new reporting path" holds only where that Terraform is already applied.

The RoPA projection is a different matter, and the honest answer is that a catalog read does not appear in it. § 2 derives RoPA from OpenLineage RunEvents — jobs moving data — and a tool call is not a job run. That is a gap only if you expected the catalog to be a processing activity. Under the scoping above it is not one, so the absence is consistent rather than missing. The reference runtime's ontology carries subject_type and retention because the pipelines that do move data need them for that projection, not because the catalog is doing anything to a data subject.

Two cautions that are not legal claims at all:

Organisational, before architectural. A works council will hear "an AI can see the data catalog" as a monitoring question long before an architecture one, so bring the tool inventory and the audit projection to that meeting rather than the protocol diagram.
Aggregation, not any single answer. One dataset name is unremarkable; the complete list of them is a map of how the business is organised — which domains exist, what they are called, where the regulated ones sit. An agent that can enumerate everything holds that map, and no individual call is the moment it was handed over. That is the argument for scoping list_domains by authorization exactly as § 5 permits: domain names are not harmless in bulk merely because each one is harmless alone.

§ 7 — Retrieval over a closed set

§ 3's IBAN exchange skipped a step. The handler turns a domain and a table into policy_curated.customer on its own — but how did the assistant arrive at domain: policy, table: customer to begin with?

The naive answer is to let the model guess a table name. The better one exploits a property of this problem — the vocabulary is closed and tiny. Five keys, twenty-one values, and dataset and column names that are enumerable.

For a closed set, embedding the vocabulary and retrieving the nearest entries beats asking a model to produce a value from memory, for a reason that has nothing to do with model quality: retrieval cannot return a value that is not in the set, and generation always can. A model asked to invent a sensitivity level will invent pii or restricted — plausible, absent from the ontology, and rejected by policy.resolve() one step later. Retrieval over the twenty-one permitted values cannot produce that failure at all.

That disposes of invalid values. It does not touch the harder case from § 1, where the model proposes residency: eu for a globally replicated copy — a permitted value, accepted by resolve(), and wrong anyway. No vocabulary check catches that one, because the vocabulary was never violated. What catches it is a second retrieval against a different corpus: the classifications already applied to comparable columns. Asking get_dataset what the other columns in that same extract carry answers from precedent, and the reply stops depending on what the phrase EU IBAN suggests.

So the shape is: embed the ontology values and the catalog's column descriptions once, retrieve candidates for "stores an EU IBAN", and let the tools do the rest. Generation belongs where the answer is genuinely open — a description, a summary of why a dataset exists — and nowhere near a policy value.

That word once is where the cost sits. An embedding index is a snapshot; the catalog it describes is not. Datasets get published, columns get reclassified, and a retrieval that was correct on Monday starts answering from a world that has moved — silently, because a stale index returns confident nearest matches rather than errors.

The protocol has an answer for the part of this it can see. List and read results alike carry ttlMs and cacheScope, so a client knows how long the answer it just received may be trusted:

{
  "resultType": "complete",
  "resources": [ /* … */ ],
  "ttlMs": 300000,
  "cacheScope": "private"   // never "public": this list varies by scope
}

Nothing about this is stored in the catalog. ttlMs is a property of the response, not of the asset — the server picks a number as it answers, from what it knows about how quickly that particular answer goes stale. The LF-tag ontology changes by governance pull request, so it can afford minutes. A dataset list changes whenever any domain publishes, so it can afford less. A single resources/read of one descriptor sits somewhere between. None of that is catalog metadata; it is a policy the server holds, and the only thing the client is told is the number.

The server's own embedding index is outside all of it. No client-side TTL refreshes something the server built for itself — that refresh is triggered by the same events described below, not by anything in a response.

private is not a preference here. It follows from the choice made in § 5 to let the lists vary by the caller's granted scopes, because the specification defines public as a response any client or intermediary may cache and serve across authorization contexts. Combine the two and a shared cache in front of the catalog becomes a way to hand one caller's view to another — the fraud engineer's dataset list served from a proxy to somebody with no fraud scope at all. A server that returned an identical list to everyone could say public honestly. This one cannot, and the field is where that consequence becomes visible to the client.

Five minutes is too long whenever a tag write lands inside the TTL window. A governance PR retags policy_curated.customer.iban from sensitivity: confidential to high at 14:03; a client that listed at 14:01 keeps serving the old classification until 14:06, and every retrieval in between answers against a grant expression that no longer holds. The protocol's answer is subscriptions: the client opens a long-lived subscriptions/listen stream naming the URIs it cares about in notifications.resourceSubscriptions, the server replies with notifications/subscriptions/acknowledged, and thereafter pushes notifications/resources/updated whenever a watched resource changes. A server declares resources: { subscribe: true } to offer it at all.

Writing that sentence is considerably easier than deploying it. What it actually costs, on AWS:

A compute target that stays up. Not Lambda. ECS, Fargate or App Runner, with the scaling, health checks and monitoring that a long-running service needs and a function does not.
idle_timeout.timeout_seconds raised on the ALB. The default is 60 seconds; a quiet subscription is closed by your own load balancer before it reports anything.
A heartbeat, if raising that timeout is not enough or not permitted.
Egress proxies you do not control. Any enterprise network may buffer or terminate long-lived responses whatever you configure at either end.
Client-side reconnect, backoff and re-subscribe — plus reconciling what changed while disconnected, which is the part that gets discovered in production.

So subscriptions are an optimisation, and for a catalog it is worth asking whether it is ever the right one. The ontology looks like the obvious candidate because every answer depends on it — but small and rarely changed are exactly the properties that make re-reading it cheap. One document, a short ttlMs, and the exposure is bounded by the window. Dataset descriptors have the opposite shape: they genuinely do change, but watching them costs a URI per table and a held stream per connected host. Neither case is a comfortable fit.

Which leaves the unglamorous default doing most of the work, including through a proxy you cannot change: a short ttlMs, plus a resources/read of the one descriptor immediately before acting on its policy value — the retrieve-then-validate shape from § 3, with the read doing what the stale index cannot. Subscriptions pay off where a resource changes often and is watched by few. A catalog is usually the other way round. Reclassification propagating without touching pipelines was already a property of the ontology in Part 1 § 6; this extends the same property to the agent's view of it.

§ 8 — What this design does not solve

Three gaps, stated rather than buried.

A catalog cannot describe what nobody registered. Asked about a dataset a domain built last week and never published, an agent correctly reports that it does not exist — indistinguishable, to the person asking, from the tool being broken. Coverage is a governance property, not a protocol one.

Annotations remain untrusted in aggregate. § 4 covers a single server. A host aggregating several inherits all their trust assumptions, and cross-server tool-name collisions are explicitly the client's problem to disambiguate. A catalog tool that is safe alone is not automatically safe beside an arbitrary third-party server in the same context window.

Half of this does not port. Everything the protocol contributes is cloud-agnostic — Tools versus Resources, output schemas, scope challenges and cache scoping hold wherever you run them — but the identity chain in § 5 is built out of AssumeRole, LF-tag grant expressions and account-per-domain, none of which exist elsewhere. On another cloud the question how does a validated caller become a credential the catalog can be read with? has to be answered again from first principles, and while the shape of that answer transfers, none of the code does.

What gets built next

Every protocol claim above is checked against the specification; none of it has served a request yet. That is worth stating plainly, and worth not leaving there.

The next piece of work is a running server in its own repository, built on the descriptor model the pipeline runtime already publishes — so that § 3's argument about one model and three consumers becomes something you can check out and run rather than something I have asserted. Until then this post is a design, and it is not meant to stay one.

Where this fits

The Governance pillar claimed governance artefacts should be byproducts of behaviour; Fabric + Mesh on AWS built the account topology and the tag ontology; Behaviour-first governance turned the emitted events into the projections an auditor asks for; the pipeline pair made a pipeline a declaration that emits its own evidence.

This post adds the read path, and the arc closes somewhere I did not expect when the first one was written. The ontology those posts spent so long keeping small, explicit and machine-checkable is what makes agentic access tractable — not because the agents are clever, but because a twenty-one-value vocabulary with a validator behind it is a search problem rather than a judgement problem. Invert it and the claim falls over: three hundred values and no validator, and no amount of protocol rescues any of this.

If you have put an agent in front of a governed catalog — or think doing so is a worse idea than I have made it sound, and can say where it breaks — I would genuinely like to hear the counter-argument. The door is open at cordata.tech/contact. Related reading: the Fabric + Mesh on AWS reference for the LF-tag ontology every tool here reads from; part 1 — a pipeline is a descriptor, not a program for the descriptor model the output schemas derive from, and part 2 for the semantic-layer tier this one sits beside; and Behaviour-first governance in practice for the audit projection every tool call lands in.

Originally published at cordata.tech.

One skeleton, many screens — React + TypeScript

László Hadházy — Tue, 25 Aug 2026 08:21:56 +0000

Every internal admin tool starts clean. One screen lists articles: a table, a few count cards across the top, a status pill. Someone writes it in an afternoon and it's good.

Then comes the pages screen. It's almost the same — a table, count cards, no status pill this time. So it gets copy-pasted and trimmed. Then a users screen, a media screen, an audit-log screen. Six months later the tool is a graveyard of near-identical table components, each with its own subtly different pagination handling, its own loading spinner, its own idea of what an empty state looks like. Fixing a bug means fixing it five times, and you'll miss one.

The reflex fix is to reach for a single mega-component — a <SuperTable> with forty props and a variant enum — and wire every screen through it. That trades a copy-paste problem for a God-object problem. The component grows a new prop every time a screen needs something slightly different, and eventually no one can change it without breaking a caller three domains away.

The better move is to change what a screen is. Not a component you configure, but data you declare.

A screen is a descriptor

The unit I settled on is a plain description of the screen — its columns, and the numbers it summarises — with no rendering logic in it at all:

export interface ColumnDef<Row> {
  field: string;
  header: string;
  flex?: number;
  width?: number;
  minWidth?: number;
  sortable?: boolean;
  type?: "date" | "number";
  /** Derive the sortable/display value from the row. */
  value?: (row: Row) => unknown;
  /** Custom cell content, e.g. a status pill. */
  render?: (row: Row) => ReactNode;
}

export interface StatCard<Row> {
  label: string;
  compute: (rows: Row[]) => number | string;
  hint?: string;
}

That's the whole contract a screen author touches. Everything about how a table paginates, sorts, shows a loading bar, or renders an error lives elsewhere — in a single generic component the domains never edit:

export function TablePage<Row extends GridValidRowModel>({
  columns,
  rows,
  getRowId,
  loading,
  error,
  statCards,
  pageSize = 10,
}: TablePageProps<Row>) {
  // maps ColumnDef -> the grid's native column shape, once
  const gridColumns = useMemo<GridColDef<Row>[]>(
    () =>
      columns.map((c) => ({
        field: c.field,
        headerName: c.header,
        flex: c.flex,
        width: c.width,
        minWidth: c.minWidth,
        sortable: c.sortable ?? true,
        type: c.type,
        valueGetter: c.value ? (_v, row) => c.value!(row) : undefined,
        renderCell: c.render ? (params) => c.render!(params.row) : undefined,
      })),
    [columns],
  );
  // ...renders the stat cards + the data grid
}

Two small translations do the real work here. A column's value becomes the grid's valueGetter — the function that turns a row into a sortable, displayable cell value. A column's render becomes renderCell — arbitrary JSX for the cell, which is how a status pill or a link gets in without the framework knowing anything about statuses or links. The domain describes intent; the framework owns the mechanics of the grid library underneath. Swap that library out one day and no descriptor changes.

Why this is a DDD boundary, not just a helper

The split isn't only about avoiding duplication. It's a domain-driven design boundary drawn in the file tree — DDD in the one sense that earns its keep here: each part of the product (articles, pages, users) is a bounded context that lives in its own module, owns its own rules, and is not allowed to reach into another. The shared machinery lives somewhere those modules can depend on, but which depends on none of them.

On disk that's two top-level folders:

src/
  framework/                 # reusable, domain-agnostic — knows no domain
    bodies/
      TablePage.tsx          # the generic, descriptor-driven list screen
  features/                  # bounded contexts (migrating to domains/)
    articles/
      ArticlesDataGrid.tsx   # a descriptor, nothing more
      StatusChip.tsx         # article-specific cell content
    pages/
      PagesTable.tsx         # a descriptor, nothing more
  utils/api/                 # typed client + query hooks

The folder is still called features/ — the conventional Next.js name — and is on its way to being renamed domains/, because bounded context is the honest label for what lives there. The name matters less than the rule it encodes: nothing in articles/ may import from pages/, and neither may be imported by framework/.

The diagram, in words: articles/ and pages/ each declare their columns against framework/'s TablePage and depend on its ColumnDef / StatCard types; framework/ depends on neither of them.

The dependency arrow only points one way. framework/ knows nothing about articles or pages — it can't, and that's enforced by it having no imports from the domains. Each domain owns its own descriptor and its data-fetching, and depends inward on the framework's types. A new domain can't accidentally couple itself to another domain through the shared table, because the shared table has no domain in it to couple to.

This is what makes "data-driven" more than a slogan. The domain layer declares what a screen is; the framework layer decides how it renders. The boundary between them is a set of serialisable-ish descriptors, not a call graph.

Here's what a domain screen actually looks like once the skeleton exists — the entire articles list:

const columns: ColumnDef<Article>[] = [
  { field: "title", header: "Title", flex: 2, minWidth: 240 },
  { field: "category", header: "Category", flex: 1, minWidth: 150 },
  {
    field: "author",
    header: "Author",
    flex: 1,
    value: (a) => a.author?.name ?? "—",
  },
  {
    field: "updatedAt",
    header: "Updated",
    width: 130,
    type: "date",
    value: (a) => (a.updatedAt ? new Date(a.updatedAt) : null),
  },
  {
    field: "status",
    header: "Status",
    width: 140,
    sortable: false,
    render: (a) => <StatusChip status={a.status} />,
  },
];

const statCards: StatCard<Article>[] = [
  { label: "Total", compute: (rows) => rows.length },
  {
    label: "Published",
    compute: (rows) => rows.filter((a) => a.status === "PUBLISHED").length,
  },
];

No JSX for the grid. No pagination. No loading state. The screen is a list of columns and two reducers over the rows. Everything else is inherited.

The test: a zero-diff refactor

Here's the part that tells you whether the boundary is in the right place.

I didn't build the skeleton first and then screens on top of it. I had a working, hand-written articles table already built and rendering live data. To validate the abstraction, I rewrote the existing screen to render through it — and watched for changes.

The proof you drew the boundary correctly is that refactoring an existing
screen through the abstraction produces zero visual diff. If the screen
changes, the abstraction is imposing opinions the screen didn't ask for.

The articles table looked and behaved identically before and after — same columns, same counts, same status pills, same sort behaviour. That's not an anticlimax; it's the whole result. A refactor that changes nothing the user can see, while collapsing a hand-written component into a ten-line descriptor, is the signal that the skeleton captured exactly the screen's structure and none of its incidental detail.

Then the second screen — pages — cost a descriptor and nothing else. Here's the whole thing — the same TablePage, fed a different descriptor:

const columns: ColumnDef<Page>[] = [
  { field: "title", header: "Title", flex: 2, minWidth: 240 },
  { field: "slug", header: "Route", flex: 1, minWidth: 160 },
  {
    field: "updatedAt",
    header: "Updated",
    width: 130,
    type: "date",
    value: (p) => (p.updatedAt ? new Date(p.updatedAt) : null),
  },
];

const statCards: StatCard<Page>[] = [
  { label: "Total", compute: (rows) => rows.length },
];

Put it next to the articles descriptor above and the entire difference between the two screens is the difference in their data: pages drop the status column — they have no lifecycle status — and keep one stat card instead of two. No new component, no second grid, no fifth reimplementation of pagination. The skeleton was already written.

Make the data typo-proof

There's an obvious objection to declaring screens as data: a descriptor is just an object literal, and an object literal is where typos go to hide. I hit exactly that. A pages column read:

{ field: "routeSlug", header: "Route" }   // the real key is `slug`

It compiled, it ran, and the column was blank for every row — the grid looked up row["routeSlug"], got undefined, and rendered nothing. No error, no crash. A silent bug that looks like a backend problem.

The fix is to make the descriptor's keys answer to the row type. Instead of field: string, bind it to the row's own property names:

// the one line that changes in ColumnDef<Row>:
field: Extract<keyof Row, string>; // was: field: string

Now "routeSlug" is not a valid field — the compiler rejects it and lists the keys that are valid. The typo becomes a build error, not a runtime mystery. Columns that genuinely have no backing property — an actions button, a row number — opt out through a separate variant, so "no field" is a decision you can see, never an accident.

Data-driven only pays off if the data is typed as tightly as code would be.
A descriptor you can typo freely is just a config file waiting to rot.

That keyof binding is small, but it's what earns "screens as data" the right to be trusted — and it's the same generated row types (straight from the API's OpenAPI schema) doing double duty: rename a field on the backend and every stale descriptor turns into a compile error on the next codegen, instead of a blank column at runtime.

The discipline: name what you did not abstract

The failure mode of every framework is that it doesn't know when to stop. "Data-driven" slides into a configuration language that reinvents the host framework badly, and now you maintain a worse React inside React.

So the useful half of this work is the list of things I deliberately left concrete:

Column headers are plain strings — even though the localisation primitive already exists in the codebase (LocalizedText, a value that's either a literal string or a reference to a translation resolved per-locale). Wiring headers through it is a one-line type change — header: string becomes header: LocalizedText — one I'd make the day a second language becomes real, not a speculative layer baked in on day one.
Stat cards recompute on every render, over the currently-loaded page of rows. For an internal admin tool with modest datasets, a filter over a few hundred rows per render is free, and the simplicity is worth more than a memoised aggregate I'd have to invalidate correctly. That's an explicit data-driven over performance call, made with eyes open — and the kind of thing to revisit the day the numbers say to, not before.
The grid library shows through in the descriptor — flex, minWidth. I chose not to invent a layout abstraction over the top of it, because a leaky one costs more than the honesty of naming the underlying grid's own knobs.

Every one of those is a place a purist would abstract further. Each would have added a layer I can't yet justify with a real requirement. The boundary is only in the right place if you can say, out loud, where it ends.

The take-home

A screen declared as data is not automatically better than a screen written as code. It's better when you have many screens that share a shape and a team that will keep adding to them — which is exactly the situation an internal admin tool is always in, whether or not anyone admitted it at the start.

The two things worth stealing, independent of the stack: draw the boundary as a one-way dependency — domains depend inward on a framework that knows nothing about them — and validate it with a zero-diff refactor of a screen you already trust. If the screen changes, you've abstracted the wrong thing. If it doesn't, you've earned the next screen for the price of a descriptor.

If you've drawn this boundary somewhere different — or think the whole descriptor move is a trap — I'd genuinely like to hear where it broke for you. The door's open at cordata.tech/contact.

Originally published at cordata.tech.

DEV Community: László Hadházy

The descriptor survived, const did not — full-stack Rust

The only data in this post is metadata

The experiment

Result one: the boundary held for values and broke at structure

Result two: the locale broke the thing I thought was the point

What that actually cost, measured

The four questions the first post promised

The take-home

The trace_id was the easy part — Rust, OTel, Loki

Four things, not one

Trap 1 — a span that never closes is never exported

Trap 2 — builder.trace_id is a root-span-only answer

Trap 3 — .instrument() on the wrong future compiles and does nothing

Trap 4 — every span closed, and they all landed in one trace

The formatter, and the advice I gave myself that was wrong

Absent, not zero

And then the pipeline was wrong too

Trap 5 — a sampler that decides per span, not per trace

The take-home

One port, three client crates — a hexagonal Rust ingester

What actually makes it hexagonal

Why a driving port hands back a channel

Which pattern is doing which job

The port in full

Three adapters

The driven side, briefly

What the port bought us

What we deliberately did NOT abstract

Honest limits

The take-home

The catalog is the API — a governed mesh over MCP

§ 1 — Why the catalog, and not the warehouse

§ 2 — Tools or Resources — the decision everything else hinges on

§ 3 — The tool set, and where its schema comes from

The IBAN question, answered

§ 4 — The safety envelope is enforced, not annotated

§ 5 — Authorization is the governance plane

§ 6 — DSGVO (the GDPR) — what the design keeps out of scope

§ 7 — Retrieval over a closed set

§ 8 — What this design does not solve

What gets built next

Where this fits

One skeleton, many screens — React + TypeScript

A screen is a descriptor

Why this is a DDD boundary, not just a helper

The test: a zero-diff refactor

Make the data typo-proof

The discipline: name what you did not abstract

The take-home

Trap 2 — `builder.trace_id` is a root-span-only answer

Trap 3 — `.instrument()` on the wrong future compiles and does nothing