DEV Community: Bilaniuc Dragos

Did it check, or did it guess? Testing a tool whose output is written by an LLM

Bilaniuc Dragos — Tue, 28 Jul 2026 08:10:31 +0000

My first AI tool was ready, and in typical AI-era fashion, I rushed to publish it. After all, it worked. Or at least when I ran it, the output looked right to me.

But one scenario kept haunting me: I recommend it to a client, they run it and challenge one of its findings. Then they ask "What makes it trustworthy? Did it check, or did it guess? How do you know?"

I didn't have a good answer. "My prompt was really good" - not quite what you'd expect from a senior architect.

This article is about building a better answer, and how building it changes the tool itself.

The subject: one niche tool, one generic pattern

I built an AI tool for auditing calcification in production auth systems. Its purpose stems from a real-world problem I encountered while working on a large-scale platform, which I described in Part 1. After dealing with it, I formulated some generalized principles in Part 2, and based on those principles, I built the tool in Part 3.

Long story short: the tool is a skill published for Claude Code. It audits a codebase and evaluates how hard-wired the auth implementation is to its vendor's defaults, across multiple axes (token storage, refresh handling, provider coupling, authorization). It then runs an interview with the user for the judgment calls, and combines everything into its final deliverables. When this story begins, those deliverables were 2 markdown files: a full report and a one-screen summary, presenting the findings, the risks and the recommended next steps.

While the tool and the previous 3 articles are good for context, this article is meant to be a standalone piece on the issues you'd usually encounter when thinking about testing AI tools and skills. The auditor is just a convenient example, since its output mixes everything that makes AI hard to test: mechanical findings, judgment calls and prose.

If your tool makes claims about something checkable (a codebase, a document base, a dataset), the pattern is the same, and the vocabulary transfers: my enums are your classification fields, my file-and-line evidence is your citations, my coverage section is your scope statement.

Where v1 left testing

In the initial version, I built 4 adversarial fixtures, meant for testing recall (does it find what's there?), precision (does it stay quiet when nothing's there?), generalization (does the methodology travel across vendors?) and edge cases. These are described in more detail here.

I ran the tool against these fixtures and manually verified the results. These already provided a good feedback loop I could iterate on. After implementing the initial batch of fixes, I ran the skill against real codebases, and real code did what real code does: it found the gaps my fixtures hadn't imagined.

I caught the skill confidently lying about findings. The most memorable one: the tool's vendor knowledge documented how token storage is configured in v6 of the vendor's library, but the codebase ran v5, where storage is configured through a completely different shape. The model searched for the v6 pattern, found nothing, and reported "default storage: localStorage", while the cookie configuration sat in plain sight, in a shape the tool had never been taught. A confident claim of absence, wrong, on the most basic axis the tool reports on (full story here).

I also caught it behaving differently from run to run, especially when switching models: the same fixture rating a boundary signal as "present" on one run and "partial" on the next, or a smaller model missing a storage pattern that a bigger one caught.

A more thorough testing layer was already on the table, but these runs made its necessity all the more obvious.

Thinking about testability changed what the tool ships

Recall from the first diagram: the tool was initially designed to produce 2 markdown files, both following a consistent structure, but in freeform text. This makes mechanical testing nearly impossible. Any relevant check immediately requires another LLM call (LLM-as-judge), which means paying for every test run.

I wanted to maximize the value of free, mechanical testing, which meant changing the output of the skill into something more testable. The immediate solution: a structured JSON as the primary output. The skill now delivers 3 artifacts, the JSON being the canonical one, with the 2 markdowns rendered from the JSON alone.

There's a second win in this ordering, beyond cheap testing. The model now has to commit its claims (statuses, classifications, evidence) into strict fields before it writes any narrative around them. A confident story can no longer gloss over an unverified claim, because the claim got pinned down first, in a checkable format.

I also introduced a rule that completed the rendering setup: if the model notices mid-render that the prose wants to say something the JSON doesn't support, the fix goes into the JSON first, and the view gets re-rendered from it; the views never fork from the source.

This was not enough to enforce the "rendered from the JSON alone" rule, though. The render step runs in the same context that produced the analysis, so nothing physically stops a detail from leaking past the JSON into the prose. The airtight version would render the markdowns in a fresh session that receives only the JSON. I deferred it, opting instead for a cheaper mechanism that polices the same risk: an agreement check between the markdowns and the JSON, coming up with the harness.

Won't structure kill the value?

A genuine question to ask. The unique value of AI sits in its ability to generate answers that are hyper-personalized to real-world contexts. A clear structure removes some of the non-determinism that makes AI hard to test, but it also risks flattening the exact specificity that made the answer worth reading. The secret lies in how you design the JSON to account for this risk.

A bad structure makes AI force reality into predefined boxes, without considering whether reality can outrun them (spoiler: it almost always does). A better structure splits the output into registers:

Some of the output is claims: statuses, classifications, evidence references. Facts that fit a strict shape and can be asserted.
Some of it is explanation: why a finding matters, what it costs you if it stays. This part stays freeform, and no test ever asserts on it.
Some of it is synthesis: the headline, the overall posture. Written freely, but recorded in the JSON next to the findings it draws on, so even the editorial layer stays anchored to claims.

The contract becomes:

The markdowns may say more than the JSON, but never different.

Designing boxes, where boxes are warranted

The JSON structure enables the next move, which actually makes AI output testable: assertable enum fields. Some parts of the output can fit in boxes that we can later use to verify the accuracy of the tool after we change something in its instructions.

For the auth auditing tool, we can say an auth boundary is either present, partial or absent, which translates into the JSON field: authBoundary: "present" | "partial" | "absent". Similarly, auth token storage can be classified and fit in a box: tokenStorage: "vendor_default" | "builtin_selector" | "custom_adapter". (Field names simplified throughout; the real schema nests these per vendor.)

At this point, there is an important distinction to be made. The boxes (or enums) can be of 2 types:

Logically exhaustive enums, that cover all cases by construction: "present" | "absent", "low" | "moderate" | "high". A claim is one or the other. Safe to use directly in the JSON.
Empirical taxonomies are catalogs of patterns observed so far. From what we've seen, token storage can either be "vendor_default" | "builtin_selector" | "custom_adapter". But nothing guarantees the wild won't produce a fourth pattern. Forcing the AI model to pick one of these values anyway produces one of the worst kinds of failure modes: a structured lie that looks even more authoritative than a prose one.

To mitigate this risk, 2 new values are helpful:

"other", with the meaning "I understand this pattern, it's just not listed in the options". To keep the hatch honest, "other" comes with obligations: a required note field where the model describes the pattern in its own words, plus (like any determined verdict) at least one cited finding.
"undetermined", carrying the meaning "I couldn't tell". This one pairs with a required entry in the coverage section, the part of the report that lists what wasn't analyzed and why. An "I couldn't tell" that doesn't surface there would be a silent shrug; paired with a coverage gap, it becomes a signal the reader can act on.

The first line of enforcement sits in the schema itself, where breaking an obligation is a shape error:

"classification": { "enum": ["vendor_default", "builtin_selector", "custom_adapter", "other", "undetermined"] },
"allOf": [
  { "if":   { "properties": { "classification": { "const": "undetermined" } } },
    "then": { "required": ["note"] },
    "else": { "properties": { "finding_ids": { "type": "array", "minItems": 1 } } } },
  { "if":   { "properties": { "classification": { "const": "other" } } },
    "then": { "required": ["note"] } }
]

The two hatches also attach at different levels, because they cover different failure modes. "other" covers a vocabulary failure: the model understood the pattern, my enum didn't. It only belongs on empirical taxonomies. "undetermined" covers an epistemic failure: the looking itself failed. That one belongs on any verdict the model has to detect, even the logically exhaustive ones (boundary status, for example, is a complete vocabulary: present, partial or absent. It still carries "undetermined", because a complete vocabulary doesn't guarantee the model can always reach a verdict).

So, our previous case now becomes: tokenStorage: "vendor_default" | "builtin_selector" | "custom_adapter" | "other" | "undetermined". This way, our tool remains testable on the fixtures (we know the calcified fixture must always produce "vendor_default"; an "other" there is a test failure, since fixtures are built from known patterns), while staying prepared for the world wild web.

Receipts, search records, and DRY fields

Three more structural rules round out the JSON, each one buying a specific kind of checkability.

Presence claims carry verbatim-quoted references. Besides anchoring the model in the reality it's analyzing, these references can be mechanically checked with zero extra LLM calls: open the file, go to the line, compare the quote.

{
  "id": "boundary-cognito-authport-interface",
  "claim": "presence",
  "statement": "An `AuthPort` interface defines the auth boundary contract used by the new Cognito surface.",
  "evidence": [
    { "file": "src/auth/port.ts", "line": 6, "quote": "export interface AuthPort {" }
  ]
}

Absence claims carry checked_patterns, their search records, which makes them auditable. Not as easy to check as grep-ing the references from the presence claims, but still mechanically valuable: a test can assert that the record covers every alternative pattern the vendor profile lists, and for live runs we get a track record instead of a vague "nothing's there" report. This is the rule aimed straight at the localStorage lie from earlier. The failure mode isn't gone (a model can still miss a pattern), but it can no longer happen in silence.

{
  "id": "boundary-no-contract-suite",
  "claim": "absence",
  "statement": "No contract test suite exercises the `AuthPort` shape; the audited scope contains no test files of any kind.",
  "checked_patterns": [
    "*.test.ts / *.test.tsx",
    "__tests__/ directories",
    "runAuthContractTests-style suite name",
    "vitest.config.* / jest.config.*",
    ...
  ]
}

Normalization: every fact is stated once. Counts are computed from the arrays, a rank is the array order, no derived fields get stored. This is the LLM flavor of the everlasting principle of DRY (Don't Repeat Yourself): anything stated twice can self-disagree within the same document.

The resulting artifact was an audit-schema.json file that specified the structure of the JSON output in great detail. You can inspect it here.

The harness: 4 free layers in front of every paid run

With the output redesigned, the testing layer could take shape. The guiding principle: cheapest feedback first.

Before writing any checks, I produced a golden file: a hand-written example of what a perfect audit JSON looks like for one fixture. I used the old, manually-approved report and translated it backwards into the new schema. Backwards-translation paid for itself before a single line of harness code existed. It exposed issues on both sides, in the old report's data and in my new schema:

roughly half the anchors (code references) were broken, citing the wrong lines from the codebase;
real findings cite multiple locations, so evidence had to become an array;
some concerns simply don't arise in a given codebase, so not_applicable had to exist;

I addressed these initial findings. Then came the checks. 4 deterministic layers, stacked in front of any LLM involvement:

Shape. The audit JSON is validated against the exact schema file that ships with the skill.

// read from the skill package directly, never copied into the harness:
// the schema the tests enforce is byte-for-byte the schema that ships
const SCHEMA_PATH = join(HARNESS_DIR, "../skill/.../assets/audit-schema.json");

export function validateShape(doc: unknown): string[] {
  const schema = JSON.parse(readFileSync(SCHEMA_PATH, "utf8"));
  const ajv = new Ajv2020({ allErrors: true, allowUnionTypes: true });
  const validate = ajv.compile(schema);
  if (validate(doc)) return [];
  return (validate.errors ?? []).map(formatError);
}

Relational invariants. Rules that connect fields across the document: every referenced finding id must exist; every evidence quote must verify verbatim against the audited code; every "other" must carry its note and finding; every "undetermined" its coverage gap. The tool's non-negotiables (never fabricate human judgment, never produce a false all-clear) live here too, encoded as cross-field implications instead of polite prompt requests.

// I4: every evidence quote must verify verbatim against the audited codebase
for (const f of doc.findings) {
  for (const ev of [...(f.evidence ?? []), ...(f.context_evidence ?? [])]) {
    // ... cited file must exist, cited line must be within the file ...
    const lines = readFileSync(join(codeRoot, ev.file), "utf8").split("\n");
    if (!lines[ev.line - 1].includes(ev.quote)) {
      const hits = lines.flatMap((l, i) => (l.includes(ev.quote) ? [i + 1] : []));
      err(
        `I4 ${f.id}: quote not at ${ev.file}:${ev.line}` +
          (hits.length ? ` (found at line [${hits}])` : " (found nowhere in file)"),
      );
    }
  }
}

// I6: one of the non-negotiables, encoded as a cross-field implication
if (interview === null && lik !== null) {
  err(`I6 axes.${axis}.likelihood is '${lik}' but interview is null (fabricated human axis)`);
}

Fixture expectations. Declarative ground truth: 87 assertions across the 4 fixtures, written in a tiny vocabulary (equals, oneOf, isNull, some), stating what each fixture must produce. The calcified fixture must classify storage as "vendor_default". And my favorite: 2 of the fixtures have byte-identical app code, differing only in the vendor adapter, so all 13 of their enum-level verdicts must be equal. "The methodology travels across vendors" used to be a marketing claim; now it's a permanent test.

{
  "fixture": "calcified-cognito",
  "assertions": [
    { "path": "axes.storage.per_vendor.<v>.classification", "equals": "vendor_default",
      "note": "THE original-bug axis: default localStorage, no custom adapter" },
    { "path": "boundary.<v>.b4_client_server_split.status", "equals": "not_applicable",
      "note": "no SSR surface — must be not_applicable, never a fabricated gap" },
    { "path": "backlog", "isNull": true, "note": "non-negotiable: no ranking without a human" },
    ...
  ]
}

(<v> resolves to the detected vendor's id at evaluation time; vendor ids are model-chosen and unstable across runs, so the paths never hardcode them.) The portability test is a spec file of its own:

{
  "type": "structural_equality",
  "fixtures": ["bounded-cognito", "bounded-auth0"],
  "paths": [
    "boundary.<v>.b1_anti_corruption.status",
    "axes.storage.per_vendor.<v>.classification",
    "axes.authorization.per_vendor.<v>.api_token_type",
    ...
  ]
}

Cross-format agreement. The markdowns checked against the JSON: every file:line link in the views must resolve to evidence recorded in the JSON, the metadata must match, a disclaimer may only appear if the JSON says it fired. This is the mechanism that gives the JSON jurisdiction over what humans actually read (and the promised policing of "rendered from the JSON alone").

// every evidence anchor recorded in the JSON, as "file#line"
const anchors = evidenceAnchorSet(doc);

for (const { display, href } of extractLinks(md)) {
  const hrefLine = href.match(/^([^#]+)#L(\d+)$/);
  if (!hrefLine) continue; // whole-file links carry no line claim
  const [, hrefFile, line] = hrefLine;

  // a line-anchored link is a claim: its anchor must be backed by a JSON
  // evidence line, regardless of how the display text reads
  if (!anchors.has(`${cleanPath(hrefFile)}#${line}`)) {
    err(`A1 ${name}: anchor ${hrefFile}:${line} is not backed by any evidence anchor in the JSON`);
  }
  // ... display/href consistency is checked only when the display parses as a path:line ...
}

All 4 layers run in about 2 seconds, cost nothing, and are wired into a pre-commit hook. Editing a fixture's source code now breaks the commit if it invalidates any committed evidence anchor.

Live runs sit on top of this. The runner copies a fixture into a fresh workspace, invokes the actual installed skill headlessly, harvests the 3 artifacts and pushes them through all 4 layers, with the quotes verified against the exact code the model just audited. In the case of the auth auditing tool, one live run costs about $3 and 8 minutes. That number is the reason the free layers exist: with regular software you run the suite on every save; when every run sends you a bill, you want most of your testing to happen with no model in the loop.

One bootstrapping problem remains: who checks the checker? A suite that has only ever shown green proves nothing; maybe it approves everything. So before trusting it, I fed it mutants: deliberately corrupted copies of the golden file, one per failure class, and asserted that the tests go red when they should. Each mutant also declares which layer must catch it; a catch by the wrong layer fails the run too, so the layering itself is tested.

const MUTATIONS: Array<{ name: string; layer: Layer; apply: (d: Doc) => void }> = [
  {
    // the committed report's actual bug class: an off-by-N anchor
    name: "wrong_line",
    layer: "invariants",
    apply: (d) => (d.findings[6].evidence[1].line = 24),
  },
  {
    // 'high' is a legal enum value; the fabrication is only visible
    // cross-field (no interview to source it from). Shape-valid on purpose.
    name: "fabricated_likelihood",
    layer: "invariants",
    apply: (d) => (d.axes.storage.likelihood = "high"),
  },
  {
    // escape hatch used silently: 'other' requires a note
    name: "other_without_note",
    layer: "schema",
    apply: (d) => {
      d.axes.storage.per_vendor.cognito.classification = "other";
      delete d.axes.storage.per_vendor.cognito.note;
    },
  },
  // ... 21 mutants in total
];

for (const { name, layer: expected, apply } of MUTATIONS) {
  const doc = structuredClone(golden);
  apply(doc);
  // ... run the full checker against the mutant ...
  if (!caught) console.log(`MISSED       ${name}: mutant passed the full check`);
  else if (actual !== expected) console.log(`WRONG LAYER  ${name}: expected ${expected}, caught by ${actual}`);
  else console.log(`CAUGHT [${expected}] ${name}`);
}

This practice turned out to be especially useful because, as we shall see, writing tests for an AI tool is much more iterative than in traditional software development. A failing test doesn't automatically mean the instructions need to be fixed; often it's the test itself that has to change. And when tests change this often, a suite that tests the tests stops being a luxury. The mutants proved it in an unexpected way: when the schema later evolved, one mutant's "wrong" value silently became the new correct one, and the run flagged it as MISSED. Mutants encode the current contract and rot when the contract moves; the MISSED detector is what makes them safe to rely on.

The pattern behind the layers

Notice where all the mechanical power comes from: the places where the model's output touches reality in a verifiable way. A quote either matches the file or it doesn't. A link either resolves to recorded evidence or it doesn't. A search record either covers the profile's patterns or it doesn't.

That's the generalizable move. Whatever your tool produces, find the spots where its free text makes contact with something verifiable (citations, quotes, identifiers, dates, links) and put your assertions there. You get deterministic tests inside prose you deliberately left free.

If the output of your tool has no such contact points, treat that as a design smell in the output itself, and add some: a source citation per claim, an input-document quote per extracted fact, an id or URL per referenced record. The receipts pull double duty: they discipline the model while it generates (a claim that must carry its source is harder to invent), and they hand you deterministic checks afterwards.

Across the fixture campaign's live runs, this one mechanism verified 72 out of 72 evidence quotes verbatim against the audited code. Zero fabricated anchors.

Who's wrong: the test or the tool?

Here's where testing AI diverges hardest from the TDD reflex. In regular software, the test is the fixed point: 2 + 2 equals 4, the assertion is correct, and if it fails, the code moves. With an AI tool, 2 + 2 sometimes comes back as "four". Or "quatro". A failing test no longer means the tool is wrong; it means the test and the tool disagree, and someone has to judge who's right. Updating the test stops being an admission of sloppy test-writing and becomes part of the design process itself.

To avoid the case where we fiddle with the tests until they pass, we need discipline in putting a clear verdict on every red run. Mine kept landing in one of 4 categories. Here they are, with real examples from the auditing tool:

The test was a guess. I had put maximum-length caps on the JSON's prose fields, sized by imagination against a single-vendor mental model. The 2-vendor fixture blew through them with dense, legitimate justification text. Verdict: the caps were guesses, calibrated by the wrong fixture in my head. Raised, with sharper field descriptions.
The checker was naive. Another test was built around the skill never estimating durations. Time estimates belong to the maintainer, that's one of the non-negotiables, so the harness flags any duration in the output. It flagged "24 months", but the string sat inside the skill's own interview question to the maintainer: "Is RBAC in your plan over the next 24 months?". Verdict: the rule was right, the checker applied it to the wrong voice. The no-durations rule binds the skill's voice; the human's recorded answers and the audited code's own strings (tokens legitimately expire in "30 days") are exempt.
The model misbehaved. Findings can be reused as evidence across different sections of the audit, and the schema originally required each finding to declare which sections cite it. The model kept reusing findings correctly while forgetting to declare the cross-reference. Verdict: tighten the prompt.

I improved the instructions and re-ran: the violations dropped from 4 to 2. Better prompting, still failing. Staring at that stubborn cross-reference bug, I realized it falls into a new category:

The schema generated the failure. The finding's declaration was a second copy of a relation the document already encoded (which sections cite the finding). Remember the normalization rule from above: I had banned stored counts and rank fields precisely because 2 copies of a fact can disagree. But then I stored this relation twice and built an invariant whose only job was checking that my 2 copies agree. The model wasn't failing the check; the schema was generating the possibility of failure. Verdict: delete the field. The relation now exists in exactly one place, the invariant shrank, and the failure became unrepresentable. No amount of prompt engineering was ever going to fix this reliably; the flaw was in the schema.

The rule that keeps the testing iteration process honest: a test only gets loosened after a verdict, never for convenience. Out of the 87 fixture assertions, exactly 1 is an allowed-set instead of an exact match: a boundary signal where 2 runs produced "present" and "partial", and re-reading the fixture showed both readings were defensible. That's the entire tolerance budget. By the end of the fixture campaign, every failure had landed in one of the 4 buckets.

Breaking it on purpose

By this point the suite was green across all fixtures, which is exactly when a testing layer is most suspicious. A net that has never caught anything is indistinguishable from a net with a hole in it. So I ran a live demolition drill.

I took the verbatim-quote rule, the flagship honesty mechanism, and inverted it in all 6 places it appears in the skill's instructions ("quotes are readable paraphrases" instead of "quotes are verbatim"). Then I ran the free test suite.

All of the mechanical tests were green, as expected. Remember: the mechanical layers check artifacts (the output JSON and MDs), not the instructions of the skill. Since the skill didn't run yet, no output artifacts were modified.

Then came the live run that actually changed the output. Quote verification failed on all 35 evidence anchors across all 17 findings. "Found nowhere in file", 35 times. The flagship check caught the flagship failure class on its first real opportunity.

That's the division of labor to remember. Static checks guard the artifacts you already committed; only a live run can tell you whether the instructions still produce honest artifacts. Prompt-level rules are enforced at run time, or they're enforced nowhere.

The real-world run

Fixtures can only take you so far. They're small by necessity (imagine committing a production-grade system as a subfolder of your test suite), and they're built from patterns I already knew about, which is precisely their limitation. The localStorage story taught me that real trouble comes in shapes nobody taught the tests. So once the fixture suite was green and drilled, I went looking for real code, and I used the AI itself to scout public repositories for a good candidate: production-flavored, actively maintained, on a current SDK version, exercising surfaces my fixtures couldn't (a real client/server split, among others).

Then I ran the whole thing the way a stranger would: installed the published plugin, pointed it at the cloned repo, ran the audit. After the audit finished, I ran the harness against the real-world output. Same schema check, same invariants, same quote verification, now against a codebase the tool had never seen. The run produced 14 findings, all of them verified verbatim.

2 things in this run were new. First, the "other" hatch fired for the first time, and fired correctly: the app constructs no Authorization header at all (token handling sits inside the vendor's data client), a real pattern outside my authorization enum. The model picked "other" and described the pattern in the note instead of rounding it to the nearest listed value. On a fixture that would be a failed test; on production code, it's the design doing its job.

Second, the agreement layer needed one last calibration. It reported 26 violations; 25 turned out to be the checker's fault (it assumed every link display is a path:line, while the model was writing richer displays over correctly backed anchors), and after the fix, the 1 that remained was genuine: a link pointing 1 line away from its recorded evidence. The tool's own honesty machinery, catching the tool's own imprecision, on real code.

After this last phase of testing and fixing, I could finally be a little more confident in the tool.

Key takeaways

Design the output for testing, and the tool improves anyway. A canonical JSON with views rendered from it forces the model to commit claims before writing the story around them.
Split the output into registers. Assert the claims, leave the explanations free, anchor the synthesis. Structure where testing needs it, freedom where the value lives.
Only close an enum that's closed by construction. Empirical taxonomies need "other" and "undetermined", and each hatch needs wired-in obligations (a note plus evidence, a coverage gap) so honesty stays cheaper than invention.
Assert where the output touches reality. Quotes, references, search records: deterministic checks living inside freeform prose.
State every fact once. When a model keeps failing to keep two copies of a fact consistent, stop storing two copies.
Put free layers in front of paid runs. Schema, invariants, expectations, agreement on every commit; the $3 live runs only for what the free layers can't see.
Mutation-test your checks. A suite that has only ever passed is unproven, and mutants rot when the contract moves, so detect that too.
Give every failed test a verdict. Guessed test, naive checker, model behavior or schema flaw. Loosen nothing without one.
Know how each rule is enforced. Prompt-level rules only exist at run time; static checks can't see them. Test each guarantee at the level where it actually lives.

Closing

The skill is open source, and everything described here (the schema, the harness, the fixtures, the mutants, the decisions log this article was distilled from) is in the repository, runnable end to end.

I started this because I couldn't hand people a tool I had only eyeballed. I ended up with a tool that is structurally harder to lie with, and with a testing layer I would now build, in some form, for any AI tool meant to leave my machine.

If you're building one of those, picture the moment this article grew out of:

A client challenges one of your tool's outputs. What can you show them, beyond "it looked right to me"?

If your team is building AI tooling and wants that kind of trust designed in, this is the sort of work I take on through Luckylabs. DMs open.

I designed an AI auth-auditing tool around honesty. On its first real run, it confidently lied to me.

Bilaniuc Dragos — Thu, 25 Jun 2026 11:24:49 +0000

How I rebuilt it so honesty and reliability were a structural constraint, not just a prompt request.

It lied to me on the first real run

I spent days building an AI auth-auditing tool that detects what it can verify and refuses to guess at the rest. The whole point was to make it reliable, grounded, and honest. The first time I ran it on real production code, it told me with no hedge that the app stored its auth tokens in localStorage.

It doesn't. That frontend keeps them in cookies. I configured it that way myself.

My next thought was the obvious one: it said localStorage, but I put them in cookies, so what else is it lying about? A tool that's confidently wrong on the basics is worse than no tool, because it spends your trust and gives you nothing safe in return. Making it truly reliable became the priority.

This piece is about why it lied, the risk in building tools like this, and how I designed around that risk. The fix had little to do with the prompt and almost everything to do with the structure around it.

How I got here

This is the third piece in a series on auth architecture in a large-scale production system. Part 1 laid out the consequences of calcification on a large auth system, which made a proper rewrite too risky and costly. In that context a deliberate, documented acceptance of a flawed state, paired with foundation work, was preferable. Part 2 turned that into a defense: own the boundary and the volatile runtime behaviors instead of inheriting the vendor's, and noted that the coding agents which generate the calcified version by default are also the cheapest way to build the resilient one.

Part 3 takes that seriously. I built a skill that reads a codebase and reports how hard-wired its auth is to a vendor's defaults, and what a future change would cost. The bar was high because the skill ships publicly alongside this series, and my gold standard was a skeptical senior engineer reading through the output of the skill and trusting it.

What the tool does

It audits one seam: the app-layer code that talks to your identity provider (Cognito, Auth0, or whatever you use), not the infrastructure or the gateway.

A system is calcified when a change that should be local becomes a cross-cutting rewrite. Swapping token storage, changing refresh, or replacing the provider should each touch one place; in a calcified system each touches dozens of call sites, making any meaningful changes to the system risky and costly.

The trouble is that code findings aren't enough. Calcification has to be evaluated in the real-world context of the project, its priorities, and the organization. That's why the tool needs to excel at detecting and classifying, but, more importantly, understand what it needs to escalate to human judgment — and do it!

Four non-negotiables enforce it:

never fabricate the human axes;
never produce a false all-clear, where "no finding" has to mean "looked and found nothing";
report and propose, don't rewrite auth code;
assess changeability, not security.

Stating them in the prompt was never going to be enough; the structure of the skill had to make them the path of least resistance, which is most of what the rest of this article is about.

Avoiding calcification in an anti-calcification tool

There was one more constraint, and it turned out to be the series eating its own cooking: no vendor could be hard-coded. Provider knowledge lives in one markdown file per vendor, and the engine never names a vendor, so adding a third is writing one more file. Hard-coding Cognito into the core would have been the calcified version of my own anti-calcification tool.

A profile is just facts the vendor-agnostic core reads:

## Token storage seam (vendors/amplify-cognito.md)

- Default storage: localStorage (in-memory fallback)
- Custom storage API: cognitoUserPoolsTokenProvider.setKeyValueStorage(storage)
- Look-alikes (NOT custom): built-in defaultStorage / sessionStorage / new CookieStorage()

I tested it before trusting it

"I tried it once and it looked right" was not an acceptable stance, so before running it on anything real I built adversarial fixtures for distinct failure modes. A fully calcified codebase tests recall (does it find what's there?), a cleanly bounded one tests precision (does it stay quiet instead of inventing findings?), and the same bounded app on a second provider tests generalization (does the methodology travel?). I made generalization structural: the app-layer files in the two bounded fixtures are byte-for-byte identical (verified with diff), and only the adapter differs.

The fourth fixture is the interesting one. It stresses four things at once: two providers in the same repo, an unknown provider with no profile (which has to surface as a coverage gap, not vanish), a mid-migration boundary (the new side bounded, the legacy side calcified), and the look-alike storage trap. That last one is a built-in selector that reads like a custom storage adapter but isn't:

// Looks like owned storage. Isn't.
// A built-in selector chooses where tokens persist; it is not a custom adapter you own.
cognitoUserPoolsTokenProvider.setKeyValueStorage(sessionStorage);

Crediting that as "owned storage" would be a confident, plausible, wrong finding. Running the fixture also forced two rules I had missed: assess multi-vendor and partial-boundary codebases per vendor instead of collapsing them to one verdict, and treat a real boundary with internal gaps as exactly that.

Then it lied on the first real codebase

The failure didn't come from a fixture. It came from real code, on the most basic axis, in a case the fixtures hadn't covered. The Cognito profile documented how token storage works in Amplify v6, where you override it by calling setKeyValueStorage:

// What the profile knew to look for (Amplify v6):
cognitoUserPoolsTokenProvider.setKeyValueStorage(storage);

The codebase was Amplify v5, where you don't call that function. Storage is a field in the config object, a completely different shape:

// What the codebase actually did (Amplify v5):
Amplify.configure({
  Auth: { cookieStorage: { domain: ".example.com", secure: true, expires: 7 } }
});

The model searched for the v6 pattern, found nothing, and concluded "default storage, localStorage." The cookie configuration was right there in the file, in a shape the profile had never been told about. One missed pattern, one confidently wrong "no finding," and it was the exact failure the tool exists to prevent, happening inside the tool.

When I fixed it, I checked Amplify's source rather than trusting memory: the default is localStorage in both versions. So the tool wasn't wrong about the default. It was wrong because it never noticed this codebase had overridden the default through a shape it didn't recognize. "The default is localStorage" and "this codebase uses localStorage" are completely different claims, and the distinction is subtle but the most valuable thing the tool can get right.

That points at the real lesson: the tool didn't lie because it was careless, it lied because the task is hard. Storage alone has two completely different shapes across two versions of one vendor; multiply that across four axes, several providers, and the open-ended variety of real codebases, and you get a huge space where "I found nothing" can mean "truly absent" or "here in a form I wasn't taught." That space is where confident fabrication lives, and the harness has to scale with it. You can't close the gap by asking the model nicely.

Designing for honesty instead of asking for it

A line in the prompt saying "be accurate, don't fabricate" does almost nothing; a model under pressure to produce a useful-looking report will produce one. The work was making fabrication structurally hard.

A small example sets the pattern. I started with "all findings should be verifiable," a wish. It became operational: every finding must carry a link to the exact file and line of evidence, one the reader can click. To produce that link the model has to have found the line, so verifiability stopped being a request and became something the output has to contain.

The same principle drives the rule that keeps the mechanical pass honest. A text match is a candidate, never a finding:

Locate: use the profile's identifiers to find candidate locations.

Confirm: open each candidate and read the surrounding code. A vendor type inside the adapter is correct; the same type in app-layer code is a leak.

The textbook case is a wrapper that looks like a boundary and isn't. The search finds it, and a shallow pass calls it owned; only reading the return type shows the vendor's shape passing straight through to every caller:

// Looks like a boundary. Isn't.
export async function getSession(): Promise<AuthSession> {
  return fetchAuthSession(); // returns Amplify's session type, unchanged
}

The rules also live in the shape of the output. The report template makes the load-bearing sections mandatory: a Coverage section stating what was and wasn't analyzed, so "no finding" can't read as "clean"; per-finding evidence so every claim is checkable; and a "Judgment calls for you" section collecting the questions the audit refused to answer. A model can still misbehave, but it has to do so against the structure, which also hands the reader a way to catch it: if the Coverage section admits it skipped a file you know is full of auth logic, that gap is itself a signal you can act on.

The interview: from evidence to meaning

In its first form the tool stopped at the evidence: finding X, finding Y, each with a file and line. Accurate, and not useful, because the reader still has to turn "you read claims inline in fourteen places" into "and for us, that means something." Having the model make that leap is where it starts inventing, because meaning depends on what it can't see: your roadmap, your org, the contracts that depend on the current choice.

So I put an interview between the evidence and the meaning. The tool presents what it found, grouped by axis, then asks the questions only the maintainer can answer, one at a time, with an explicit "don't know" that routes that axis into "Judgment calls" rather than forcing an answer. These were the guiding questions, adaptable to different contexts:

Token storage: "Is a storage change on the table, like a move to HttpOnly cookies?"

Refresh: "Is a change to refresh coming, often downstream of a storage move?"

Identity provider: "Is a provider swap realistically possible in the next year or two? Roughly how many call sites would it touch?"

Authorization: "Is an authorization-model change planned (RBAC, ID to access token)? What backend contracts depend on the current choice?"

Their answers supply likelihood, the mechanical pass supplies the cost evidence, and only together do they produce a ranking, with every rank traceable to which input was the human's. With no human in the loop, the tool refuses to rank and stops at the findings and open questions. The refusal is a feature.

Building the interview into the skill changed how it analyzes, not just how it reports. Because the run is oriented toward an interview from the start, the mechanical pass is already asking "what here is a judgment call I will have to escalate, and what is the most useful question to ask about it?" instead of "what answer can I produce?" A model planning to ask is a model less inclined to invent; the interview is an output stage and an anti-fabrication mechanism at once.

The broader lesson holds past auth: between a finding and its meaning is a set of things only the human knows, and designing the tool to ask for them sharpens its own detection too.

Distilling without diluting

Running it on real codebases surfaced a quieter failure: the reports were thorough and far too long to use. A valuable finding would sit in paragraph nine, indistinguishable from six routine ones. A finding nobody reads might as well be one the tool never made.

So I added a one-screen summary, distilled from the full report rather than written separately. That ordering matters more than it looks. Because the full report already carries the harness (the coverage section, the per-finding file links, the refusal to score what only the human knows), the summary inherits that reliability and doesn't have to re-earn it. So the summary can play by lighter rules: focus on finding and presenting the most relevant aspects of the report, instead of validating and double-checking.

With the right structure, you can get both the actionable clarity of a concise summary and the proven reliability of a full report behind it.

## Auth Calcification Summary: <repo>

Vendor: Amazon Cognito (Amplify v5). Boundary: a leaky facade, not a real seam.

Most important: RBAC is on your roadmap, but the API is authorized with the ID token
while permissions arrive in the access token. The plan has an unnamed dependency.

| Axis          | Today                 | If you change it                          |
| ------------- | --------------------- | ----------------------------------------- |
| Token storage | v5 cookieStorage      | Low, once the facade becomes a real seam  |
| Refresh       | Inherited from vendor | Medium; no single-flight, no failure path |
| Provider      | Cognito-specific      | High; groups/claims read in 14 call sites |
| Authorization | ID token, inline      | High; blocks the RBAC move above          |

Full evidence and file links are in the report.

Where it is now

The skill is live, open source, and installable as a Claude Code plugin. The repository is here.

Add the repo as a plugin marketplace, install from it, then point Claude at a project:

# in Claude Code (exact names are in the repo README)
/plugin marketplace add dragosbln/auth-calcification
/plugin install auth-calcification-audit@auth-calcification

From there, ask it to audit your auth layer. It runs the mechanical pass, asks you the short interview, and writes three things: the full report, the one-screen summary, and an actionable backlog ordered by priority using your answers.

Right now it ships with Cognito and Auth0, but the ports-and-adapters design means you can extend it. Clone the repo, hand references/vendor-profile-schema.md and your provider's docs to Claude Code (or your AI IDE) to draft a new profile, drop it in vendors/, and run the skill from your fork. No core changes; that's the whole point of the split. If you do, I'd welcome the findings, the misses, and the suggestions.

Key takeaways

The harder and broader the task, the more the harness matters. Confident fabrication concentrates where the work is complex and the inputs vary.
Prompting for honesty isn't enough. A clickable file-and-line link, a mandatory coverage section, and a refusal to rank without human input are structure, not requests.
Don't fabricate judgment; escalate it. Detect mechanically, then hand back what only the maintainer can answer.
Tools over-produce, so distillation is part of the product. A correct finding buried on page three fails like a wrong one.
An interview built into the tool disciplines the model as much as it informs the human.
The boundary discipline you would apply to auth applies to the tool itself: keep the vendor-specific parts swappable.

Conclusion

Every one of these fixes came from the same root: a tool that has to reason about a wide, nuanced space will, sooner or later, hit a corner it wasn't taught and answer anyway, with total confidence. You don't fix that by asking it to be honest; you fix it with a harness that makes the honest output the easy one to produce, and the harness has to scale with the task, because that complexity is exactly where fabrication breeds.

In the agentic era, the typing gets outsourced and the judgment doesn't. The mechanical work that used to make teams skip the boundary is nearly free now; what's left, and what's worth your attention, is the judgment about what any of it means. A good tool automates that detection, escalates the judgment, and presents the findings with clarity — baked into its structure, not asked for.

If your team is about to make an expensive-to-reverse decision in auth, cloud, or platform architecture and wants an independent second pair of eyes, that's the kind of work I take on through Luckylabs. DMs open.

Keep the steering wheel: 3 ways to future-proof your auth

Bilaniuc Dragos — Fri, 12 Jun 2026 09:36:17 +0000

Your auth library's defaults quietly become your architecture. Here's how to stay in control while still renting the hard parts from people who specialize in them.

Auth is the ultimate hot path

Most components in your system sit on one critical path. Authentication sits on three at once.

It's the entry point: the first thing a user touches, where friction costs you people before they're even inside the product. It's on the critical path of every authenticated API call: every request carries a credential, so auth's latency, reliability, and cost are levied on all of them. And it's the highest-value security target in the system, the one component where a single flaw doesn't degrade something — it compromises everything.

Picking the right vendor for your requirements matters, obviously. But even with the ideal vendor, there's a hidden risk. When you adopt a vendor auth library, you outsource the implementation, which is good — you should. But you also silently outsource the architecture. The library's defaults become load-bearing walls you never decided to build. By the time you want to move one, you discover the whole house is resting on it.

A note on scope before we start: this piece is about the client side of auth — SPAs and micro-frontends, token storage, refresh, the seam between your frontend code and a managed identity provider. The principles travel, but the examples live in the browser.

How auth calcifies

In a previous article I walked through a real situation: a penetration test flagged that our auth tokens were stored where client-side JavaScript could read them, and we deliberately accepted the vulnerability rather than fixing it immediately. Not out of negligence. The auth system hadn't been built with change in mind, so what should have been a configuration change had calcified into a months-long re-architecture touching multiple micro-frontends and 100+ backend services that consume those tokens.

I made a claim in that piece: that "do nothing for now" actually meant doing a lot of quiet foundation work — the kind that turns a future migration from a re-architecture into a flipped switch.

This article is that foundation work: three ways to keep auth changeable. It applies whether you're starting greenfield, where building it in costs almost nothing, or running a large production system, where you do it proactively, in the windows where you have bandwidth, ahead of a move you can see coming. In the second case there's a bonus: making auth changeable forces you to understand the system you already have, in ways that pay off well beyond auth.

Way 1: Own the boundary (and make it more than a facade)

Create your own auth interface — but make sure it isn't a leaky facade. That's the common trap: a lib/auth.ts that re-exports the vendor's functions but still returns the vendor's session object, throws the vendor's errors, and uses the vendor's claim names. The vendor's shape passes straight through to every call site. You've added a file, not a boundary.

A boundary that actually future-proofs you does three things a facade doesn't.

1. It's an anti-corruption layer. It speaks your domain's language — your Session, Principal, AuthError — and the vendor's types never cross it.

// ❌ Leaky: the vendor's shape escapes into every caller.
export const getSession = () => fetchAuthSession(); // returns Amplify's AuthSession

// ✅ Bounded: callers only ever see your types.
export interface AuthPort {
  getAuthHeaders(): Promise<Record<string, string>>;
  getPrincipal(): Promise<Principal | null>;
  onRefresh(): Promise<boolean>;
  onSessionExpired(): void;
}

2. Capabilities are injected, not imported. The common wrapper is an imported singleton, so every call site is hard-wired to one implementation. That makes it impossible to swap per context and untestable without mocking modules.

// ❌ Imported singleton: every call site hard-wired to one implementation.
import { getToken } from "@/lib/auth";
const res = await fetch(url, { headers: { Authorization: `Bearer ${await getToken()}` } });

Inject the capability instead. Now you can swap it per execution context (client, server, test) and stand up your whole data layer against a fake in tests, with zero network:

// ✅ Injected: the client depends on the AuthPort contract, not a concrete import.
export function createApiClient(auth: AuthPort) {
  return async function apiFetch<T>(url: string, init: RequestInit = {}): Promise<T> {
    const call = async () =>
      fetch(url, { ...init, headers: { ...init.headers, ...(await auth.getAuthHeaders()) } });

    let res = await call();
    if (res.status === 401 && (await auth.onRefresh())) res = await call(); // refresh → retry
    if (res.status === 401) { auth.onSessionExpired(); throw new Error("Session expired"); }
    return res.json();
  };
}

(What onRefresh actually does is the subject of Way 3. For now, notice that it's a slot. The call sites don't know or care.)

3. The boundary is contract-tested. This is the part almost nobody does, and it's what makes "future-proof" verifiable instead of aspirational. Because the boundary is an explicit interface, you write one conformance suite, and every adapter you ever write must pass it:

// auth-port.contract.ts
// Run the SAME suite against every adapter: Amplify today, Auth0 tomorrow,
// the in-memory fake your tests use forever.
export function runAuthContractTests(
  name: string,
  makeAdapter: (scenario: TestScenario) => AuthPort,
) {
  describe(`AuthPort contract: ${name}`, () => {
    test("returns a bearer header when authenticated", async () => {
      const auth = makeAdapter({ tokenState: "valid" });
      const headers = await auth.getAuthHeaders();
      expect(headers.Authorization).toMatch(/^Bearer /);
    });

    test("returns no principal when unauthenticated", async () => {
      const auth = makeAdapter({ tokenState: "absent" });
      expect(await auth.getPrincipal()).toBeNull();
    });

    test("recovers from one expired token via onRefresh", async () => {
      const auth = makeAdapter({ tokenState: "expired", refreshOutcome: "success" });
      expect(await auth.onRefresh()).toBe(true);
      const headers = await auth.getAuthHeaders();
      expect(headers.Authorization).toMatch(/^Bearer /); // new token, same contract
    });

    test("signals expiry exactly once when refresh fails", async () => {
      const onExpired = vi.fn();
      const auth = makeAdapter({
        tokenState: "expired",
        refreshOutcome: "failure",
        onSessionExpired: onExpired,
      });
      expect(await auth.onRefresh()).toBe(false);
      auth.onSessionExpired();
      expect(onExpired).toHaveBeenCalledTimes(1);
    });

    test("dedupes concurrent refreshes into a single underlying call", async () => {
      const scenario: TestScenario = { tokenState: "expired", refreshOutcome: "success" };
      const auth = makeAdapter(scenario);
      await Promise.all([auth.onRefresh(), auth.onRefresh(), auth.onRefresh()]);
      expect(scenario.refreshCallCount).toBe(1); // see Way 3
    });
  });
}

// Then each adapter is one line of registration:
runAuthContractTests("amplify", (s) => makeAmplifyAdapter(s));
runAuthContractTests("in-memory fake", (s) => makeFakeAdapter(s));

This changes what a migration is. It stops being "swap it and pray across 100 call sites" and becomes "make the new adapter green." Your migration now has a definition of done — and your test fake is held to the same standard as production, so green tests against the fake actually mean something.

The boundary is the shape of future-proof auth. The next two ways are about drawing it correctly and filling in its hardest piece.

Way 2: Know your tools deeply enough to replace them

You can't draw a good boundary around something you don't understand. And you don't understand an auth library by reading its quickstart — you understand it by extending it until its machinery is visible.

One high-leverage exercise: replace its token storage. Even if you only reproduce what the library already does, the act forces every question that matters. What's holding my tokens right now? Where else could they live — localStorage, a cookie, an HttpOnly cookie, memory? What are the tradeoffs? How would I switch later? Done with a clean, well-documented adapter, the result is a concise artifact your whole team benefits from: the storage mechanism explicit and in front of you, instead of buried in vendor docs and guesswork.

Every serious library exposes this seam, and learning where is half the lesson:

// Amplify v6 — pluggable key-value storage for tokens.
import { cognitoUserPoolsTokenProvider } from "aws-amplify/auth/cognito";

cognitoUserPoolsTokenProvider.setKeyValueStorage({
  async setItem(k, v) {/* cookie? memory? your call */},
  async getItem(k) {/* ... */ return null;},
  async removeItem(k) {/* ... */},
  async clear() {/* ... */},
});

Auth0's SPA SDK takes a custom cache implementing get/set/remove.
Firebase lets you select among persistence strategies (browserLocal, browserSession, indexedDB, inMemory) via setPersistence.

Different surfaces, same underlying seam. Find it in one library and you know what to look for in the next.

Storage is just the lead example. The same "extend it to understand it" move applies elsewhere: swapping in a custom HTTP client, hooking the token-refresh cycle, subscribing to the library's auth lifecycle events, plugging in a custom credentials provider. Each one makes a different part of the machine visible.

The deliverable isn't the adapter. It's the understanding — the thing that lets you write an adapter you trust, and recognize in advance which of the vendor's defaults are about to become your walls.

Way 3: Own your critical runtime behaviors, starting with refresh

Token refresh is where vendor magic most quietly becomes load-bearing. In our Amplify setup, calling fetchAuthSession() refreshed tokens silently and automatically. Very convenient — until you look closer and notice the convenience depends on the tokens living in storage the browser's JavaScript can read, the same client-accessible storage that's open to XSS exfiltration. (That's the root problem behind the first article.) The moment you move tokens into HttpOnly cookies to close that hole, client code can no longer read them, and the silent client-side refresh you built everything on stops working. All at once, everywhere.

Own the behavior instead, so a change like that touches one place. Refresh has three parts worth understanding rather than inheriting: when you refresh, how you avoid stampedes, and what happens when it fails.

The robust shape is reactive — refresh on a 401, then retry — with single-flight deduplication, so that ten concurrent 401s trigger one refresh, not ten:

let inflight: Promise<boolean> | null = null;

export function refreshOnce(doRefresh: () => Promise<boolean>): Promise<boolean> {
  inflight ??= doRefresh().finally(() => { inflight = null; });
  return inflight; // every concurrent caller awaits the same refresh
}

That's the onRefresh slot from Way 1, finally filled in. Owned by you, swappable behind the boundary, and held to the contract suite (that's what the "dedupes concurrent refreshes" test was checking).

Refresh is the headline example, but the principle generalizes to every behavior the vendor does invisibly: multi-tab session sync, sign-out propagation, silent re-auth. None of these are wrong to inherit. On a hot path, though, you want to know you've inherited them, and be able to take the wheel when you need to.

Where the three ways point

Notice that none of the three is about swapping your identity provider, yet together they push you toward it. A boundary that speaks OIDC and OAuth2 rather than vendor-isms, an understanding deep enough to write a conforming adapter, and volatile behaviors you own behind the seam all pull in the same direction: making the IdP closer to configuration than architecture.

You won't get there entirely for free. Vendor-specific features (Cognito groups, custom attributes, admin APIs) leak, and the more you lean on them, the more coupling stays behind. But these steps localize that coupling to one adapter instead of spreading it across the codebase. If you do reach the point where switching providers means writing a new adapter and making it pass your contract suite, that's one of the most valuable decoupling points a system can have.

Where the line sits: rent vs. own

If you're doing all this, a reasonable question is whether the vendor is earning its keep at all. Why not drop it and own the whole thing?

Because the vendor is doing the genuinely hard, genuinely dangerous work you do not want to own:

Cryptographic correctness — password hashing, token signing, PKCE, constant-time comparisons, the dozen ways to get this subtly and catastrophically wrong.
The edge-case swamp — MFA, account recovery, brute-force lockouts, credential-stuffing detection, bot defense. Each is a project.
Compliance and certification — SOC 2 and the security audits a managed provider passes so you don't have to.
The 24/7 operational burden — auth is a hot path, so if your hand-rolled auth goes down, everything goes down. Now it's in your on-call rotation forever, against an adversary whose techniques evolve weekly.

In the previous article, the single most decisive argument against one of the candidate architectures was exactly this: it would have meant owning the entire auth substrate — a new system of record, patched and scaled and on-call'd by us instead of by people who do only that. That cost sank the option.

Rent the engine. Own the steering.

The line has another side, and falling off it is just as expensive. Take "future-proof" too far — abstract everything, build seams nobody will ever flex, reproduce half the vendor's library "just in case" — and you land somewhere worse than where you started: over-built for today and still wrong for tomorrow, because the future arrived in a shape you didn't predict.

The discipline is to build seams only where change is both likely and expensive to retrofit. Notice what you're betting on: which axes will move, not how. You don't need to know whether you'll move to HttpOnly cookies or switch providers, only that those axes are live and worth making cheap to change in any direction.

A seam is cheaper than a prediction.

Flexibility everywhere is its own kind of rigidity. The skill is choosing where to be flexible.

The agentic era cuts both ways

The calcification trap predates AI, but coding agents make it faster and more pervasive, for a structural reason. An agent reaches for the vendor's documented happy path: import { signIn } from "vendor", used in-line, no boundary, vendor-magic refresh. It optimizes for "compiles and ships," not "still changeable in two years," and it won't weigh changeability unless you tell it to. The patterns that calcify now get generated at volume, often by people scaffolding auth they don't fully understand. What used to happen one developer at a time now happens across a codebase before anyone makes a conscious architectural decision.

But the same agent that defaults to the calcified version is also the cheapest way you've ever had to build the resilient one. The AuthPort, the adapter, the conformance suite, the single-flight refresh — the mechanical work that used to be the reason teams skipped the boundary ("no time for all that abstraction") — is now nearly free to generate. The cost that justified cutting the corner has collapsed.

So the balance tips toward doing the three ways, not away. The judgment stays yours: where the seams belong, which axes are live, what's likely and expensive enough to bound. The typing is the agent's. The newly essential discipline is encoding your intent so the agent stays inside it — a boundary spec it builds against, the contract suite as its target, lint rules that fail the build when a vendor type leaks past the line.

Takeaways

A wrapper isn't a boundary. If the vendor's types, errors, and claim names cross your abstraction, you've built a leaky facade. Speak your own domain's language and keep the vendor on one side of the line.
Inject auth as a capability, not an imported singleton, so it's swappable per context and your data layer is testable with zero network.
Contract-test the boundary. One conformance suite per interface turns "future-proof" into a definition of done for any future migration.
Understand your library by extending it. Replacing token storage is the highest-leverage exercise, and the result is an explicit artifact your team can read instead of guess at.
Own the volatile runtime behaviors, refresh first, with single-flight dedup and an explicit failure path. Know what you've inherited even when you choose to keep it.
Future-proofing is not building auth yourself. Rent the crypto, the edge cases, the compliance, the on-call. Own the seams and the judgment.
Don't over-rotate. Build seams only where change is both likely and expensive to retrofit. Bet on which axes move, not how.
Agents accelerate calcification — but also the fix. They default to the vendor's calcified happy path at scale, but make the boundary nearly free to build. Supply the architectural intent; encode it as specs, contract tests, and lint rules the agent can't route around.

Closing

This is part two of a short series on auth architecture in production. Part one is the narrative: how an under-designed auth system cornered a large platform into accepting a known vulnerability, and why that was the right call given the constraints. This piece is the constructive flip side — the foundation work that keeps you out of that corner in the first place.

Next in the series: I'm building an agent skill that audits a codebase for exactly the failures described here — missing boundaries, leaking vendor types, inherited refresh — and proposes a backlog to fix them. Follow along if that's useful.

And if your team is about to make an expensive-to-reverse decision in auth, cloud, or platform architecture and wants an independent second pair of eyes, that's the kind of work I take on through Luckylabs. Reach out if it sounds useful.

Securing auth in a large-scale production system: three industry-standard architectures — and why none survived a closer look

Bilaniuc Dragos — Tue, 26 May 2026 09:47:36 +0000

A case study in why the verdict on an architecture decision can shift entirely once you dive into implementation details — using auth security on a large Next.js + AWS system as the vehicle. This article walks the three architectural paths we considered — plus a fourth option that, in this context, won.

Context: a sensitive problem in a large scale system

The system: a large multi-frontend production stack. Multiple Next.js apps spread across subdomains, 100+ backend microservices, AWS Cognito for identity, API Gateway in front of the backend, downstream APIs expecting Authorization: Bearer <id_token> on every request.

The finding: a penetration test flags that our Cognito tokens — ID, access, refresh — are stored in non-HttpOnly cookies. Client-side JavaScript can read them. Any successful XSS exfiltrates them. Any leaked refresh token gives an attacker effectively indefinite access.

Here's what that looks like in flow form — the current client-side API call pattern, with the moment of vulnerability highlighted:

Three things worth noticing before we go further. First, this flow is cheap — the client calls the backend directly, no proxy layer in the path, every component doing the minimum it has to. Second, the entire vulnerability is concentrated in the single step where browser-side code reads the token out of document.cookie to attach it to the outbound request. Third, every alternative we'll walk through changes the shape of that single moment — and in doing so, ripples through every other component on the diagram.

The shape of the fix is well-known: tokens should not be JS-readable. Move them off the client.

So why are we writing a 3,500-word article about it?

Because auth in a system like this isn't a feature. It's a hot path — twice. It's the entry point to the system, and then it's a participant in every API call that follows. The architecture choice you make doesn't just impact "the login screen." It impacts every request, every page render, every cost line item, every incident postmortem for the foreseeable future.

That's the kind of problem where the obvious architectural answer is the dangerous one — not because it's wrong, but because it's incomplete. The shape is right; the implementation cost can take you over the edge.

This article walks four architectural paths to fix the same vulnerability in the same system. Each path is sound in isolation. Each path's verdict shifts — sometimes inverts — once you stop thinking at the architecture level and start thinking at the implementation level.

The stack details are the vehicle. The cargo is the thinking pattern: architectural thinking alone won't tell you which of these four paths is right. Implementation depth will.

The "obvious" answer, and why to distrust it

The chain of obvious reasoning:

Tokens accessible to JavaScript are vulnerable to XSS exfiltration.
Therefore, move tokens out of JavaScript's reach.
Therefore, use HttpOnly cookies, set by the server, never read by the client.
Done.

At the shape level, this is correct.

The trouble is that step 4. "Done" is not done — it's actually doing about 90% of the work in that argument. "Done" implies the implementation falls out cleanly. It almost never does.

The architectural move is to refuse "done" as an output until you've answered:

How does the client-side code now authenticate API calls it used to authenticate directly?
How does the backend now extract a credential it used to receive in a known format?
How does the system refresh tokens that can no longer be seen by the code that previously refreshed them?
What's the new attack surface? (HttpOnly cookies close XSS; they open CSRF.)
What does the cost model look like at production scale?

Each of those questions has multiple architecturally-valid answers. Each answer creates a different system. The point of the analysis isn't to find an answer — it's to walk each answer to its third-order consequences before picking.

Let's do that.

The constraint stack

The constraints below are real — pulled from the actual system. They're what every architectural option gets weighed against. The discipline here is map options against constraints, not against feature checklists.

1. Frontend framework. Next.js App Router, heavy use of Server Components, hybrid client-server data fetching.

2. Downstream contract. ~100+ backend microservices, all expecting Authorization: Bearer <id_token>. The API Gateway in front of them validates JWTs via its built-in authorizer reading the Authorization header. Changing this contract means changing it across all 100+ services in a coordinated manner. Doing it in fewer than all of them is not an option — the contract is uniform on purpose.

3. Hosting cost model. The Next.js apps run on Vercel. Vercel bills serverless functions by the wall clock time of every request — including the time spent waiting on a slow downstream call. A 5-second hung backend response gets billed for 5 seconds of compute, every time it happens.

4. Client/server call ratio. Roughly 65-70% of API calls in the system are made directly from the browser (client components, after-hydration data fetches). The remaining 30-35% happen server-side during SSR. Any architecture that proxies "all client calls through the server" effectively doubles the proxied request volume.

5. Token characteristics. Cognito JWTs are large — ID tokens commonly run 1-2KB, sometimes more depending on groups and claims. Browser cookie size limits sit at 4KB per cookie, and total request header limits (the source of HTTP 431 errors) sit at 8KB on many gateways. Putting multiple Cognito tokens in cookies and sending them on every request is a known production failure mode.

6. Vendor library maturity. AWS Amplify v6 supports HttpOnly cookies for Next.js — but only via an experimental server-side auth feature, and only when paired with Cognito Managed Login.

That's the stack. Every option below has to survive it.

Path 1: Next.js BFF proxy

The pattern most commonly recommended in the Next.js ecosystem, and the one AWS Premium Support proposed in the first round of their guidance. It's also the default shape that libraries like Auth.js (formerly NextAuth) push you toward when wrapping OAuth providers — manage session cookies on the server, proxy token-bearing calls through Next.js route handlers.

The shape: All client-side API calls stop going directly to the backend. Instead, they hit Next.js route handlers (or Server Actions). Those handlers read tokens from HttpOnly cookies — which they can, because they're server-side — and forward the request to the backend with the Authorization: Bearer header attached.

The architectural appeal. Clean. Well-documented. Vendor-blessed. The pattern most public material recommends. Tokens never touch the client; the downstream contract doesn't change; the API Gateway's existing JWT authorizer keeps working.

At the architecture level, this looks like the right answer. It is, in fact, the right answer for many systems.

Where it stops looking right — implementation level. Two things surface as you zoom in:

The cost model breaks. Recall constraints #3 and #4. We have a hosting model that bills for wall-clock wait time, and we'd be putting 65-70% of API traffic that previously bypassed Vercel directly through Vercel. Every slow backend call now costs us twice — once for the original execution time, once for the time the Vercel function spends waiting. A backend incident where calls take 30 seconds to time out becomes a Vercel bill incident.

Vercel lock-in becomes structural. In the BFF model, 100% of API traffic flows through Next.js route handlers on Vercel — making Vercel part of every API call's critical path, and shaping the architecture around Vercel-optimized patterns. Any future migration to a different host (for cost reasons, feature reasons, or just optionality) stops being a packaging change and becomes a real re-architecture. Vercel's billing model has, at this scale, already produced unwelcome surprises; locking the auth hot path to a single hosting vendor compounds that exposure rather than reducing it.

Verdict shift. Strong default. Wrong for this stack — the cost surface and the vendor lock-in are the real disqualifiers; everything else is downstream of them.

Path 2: Token-broker BFF with session DB

A variant that escapes Vercel by introducing a dedicated AWS-native proxy. The browser holds an HttpOnly session cookie (opaque); a Lambda sits in front of the backend, reads the cookie, looks the session up in a database, retrieves the Cognito token, and forwards the request with the Bearer header attached.

This is essentially the architecture that Better Auth promotes when it's wired up to an external OAuth provider like Cognito: a server-side session store with the provider's tokens held inside it, and the browser carrying only an opaque session reference. Better Auth also supports a stateless variant — where the session is encrypted directly into the cookie and the DB lookup disappears — which trades the database hot path for cookie-size and key-rotation problems. We'll focus on the DB-backed shape, as it's the one most production deployments converge on.

The architectural appeal. Full session control — you can revoke individual sessions, track devices, enforce idle timeouts, support multiple identity providers without each one polluting the cookie payload. Vercel cost goes away. The downstream contract is preserved.

Where it stops looking right — implementation level.

We now own the entire auth substrate. The biggest hidden cost in this path isn't in any individual diagram; it's in the operational total. A new database layer to provision, scale, and back up. A new session mechanism to write, test, patch, and version. A new piece of critical infrastructure to bring into the on-call rotation. Security updates and resilience improvements that previously came "for free" from the vendor stack — Amplify patches, Cognito-side improvements — now have to be applied (or replicated) by us. Auth is the part of a system you most want operated by people who specialize in operating auth; this path moves you in the opposite direction.

The session DB becomes a hot path. Every API call now pays a DB lookup latency. Even at with high-throughput DBs (like Dynamo) that's added to every request that crosses the system boundary, and the database becomes the system's availability ceiling for any authenticated traffic.

Database choice is a one-way door. DynamoDB is the AWS-native default, but if you ever need a relational session model — joins, audit trails, complex revocation rules — you're either retrofitting an inappropriate store or migrating. ElastiCache buys sub-millisecond reads but adds VPC complexity. Aurora + RDS Proxy buys SQL but needs explicit guard-rails against connection storms. Each is sticky once chosen.

Observability fragments. There's now an extra hop, possibly in a different language, in a different deployment unit. Distributed tracing has to follow it. Failures in the broker look different from failures in the backend — and that distinction matters during the worst incident calls, when you need to localize a problem in under five minutes.

This is where it's worth distinguishing stateful complexity from feature complexity. Both Path 1 and Path 2 add complexity, but in different shapes. Path 1's complexity is operational — a new layer that needs to scale and stay up. Path 2's complexity is stateful and ownership-shaped — a new system-of-record whose failure modes are harder to reason about, whose data has independent lifecycle from your application, and whose operation you now own end-to-end. Stateful complexity tends to be more expensive in the long run because state attracts more state, and ownership tends to be more expensive than people estimate at decision time.

Verdict shift. Justified only if you actually need the session-control features that come with it (multi-IdP, fine-grained revocation, device tracking as a product feature). Otherwise, you've taken on durable architectural debt — and a permanent operational burden — to solve a hosting-cost problem.

Path 3: API Gateway Lambda authorizer reading cookies

The cleanest path if you control the gateway. Skip the BFF entirely. Configure the API Gateway with a custom Lambda authorizer that reads the auth credential from a cookie instead of the Authorization header. Browsers send the cookie automatically on requests with credentials: 'include'; the authorizer extracts and validates it; downstream services receive a Bearer token injected by the gateway.

The shape: HttpOnly cookie carries the Cognito JWT directly. The Lambda authorizer reads the cookie, validates the JWT against Cognito's JWKS endpoint (a library like aws-jwt-verify handles the signature and claims validation cleanly), and tells the gateway to inject the token into the Authorization header for the downstream call.

On the architecture level, this is the most elegant of the four paths. No BFF. No session DB. No Vercel cost increase. Tokens never reach JS. Downstream services keep their existing JWT-from-header contract because the gateway re-injects it. Lambda authorizer responses are cached by API Gateway (default 300s), so the validation cost amortizes.

Where it stops looking right — implementation level.

Cookie size limits. Constraint #5 was waiting for this option. A full Cognito ID token plus an access token plus a refresh token can easily exceed the 4KB per-cookie browser limit. And on every request the browser sends, those cookies sit inside the total header size budget — most API Gateways cap around 8KB. Hit the cap, and the gateway returns HTTP 431 before any of your code runs. This isn't a hypothetical; it's a documented production failure mode for Cognito users who try this naïvely.

Refresh token handling migrates into the auth infrastructure. This is the surface most underestimated at decision time. In the JS-readable token world, the auth library handles refresh transparently from the client: the client asks for a session, the library checks expiry, refreshes if needed, returns the token. The client never sees the mechanics. In this model, the client can't see expiration at all — and the authorizer's job is structurally "allow or deny," not "issue new credentials."

So who refreshes? Three sub-options, each with cost:

The authorizer itself, on every expired-token detection — but it then needs to write new cookies onto the response, which is awkward in an authorizer's lifecycle, and concurrent requests can stampede the refresh endpoint.
A dedicated refresh route handler the client calls when it gets a 401 — this is the cleanest of the three, and the future state we ended up optimizing toward. It has well-defined seams, it's easy to test, and the client-side 401-then-refresh-then-retry pattern is a standard piece of HTTP plumbing when implemented carefully (with refresh deduplication so concurrent 401s don't trigger N parallel refreshes).
Pre-emptive refresh from middleware — but middleware doesn't have a natural seam to mutate cookies on outbound responses in a way Next.js, the browser, and the gateway all agree on.

None of these are blockers. All of them are real work. And refresh is the part of auth where bugs become production incidents fastest, because they manifest as intermittent 401s that are hard to reproduce.

CSRF surface opens. When the credential travels in an Authorization header set by your code, CSRF is structurally impossible — the attacker's page can't make the browser attach a custom header. When the credential travels in a cookie automatically attached by the browser, CSRF re-enters the threat model. You now need SameSite=Lax (or Strict), explicit anti-CSRF tokens for state-changing operations, or both.

The non-technical reason this path didn't win — for us, this round. Even with the refresh-route-handler model as the cleanest variant, we didn't pick Path 3, and the reason wasn't technical. The Lambda authorizer sat in another team's ownership, and that team had an active RBAC initiative consuming most of their bandwidth. Kicking off the cookie-authorizer rewrite at that moment would have meant either derailing their roadmap or eating cross-team coordination overhead that would have stalled both efforts. The right time for Path 3 was after the RBAC work landed — which gave us a natural future state to optimize toward, rather than a path to force through now.

Worth naming: architecture is org-shaped. The "right" technical path is sometimes blocked not by technology but by ownership and timing — another team owns the relevant component, that team has competing priorities, kicking off a cross-team effort would derail their roadmap. That's not a failure mode; it's a constraint. Name it explicitly in the analysis, and shape today's decisions around the future state where the timing flips — rather than pretending the org-state is somebody else's problem.

Verdict shift. Cleanest of the active fix paths if downstream services and gateway are yours to modify and the timing is right with the team that owns them. Costly enough in implementation work — particularly the refresh flow — that it should be entered into with eyes open, not signed up for at the elegant-diagram stage.

Path 4: Conscious deferral + foundation work

The "boring" option. Presented last because it only earns its place after the first three have lost some shine.

Keep the current non-HttpOnly cookie storage. Mitigate the XSS surface via the usual hardening (strict CSP, short-lived access tokens, careful XSS posture in dependencies). Document the risk acceptance explicitly. Define what triggers the eventual migration. Do the foundation work now that makes the eventual migration cheap when the trigger fires.

This is not "do nothing." It's a substantial decision with substantial work attached.

What the foundation work actually looked like:

Migrated from an older Amplify version that used self-hosted login pages — which structurally could not support HttpOnly cookies — to Amplify v6 with ssr: true and Cognito Managed Login. That's the version whose experimental HttpOnly feature exists. Moving to it converts the eventual migration from a six-month re-architecture into a config flag.
Re-pointed the application's login UX from self-hosted pages to Cognito Managed Login, which is a prerequisite for Amplify's HttpOnly mode and also de-risks the future migration (no custom login UI to retrofit).
Wrote an explicit risk acceptance — not a passing mention in a Slack thread, but a stakeholder-signed document describing the vulnerability we were knowingly keeping live, the mitigations in place, and the triggers that flip the decision.
Defined the migration trigger. The primary trigger was the sunset of legacy frontend systems that were structurally incompatible with the new auth flow. Until those were gone, migrating the new system wouldn't actually have eliminated the vulnerability at the org level. A secondary trigger was a parallel RBAC initiative that would create a natural alignment point for backend/infra teams to revisit authentication anyway.
Captured the full decision and option analysis — including the work that went into Paths 1-3 — so the next architect picking this up doesn't redo months of research.

The architectural appeal. No disruption to a 100+ service system. Full understanding captured. The eventual fix is now cheap, not prohibitive. The decision is reviewable, with clear conditions under which it should be revisited.

This is the work that separates "ship the broken pattern" from "defer the fix with intent." When the right answer is "don't change the externally-visible behavior yet," the work that earns that answer is invisible from the outside — but it's the work that determines whether the eventual change is a flag flip or a re-architecture.

Verdict. In this constraint stack, with these specific triggers visible in the roadmap, the right answer. Anchored explicitly: in a different stack (no Vercel, smaller backend, fewer downstream services, no upcoming roadmap alignment point), the verdict could land on any of Paths 1-3.

The decision matrix

Compressing everything above into a comparison:

	Frontend impact	Backend impact	Cost surface	New attack surface	Reversibility
Path 1 — Next.js BFF	High (all client API calls re-routed through Next.js)	None	Vercel compute doubles on proxied traffic	CSRF if cookies auto-attached	Medium
Path 2 — Broker + session DB	Medium	Low (gateway untouched)	DB hot path; new infra cost	CSRF + new failure mode	Low (DB choice is sticky)
Path 3 — Gateway authorizer w/ cookies	Medium (refresh flow re-architecture)	Medium (new Lambda authorizer + gateway config)	Modest (Lambda invocations, amortized by cache)	CSRF + cookie size risk + new refresh failure modes	High
Path 4 — Defer + foundation	Low (substrate upgrade)	None	None	Unchanged (deferred)	High

The matrix shifts if specific constraints flip:

Not on Vercel? Path 1's cost surface mostly disappears. It becomes a strong default again.
Don't control the API Gateway? Path 3 evaporates.
Need session revocation as a product feature? Path 2 moves from over-engineered to required.
No legacy frontend dependency? Path 4 loses its trigger and you go pick a real fix.

This kind of sensitivity is the point. The verdict isn't intrinsic to the path; it's intrinsic to the path plus the constraint stack.

Where architectural thinking ends and architectural judgment begins

The meta-point of this article, explicitly:

Architectural thinking, by itself, can absolutely tell you that Cognito tokens shouldn't be readable in JS. It can tell you that the broad shapes of the fix involve some combination of server-held state, BFF proxying, or gateway-level credential extraction. It can sketch you a clean diagram for each path and identify their structural trade-offs.

Architectural thinking alone cannot tell you:

That Vercel's billing model makes Path 1 expensive in a way that doesn't show up on the diagram.
That Cognito tokens are too big to fit inside cookies the way Path 3 wants them to.
That moving from one auth library version to another is itself the substrate work that makes the eventual fix cheap.
That vendor support (AWS Premium Support in our case) can give you two contrary answers if you ask twice — and that both can be correct under different assumptions.
That your stakeholders can accept "we're deferring the fix" if you give them an artifact, instead of just giving them a meeting.

These details don't live in the architecture diagrams. They live in implementation specifics, vendor docs, billing models, threat models, organizational state, and roadmap context. The job is to go get them — read the vendor docs to the bottom, run the cost numbers, sketch the migration in enough detail that the unknowns become visible — and feed them back into the architectural choice.

That's the thing worth leaving you with. Architecture is a discipline you do all the way down to implementation depth, or it's not architecture — it's a slide deck.

Acceptance of an existing flawed pattern can be a legitimate architectural outcome. So can adopting an experimental vendor feature, or building a new infrastructure component, or kicking the decision to the team that owns the gateway. What separates a good outcome from a bad one is whether the choice was made with full visibility into its second- and third-order consequences, or without.

Takeaways

Surfaced throughout the article, collected here for reference:

Map options against constraints, not features. A pattern's quality is a function of context; the same pattern can be right or wrong in different stacks.
Distinguish stateful complexity from feature complexity. Both are complexity; they fail differently and accumulate differently.
Use foundation moves. Modernize substrate now to make future fixes cheap. The work is invisible from outside but determines the cost of the eventual decision.
ADR-as-artifact. When you defer a decision, the deliverable is the documented decision and risk acceptance — not the absence of the change.
Architecture is org-shaped. The "right" technical path is sometimes blocked by ownership and timing rather than technology. Name it explicitly in the analysis; design today's decision around the moment the timing flips.
Vendor support is an input, not a conclusion. Useful, often correct, never the last word. Ask twice; you'll get different answers when you clarify constraints.
When deferring, write the trigger. A deferral without a defined trigger is procrastination. With one, it's a roadmap entry.
Walk implementation details for every option. Without it, architecture is artificial in the best case, severely misleading in the worst.

Closing

I wrote a companion piece on LinkedIn about the narrative side of this — what it felt like inside the team to spend weeks researching a security fix and then deliberately ship the pattern the pentest had flagged. [Link soon to come]

The technical depth here is the substantiation. The thinking pattern — implementation-aware architecture — is the actual point.

If your team is about to make an expensive-to-reverse architecture decision and wants an independent second pair of eyes, that's exactly the kind of work I run as part of architecture and cloud advisory engagements through Luckylabs. The analysis above is roughly the format of a typical engagement, applied to one specific problem. If it sounds useful, the fastest way to reach out is dragos@dbln.me

Sketching out the extremes: an approach to designing software architectures in highly unpredictable projects

Bilaniuc Dragos — Mon, 12 Aug 2024 07:29:23 +0000

Imagine a scenario: a fresh, ambitious project with enormous potential but limited resources. The stakes are high, and the product needs to hit the market as soon as possible to validate its potential and seize the opportunity. The roadmap is hazy, making flexibility crucial.

Classic story until now. But here’s where things get spicy: the company that started the project has a huge community ready to jump on this new product they’ve been advertising for months. Thus, at the initial beta release, the system must be prepared to jump from 10 internal people testing it to anywhere from 20-100k daily active users and millions of events per hour.

This mix of conditions creates a particular set of requirements:

the system must be simple enough to be handled by a few devs at the beginning, with short iteration time and cost-effectiveness being key
it must be scalable enough to handle the spike of users around the initial release date
it must be flexible enough to permit a rapid move toward scalability later on, delivering more and more complex features along the way

The process

The first step was to create an architecture that would satisfy all these requirements.

As a preparation, I read and re-read the following books:

Drawing on the insights from these resources, along with my prior experience, I knew that some form of distributed architecture would be essential to handle the expected traffic.

Roughly speaking, I followed these steps to arrive at an initial sketch of how the system would look like:

Analyse requirements and gather events that will need to be handled by the system
Group those events into domain boundaries
Identify architecture characteristics (scalability, security, robustness - these kinds of things)

That initial sketch looked like this (in order to protect the intellectual property of the company, I anonymized all event and module names, keeping only the shape of the sketches):

Readable version here

After lots of refinement and discussion sessions, two things became clear:

one big chunk of code (i.e a monolithic architecture) wouldn’t fit our scalability and flexibility needs
a full-blown microservice architecture would be too complex and costly at this stage of the game

So, as with most things in life, the solution was somewhere in the middle. Finding that middle was now the challenge.

Here’s where the approach that I’m proposing comes into play:

Sketching out the extremes

Initially, the mindset was to design the simplest architecture that would satisfy our requirements and see if I noticed any specific areas of risk/improvement.

The simplest architecture I could think of was a modular monolith, where those modules would be as separated and clearly defined as possible. It looked something like this:

Readable version here

The thing that I thought would be the “secret sauce” here was the data access library, a library shared between all modules, it would sit in front of the database layer and would be designed in such a way that extracting a module into its own microservice would take as little effort as possible, and we would only have to tamper with the shared library.

As a bonus, and to be prepared “even more” for the move towards microservices, I was planning to deploy each module into its own independent function on the cloud infrastructure we were planning to use.

After analyzing the approach a little more, two things became obvious:

implementing the “secret sauce” library in the way that I imagined wasn’t realistic, and it would lead to lots of complications
deploying modules in different containers that accessed each other's databases risked creating a "Distributed Monolith"—a problematic architecture that combines the worst aspects of monolithic and distributed systems.

So, since I went to one extreme, i.e. the cheapest/fastest one, I thought it would be interesting to go to the other one as well:

What would the architecture look like, if we had all the resources we needed to create it?

After some more sketching, I arrived at this rough design of a microservice architecture for our system (I only highlighted some the inter-service events here):

Readable version here

In this version, we have some clear advantages:

individual scalability of each module, based on loads
clear ownership of the data
each module can be independently deployed

However, these advantages come at a cost: complexity.

Most of that complexity came from the inter-service communication. In V1 of the architecture, if module C needs data that is stored in database E, it can directly access it via the shared library. In this version, it has to request the data from Module C over the network - which adds more points of failure and makes tracking and debugging more difficult.

But even though this wasn’t the best solution for our current situation, it led us to some important insights based on which we built the actual architecture of the system.

The solution

After taking a closer look, we noticed that most of the inter-service communication was happening between 3 of the 6 modules: Module C, Module D, and Module F. If the network communication between those services was somehow taken out of the picture, the remaining microservices would have a reasonable number of events and requests to share between them.

So that is exactly what we did: compressed the 3 highly interrelated modules into a single modular microservice, and transformed the other modules into their own microservices.

After some more tweaking and refining, this is the architecture we’ve come to:

Readable version here

Here are the advantages of this architecture:

it retained the scalability advantages of microservices
the downsides of inter-service communication were reduced to a manageable level, enough so that our small initial team could implement the architecture in an efficient manner
the "big" microservice was implemented modularly, allowing for future extraction of individual modules if needed
the established microservice framework meant there would be less friction when splitting additional components into new microservices

Conclusion

Oftentimes, as software developers and architects, we are expected to come up with the ideal solution for any set of circumstances, requirements, or problems. We are aware, however, that everything in software development is a tradeoff.

While this little story isn’t about promoting any specific type of software architecture, its purpose is to showcase a promising strategy for building software architectures, and it sounds like this:

When trying to figure out where exactly to stop between two extremes, a good starting point is to sketch out how both of these extremes would look like.

In terms of software architecture, designing the cheapest and fastest architecture that would sort of satisfy the requirements, and putting it next to the most comprehensive and scalable solution can be very insightful. It can bring a better understanding of the system, its risks, and possibilities - as well as provide a clearer general direction for the product as a whole.

In our case, this strategy showed us how a little tweak in the way we were organizing our components allowed us to aim much closer to the “ideal” microservice architecture than we had initially thought was possible.

What insights would such an approach bring to your project and team?

How I stopped freaking out when speaking to people | A journey with 4 lessons

Bilaniuc Dragos — Fri, 23 Jul 2021 16:33:27 +0000

Imagine: You're in 6th grade. Your English teacher gives you the main role in a poetry event that will appear on TV. On the big day, you walk on the stage. The camera turns upon you. Everyone's eyes turn upon you. You take a deep breath, and… blank.

I always like to point to that story as the reason for my excessive fear of speaking in front of people. Of course, there must be lots of other reasons for it, but having a brain fart at 11 years old - on TV and in front of everyone - sounds dramatic enough, so we'll stick to that. The fact is, this became one of my deepest fears.

Learning to live with this fear - and (spoiler alert) going from that panicked little kid to winning the Speaker of the Year award in front of over 300 people - was the most transformational journey of my life. It taught me some of the most important lessons I know, so I decided to write about them. Maybe they'll help you, too :)

Denial

For as long as I can remember, whenever I'd have to speak in front of people, my heart would start thumping, my mind would start racing and I'd start blurting out words rapidly and with no coherence. And the fact that I was in complete denial of this fear didn't help at all.

I quickly fell into a very nasty loop: I'd try to mask my insecurities with "confidence hacks" from youtube, people around me would smell that and become even more repulsed, which confirmed my insecurities, which in turn made me seek even more tricks and hacks… and so on.

And this loop led me into a really dark place. I couldn't feel an authentic connection with anyone - not because there was anything wrong with the people around me, but because I never allowed them to see who I really was.

All I was doing was keeping up this facade of fake confidence, and getting disappointed and frustrated every time it would inevitably fail to solve my problems.

Change

The trigger for change came at a leadership event that I participated in. I was my usual self, feeling awkward and trying to mask it with tricks and hacks.

And then, there was this guy - speaking calmly, telling the right jokes at the right times, never really "trying" anything, but making everyone feel good - especially when he got in front of the group. Everyone wanted to be around him. And I wanted to be him.

Following his advice, I joined Toastmasters, an organization that had regular meetings for practicing public speaking - and this is where the real pain of confronting my fears and insecurities began.

At first, I was obsessively practicing my speeches (30–40–50 times!), just to make sure what happened in 6th grade would never happen again. And it went quite well, insofar as blocking on the stage went.

But the painful part came after a speech was over, and I would look over the recording. I could barely stand to look for more than 2 minutes - that's how unnatural and awkward I saw myself on the stage. All my insecurities were exposed in front of everyone. And that was a hard pill to swallow.

The only thing that kept me going was gradually learning the lessons I'll talk about. The made me stick to this journey, one painful step at a time, until I was holding this speech:

that ended up with me holding this beauty, as one of the biggest achievements of my life:

Now, on to the lessons…

1. The most powerful thing to know is that it's okay to fuck up

And I'm not saying it in a make-me-feel-better way. This is actually the best way to minimize the mental chatter that's such a big part of fear.

Most times, fear would begin with the question "what happens if I fuck up?", which would trigger an endless loop of apocalyptic scenarios in my head - scenarios that were totally ungrounded in reality. Once I accepted that it's okay to fuck up, I could short-circuit the loop by answering that question with a realistic "nothing much".

Toastmasters taught me that environment can play a crucial role here. If the feedback for my first speech was an honest "you suck, just go home" - you can bet I'd still be filling the pockets of "hack your confidence" gurus on youtube.

Instead, the message was always: it's not a big deal - it happens to everyone. And that's about the most empowering message you can get when confronting fear and insecurities.

Another thing that really helped me with this point was seeing other people around me awkwardly trying to confront the same fear and fucking up themselves. Knowing that I was not the only one struggling (and failing) was what kept me going, during some of my most intense moments of self-criticism.

And this brings me to my second eye-opening lesson:

2. Encouraging others is the best way of dealing with your own fear

And this is something I didn't expect at all: the most effective method of dealing with my own fear didn't have much to do with me. It was all about encouraging others to confront their own version of this fear of public speaking.

And not just because it's the right thing to do in a higher, humanitarian way. It's actually the best thing you can do, egotistically speaking. Because when you see someone else dealing with the same kind of fear you're dealing with - and start encouraging them - something amazing happens:

that fear becomes smaller for both of you

This was another huge advantage at Toastmasters: being a member, I didn't only give speeches. I would also have to evaluate others, encourage them and give them the best feedback I was capable of. And telling others that they can do it, I suddenly found myself believing it, too!

Find someone that's dealing with the same kind of fear. Encourage them and let them know that it's okay to struggle and even to fail. Then feel the empowerment this simple act brings upon you, in your own struggles.

And this lesson kept me fighting, until I finally realised that:

3. You don't conquer fear. You learn to live with it

About 2 years passed since I won that trophy that marked the success of my "public speaking project". Since then, I've had quite a lot of events to moderate and speeches to present. In short, I kept gaining experience.

But just the other day, I was going to this salsa&bachata party where I didn't know anyone. I arrived a little earlier, and there were a few people gathered up in a discussion. As I approached them, my heart started thumping, my thoughts started racing - all the good old friends were right back.

And this is the biggest takeaway from my journey:

My fear never went away. I just learned to live with it.

More specifically, I learned to expect fear, without expecting that it will ruin everything. To leave it in the background, and keep going anyways.

I believe I learned to do this simply through experience. Speech after speech - no matter how much I rehearsed or how small the audience was - the same feelings of fear would arise. And, speech after speech, they would prove to be less tragic than I imagined.

Eventually, those feelings became just part of the landscape - not going away, but not consuming all my attention, either. And that's how I started to actually enjoying public speaking. That's how my gruesome fight came to an end- not because I won or lost, but because I stopped fighting.

What I wish I knew then is how much mediation can help with this - since its core training is leaving thoughts and emotions in the background. And it's such an effective way of doing it that, in hindsight, I believe it would've saved me months of pain and frustration I had to go through, just to begin to learn this lesson.

Just for the record, after trying out lots of apps and techniques, the most effective way of meditating for me right now is through the guided sessions of Jeff Warren, available on Calm. If you have a hyperactive mind like mine, he might resonate with you, as well :)

And the last big lesson this journey had to teach me was that:

4. It's absolutely worth it

I believe that one of the most perverse things when it comes to dealing with fear and insecurities is how easily the battle can be lost before it even begins.

For such a long time, all I did was find circumstances and excuses. I'd tell myself it's alright, I'll eventually cover those insecurities up with confidence hacks. Or I'd have a little fuckup and see it as confirmation that I'll never be able to do it. Or, most often, I'd just delay things, expecting some enlightenment moment where all my fears and insecurities would disappear, just like magic.

And now, I can confidently say that these were the worst ways in which I was sabotaging myself. Because learning to live with only this one fear, changed the way I looked at every other area of my life.

More and more often, when faced with insecurities, fears and uncomfortable situations, I found myself thinking:

"if public speaking didn't kill me, this won't kill me either"

And soon enough, this automatic way of thinking made me take on opportunities that would've terrified me before - opportunities that, among others, ended up sending my professional growth through the roof.

I gained the courage to begin freelancing, to assume the role of Tech Lead in my projects, and, soon after, to become the CTO of a very promising startup.

All of these moves came with lots of doubts, fears and insecurities packed in. And the only way I could get over them was through the lessons learned in this seemingly unrelated journey of confronting my deep fear of public speaking.

Final thoughts

So, that's about it for my journey. Confronting my fear taught me some of the most valuable lessons I know: that it's okay to fuck up, that encouraging others pays huge dividends, that you can live with fear and that it's totally worth doing it. I hope you found some value in those lessons, as well.

But, at the end of the day, this is just one journey among millions. Fear and insecurities are everywhere, and each of us is trying our best to deal with them. If you're struggling, rest assured that you are not alone, not by a long shot.

And if my specific lessons don't work for you, what will definitely work is what you're already doing now: keeping an open mind, seeking out new ideas and enlarging your own perspectives. A little bit from here, a little bit from there, and that's how you're building up your own recipe for dealing with this existential human condition.

Kudos to you, for keeping an open mind in front of so many fears and insecurities!

I tracked every minute of my time for the last 4 months. Here are 7 totally unexpected results

Bilaniuc Dragos — Wed, 14 Jul 2021 13:11:09 +0000

A 30 minutes lunch break? Time entry. 12 minutes of arguing with mom? Time entry. 7 minutes of stalking my ex on Instagram? You guessed it!

That was my life over the last 4 months, being my own time tracking cop while carrying out the most outrageous experiment of my life so far. The results amazed me in ways that I would never have expected, so I decided to write about them. Who knows, maybe you’re also going to find something of value in my journey.

But first…

What the hell made me do it?

To be honest, it was a combination of frustration, guilt and fear — all generated by the fact that I kept setting big goals for myself, but never seemed to make any progress towards them. Time was passing, and I had nothing to show for it.

All I had were vague reports at work, where “working” covered everything from writing code, to talking to people, to watching Russian slapping contests on youtube.

I knew I had to do start tracking my time if I wanted to keep my sanity. And I’ve seen how something casual/half-hearted would quickly become too vague to be of any use. If I wanted to do it, I had to go all-in.

So…

How exactly did I do it?

I used an app called Toggl to do all my tracking. I chose this specific app because:

it’s available on all platforms and syncs very nicely between them
it has an intuitive UI, with suggestions that become more and more accurate over time
it allows for very fine customization and reporting

At first, I devised 7 main areas where all my entries would fall, using tags and colors to mark them in the app:

Then, I created some high-level projects that I knew I was already spending time on:

After the setup was done, I would simply start a new time entry, every time I would move from one activity to the other.

One important note here was that, because I decided to go all-in, I had to be particularly careful not to have any entries “cover up” unrelated things that I was doing. For example, if I would be writing code on my project and I’d receive a call, I’d have to create a separate time entry for it, even if it only took 5 minutes.

This level of strictness was a bit taxing at the beginning, but after a while, Toggl’s UI came in very handy. I rarely had to create new entries. Most of the times when I’d switch to something else (even just short interruptions), I would just have to press “continue” on a previous entry.

At the end of the week, here’s how my timetable would look like:

And now that the “how” is also out of the way, let’s move on to the exciting stuff. Here are the biggest changes I noticed, over these 4 months:

1. Drastically improved my focus and productivity

Yep, the biggest impact of this time tracking journey didn’t come in the form of a mind-blowing insight at the end of the week, when I was looking over my timetable. The biggest impact was in how I was spending my time while I was doing the tracking.

At first, nothing big happened. But after a while, I found myself more and more often in an interesting situation. Whenever I was working and felt the need to “zone out” by jumping “just a bit” on youtube or Instagram, I’d find myself thinking “nah, I don’t want to start a new entry now. I’ll just get this thing done, and then take a bigger break for that”.

And, more often than not, I wouldn’t even feel that need for youtube or Instagram anymore, once I actually finished the task. The feeling of accomplishment was enough to give me the refreshment I needed.

This increase in focus sent my productivity through the roof. Yes, I’ve always read and heard of how context switching is a very bad idea. But, until I tracked it down, I never knew how often I was doing it — and I never knew that getting rid of it would make me get things done literally twice as fast! After seeing all this for myself, I felt like I somehow hacked my own life.

2. Saved a lot of time from unexpected places

One of the most shocking insights that I had in the beginning, after looking at my reports for a week, was that I was spending over 20 hours a week on food-related activities (buying food, cooking, eating).

Every day, 3 times a day, I would take some time off to cook. And, since eating healthy was always an important thing for me, I didn’t feel bad if it sometimes took a little longer. But I never knew I was doing the work of a part-time cook out there! At least if I’d learned to cook something more exciting than this boring salad:

Once I became aware of this, I automatically started thinking about how I could make it a bit more efficient. A simple thing that popped into mind was that, instead of cooking everything from scratch in each of the 3 breaks, I could simply prepare everything in the evening before, and have my food ready to go for the next day, with minimum adjustments.

This trick solved all my cooking for the day in under 1 hour, instead of 2–3, which amounted to over 10 hours of saved time, per week. And the best part was that I didn’t have to sacrifice anything in terms of how healthy or tasty my food was!

Such a simple solution. However, I would never have thought of it, if there wasn’t this table that was telling me straight to my face: “here’s how much time you’re actually spending on this thing”.

3. Formed habits & routines much easier

Turns out time tracking helped with my habits and routines, as well, in two not-so-obvious ways:

1 - If I would have the same entries at roughly the same hours, Toggl’s mobile app would start suggesting them, in those time intervals. This made my tracking way easier, and turned the clock into a kind of trigger for the habits I was trying to form:

2 - There was a deep satisfaction when I would look at my weekly timetable and see all the entries nicely aligned. Here’s one of the weeks that I’m most proud of. Just look at how satisfying those morning entries look:

It might sound trivial, but just these two simple tricks were enough to make me form and cement very helpful habits, without thinking too much about them.

4. Enjoyed my “chill time” more than ever

Here’s something interesting. This experiment, which was chiefly meant to increase my productivity, had another very unexpected — but very welcomed — effect: it increased the quality of my “chill time”, as well!

Firstly, as I was doing my work much more efficiently and saving time from all kinds of unexpected places, I found myself with more and bigger blocks of time on my hands. That meant I could fit in some leisure activities that were more time-consuming — but that I really enjoyed doing.

Secondly, all this increased amount of “chill time” came without the usual guilt I felt before, when I was “chilling” by scrolling through the Instagram feed, while I knew I had to fix that stupid programming bug at work.

This made me enjoy my free time like never before, and rekindle old passions like salsa & bachata —passions that I forgot how good they made me feel, amongst all this clutter of work and guilt and continuous distractions.

5. Made better decisions, overall

one of the toughest decisions in life

Since I was being as honest as I could with those time entries, weighing my decisions wasn’t that complicated anymore: I would look over my timetable, see exactly how much time I spent on something I decided to do, and look at how much I got out of it.

This way, “Is it a good idea to write that article for my 23rd birthday?” became “Is it a good idea to write that article for my 23rd birthday, given the fact that last week I spent 11 hours, and got nowhere with it?”. Not so hard to see the issue, from that perspective.

As a bonus, tracking my time solved another one of the biggest problems I had, in terms of decision making. I always had the tendency to overestimate my abilities and take on much more than I could actually handle. This obviously had some very bad consequences, like frustrating other people that were working with me.

Well, once I had this sheet that was telling me “this is how much it actually takes you to do it” and “this is how much free time you have on your hands right now”, deciding whether I could or could not handle something new became a factual decision, rather than a whim of my arrogant ego.

6. I finally felt in control of my time

As I said in the beginning, the irreversible passing of time was one of the biggest sources of stress and anxiety for me. And, while I’m still having my fair share of existential crises about it, doing this experiment drastically reduced that stress and anxiety. The main reason was that, again,

I finally felt in control of my time.

Yes, I was still binge-watching cat videos on youtube, probably more than I should. But at least it was a conscious decision, rather than a reaction to some algorithms designed by smart people to make me mindlessly spend hours on end on their website. And this “conscious decision” part was what made all the difference, for me.

And the coolest thing was that I didn’t have to change anything in the way I was spending my time, in order to feel better about it. Simply having it noted down was enough to give me that empowering feeling of control.

7. Felt more tense, uneasy and guilty (at first)

Surprise-surprise: this is not a silver bullet to shoot down all of life’s problems. It actually came with some not-so-nice consequences that I have to mention, if I am to paint an honest picture of my journey.

First up, I became more tensed and uneasy, especially in the beginning. I think that’s not too unexpected — just imagine having a cop on your back, 24/7, that sees every single thing your doing, and tracks it down on a sheet that’s gonna stare you in the face every Sunday. Quite a reason to feel uneasy…

shoutout to Mr. policeman, for not arresting me when I randomly asked him to pose for my blog

The other thing was guilt. Again, especially in the beginning. After the first few weeks, I was confronted with a hard truth: I was wasting time. Lots of it. And not in a vague way. I could actually count the hours I spent on things that were totally unproductive, projects that failed and times when I did something completely different than what I had to. Talk about pills that are hard to swallow…

I eventually managed to keep those bad sides under control through other habits, like meditation. That being said, this journey was definitely not all fun and rainbows.

So, that’s about it for my journey. Currently, I got so used to this system and it seems to work so well for me, that I’m gonna keep doing it — as long as I’ll be able to keep those negative sides under control.

Afterthought: it’s not as hard as it seems

I’ve been showing this article to some friends, to get some feedback before publishing it. A common response was: “Wow, that’s so impressive! You must be so disciplined! I could never do that.” And while it tickled my ego to hear this, the truth is, it’s really not that hard.

Yes, like all things, it requires effort in the beginning. But after you got it all set up and run it for a few days, it becomes pretty easy. 99% of the time, you’ll just press “continue” on a previous entry, which only takes 5 seconds. And since you’ll have to do it dozens of times a day, it will become instinctive much sooner than the usual “3–8 weeks to form a new habit”.

Even now, my timer runs “Blog — time tracking” for 9 minutes and 20 seconds. I don’t remember having made a conscious decision to start it.

Tips & tricks

If you like the idea and consider trying it for yourself, here are some things I wish I knew when I started, and that I hope will help you in your journey:

Try to be as honest as possible. The benefits are gone if you find yourself scrolling through your Instagram feed, while your timer says “preparing yearly report”
Keep it personal. Don’t mix it up with any tracking software from work. Don’t tell anyone about it. Basically, don’t do anything that would make you feel even remotely uncomfortable about recording those 2 hours of binge-watching Japanese eating contests on youtube.

Keep it simple. As simple as possible. Don’t overkill it with tags and projects (the way I did, in the beginning). You can always add those later. The whole system should be a pleasure to use, or you’ll have trouble using it at all.
One step at a time. At first, don’t change anything in how you spend your time. Just get used to tracking down what you’re already doing. The main focus is to get used to pressing “play” every time you’re doing something new.
Be patient with yourself. Once you start doing this in full honesty, you’ll probably see that your time isn’t really spent the way you imagined. That’s an easy way for guilt to creep in. But don’t despair. Once you become aware of this, things will get better by themselves. And keep in mind that, just by trying it out, you’ve already done something very courageous that’s a huge achievement in and of itself.

Conclusion

Tracking every minute of my time is, by far, the most effective thing I’ve ever tried, in terms of becoming more focused, getting more done and improving the quality of my time.

But, at the end of the day, this is just another way in which someone on this planet is trying to deal with the fact that life is hard. And if my trick doesn’t work for you, what will definitely work is keeping this healthy mindset that makes you read these kinds of articles, in the first place. Kudos to you, for keeping an open mind in the face of life’s hardships!

We’re all trying to make the most out of our time. Thank you for spending some of yours with me, today!

PS: This is the first article I’ve ever written, so I think you can imagine how anxious I am about it 😬 Any though (positive or constructive), clap, comment, share, or feedback in any form would be very, very much appreciated!