DEV Community: James Sanderson

Build the Retrieval Layer Before the Sales Agent

James Sanderson — Mon, 07 Sep 2026 06:31:40 +0000

The agent is the easy part. A research agent that pulls context, reasons over it, and drafts a brief is maybe two weeks of work if the context is available.

The context is never available. That is the project, and it is roughly three months.

Teams that invert this order build an impressive demo on curated data in week two, then spend six months discovering that production data does not resolve, does not chunk usefully, and does not have access controls the assistant can respect. Here is the build order that works.

Layer 1: Ingestion

Connectors for CRM, email, calendar, call recordings, support, product telemetry, and billing.

Design choices that matter more than they look:

Incremental sync with a replayable event log. You will change your chunking strategy. You will change your embedding model. Both mean reprocessing everything, and you do not want to re-pull from every source API to do it. Land raw payloads first, transform downstream.
Preserve raw payloads. Whatever you discard now is what you need in month four. Storage is cheaper than a re-integration.
Rate limits are the schedule. CRM APIs will not let you backfill three years of activity quickly. Plan the backfill as a multi-day job with checkpointing, not a script someone runs on a Friday.
Diarised transcripts, not merged text. A transcript that does not distinguish the rep from the buyer is close to useless for the questions you actually want to ask. Speaker attribution is not a nice-to-have.

Layer 2: Entity resolution

This is where the schedule goes wrong, and it goes wrong for everyone.

The same account is Acme Corp in the CRM, acme.com in product telemetry, Acme Corporation Ltd in billing, and ACME in the support system. The same person is a user ID, an email hash, a phone number, and a LinkedIn URL.

Retrieval quality is bounded by this mapping. A brief that misses half an account's history because two records never joined is worse than no brief, because the rep trusts it.

What actually helps:

Deterministic matching first — exact domain, verified email, external ID. Cheap, high precision, and it handles most of the volume.
Probabilistic matching for the remainder, with a review queue rather than an auto-merge. Silent bad merges are the worst failure mode here, because they surface as one customer seeing another customer's information.
Watch transitive merges. A resolves to B, B resolves to C, and the A-to-C link is wrong. Cap chain depth and require higher confidence for transitive links.
Persist the resolution as a graph, not as a one-off join. You will need it for privacy work too — rights requests need exactly this mapping.

Layer 3: Chunking and indexing

Generic chunking strategies perform poorly on sales data, because the semantically meaningful unit is not a fixed token window.

For call transcripts, chunk by topic segment, not by turn or by token count. An objection spans several exchanges — the buyer raises it, the rep responds, the buyer clarifies. Split that across a boundary and retrieval returns half an objection, which reads as agreement.

For email threads, the thread is the unit. A single message out of context is frequently misleading, especially short ones.

For CRM records, generate a natural-language summary at write time and embed that alongside structured filters. Embedding a raw record dump wastes most of the context window on field names.

Hybrid retrieval — semantic plus keyword — outperforms pure vector search noticeably here, because sales queries contain exact tokens that must match: competitor names, product SKUs, contract terms. Semantic similarity will happily return a passage about a different competitor.

Layer 4: Access control at query time

Non-negotiable, and much easier to build now than to add later.

A rep's assistant must not surface a deal they are not entitled to see. Enterprise customers will ask about this in security review, and "the UI filters it" is not an answer, because the UI is not where the retrieval happened.

Filter at the retrieval layer, before results enter the context window. Once a passage is in the prompt, it is in the answer, and no amount of instruction reliably prevents the model from using it. Post-hoc filtering of generated output is not a control.

Practically: encode ACLs as metadata on every chunk and apply them as a pre-filter in the vector query, not as a post-processing step.

The validation gate

Before building anything user-facing, ask the system this:

What were the three most common objections in lost deals last quarter, in the buyer's own language?

If it cannot answer, do not proceed to the agent. The answer requires working transcript ingestion, correct entity resolution across CRM and calls, chunking that keeps objections intact, and retrieval that can filter by outcome and date.

It is a single query that exercises every layer. Teams that pass it ship agents that work. Teams that skip it ship agents that hallucinate confidently, which is worse than shipping nothing.

Then, and only then, the agent

Once retrieval works, the patterns come quickly: pre-call research briefs with citations, constrained outreach drafting against an approved evidence library, questionnaire response generation, deal-risk signals.

Two things to carry into all of them. Every factual claim about the buyer needs a citation the rep can check in two seconds — an unsourced claim that turns out wrong costs more trust than ten correct ones earn. And log every prompt, retrieval set, and completion, because after your first model upgrade you will need to explain why quality changed.

The full engineering treatment — the five production patterns, guardrails, measurement, cost ranges, and a ninety-day rollout — is here: Generative AI for Sales: The Engineering Guide for 2026.

We build these as LLM integration and agentic workflow engagements.

Frequently Asked Questions

Why build retrieval before the agent?

Because the agent is roughly two weeks of work and the retrieval layer is three months. Building the agent first produces a demo on curated data, followed by months of discovering that production data does not resolve, chunk, or filter the way the demo assumed.

What makes entity resolution hard in sales data?

The same account and person appear under different identifiers in every system — CRM, billing, telemetry, support. Retrieval quality is bounded by that mapping, and a silent bad merge surfaces as one customer's data appearing in another customer's context, which is a confidentiality incident rather than a quality bug.

How should call transcripts be chunked for retrieval?

By topic segment rather than by turn or fixed token count. An objection spans several exchanges — raised, answered, clarified — and splitting it across a chunk boundary returns half an objection, which frequently reads as agreement.

Is vector search alone enough for sales retrieval?

No. Hybrid retrieval combining semantic and keyword search performs noticeably better, because sales queries contain exact tokens that must match — competitor names, SKUs, contract terms. Pure semantic similarity will return a passage about a different competitor.

How do you enforce access control in a RAG system?

Filter at the retrieval layer, before results enter the context window, using ACL metadata as a pre-filter on the vector query. Once a passage is in the prompt it is available to the answer, and instruction-based restrictions are not a reliable control.

What is a good readiness test for a sales retrieval layer?

Ask it for the three most common objections in lost deals last quarter, in the buyer's own words. Answering requires working transcript ingestion, correct entity resolution, objection-preserving chunking, and outcome filtering — one query that exercises every layer.

How to Delete Data From an Immutable Backup

James Sanderson — Mon, 07 Sep 2026 06:31:32 +0000

Here is a contradiction that shows up in almost every architecture review with a privacy component.

The right to erasure assumes data can be removed on request. Modern data architecture is built on the opposite assumption. Append-only event logs. Immutable object storage with object-lock enabled. Replicated backups across regions. Warehouse snapshots retained for year-over-year analysis. Derived models that encode information about individuals without storing a row.

Deleting a user row from Postgres takes one statement. Deleting that user from the system takes an architecture decision you probably made three years ago without knowing it.

Three patterns actually work. They are not equivalent, and the cost of picking one is almost entirely a function of when you pick it.

Pattern 1: Crypto-shredding

Encrypt every subject's personal data under a key unique to that subject. Store keys in a KMS. On erasure, destroy the key.

Every copy of the ciphertext — the primary database, last quarter's backup, the object-locked archive, the warehouse snapshot nobody remembers creating — becomes permanently unreadable. You did not delete the data. You deleted the ability of anyone, including you, to read it, which is functionally equivalent and vastly easier to prove.

What it actually requires:

A per-subject key hierarchy. Typically a per-subject data key wrapped by a rotating master key, so key rotation does not mean re-encrypting everything.
Envelope encryption at the field or record level, not full-disk. Disk encryption gives you nothing here — one key protecting everything means erasing one subject erases all of them.
A key-destruction audit trail. The destruction event is your evidence of erasure, so it must be immutable and timestamped.
Query planning that survives encryption. This is the real cost: you can no longer filter or join on encrypted fields. Deterministic encryption for equality lookups helps and leaks frequency information; blind indexes are usually the better tradeoff.

Adopt early or regret it. Greenfield, this is a week of design and mostly ordinary work. Retrofitted onto a system with ten years of plaintext personal data across forty tables, it is a multi-quarter migration where the hard part is not encryption but finding every field that needs it.

Pattern 2: Tombstone-and-compact

For event streams, deletion is a write, not a removal.

Publish a tombstone — a null-payload record keyed to the subject — and let a compaction process physically remove prior records for that key on a defined cycle.

Kafka's log compaction does exactly this natively. The failure mode is entirely operational:

Compaction is not immediate. There is a window between the tombstone and physical removal, and your retention documentation needs to state it honestly rather than imply instant deletion.
Every downstream consumer must handle tombstones correctly. A consumer that ignores null payloads and keeps its own materialised view has silently defeated the whole mechanism. This is worth an integration test per consumer.
Topics without a compacted cleanup policy will not compact. Someone will create one. Enforce the policy in your topic provisioning code, not in a wiki page.

Tombstoning composes well with crypto-shredding. Shred the key for correctness and immediate unreadability; tombstone-and-compact for actual space reclamation and to keep the streams honest.

Pattern 3: Reference indirection

Keep personal data in exactly one deletable store. Everywhere else — events, logs, warehouse, caches, search indices — holds only an opaque subject identifier.

Erasure becomes a single operation in a single place. Every other system keeps referring to a subject ID that no longer resolves to anything, which is usually fine: your analytics still count the event, it just cannot be tied back to a person.

This is architecturally the cleanest option and the one with the largest ongoing tax. Every read path that needs personal data becomes a join or a service call. Latency goes up. You will be tempted to denormalise "just this one field" into the event payload, and the moment someone does, the guarantee is gone.

The mitigation is enforcement rather than discipline: schema validation in CI that rejects event definitions containing personal-data-typed fields. Make the wrong thing fail the build, because it will not fail review reliably.

Picking one

Rough decision rule:

Greenfield? Crypto-shredding plus reference indirection. The combination is cheap now and eliminates most of the cost of every other privacy capability later.
Existing system, event-stream heavy? Tombstone-and-compact first, since it delivers immediate improvement, then reference indirection for new event types.
Existing system, monolith and warehouse? Reference indirection for new writes, crypto-shredding scoped to the highest-risk fields. A full retrofit is rarely worth it; a targeted one usually is.

The model layer, honestly

None of these patterns cover trained models. If a model was trained on a subject's data, deleting the source record does not remove their influence on the weights. Machine unlearning research exists; production-grade machine unlearning largely does not.

The currently defensible position is documented retraining cadences plus exclusion lists, stated plainly in your impact assessment. Do not claim erasure from model weights. Regulators are increasingly informed on this point, and a claim you cannot substantiate is worse than a limitation you disclosed.

Deletion as a tested contract

Whichever pattern you choose, make deletion a first-class API on every service that stores personal data, and test it in CI like any other contract.

A service that cannot delete becomes permanent compliance debt, and you want to discover that at merge time rather than during a rights request with a statutory deadline attached.

The wider engineering context — discovery, consent enforcement, rights-request pipelines, generated registers, and where this sits in a full stack — is in the complete guide: GDPR Software in 2026: A CTO's Build vs Buy Playbook.

We build this as part of custom software development work, usually alongside the cloud architecture decisions that constrain which pattern is even available.

Frequently Asked Questions

What is crypto-shredding?

Storing personal data encrypted under a key unique to each data subject, then destroying that key on an erasure request. Every copy of the ciphertext — including immutable backups and object-locked archives — becomes permanently unreadable without deleting or rewriting the copies themselves.

Does full-disk encryption satisfy the right to erasure?

No. One key protecting everything means you cannot erase a single subject without destroying access to all data. Crypto-shredding requires per-subject keys with envelope encryption at field or record level.

How do you delete from a Kafka topic?

Publish a tombstone — a null-payload record with the subject's key — on a topic configured for log compaction. Compaction physically removes earlier records for that key on its cycle. Verify that every downstream consumer honours tombstones, since a consumer maintaining its own materialised view will otherwise retain the data.

Can you delete personal data from a trained machine learning model?

Not reliably in production today. Deleting the source record does not remove its influence on the weights. The defensible approach is documented retraining cadences plus exclusion lists, disclosed as a limitation in your impact assessment rather than claimed as erasure.

What is the performance cost of encrypting personal data per subject?

The encryption itself is negligible; the query cost is not. You cannot filter or join on encrypted fields directly. Deterministic encryption enables equality lookups but leaks frequency information; blind indexes are usually the better tradeoff. Budget for query redesign, not for CPU.

Should deletion be tested in CI?

Yes. Treat deletion as a contract on every service that stores personal data and test it like any other API contract. Services that cannot delete become permanent compliance debt, and merge time is a much cheaper moment to discover that than a rights request with a statutory deadline.

Offline-First POS — Event Streams, Client IDs and Merging Divergent Terminals

James Sanderson — Wed, 02 Sep 2026 16:03:03 +0000

A restaurant POS is one of the few consumer-scale products where offline operation is not a nice-to-have. The venue's network is consumer broadband shared with a card terminal and a guest wi-fi network, and it will fail during service. When it does, the software has to keep taking orders and firing tickets or the business physically stops.

This post covers the concrete architecture: what lives locally, how identifiers work, how divergent terminals merge, and which operations genuinely cannot be merged.

Local-first writes

The terminal holds a local database containing the current menu, open checks, the session's orders, and an outbound operation queue.

Every user action writes locally and synchronously. Nothing in the interaction path waits on a network call — an action that blocks on a round trip is a defect that will be discovered at peak service, when a queue of actual people is forming.

Synchronisation is a background process reading the outbound queue and reconciling with the server. It has no bearing on whether the user can proceed.

Client-generated identifiers

All identifiers are generated on the terminal. Universally unique values, never server-assigned sequences.

This matters more than it first appears. A server-assigned identifier means an order created offline has no identity until it syncs, so every local reference is provisional and must be rewritten afterwards. Every relationship becomes a two-phase problem, and the bugs that result are subtle and hard to reproduce.

Client generation makes an offline-created order a first-class entity from the moment it exists. Human-facing order numbers can still be assigned per-venue per-day locally, with a terminal prefix to avoid collisions.

Checks as operation streams, not documents

Here is the core modelling decision.

Do not represent a check as a mutable document that terminals overwrite. Represent it as an append-only sequence of operations:

{ id, checkId, terminalId, seq, ts, type: 'ADD_ITEM',    payload: { itemId, qty, seat, modifiers } }
{ id, checkId, terminalId, seq, ts, type: 'VOID_ITEM',   payload: { lineId, reason } }
{ id, checkId, terminalId, seq, ts, type: 'APPLY_DISCOUNT', payload: { ... } }

The current state of a check is a fold over its operations. Two terminals that diverged offline are merged by unioning their streams and replaying, which is commutative for most operation types.

Compare this with the naive approach. Last-write-wins on a whole check produces a specific, expensive bug: the bar terminal added a drink, the till voided a starter, and whichever version arrives second silently discards the other terminal's work. One direction is uncharged revenue; the other is a customer billed for something they sent back. Both destroy operator trust faster than nearly any other defect.

Operations that cannot be merged

A small set genuinely requires arbitration, and it is better to be explicit than to pretend otherwise:

Settling a check. Two terminals settling concurrently must not both succeed. This requires a server round trip, or a venue-local coordinator when the cloud is unreachable.
Voiding an item already served. A void that arrives after the kitchen has fired and served the item is a business decision, not a data merge.
Anything with an external side effect — refund issued, loyalty points redeemed — where reversing costs real money.

Define this list explicitly, surface the requirement in the interface, and design the fallback deliberately. Telling a server that payment needs connectivity is much better than allowing two payments to land and reconciling through support tickets a week later.

Visible failure beats silent failure in this environment. Staff need to know the terminal is offline and orders are queued; ambiguity produces double-fired tickets.

The local real-time layer

Kitchen displays are a local problem, not a cloud one. A fired course needs to appear on the correct station within about a second.

Terminals and kitchen screens on the same venue network should communicate directly, with the cloud as a synchronisation channel rather than as the message bus. Routing a ticket from a till to a screen four metres away via a remote data centre is one dropped connection away from a service failure, and that is a design choice rather than bad luck.

Also budget for printers. Every venue has thermal printers, and escape codes, network discovery, per-model quirks and paper-out handling form a deeper body of work than anyone plans for.

The data model decision that matters later

Store operations, not summaries. Item-level, timestamped, with modifiers and voids preserved rather than collapsed into settled totals.

The reason is that the highest-value features arrive later and all depend on this history: demand forecasting at item level to drive prep and ordering, labour scheduling derived from those forecasts, and anomaly detection on voids and discounts. A team that stores only check totals has thrown away the input those features require, and it cannot be reconstructed.

This is the single cheapest decision to make correctly in week three and the most expensive to regret in year two. More on the applied-model side in our work on AI development services.

Full build guide covering the domain model, modifier design, payments, tenancy, hardware and costs: Cloud Based Restaurant POS Systems: The 2026 Build Guide.

Frequently Asked Questions

Is this just event sourcing?

It shares the mechanism but applies it locally for a specific reason: enabling deterministic merges between devices that diverged without coordination. You do not need a full event-sourced backend to benefit — the operation stream can be a local concern that projects into conventional server-side storage.

What storage should the terminal use?

Anything durable with transactional writes and reasonable query support — SQLite on native platforms, IndexedDB in the browser. What matters far more than the choice is that writes are synchronous from the user's perspective and never block on the network.

How do you handle clock skew between terminals?

Do not rely on wall-clock time for ordering. Use per-terminal sequence numbers plus a logical clock, keeping timestamps for display and audit only. Terminal clocks drift, and merges that depend on them produce non-deterministic results.

How large can an operation stream get before it needs compaction?

A single check rarely exceeds a few dozen operations, so per-check folding is trivial. Compaction matters at the venue-day level for archival — snapshot the settled state and retain the operation history separately, keeping it rather than discarding it, since it is the input to later analytics.

What happens if a terminal is offline for days?

Bound it. Define a maximum divergence window, typically a single service day, after which the terminal requires reconciliation before accepting new work. Unbounded divergence produces merges nobody can reason about and staff cannot explain to a customer.

Seven Technical Questions I Ask Every Fintech Development Agency

James Sanderson — Wed, 02 Sep 2026 16:00:02 +0000

Every agency has a fintech case study now, because almost every app touches money somewhere. That makes portfolios close to useless as a filter. What does work is a twenty-minute technical conversation with the engineers who would actually do the build — not the account team — built around questions that cannot be answered from a sales script.

Here are the seven I use, and what the answers tell you.

1. How do you represent money?

The correct answers are minor units as integers, or a fixed-scale decimal type, with a reason attached. What you are listening for is whether they have opinions about rounding, about where currency conversion happens, and about how fees are represented relative to the principal amount.

Floating point ends the conversation. It sounds obvious. It still shows up.

2. A payment succeeds at the provider and your service times out before recording it. What happens?

This is the single best question on the list because there is no way to answer it plausibly without having lived through it.

The answer you want includes an idempotency key generated before the outbound call, a persisted intent record written before the request rather than after the response, a reconciliation job that reads the provider's settlement file and closes the gap, and a bounded window with alerting for anything unresolved.

The answer you do not want is any variation on retry logic alone. Retry without idempotency is how duplicate charges happen, and it happens precisely under the network conditions you have no control over.

3. How does a support agent correct a wrong transaction?

If the answer involves updating a row, they have not operated a regulated system.

The correct pattern is a compensating entry: a new record that reverses the effect of the original while leaving the original intact and linked. The original transaction remains queryable forever. The correction has its own actor, timestamp and reason code.

This matters because destructive corrections are invisible in aggregate and catastrophic in an audit. You cannot explain a balance whose history has been edited.

4. Where does the ledger live, and is the balance stored or derived?

There are several defensible architectures — ledger inside the service, ledger as a separate service, ledger as an event-sourced projection. Having no view is not defensible.

The property to insist on is that balances are derivable from immutable double-entry records. Storing a balance as a cache is fine. Storing it as the truth is not, because it means a lost update is silently permanent.

Watch specifically for how they handle the gap between authorisation and capture, and how a currency conversion produces entries. Naive designs fall apart at both.

5. Which parts of PCI scope do you keep, and which do you push out?

The right answer draws a boundary. Tokenising provider or semi-integrated terminal handles card data; your application never sees a PAN; your scope collapses to a much smaller self-assessment.

Teams who cannot draw that boundary on request will discover it during an assessment, at which point moving it is a refactor rather than a design choice.

6. How do you test a risk or fraud model before it goes live, and how do you roll it back?

Baseline answers: shadow deployment where the new model scores traffic without acting, champion-challenger comparison on live traffic, a kill switch that reverts to the previous version or to rules alone, and stored decision records containing inputs and model version so any decision can be replayed.

The follow-up is more revealing: how long would it take, in your architecture, to add a new signal to the fraud model and get it safely into production? If the answer is weeks, then during an actual attack your losses accrue for weeks.

This is also the question that separates AI-as-architecture from AI-as-vendor-logo. Anyone can call a scoring API. Owning the features, versioning the model and being able to explain a decline is engineering, and it is what the LLM integration and applied-AI side of this work actually consists of.

7. What have you had to rebuild on a previous fintech project, and why?

An engineer who has shipped will have an answer immediately, usually with some feeling attached. The most common honest answers are the ledger, the onboarding state machine, and the notification system.

No answer means either not enough shipping, or not enough candour. Both are disqualifying for different reasons.

Why the AI answers matter more than they used to

The operational economics of fintech shifted over the last two years, and it is worth being precise about how. Document review, sanctions alert triage, transaction monitoring and case handling used to scale linearly with headcount. They now scale substantially with compute, provided the system is built so automated decisions carry a stored input, a model version, a confidence score and a defined escalation threshold.

That is an architectural requirement, not a procurement one. A case management layer that treats model output as evidence — with a human adjudicating the flagged minority — is what turns automation into a saving rather than a compliance exposure.

The rest of it

The full write-up covers how to read a fintech portfolio, ledger design in more depth, the four-part cost model, engagement structures and the proposal red flags: How to Choose a Fintech App Development Company in 2026.

Frequently Asked Questions

Should these questions go to the sales team or the engineers?

The engineers who would be assigned to your build, named in the contract. Answers from an account lead are second-hand and tell you nothing about who writes your data layer.

What if an agency answers well but has no fintech portfolio?

Strong answers to questions 2, 3 and 4 are worth more than a portfolio. The failure modes described in those answers only get internalised by teams who have operated systems under real conditions, whether or not the domain was labelled fintech.

Is idempotency really necessary if we use a major payment provider?

Yes. Providers give you the mechanism — idempotency keys — but nothing enforces that you use it correctly, and the failure happens on your side of the call when a response is lost. The provider's reliability does not cover your write path.

How do we verify a claimed audit trail?

Ask to see a query that answers a concrete question: who accessed a specific customer record on a given date, and what changed. If the answer requires log grepping rather than a queryable event store, the audit trail is aspirational.

Do we need our own ledger if the payment provider tracks balances?

Yes. The provider records movement through their system, not your fees, holds, splits or accounting treatment. More importantly, with only one record you cannot reconcile — a discrepancy is undetectable by definition.

How much does the ledger design actually cost upfront?

Typically a few weeks of senior engineering time in the first phase. The alternative is a four-to-six month rebuild later, run in parallel with production, reconstructing history from settlement files. The ratio is not close.

Modelling a Music Catalogue Without Regretting It Later

James Sanderson — Tue, 01 Sep 2026 16:11:18 +0000

If you are building anything that plays licensed music, the schema you choose in week two determines whether accurate royalty attribution is even possible. Not difficult — possible. You cannot report a split you have nowhere to store.

This is a walkthrough of what breaks in the obvious model and what to do instead.

The model everyone starts with

tracks(id, title, artist_name, album_name, duration, audio_url, isrc)

It is clean, it demos beautifully, and it fails on contact with a real catalogue. Here is the sequence, which happens in roughly this order every time.

Week 3. A track has three writers, one performer and a producer, all with different rights and different splits. There is one artist_name column. You add artists as a many-to-many and feel clever.

Week 5. The same recording arrives from a second distributor with slightly different metadata and a different local identifier. You now have two rows for one recording, and any play count you report is split across both.

Week 7. A compilation arrives. The recording already exists on its original album. Your model conflates recording with release, so you either duplicate the recording or lose the compilation context. Both are wrong.

Week 9. Explicit and clean versions. Different audio, same composition, sometimes the same ISRC in practice because someone was careless upstream.

Week 12. Territory restrictions differ per release, per recording and per agreement. There is no column for it, and geo-enforcement is now a client-side check somebody can bypass.

Month 6. A rights split is corrected and backdated. You have stored aggregates. You cannot recompute.

The model that survives

Separate the concepts that the industry separates. The vocabulary exists for a reason.

Work — the underlying composition. Writers and publishers attach here.
Recording — a specific performance of a work. Performers, producers and the master rights holder attach here. This is what actually gets streamed.
Release — a package containing recordings: album, single, compilation. Territory and availability windows attach here, not to the recording.
Contributor — a party with a role (writer, performer, producer, publisher, label) and a time-bounded rights share on a work or a recording.
Identifier — an external id (isrc, iswc, distributor-local) attached to a work or recording, with the source recorded, many-to-one.

Three properties fall out of this that you cannot get from the flat model:

Deduplication becomes possible. Two deliveries of the same recording resolve to one recording entity with two identifier rows, rather than two competing rows.
Attribution becomes expressible. A split is a relationship with a validity window, not a column that gets overwritten.
Territory lives at the right level. Availability is a property of a release in a market, which is how the agreements are actually written.

Splits must be time-bounded

This is the single most commonly missed detail, and it is what makes retroactive corrections survivable.

Do not store a contributor share as a current value. Store it as a record with valid_from and valid_to. When a split is corrected and backdated, you insert a new period rather than overwriting anything, and the historical calculation for any past month remains reproducible from the data as it applied then.

If you overwrite, you have silently destroyed your ability to explain a past statement. That conversation with a rights holder does not go well.

Events, not aggregates

The corollary on the reporting side: plays are immutable events, and every aggregate is a derived artefact that can be rebuilt from them.

play_events(id, recording_id, user_id, started_at, ms_played,
            qualified, client, country, session_id, ...)

Store enough to apply whatever definition of a qualifying stream your agreements specify — including one you might renegotiate later — and enough for fraud detection to distinguish genuine listening from synthetic patterns. You cannot reconstruct signal you never captured.

Then every monthly statement is a function over events plus the rights model as it stood, which means:

A backdated split correction is a recomputation, not an archaeology project
Fraud identified after payout can be excluded and the period restated
A disputed statement can be reproduced line by line

Teams that store aggregates and drop raw events save a little storage and lose the ability to answer the only questions that ever really matter.

Practical notes

Do not trust ISRCs to be unique. They are supposed to be. In real catalogues they are not. Treat them as one identifier among several, with a source attached.
Model unknown contributors explicitly. Deliveries arrive with missing credits constantly. A null is not the same as "we know there is a writer we cannot identify", and only one of those can be resolved later.
Keep qualified as a derived, recomputable flag, not something written at ingest. The definition changes when agreements change.
Version the rights model itself. Not just the splits — the rules for applying them.

Why this is worth the upfront cost

Remodelling a catalogue after launch means migrating every playlist, every recommendation index and every historical royalty record simultaneously, while continuing to report accurately throughout. It is one of the more painful migrations available in consumer software, and it is entirely avoidable by spending a fortnight on the model before writing the player.

When you evaluate a build partner for this kind of product, ask to see their catalogue model and ask what happens when two distributors deliver the same recording with conflicting credits. The answer tells you whether they have ingested a real catalogue or a test fixture.

Full guide, including build budgets, discovery cost economics and the questions to ask on a shortlisting call: Music Streaming App Development Company. Related: how we approach custom software development for systems like this.

Frequently Asked Questions

Is a relational database the right choice for a music catalogue?

Generally yes for the catalogue and rights model, where relationships and constraints matter and correctness beats flexibility. Play events are usually better served by an append-only store or event log, with aggregates materialised into whatever your reporting layer queries.

How do I dedupe recordings arriving from multiple distributors?

Match on identifiers first, then fall back to fuzzy matching on title, contributors and duration, with a human review queue for anything below a confidence threshold. Store every source identifier rather than picking a winner, so a bad merge is reversible.

Should territory restrictions live on the recording or the release?

On the release, in almost all cases, because that is how agreements are written. A recording can be available in one market on one release and unavailable there on another.

What is the minimum I should capture in a play event?

Recording, user, start time, duration played, client, country, session, and enough context to apply a qualifying-stream rule you might change later. Fraud detection needs pattern signal too, so do not sample aggressively.

How do I handle a split correction backdated a year?

Insert a new time-bounded split period rather than updating the existing one, then recompute affected statements from play events. This only works if you never overwrote history and never discarded the events.

Do I need this level of rigour for a small niche platform?

The catalogue and rights model, yes, because the cost of getting it wrong scales with how long you run rather than how large you are. Discovery sophistication and platform tooling can absolutely wait.

Five Interview Questions That Separate AI Engineers From Demo Builders

James Sanderson — Tue, 01 Sep 2026 16:09:58 +0000

If you are running interviews for AI roles and your process is a LeetCode round plus a chat about transformers, you are measuring almost nothing that predicts performance. Attention mechanics are memorisable and rarely relevant to application work. Graph traversal has never told anyone whether a candidate can keep an inference bill under control.

What actually predicts success is judgement under production constraints. Here are the five questions we use, what a strong answer sounds like, and the take-home format we have found genuinely discriminating.

1. Describe an AI feature you shipped that did not work

You are listening for a failure signature, not an apology.

Strong answers are specific about how the failure showed up: a class of inputs that silently degraded, a retrieval step that returned plausible but wrong documents, latency that only broke under concurrency, a tool call that fired on untrusted user input. They usually include what the fix cost and what they would do differently.

Weak answers are either "we iterated on the prompt" or a story about something that was really a product disagreement. Neither tells you the candidate has watched a probabilistic component misbehave at scale.

2. How did you know it was getting better between releases?

This is the highest-signal question in the set, and most candidates have no real answer.

What you want to hear, unprompted: a golden dataset of representative cases including deliberately hard ones, an automated scoring method appropriate to the task, a regression run wired into CI, and traces in production that link a user complaint back to exact inputs and model version.

Follow up on the details, because that is where experience shows. How many cases? Who labelled them? What did you do when two labellers disagreed? What happened when the automated score and human judgement diverged? Candidates who have really built this have opinions and war stories. Candidates who have not will describe an observability vendor.

3. Walk a request through your system, including the timeout

A straightforward architecture question with a specific trap in it. Most people describe the happy path fluently and stop.

Push on failure. What happens when the model call times out — retry, fallback model, degraded response, queue? Is the retry idempotent? If the request had already triggered a tool call with a side effect, what undoes it? What does the user see meanwhile, and how does streaming interact with a mid-stream failure?

This is ordinary distributed systems thinking applied to a component that fails in unusual ways, which is why strong backend engineers often outperform AI specialists here.

4. What did it cost per thousand requests, and what did you do about it?

The economics question, and close to binary in its discriminating power.

In conventional software the marginal cost of a request rounds to zero, so engineers are trained not to think about it. In AI products it is real money, varies by an order of magnitude with design, and scales with success. Routing everything to a frontier model is the default that quietly destroys margins.

Good answers reach for cheap intent classification, caching, small models for the easy majority, escalation only where it matters, and trimming what goes into the context window. They usually come with a number and a story about the optimisation that was rejected as premature and later turned out not to be.

5. Where did you decide not to use a model?

The maturity question, and my personal favourite.

Experienced engineers have a ready list: a rules engine that was cheaper and testable, a database query that was exact, a well-designed form that removed the ambiguity entirely, a search index that beat a retrieval pipeline on both latency and accuracy. They explain it without defensiveness.

Candidates who answer every problem with an LLM, including ones a SQL query would solve, will build you an expensive and fragile architecture. This question surfaces that in about ninety seconds.

The take-home that actually works

Do not ask for a chatbot. Everyone can build a chatbot and you will learn nothing.

Give them thirty real examples of a classification or extraction task from your domain, including several genuinely ambiguous ones and — this is the important part — a few that are mislabelled. Ask for a working solution, an evaluation harness, and a short written analysis of where it fails. Cap it at four hours and say so explicitly.

The accuracy number barely matters. What you are grading is the analysis. Did they notice the bad labels and say so? Did they characterise the failure modes rather than just reporting a score? Did they explain what they would need to push it further?

Assume AI coding assistants will be used, because that is how the job is done now. Screening for their absence selects for the wrong thing. What you are testing is whether the candidate can critically review generated output — whether the harness measures the right thing, whether edge cases are covered, whether they caught the plausible-looking helper that quietly drops nulls.

Red flags worth encoding into your process

A portfolio of demos with nothing operated behind any of them
No answer at all to how quality was measured
Framework fluency substituting for systems reasoning
Reflexive model-first thinking on problems that do not need a model
Vagueness about cost, which usually means nobody ever asked
No instinct around prompt injection, cross-tenant leakage, or tool calls triggered by untrusted input

The full guide — including rate benchmarks by market, the four AI developer archetypes, and a framework for choosing between in-house hiring and a build partner — is here: Hire AI Developer in 2026. If you would rather not run the search at all, this is roughly how we structure LLM integration work for teams in that position.

Frequently Asked Questions

Should I still run an algorithms round for AI roles?

A short one is fine as a general coding filter, but do not weight it heavily. It measures nothing specific to AI engineering. The take-home plus the five questions above will give you far more signal per hour of interviewer time.

How do I test for agentic systems experience?

Ask what happens on step seven of a multi-step workflow when a tool returns something unexpected. Look for timeouts, idempotency, compensating actions and human handoff. It is distributed systems failure design, so engineers from that background pick it up fast.

Is prompt engineering still a real skill to hire for?

Not as a standalone one. Modern models are robust to phrasing and structured output modes removed most of the parsing tricks. It has collapsed into ordinary craft, the way writing good SQL did.

What if a candidate has no production AI experience at all?

Frequently fine, sometimes preferable. A strong senior engineer with cost discipline and a measuring habit ramps up quickly, and their fundamentals do not decay the way model-specific knowledge does. Screen the fundamentals hard and be explicit that ramp-up time is budgeted.

How long should the take-home be?

Four hours, capped and stated. Longer exercises select for people with free time rather than people with skill, and they materially reduce your acceptance rate among strong senior candidates who are already employed.

What is the most common hiring mistake you see?

Writing a job description that lists all four AI archetypes at once — product engineer, ML engineer, platform engineer, agentic systems engineer. It attracts generalists who are shallow in each. Pick one, and hire a team or a partner if you genuinely need coverage across several.

Building a Local Evaluation Harness for Clinical AI

James Sanderson — Mon, 31 Aug 2026 08:12:25 +0000

Every clinical AI vendor shows you validation results. Almost none of them were produced on your population. This post is about the infrastructure that closes that gap, and why it has to be yours rather than theirs.

Why vendor validation does not transfer

Model performance in healthcare is unusually sensitive to three things that vary enormously between institutions.

Case mix. Prevalence changes the positive predictive value of a fixed-sensitivity model directly. A sepsis model validated in a tertiary referral centre encounters a different base rate at a community hospital, and the same threshold produces a different alert burden.

Documentation practice. Models trained on free text or structured fields inherit the conventions of the site that produced the training data. Different templates, different macro usage, different specialty conventions — all shift the input distribution in ways that are invisible until measured.

Coding behaviour. Local code sets, mapping decisions, and the ten percent that did not map cleanly and got assigned something adjacent. This is the least discussed and frequently the largest source of divergence.

None of these are vendor failures. They are why local evaluation is not optional.

The components

1. A held-out local sample with defined ground truth

The hard part is ground truth, not sampling. For some tasks a downstream coded outcome is adequate. For most, you need adjudicated labels — clinicians reviewing a sample against a written rubric, with a second reviewer on a subset to measure agreement.

Budget for this honestly. A few hundred adjudicated cases is usually enough for a first read and is typically the most expensive line in the whole exercise. It is also the line most likely to be cut, which is how organisations end up evaluating against a proxy outcome that answers a different question.

2. Stratified performance reporting

Report performance overall and separately for the subgroups where failure would be most consequential. At minimum: age bands, sex, race and ethnicity as recorded, primary language, payer class, and site of care.

This is not a compliance exercise. Aggregate performance can be entirely acceptable while a subgroup with a different base rate or different documentation patterns is being systematically failed, and the aggregate number will never show it.

3. Operating point selection with the alert burden attached

A model is deployed at a threshold, not at an area-under-curve. Pick the operating point against the burden it produces — alerts per clinician per shift — and involve the people who will receive those alerts in choosing it.

Alert fatigue is the dominant failure mode for clinical decision support, and it is a threshold decision made by someone who never sees the consequence.

4. Scheduled re-evaluation

Model drift in healthcare is real and its failure mode is silent. Performance degrades gradually while clinician trust remains at the original level, because nothing is designed to notice.

Schedule the re-run — quarterly is a reasonable default — on a fresh local sample, with the same stratification. Define in advance what degradation triggers a review and who has the authority to withdraw the tool. If nobody has that authority, the monitoring is decorative.

5. Data plumbing

Practically, this means a reproducible extract, a stable identifier strategy that survives de-identification, and a mapping layer that reconciles local codes to standard terminologies with the unmapped remainder made explicit rather than silently dropped. Bulk FHIR export handles a lot of this now, with the caveat that any extract without a defined retention policy, access model and de-identification standard is a liability that grows with every run.

What to demand at contract time

Put these in the agreement, because retrofitting them is much harder:

The right to evaluate on your own data before full deployment
Access to model outputs at a granularity that supports subgroup analysis
Notification when the vendor updates the model, with a defined re-evaluation window
A documented statement on whether your inputs are retained or used for training
An exit path that returns your data and does not strand the workflow

The fourth is a business associate agreement question and is frequently answered vaguely. Push for specificity.

Longer piece covering data readiness, interoperability, governance and engagement models: Healthcare IT Consulting Services: What Actually Moves the Needle in 2026. General ML engineering context in our machine learning development services guide.

Frequently Asked Questions

How many labelled cases do we need?

For a first read on a binary classification task, a few hundred adjudicated cases usually suffices, with more needed for rare outcomes or for meaningful subgroup precision. Power the subgroup analysis deliberately rather than hoping the sample covers it.

Can we use retrospective data or does it need to be prospective?

Retrospective is fine for initial evaluation and much cheaper. Prospective silent-mode running — the model produces predictions nobody acts on — is the stronger design for the deployment decision, because it exercises the real data pipeline rather than a curated extract.

Who adjudicates the ground truth?

Clinicians in the relevant specialty, working from a written rubric, with a second reviewer on a subset so you can report inter-rater agreement. Without that agreement number, a disappointing model result cannot be distinguished from noisy labels.

What drift threshold should trigger a review?

Define it as a relative degradation from your local baseline rather than an absolute value, set it before deployment, and set it separately for the subgroups you care about. The specific number matters less than agreeing it in advance and naming who acts on it.

Does this apply to ambient documentation tools too?

Yes, with different measures. Note quality against a rubric, blinded review, and error categories that matter clinically — fabricated content, omitted findings, incorrect attribution — rather than a single accuracy score.

Is bulk FHIR export enough to build this?

It covers a lot of the extraction, but not terminology reconciliation, ground truth, or governance. Treat it as the plumbing rather than the solution, and do not build an extract pipeline before deciding its retention and access policy.

Your CI Bill Tripled Because Your Team Started Using AI

James Sanderson — Mon, 31 Aug 2026 08:12:19 +0000

If your continuous integration spend jumped in the last eighteen months and nobody changed the pipeline, the explanation is probably sitting in your editor. AI assistants made producing code cheap. They did nothing for the systems that verify and ship it, and those systems bill by the minute.

This is a practical post about what breaks, in what order, and what to change first.

The arithmetic

CI cost tracks merge volume, not headcount. If merges doubled, expect build spend to roughly double even with an unchanged configuration. That part is expected.

What is not expected is that per-merge duration also degrades. More merges means more contention for runners, more queueing, and — critically — more encounters with tests that fail intermittently. A test failing one run in twenty is background noise at ten merges a day. At forty it fires twice a day, someone re-runs the pipeline, and you have paid twice for the same verification.

So the bill grows faster than the linear expectation, and the loop gets slower at the same time.

What to fix, in order of return

1. Test selection

Running the full suite on every commit is the single largest recoverable cost in most repositories. Map which tests exercise which modules and run only the affected set on pull requests, keeping the full suite for the main branch and a nightly run.

Most language ecosystems have tooling for this now, and even a crude version — directory-based mapping — captures most of the benefit. The reason it does not get done is that it is a week of unglamorous work, not that it is hard.

2. Verify your cache actually helps

Dependency caching is frequently configured once, never measured, and occasionally slower than a cold build because the restore is fetching a large archive over a slow link. Time a cached run against a cold run before assuming the cache is helping. This takes an hour and surprises people regularly.

3. Right-size the runners

Runner class is typically chosen during a debugging session and inherited forever. Compile-heavy jobs may want more CPU; test jobs are often memory-bound and over-provisioned on cores. Measure a representative job at two or three sizes and pick deliberately.

4. Prune the matrix

Every matrix build multiplies cost. Check which combinations of runtime version and OS your customers actually run and which your support policy actually covers. The intersection is usually smaller than the matrix.

5. Shut down idle environments

Non-production environments running twenty-four hours a day for a team that works eight is a scheduled-shutdown problem, not an architecture problem. It is an afternoon of work.

The problem that is not a cost problem

Once the pipeline is efficient, the remaining bottleneck is usually human: pull request review capacity did not scale with generation capacity.

Measure two numbers separately — time from PR opened to first review, and time from first review to merge. If the first dominates, you have a capacity problem, and no amount of pipeline tuning will fix it. Options that actually work:

Automatically splitting large changes, since review time scales worse than linearly with diff size
Defining categories of change that do not require two human reviewers, explicitly rather than by convention
Using model-assisted review for the mechanical pass — style, obvious null handling, missing test coverage — so humans spend attention on design and correctness
Setting an explicit queue-length threshold that triggers a team conversation, so the constraint is visible rather than ambient

Dependency drift

Worth a specific note because it is new. Generated code adds dependencies more casually than humans do — asked to parse a date, a model reaches for a library rather than writing four lines. It has no opinion about maintenance status, licence compatibility, or whether the package was published last week by an anonymous account.

Enforce this at merge time rather than in an audit: policy on new dependencies with a clear exception path, a software bill of materials generated on every build, and secrets scanning that runs before the commit lands rather than after.

Measuring whether it worked

Track cost per merge and duration per merge as trends. Track PR cycle time split into the two phases above. Track flake rate honestly — count re-runs, not just failures. And keep the four delivery metrics as the spine: deployment frequency, lead time from merge to production, change failure rate, and time to restore.

The longer version, covering platform engineering, engagement models, and how to vet a vendor for this work, is at DevOps Services Company: How to Choose the Right Partner in the AI Era. If you are building the agent-assisted side of this, our agentic workflow development work covers how to scope those loops safely.

Frequently Asked Questions

How do I find which tests to skip on a pull request?

Start with a directory-based mapping from changed paths to test suites — crude but effective. Language-specific tooling for precise coverage-based selection exists in most ecosystems and is worth adopting once the crude version proves the value.

Is model-assisted code review actually useful?

For the mechanical pass, yes: style violations, missing error handling, absent test coverage, obvious null cases. It is not reliable for design judgement or for spotting the subtle logic error, so treat it as a first filter that reduces what a human must attend to, not as a replacement reviewer.

Should flaky tests be quarantined or fixed?

Quarantine to unblock, then fix on a schedule with an owner. Permanent quarantine is how a suite loses its meaning — after a year of accumulation, nobody trusts a failure and the suite stops functioning as a gate.

Does self-hosting runners reduce cost?

Sometimes, but it converts a variable cost into a maintenance obligation. Exhaust configuration wins first; they are cheaper, reversible, and do not add a system someone has to own.

How much CI spend is typically recoverable?

Thirty to fifty percent on a neglected pipeline, mostly from test selection and caching. Ten to fifteen on a well-maintained one. Measure your own baseline before believing any number, including this one.

What is a reasonable pipeline duration target?

Under ten minutes for the feedback a developer actively waits on. Beyond that, people context-switch away and return to failures they have to reconstruct, which degrades change size and recovery time in ways the invoice never shows.

Your Eight-Step Agent Is Wrong a Third of the Time

James Sanderson — Sat, 29 Aug 2026 05:55:30 +0000

Let's start with arithmetic, because the arithmetic is the entire post.

Single step accuracy:  0.95
Chain of 8 steps:      0.95 ^ 8  =  0.663

A model call that is right 95% of the time is a genuinely good component. Chain eight of them, each consuming the previous output, and your end-to-end success rate is about 66%.

That is a coin flip with extra latency. And it gets worse, because the failures are silent — every individual step returned something structurally valid and semantically plausible. Nothing threw. Your logs look clean.

This is the failure mode that caught a lot of teams in the middle of 2026, after agentic scaffolding matured enough that people started shipping chains rather than single calls.

Why the failures are silent

In deterministic software, a broken step throws. You get a stack trace pointing at the line.

In an agent chain, step 3 misidentifies which customer record is relevant. It returns a perfectly well-formed record. Steps 4 through 8 then execute flawlessly against the wrong record. The output is coherent, confident, correctly formatted, and about a different customer.

There is no exception anywhere in that trace. Everything succeeded. The system is simply wrong, and it will be wrong at scale until a human notices something odd about a specific case.

Four things that actually help

1. Shorten the chain

The single most effective intervention, and the least fashionable, because "we replaced the agent with three deterministic steps and one model call" does not make a good demo.

Every step you can make deterministic is a step that cannot compound error. If step 4 is "look up the customer by ID", that is a database query, not a model call. A surprising proportion of agent chains in the wild contain model calls doing work a function could do — usually because the agent framework made it easy.

0.95 ^ 8 = 0.663
0.95 ^ 3 = 0.857   ← same task, 5 steps made deterministic

2. Validate between steps

Not "did it return JSON" — schema validation is table stakes and catches almost none of the real failures. Validate the semantics at each boundary:

Does the returned ID exist in the source system?
Is the amount within the range this account can produce?
Does the retrieved record's owner match the requesting user?
Is the classification one of the fourteen permitted values, or did the model invent a fifteenth?

Each check is a few lines and converts a silent wrong answer into a loud failure you can handle.

3. Constrain the action space hard

The instinct when building an agent is to give it every tool it might need. This is backwards.

Fewer tools, tighter schemas, narrower parameter ranges. Every tool you expose is a branch the agent can take incorrectly, and the failure rate scales with the branching factor rather than with model quality. The teams that succeeded with agents this year all did the same unglamorous thing: they cut the tool list down and made the remaining schemas strict.

4. Human checkpoint at the irreversible step

Find the point in the chain where an action becomes hard to undo — money moves, an email sends, a record is deleted, a customer-visible state changes — and put a human there.

This is not a failure of ambition. It is the same design principle as a payments system, and it is the correct mental model for agents generally: closer to how you design a transaction boundary than how you design a chatbot.

The heuristic for whether to build an agent at all

An agent is a good fit where the task has a verifiable output and a bounded set of actions.

Reconciling invoices against a ledger — verifiable. The numbers either match or they do not.
Triaging support tickets into a fixed taxonomy — verifiable. There are fourteen categories and a right answer.
"Managing" a process with no definition of correct — not a project. An aspiration.

If you cannot state, in one sentence, how a program would check the output was right, you do not have an agent task. You have a demo.

What to instrument from day one

Per-step success rate, not just end-to-end. You cannot fix a chain you cannot see inside.
Where chains terminate, and why. Distinguish "completed" from "gave up" from "hit the retry ceiling".
Cost per resolved task, not per token. A chain that retries three times is expensive regardless of how cheap the model is.
Divergence between validation failures and user complaints. If users complain about cases your validators pass, your validators are checking the wrong things.

Frequently Asked Questions

Does a better model fix compounding error?

Only marginally. Going from 95% to 97% per step takes an eight-step chain from 66% to 78% — better, still not shippable for most purposes. Shortening the chain from eight steps to three does more than any model upgrade available.

How do I know how accurate each step is?

Build an evaluation set per step: 200 or so real inputs with known correct outputs at that boundary. Most teams evaluate only end-to-end, which tells you that something is broken but not where.

Are agent frameworks the problem?

Not inherently, but they make adding steps and tools frictionless, and friction was doing useful work. Whatever framework you use, treat every additional model call and every additional exposed tool as a cost that must be justified.

Where exactly should the human checkpoint go?

At the last reversible point before an irreversible action. Not at the start (you gain nothing) and not after the action (too late). If the chain has several irreversible actions, that is itself a design smell — batch them.

What about multi-agent systems?

The same arithmetic applies, usually worse, because inter-agent communication adds boundaries where meaning degrades. Multi-agent architectures are justified when subtasks are genuinely independent, not when a single chain felt too long.

How do we validate outputs without a ground truth?

Use constraint checks rather than correctness checks: does the ID exist, is the value in range, does the referenced entity belong to this user, is the category in the permitted set. These catch most real failures without needing a known right answer.

The wider 2026 picture — cost curves, evaluation, retrieval, regulation, org structure — is here: Top AI Developments for Business in 2026: What Actually Changed.

We build agentic workflows and LLM integrations for production use. Get in touch.

What an Architecture Review Should Actually Hand You

James Sanderson — Sat, 29 Aug 2026 05:54:18 +0000

Someone above you has decided to bring in consultants to review your system. You have opinions about this. Most of them are correct.

But you also have leverage you probably are not using, because the deliverable specification is usually written by someone who does not know what a good one looks like. This post is the list I would hand any engineer whose employer is about to buy an architecture review — the artefacts to demand up front, while the statement of work is still editable.

1. Findings that are specific enough to be wrong

This is the whole test and everything else follows from it.

❌ "The data layer presents scalability concerns."
✅ "The single Postgres primary serves both transactional writes and the
    finance team's month-end analytics queries. At the projected 4x volume
    this becomes a write-latency incident in ~2 quarters. Fix: read replica
    + query routing at the ORM layer. Est. 3 engineer-weeks."

The first is unfalsifiable. Nobody can check it in six months. It survives any outcome, which is precisely why it gets written.

The second stakes a position. It might be wrong — the volume projection could be off, the finance query might get killed for unrelated reasons — and that is what makes it worth the fee. Someone put judgement on the table where reality can hit it.

Since 2023 this test has become the only one that works. Fluent technical prose is free now; anyone can generate forty pages of well-organised assessment about a codebase they skimmed. Count the falsifiable claims per page and you have a quality metric that AI cannot inflate.

2. A blast radius per finding, not a severity label

HIGH / MEDIUM / LOW is a way of avoiding the question. What you want per finding:

What breaks first
Under what condition (load, data volume, concurrency, a specific date)
Who notices — internal team, one customer segment, everyone
Whether it degrades or fails hard

A finding without a blast radius cannot be prioritised against the rest of your backlog, which means it will not be actioned, which means the review produced nothing.

3. Effort estimates attached to every recommendation

Not T-shirt sizes. Engineer-weeks, with the assumptions written down.

This one clause does more work than any other, because it forces the reviewer to have thought concretely about implementation instead of gesturing at a target state. "Adopt event-driven architecture" survives contact with a slide. It does not survive "estimate that in engineer-weeks against this repository, listing what you assumed."

Expect and accept ranges. Reject the absence of numbers.

4. An explicit list of what is fine

Demand this one. Ask for the section titled decisions we would not change.

A review returning twenty problems and zero acceptable decisions is not a review, it is a scoping document for the remediation contract that follows it. Real systems contain choices that were correct, and choices that are wrong in ways that will not matter for three years. Naming them proves the reviewer is ranking by impact rather than cataloguing against a reference architecture from a cloud vendor's marketing site.

It is also the part your team most needs. Being told the thing you built is defensible, in writing, by an outsider, is worth something the remediation list is not.

5. Proof they read the incident history

Ask directly: which post-mortems did you read, and what did they change in your conclusions?

Production incident history is the most honest artefact any engineering org produces, because nobody writes a post-mortem to impress a buyer. A reviewer who has read the code but not the incidents has read the fiction and skipped the documentary. Their findings will mirror the architecture diagram rather than the system's actual failure pattern — and those two things diverge in interesting ways in every codebase over about two years old.

The same applies to the people who get paged. If nobody in the on-call rotation was interviewed, the review is incomplete regardless of how thorough the static analysis was.

6. The raw artefacts

Scripts, query outputs, interview notes, the dependency graph in a machine-readable format. You paid for them.

More importantly: the review is a snapshot and your system moves. If you have the scripts, you can re-run the analysis in six months for free. If you have only the PDF, you have bought a one-time observation and you will be buying it again.

7. A walkthrough where you are allowed to argue

Not a presentation to leadership. A working session with the engineers, scheduled, with time for objection.

Two reasons. First, a finding that cannot survive informed challenge from the people who built the system was never solid — and you will find out in that room, not in production. Second, findings your team has argued with become findings your team owns. Findings delivered as a document from outside get treated as an external opinion, and external opinions do not make it into sprint planning.

If a consultancy resists the challenge session, that is your answer about how much the findings are worth.

The one-line version

An architecture review is an engagement to buy a decision, not a document. If the artefacts above are not in the statement of work, what arrives will be a summary of things your team already knew, written more confidently than any of you would have dared.

Frequently Asked Questions

How long should an architecture review take?

Two to four weeks for a system of ordinary complexity. Under two weeks generally means code was read but incident history was not and operators were not interviewed. Beyond six weeks the findings start going stale against a team that is still shipping.

Who should be interviewed during a review?

The on-call rotation first, then the people who wrote the oldest surviving modules, then product. Interviewing only tech leads and architects produces a review of the intended system rather than the real one.

What if we disagree with a finding?

Argue it in the scheduled walkthrough with evidence. A good reviewer will change a conclusion when shown data, and will label which findings are high-confidence versus judgement calls under uncertainty. If nothing is ever labelled uncertain, the confidence is decorative.

Can we just run an AI model over the codebase instead?

For the inventory layer, yes, and you should — dependency mapping and dead-code detection are close to free now. What a model cannot do is rank findings against your organisation's actual capacity, politics and roadmap, because none of that is in the repository.

Should the reviewing firm also do the remediation?

It can, and the context transfer is already paid for. Just make sure the assessment was contracted and priced separately, so the size of the recommended fix does not influence the fee for the diagnosis.

What is the single best clause to add to the SOW?

Named individuals with a committed time percentage. The gap between the person who sold the engagement and the person who performs it is the oldest complaint in professional services and the easiest to close on paper.

Full write-up with the other four engagement types, pricing models and the 30-day test: Software Development Consulting Services: What You Actually Get for the Money.

We do this work at TechCirkle alongside building products — get in touch if you want a scoping call.

Build the Pilot Like It Has to Survive

James Sanderson — Wed, 26 Aug 2026 15:59:34 +0000

The pilot-to-production gap in AI projects gets discussed as if it were mysterious, or organisational, or a matter of change management. It is mostly none of those. It is five missing engineering artefacts, and they are the same five almost every time.

A demo runs on curated data, handles the happy path, and is judged by a sympathetic audience in a meeting. A production system handles malformed input, upstream schema changes, adversarial users, latency budgets, cost ceilings, model deprecations, and the specific edge case that represents four percent of volume and eighty percent of your support tickets.

Here is what to build from the first commit so the second thing is reachable from the first.

1. An evaluation harness

This is the one that matters most and gets skipped most.

You need a held-out set of representative examples and an automated method of scoring output against them. Not a notebook where someone eyeballs twenty results — a scored, repeatable, version-controlled comparison you can run on every change.

Without it, every subsequent decision about prompts, model versions, retrieval strategy, chunking, or thresholds is made on anecdote. Teams end up in an endless loop of subjective adjustment where "is it better now?" is genuinely unanswerable, because these systems fail in ways that are invisible without measurement.

Build the evaluation set before building the system. It forces you to define what correct means, which is frequently where the real disagreement surfaces — and it is far better to surface it in week one than in month five.

2. A real data connection

Pilots run on CSVs. Production runs on a pipeline.

The gap is not the extraction — it is everything the manual extract silently removed. Nulls someone dropped. Encoding problems someone fixed. Records that failed a filter nobody documented. Schema variations across the three years of history the sample did not span.

Connect to the actual source early, even at reduced volume. Discovering that eleven percent of production records have a field format the pilot never saw is a week-two problem or a month-five problem, and it is exactly the same problem either way.

3. Explicit failure behaviour

What does the system do when the model returns malformed output? When the provider times out? When input exceeds the context limit? When retrieval returns nothing relevant?

A demo has no answer because these did not happen. A production system needs a defined answer for each, and the answer is often "escalate to a human" — which requires a path for that to happen, and someone on the other end of it.

Define the confidence threshold below which the system declines to act. Systems without a decline path produce confident wrong output on inputs they should have refused, and that failure mode is more damaging than an outright error because nothing signals it.

4. A cost model

Token costs at a hundred requests a day tell you nothing about ten thousand.

Instrument cost per request from the beginning, broken down by component — retrieval, generation, re-ranking, any evaluation running in production. Then project to realistic volume before committing to an architecture.

This routinely changes design decisions. A pattern that makes three model calls per request is fine at demo volume and can be untenable at scale, and discovering that after the architecture is settled is expensive. Caching strategy, model tier selection, and whether to re-rank at all are all cost decisions disguised as quality decisions.

5. An owner

Not an engineering artefact, and it kills more pilots than the other four combined.

Systems built by a team that then disperses have no one to handle the model deprecation notice, the gradual quality drift, the new input pattern that started appearing in March. Name the owner before launch, not after.

The drift problem specifically

One production concern has no demo equivalent and deserves separate attention.

These systems degrade without erroring. The input distribution shifts. An upstream system changes format. The provider updates the underlying model. Output quality drops and nothing raises an alarm, because from the infrastructure's perspective everything is healthy — requests are being served, latency is fine, error rate is zero.

Uptime monitoring cannot see this. You need quality monitoring: run the evaluation set on a schedule against production configuration, track the score over time, and alert on degradation. This is why the evaluation harness is not merely a development tool. It is a production control, and it is the reason skipping it early is so hard to recover from.

What this costs

Roughly thirty percent more than a demo.

That is the entire trade. Thirty percent more up front against a rewrite when the demo needs to become real — and the rewrite is where second-year AI budgets tend to disappear, usually while the team is also trying to deliver the next thing.

The wider context — engagement models, use-case selection, build versus buy, governance, and cost ranges — is at techcirkle.com/blog/ai-business-consulting. Most of what makes this work in practice is LLM integration architecture: retrieval, evaluation, caching, and fallbacks.

Frequently Asked Questions

What is an evaluation harness for an AI system?

A held-out set of representative inputs with expected outputs or scoring criteria, plus an automated way to run the system against them and produce a comparable score. It is version-controlled and runs on every change. Without one, decisions about prompts, models, retrieval, and thresholds are made on anecdote, and quality becomes unmeasurable rather than merely uncertain.

How do you detect AI model quality drift in production?

Run your evaluation set on a schedule against the production configuration and track the score over time, alerting on degradation. Standard uptime and error-rate monitoring cannot detect drift, because a degrading model still returns well-formed responses quickly. This is why the evaluation harness functions as a production control rather than only a development tool.

Why does connecting to real data early matter so much?

Because manual extracts silently remove exactly the problems production will surface — dropped nulls, fixed encodings, filtered records, schema variation across years the sample did not span. Connecting to the actual source at reduced volume surfaces these in week two rather than month five, at which point the architecture has usually settled around assumptions that no longer hold.

What should an AI system do when it is uncertain?

Decline and escalate, which requires a defined confidence threshold and a human path on the other side. Systems without a decline path produce confident wrong output on inputs they should have refused. That failure mode is more damaging than an outright error because nothing signals it — the output looks exactly like a correct one.

How much more does building a pilot properly cost?

Roughly thirty percent more than a demo, covering the evaluation harness, a real data connection, explicit failure behaviour, and cost instrumentation. The comparison is against rewriting the system when the demo needs to become production, which typically costs considerably more and happens while the team is also committed to delivering the next thing.

Ten Minutes or Nobody Reads the Result

James Sanderson — Wed, 26 Aug 2026 15:57:38 +0000

Pipeline duration gets treated as a performance metric — nice to improve, competing with other priorities. It is better understood as a hard design constraint, because there is a threshold past which continuous integration stops doing the job it exists to do.

Under roughly ten minutes, a developer opens a pull request, stays in context, and acts on the result. Past twenty, they switch to something else. When they come back, the change is no longer loaded in their head, the fix costs a full re-context, and the tight feedback loop that justified building CI in the first place is gone.

You still have a pipeline. You no longer have continuous integration in any meaningful sense.

Where the minutes actually are

Four things account for most slow pipelines, roughly in order of how much they cost and how rarely they are addressed.

Dependency installation on every run. Still remarkably common. If your pipeline resolves and downloads the full dependency tree on each execution, that is often two to four minutes of pure repetition. Lockfile-keyed caching is a small change with an outsized return.

Serial test execution. Test suites parallelise well and frequently are not parallelised at all, because the suite grew gradually and nobody ever revisited the assumption. Sharding across runners is usually the single largest available win.

Running everything on every change. A one-line documentation edit triggering the full integration suite is waste with a straight face. Test selection based on the dependency graph — running only what the change could plausibly affect — cuts typical run time substantially, with the caveat that the selection logic itself needs to be conservative and tested.

Slow work in the blocking path. Anything genuinely slow — full end-to-end suites, performance benchmarks, container image scanning of large images — does not belong in front of a merge. Move it after merge, run it against a real environment, and page the owning team on failure.

What should actually gate a merge

The instinct is to gate on everything, and it backfires in a specific and predictable way.

Gate on everything and the pipeline becomes slow and flaky. Slow and flaky means engineers learn that red does not necessarily mean broken. Once that is learned, the standard response to a failure is to re-run rather than investigate. At that point the gate is theatre: it delays every change and catches nothing, because failures are dismissed by default.

A defensible policy gates on signals that are fast and deterministic:

Unit tests
Linting and formatting
Type checking
Dependency and secret scanning
Contract tests against consumer expectations

Everything slow or inherently flaky runs post-merge against a real environment. This feels like lowering standards and does the opposite — the gate that remains actually means something, so people respond to it.

Flaky tests are the real emergency

Flakiness deserves more urgency than it usually gets, because it is not a nuisance — it is corrosive to the entire mechanism.

One test that fails intermittently teaches the team that failures are sometimes noise. That lesson generalises immediately. Within a few weeks people are re-running failures reflexively rather than reading them, and the day a red build represents a genuine regression, it gets re-run too.

The control that works is automated quarantine. Detect tests that fail non-deterministically across runs on the same commit, automatically move them out of the blocking path, and open an owned ticket. The suite stays green and meaningful while the underlying problem gets fixed properly rather than urgently.

What does not work is asking people to be more diligent about investigating failures. That is a request to spend attention on a signal that has been demonstrated to be unreliable, and it loses to incentives every time.

Deploy and release are different events

The last structural change worth making is separating deployment from release with feature flags.

Once shipping code and activating behaviour are decoupled, deploying stops being a decision that carries risk. Code goes to production dark; the behaviour turns on when someone chooses. Two things follow. Deployment becomes routine enough that continuous deployment is politically achievable in organisations that would otherwise refuse it. And rollback becomes a flag change measured in seconds instead of a redeploy measured in pipeline duration.

The cost is flag debt. Long-lived flags multiply into state combinations nobody can reason about. The control is simple and must be enforced: every flag gets an owner and an expiry date, and a scheduled audit removes anything past it.

A pragmatic order of operations

Cache dependencies keyed on the lockfile.
Shard the test suite across runners.
Move slow and flaky suites out of the blocking path, with quarantine automated.
Add change-based test selection once the suite is fast enough that correctness of selection is the main risk.
Introduce feature flags and decouple deploy from release.

Steps one to three usually get a typical pipeline under ten minutes without any architectural change, which is why they come first.

The full treatment — the seven layers of a delivery stack, infrastructure drift, where AI agents genuinely help, costs, and a 90-day roadmap — is at techcirkle.com/blog/devops-automation. We build delivery platforms alongside our custom software development work, which is why the advice here is shaped by systems we have had to operate.

Frequently Asked Questions

Why is ten minutes the threshold for CI feedback?

It approximates how long a developer will wait before switching tasks. Under ten minutes they stay in context and act on the result immediately. Past twenty they move on, and returning to the change costs a full re-context — which is what destroys the tight feedback loop CI exists to provide. The exact number varies by team, but the cliff is real and sits in that range.

Should end-to-end tests block merges?

Generally no. End-to-end suites are slow and inherently more flaky than lower tiers, and putting them in front of a merge is the most common cause of teams learning to re-run until green. Run them post-merge against a real environment with failures paging the owning team. The merge gate should hold only fast, deterministic checks.

How do you detect a flaky test automatically?

Re-run failures against the identical commit. A test that passes on retry with no code change is non-deterministic by definition. CI platforms increasingly support this natively; where they do not, a retry-and-compare job plus a small history store is enough. The important part is that detection triggers automatic quarantine rather than a request for someone to investigate later.

Does test selection risk missing real failures?

Yes, which is why the selection logic must be conservative — err toward running too much — and why it belongs after the basics rather than first. It also pairs naturally with a post-merge full run, so anything selection missed is caught shortly afterward rather than never. Do not adopt it as the first optimisation; adopt it once caching and sharding have already been done.

What is the downside of feature flags?

Flag debt. Flags that are never removed accumulate into combinations of state that nobody can reason about or test, and this compounds quietly. The mitigation is treating every flag as temporary by default: an owner, an expiry date, and a scheduled audit that removes expired ones. The debt is real and manageable; coupling deployment to release is worse.