DEV Community: Ivan Zdravkov

Your model is not the product. The pipeline around it is.

Ivan Zdravkov — Wed, 28 Oct 2026 06:59:15 +0000

Swap the model and almost nothing changes. Swap the masking layer, the permission model, or the monitoring, and the system stops running, starts leaking, or quietly drifts wrong. The part everyone demos is the part that matters least.

Key takeaways

Model choice sits behind an abstraction in a well-built system, which means it is the cheapest part to change and the least differentiating.
The expensive 90% is plumbing: masking before any prompt, permissions that travel with the person, audit trails, kill-gated evaluation, drift detection, none of which shows up in a demo.
Research on production ML lifecycles increasingly measures workflow completion, artifact traceability and drift recovery, not just predictive accuracy, because that is where systems actually fail.
A team that can name its out-of-scope cases and its monitoring signal is pipeline-first; a team that can only name its model is not, yet.
None of this is an argument against caring about models. It is an argument about where the engineering hours actually need to go.

What is "the pipeline" here, concretely?

Everything between a request arriving and an answer leaving, minus the forward pass itself. Ingestion and normalisation of whatever format the input actually shows up in. A masking boundary that decides what a model is ever allowed to see, the subject of why the masking layer matters more than the model. An authorisation layer that scopes what an assistant can reach, described in what an internal MCP server actually solves and permissions that travel with the person. Logging, evaluation and the discipline of registering a kill-gate before trusting a number, covered in why I pre-register a kill-gate for every experiment. None of that is the model. All of it is what makes the model usable inside a real company.

The model sits in the middle of this as one component with a clean interface: text in, text out. Everything upstream and downstream of that interface is where the actual engineering risk lives, because it is where identity, authorisation, sensitive data and operational failure modes are handled.

Why does swapping the model change so little?

Because a well-built pipeline puts the model behind an abstraction on purpose. One of the four principles I hold myself to on this site is no vendor lock-in: open models behind an interface, swappable without rewriting anything around them. If swapping the model breaks the system, the system was never really decoupled from it, and that coupling is a design defect in the pipeline, not a fact about models.

This is not just a personal preference. A 2026 framework for LLM-orchestrated data pipelines measures success across workflow completion, artifact traceability, deployment readiness, reproducibility and drift recovery, deliberately alongside predictive performance rather than instead of it. Lifecycle reliability, not raw model quality, is the axis the field is now building evaluation around, because that is the axis production systems actually fail on. A model a few points better on a leaderboard rarely rescues a pipeline with no masking, no permission model and no drift detection; a mediocre model behind a disciplined pipeline usually survives long past the leaderboard's relevance.

Where does the 90% actually go?

Concretely, in the systems I have built and measured this year: designing the masking boundary took longer than building the AI features it protects, because free text detection has real failure rates that have to be measured, not assumed, as the entity-level numbers in detecting PII before the model sees it show directly. The permission model, deciding that an assistant inherits its user's rights rather than owning any of its own, is infrastructure work with no model in it at all. Kill-gate discipline, writing a threshold down before a single experiment runs, is process, not code, and it is the difference between a result and an impression. None of these three things appears in a five-minute demo. All three decide whether the demo's system survives a normal Tuesday.

The pattern generalises past my own work. Structured evaluation of LLM agents keeps finding that failures cluster in tool invocation, planning under constraints and long-horizon coordination, categories that live entirely in the pipeline around a model, not in the model's raw capability. Scaffolding does not reliably fix this either; the same research finds that additional orchestration does not consistently improve reliability, which means the fix is disciplined engineering around the model, not more layers stacked on top of it.

FAQ

Does this mean model choice does not matter at all?

No. It means model choice is the reversible, comparatively cheap decision, while masking, permissions and monitoring are expensive to retrofit and dangerous to skip. Spend engineering hours where mistakes are costly and hard to undo; treat the model as the part you can change your mind about later.

What is the fastest way to tell if a team is pipeline-first or model-first?

Ask what happens on malformed input and who gets paged when output quality drops silently. A pipeline-first team answers with a specific mechanism. A model-first team answers with the name of the model, which is not an answer to either question.

Isn't "the pipeline" just a rebrand of MLOps?

Overlapping, not identical. MLOps usually centres training and deployment tooling; the pipeline argument here includes things MLOps tooling rarely owns, such as the masking boundary and the permission model, both security decisions as much as operational ones.

How do you get budget for work that produces no visible feature?

Frame it by what it prevents, not what it adds: a masking failure is a data breach, a missing permission model is unlimited blast radius from one prompt injection, a missing kill-gate is a metric nobody can trust. Those are costs a stakeholder already fears; the pipeline work is the line item that keeps them from materialising.

References

Aueaphum Aueawatthanaphisut, Badri Raj Lamichhane. Trustworthy Self-Composable Big-Data-as-a-Service: An LLM-Orchestrated Multi-Agent Framework for Automated Data Engineering, AutoML, MLOps Deployment, and Drift-Aware Lifecycle Optimization, 2026. arxiv.org/abs/2606.17915
Wael Albayaydh, Rui Zhao, Ivan Flechais. Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents, 2026. arxiv.org/abs/2607.05775

What the EU AI Act actually requires of a masking layer

Ivan Zdravkov — Wed, 28 Oct 2026 06:58:39 +0000

The Act never uses the word masking. It does not need to. Three articles, on data governance, record-keeping and human oversight, each describe a property a masking layer either has or does not, and retrofitting any of the three after a system is running costs far more than building toward them from the start.

Key takeaways

Article 10 permits processing special category personal data only for bias detection, and only under conditions that already sound like a masking layer: strict security measures, no transmission to third parties, deletion once the purpose is served.
Article 12 requires automatic logging over the lifetime of a high-risk system, with no specified retention period, which means the logging capability has to exist before anyone decides how long to keep it.
Article 14 requires a human overseer to be able to interpret, override and stop a system, which fails silently if nothing in the pipeline explains what data reached a decision.
Recent research on regulatory compliance argues that audit-time, binary compliance is structurally the wrong model for a running system; the requirement is a continuous signal, not a certificate filed once.
None of this depends on whether a given system is formally classified as high-risk. The three properties are worth building toward regardless, because they are expensive to add later and cheap to design in now.

What does the Act actually say about data?

Article 10 governs training, validation and testing data for high-risk systems, and most of it reads like ordinary data-quality discipline: datasets have to be relevant, sufficiently representative and, to the best extent possible, free of errors. The part that matters for a masking layer is narrower and stricter. The Act allows processing of special category personal data, health, ethnicity, and similar sensitive fields, but only for the specific purpose of detecting and correcting bias, and only when a list of conditions holds: no alternative data would serve the purpose, technical limits restrict reuse, strict security measures apply, the data is never transmitted to third parties, and it is deleted once the bias has been corrected.

Read as an engineering requirement rather than a legal one, that is a masking and lifecycle policy with a narrow, purpose-bound exception carved into it. A system that cannot say which fields are sensitive, cannot restrict a specific processing purpose to a specific dataset, and has no deletion trigger tied to that purpose ending, has no way to satisfy this even if the underlying use case is legitimate. The masking boundary I described in why the masking layer matters more than the model is the component that makes the distinction between sensitive and non-sensitive fields exist at all; Article 10 is one concrete reason that distinction has to be enforced, not merely documented.

What has to be logged, and why does that matter for masking?

Article 12 requires that high-risk systems technically allow automatic recording of events over their lifetime, so that the system's functioning is traceable. Notably, it does not specify a retention period. That omission is easy to misread as leniency; in practice it means the logging capability has to exist unconditionally, because nobody can decide how long to keep records that were never captured in the first place.

A masking layer sitting at the boundary between internal systems and a model is naturally the place this logging belongs. It already sees every field that leaves the boundary, so it is the cheapest point to record what left, in what form, and under what placeholder mapping. Building this after the fact means reconstructing months of decisions from application logs that were never designed to answer a compliance question. Building it into the boundary from day one means the audit trail is a side effect of normal operation, which is the only version of record-keeping that survives contact with an actual audit.

What does human oversight mean for an automated pipeline?

Article 14 requires that a human assigned to oversee a high-risk system can understand its capacities and limitations, remain aware of the tendency to over-rely on its output, correctly interpret what it produced, and override, disregard or stop it. None of that is possible if the system presents a final answer with no visible path to the data behind it.

This is where masking and oversight pull in the same direction rather than against each other. A masking layer that replaces sensitive values with stable placeholders, the design I described in the same earlier article, gives a human reviewer something concrete to check: which fields were substituted, whether the substitution was correct, and whether the model's answer still makes sense once the real values are restored. An oversight requirement bolted onto a system with no visibility into what the model actually saw reduces to a person clicking approve on a black box, which satisfies the letter of the article and defeats its purpose.

Why does audit-time compliance fail for a running system?

A 2026 paper on regulatory compliance for deployed LLM systems argues that treating conformity as a binary, audit-time verdict is structurally mismatched to a requirement built around ongoing oversight and the detection of behavioural drift. The authors propose governance from metrics: compliance as a continuous signal derived from runtime observability, not a static assessment filed once and forgotten. Their own validation, a panel of small on-premise models scoring compliance criteria, found agreement rates between 51.5% and 69.1% depending on the model, with no single model dominating across every criterion, which is itself evidence that compliance judgement is not a solved, one-shot problem even with dedicated tooling.

The relevance here is architectural rather than a specific product recommendation: whatever machinery ends up checking compliance, it needs a continuous stream of real signal to check, not a document written before launch and never revisited. A masking layer with logging built in produces exactly that stream as a by-product of running. A compliance binder produced once for an audit does not, and updating it by hand every time the system changes is precisely the kind of expensive retrofit the rest of this article has been describing.

FAQ

Does this mean my system is legally classified as high-risk under the Act?

Not necessarily, and that classification depends on specifics this article does not cover, listed in the Act's Annex III. The three properties discussed here, purpose-bound handling of sensitive data, unconditional logging and genuine human oversight, are worth building regardless of classification, because the cost of retrofitting them after a system is running is far higher than the cost of designing them in.

Is a masking layer enough to satisfy Article 10 on its own?

No. Article 10 is about the training, validation and testing data pipeline as a whole, including bias examination that a masking layer does not perform. What a masking layer provides is the enforcement mechanism for the purpose-bound, security-conditioned handling that the special category data exception requires; the bias assessment itself is separate work.

Do I need a dedicated compliance tool like the one in the cited paper?

Not necessarily this specific one; it is research software, not a product recommendation. The useful idea to take from it is the framing: compliance as a continuous signal your system produces, not a document written once. Whether that signal comes from a purpose-built tool or from disciplined logging at your masking boundary is a separate decision.

Is this legal advice?

No. I am reading the Act's text as an engineer deciding what to build, not as a lawyer assessing legal exposure. Treat the article numbers as a starting point for a conversation with counsel, not as a substitute for one.

References

Regulation (EU) 2024/1689 (Artificial Intelligence Act), Official Journal version of 13 June 2024. Articles 10, 12 and 14. Article 10, Article 12, Article 14.
Jehanne Dussert. Who judges the judges? Governance from metrics: a runtime framework for continuous LLM compliance monitoring, 2026. arxiv.org/abs/2605.24737

What EWC buys you that flat L2 does not

Ivan Zdravkov — Wed, 28 Oct 2026 06:52:48 +0000

A flat L2 penalty can match EWC's exact Task A retention number on Permuted MNIST: 97.63% against 97.63%. It does this by collapsing Task B accuracy to 18%, against EWC's 97%. The same headline number hides two very different costs.

Key takeaways

A registered gate asked whether flat, uniform L2 regularisation matches EWC's Fisher-weighted penalty at comparable cost. The cheapest tested setting landed within 0.86 points of EWC's Task A accuracy, inside the registered 3-point bar - the gate fired.
No tested setting met the registered "fair budget" of costing at most 1 point of Task B accuracy. The cheapest one still cost 2.70 points, over six times what EWC paid.
At a stronger setting, flat L2 hits EWC's exact Task A number, 97.63%, while Task B accuracy collapses to 18.42% against EWC's 97.43%.
Fisher weighting is not required to match EWC's retention number in this setup. It is required to match it without wrecking the model's ability to learn the new task.

What did the registered gate actually test?

An earlier experiment found that Elastic Weight Consolidation (EWC, Kirkpatrick et al., 2016) recovers 98.0% of the accuracy a small network loses to catastrophic forgetting on Permuted MNIST. EWC works by penalising drift away from Task-A-optimal parameters, weighted by a diagonal Fisher information estimate: parameters that mattered more to Task A get pulled back harder. That estimate costs something to compute, 1,000 samples and a forward-backward pass through each. The open question this experiment registered: does the Fisher weighting itself earn that cost, or would any quadratic pull-back toward the old parameters, uniform and uninformed, do just as well?

The registered kill-gate compared a flat L2 penalty, same shape as EWC's but with every parameter weighted equally, against EWC's actual result. To keep the comparison honest, the plan fixed a rule before running anything: search a range of L2 strengths, and report the one whose cost to Task B accuracy sits closest to, but under, 1 percentage point, matching EWC's own 0.41-point cost. If Task A accuracy at that point comes within 3 points of EWC's, the claim that Fisher weighting matters would be rejected.

Where did the gate fire, and what did that hide?

The registered gate fired, literally. Across six tested strengths (0.01 to 1000, log scale), the cheapest one, lambda=0.01, reached 96.77% Task A accuracy, 0.86 points under EWC's 97.63%, comfortably inside the 3-point bar. By the letter of the pre-registered rule, the hypothesis that Fisher weighting provides a measurable advantage is rejected.

But the fair-budget rule that gate was supposed to respect was never actually satisfied. That same lambda=0.01 setting cost 2.70 points of Task B accuracy (97.84% down to 95.14%), not the 1 point the plan called "fair." Every cheaper option would have cost less on Task B but also recovered less of Task A; every stronger option recovered more of Task A at a steeper Task B cost. No point in the sweep lived inside the budget the plan defined as comparable to EWC. fair_pick_in_budget: false sits in the raw metrics file precisely so this gets reported rather than smoothed over.

What does the lambda=10 result reveal?

One point in the sweep makes the trade-off impossible to miss. At lambda=10, flat L2 reaches 97.63% Task A accuracy, identical to EWC to two decimal places. Read as a single number, EWC and flat L2 look interchangeable. Task B accuracy at that same setting is 18.42%, against EWC's 97.43%. The network did not learn to protect what mattered on Task A while staying free to absorb Task B. It nearly stopped learning Task B at all, and freezing a network is a cheap way to protect old knowledge from any source.

This is the shape a naive comparison misses: pick one number, retention on the old task, and two very different mechanisms can look tied. Add the number nobody asked for and the picture reverses completely. It also explains why the sweep never found a point inside the fair budget: Task A accuracy rises smoothly toward EWC's figure as lambda climbs, but Task B accuracy falls off a cliff well before Task A catches up. There is no gear between "too weak to matter" and "strong enough to freeze the network," at least at this granularity of search.

So does Fisher weighting matter?

Both, depending on which question is being asked. If the question is "can a simpler method hit EWC's retention number," this experiment says yes, at least in this mild Permuted MNIST setting, with enough uniform pressure. If the question is "can a simpler method hit that number while leaving the network able to learn the next task," the answer across every tested setting is no. Fisher weighting is not a hard requirement for the headline number. It is the reason EWC can post that number while still being useful for anything after it.

The distinction matters beyond this one benchmark. Any report that states only "method X preserved Y% of old-task accuracy" is stating half a result. The other half, what the method spent to get there, decides whether it is a usable continual-learning method or a network that has quietly stopped learning. A retention number without its corresponding new-task cost is not a false claim, but it is an incomplete one, and this experiment is a case where the missing half changes the conclusion entirely.

FAQ

Does this mean EWC is unnecessary, and flat L2 works just as well?

Not for the actual goal of continual learning. Flat L2 can copy EWC's Task A number, but only by paying far more of the model's capacity to learn new tasks, an option that defeats the point of learning continually in the first place. The two methods are not interchangeable once both accuracy numbers are read together.

Why sweep several L2 strengths instead of tuning one directly to match EWC?

Tuning a single value to match a known target after seeing the result is exactly the hindsight pre-registration exists to prevent. The sweep and the fair-budget rule were both fixed before any run, so the reported pick is the honest output of that rule, not a value chosen because it looked good afterward.

What prompted this specific comparison?

A correspondent, following up on the earlier EWC experiment, asked directly whether a flat L2 baseline had been tested. It had not. Answering from intuition would have been easy and probably wrong in an interesting way, given how the lambda=10 result turned out, so the honest option was to register a plan and run it rather than guess.

What would make this comparison more convincing?

A finer sweep near the two extremes that bracket the trade-off, more task pairs than one, and ideally a network trained from scratch by the same author, since this Permuted MNIST setup is a mild benchmark where forgetting itself was measured, not catastrophic, in the original experiment.

References

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, Raia Hadsell. Overcoming catastrophic forgetting in neural networks, 2016. arxiv.org/abs/1612.00796

What an internal MCP server actually solves

Ivan Zdravkov — Wed, 28 Oct 2026 06:52:09 +0000

An internal MCP server does not make a model smarter. It solves a governance problem: without one, an assistant either guesses from its training data or receives internal data through copy-paste, and neither path has authentication, permissions or an audit trail. The server is where those three things finally get a place to live.

Key takeaways

Before any tool access, an assistant answers from model memory, and employees fill the gap by pasting internal data into chat windows, which is an unmanaged leak channel.
The Model Context Protocol standardises how assistants discover and call tools, turning N assistants times M systems into N plus M.
Typed schemas matter more than clever prompts: the server validates every call against a contract and rejects what does not fit, instead of hoping the model behaves.
A survey of 247 papers on agent security finds prompt injection and tool-mediated control-flow hijacking dominate, and that current defences compose poorly, so enforcement belongs server-side.
Per-person authentication and role-scoped tool lists turn an assistant into something that inherits its user's rights instead of having rights of its own.

What breaks before there is an MCP server?

Two failure modes appear in every company that gives employees AI assistants without giving those assistants data access. The first is confident guessing: the model answers questions about internal state from its training data, which contains nothing about the company, so the answers are fluent and wrong. The second is worse: people close the gap themselves by pasting customer records, contracts and internal reports into whatever chat window they use. Copy-paste is an integration layer with no authentication, no scoping and no log, and it appears wherever an official path is missing.

Building official paths one at a time does not scale either. Each assistant product has its own plugin format, each internal system has its own API, and every combination needs custom wiring. The integration count grows as the product of the two, and every integration is another place where credentials and permissions are handled slightly differently. What is missing is not capability but a standard boundary, one place where access is granted, scoped and recorded.

What does the protocol actually standardise?

MCP, the Model Context Protocol, standardises the conversation between an AI client and a tool server: how the client discovers what tools exist, what parameters each tool takes, and how calls and results travel. A server written once works with any client that speaks the protocol, which collapses the integration problem from N times M to N plus M. The protocol carries typed tool definitions, so every tool declares a schema for its inputs, and the server validates calls against that schema before anything executes.

How far this stretches is best shown by a hard case. A 2026 paper describes controlling scientific instrumentation, including electron microscopy workflows, through MCP: a reference server exposing 30 typed tools, with a schema-bound surface that validates requests against physical bounds before any command reaches an instrument, precisely because natural-language interfaces "can emit physically unreasonable arguments". If schema-bound MCP control is trusted to keep an agent from sending a physically impossible command to a microscope, the same pattern covers reading order data or querying a contract database. Business systems are the easy case.

Why do typed schemas matter more than clever prompts?

Prompt instructions are requests; schemas are contracts. A system prompt that says "only query orders from your own region" depends on the model honouring it under every phrasing a user can produce. A tool whose schema requires a region parameter that the server fills in from the authenticated user's profile does not depend on the model at all. The difference decides what an attacker or a careless user can reach.

The instrumentation paper cited above is built on that observation: arguments are validated against declared bounds before dispatch, and invalid calls fail at the boundary instead of reaching the instrument. The business translation is direct. Models produce plausible parameters, and plausible is not the same as valid: a date range spanning ten years, an identifier from the wrong system, a status value that sounds right and does not exist. Server-side validation catches all of it in one place, and the assistant gets a structured error it can correct from. My rule for the internal server is that nothing the model sends is trusted: every call is checked against the schema, and everything the server-side context already knows, identity, role, scope, is filled in by the server rather than requested from the model.

Where does security actually live?

Research on agent security keeps arriving at the same place. A 2026 survey synthesising 247 papers finds that prompt injection and tool-mediated control-flow hijacking still dominate the threat landscape, and its verdict on defences is sobering: current mechanisms are useful building blocks but remain weakly compositional. Stacking model-level mitigations does not add up to a guarantee. The authors argue for explicit trust boundaries, principled privilege control and provenance-aware state management, which are properties of infrastructure, not of prompts.

An internal MCP server is where those properties become enforceable. Each person authenticates with their own key; the design leaves no shared service account behind which everyone could hide. The tool list an assistant sees is derived from its user's role: a tool that is not listed cannot be called, which is a stronger statement than a tool that is listed but forbidden. Every call is logged with who, what and when, which turns incident response from archaeology into a query. And the same boundary is the right place for the masking layer I described in why the masking layer matters more than the model: tool results pass through masking before they reach an external model, so grounding the assistant does not become a new leak.

The permission model behind this is the subject of its own article: permissions that travel with the person.

What does this look like in practice?

In the systems I work on, an internal MCP server is currently in testing. It sits in front of the company's data, and the design is that colleagues connect their own assistants to it: each person authenticates with a personal key, their role decides which tools their assistant can list and call, and answers come from the systems instead of from the model's imagination. Read-only tools come first, because a wrong read costs a correction while a wrong write costs an incident.

Not everything is solved. Injection through retrieved content remains the open flank: a tool result can carry text that tries to steer the model, and the survey's finding that defences compose weakly matches what I see in testing. The mitigations in the design are scope and privilege, not cleverness: an assistant reads what its user could read, writes almost nothing, and the interesting attack surface stays as small as I can keep it. I do not yet have a monitoring signal that would reliably tell me an injection attempt happened rather than a user asking something odd. That is measurement work of the kind I described in the demo is the easy part, and it is next on the list.

FAQ

Is an MCP server just another API wrapper?

Structurally yes, practically no. The difference is discovery and client support: any MCP-speaking assistant can list the tools, read their schemas and call them without custom integration work per client. A REST API needs bespoke wiring for every assistant product; an MCP server needs none, which is what makes the N plus M economics work.

Do we need MCP if our systems already have good APIs?

Yes, if you want assistants to use those systems. Existing APIs were designed for developers, not for language models: no discovery, error formats models cannot correct from, and authentication schemes assistants cannot complete. The MCP server is the adapter layer that presents your systems in a form a model can use safely.

Is it safe to let an assistant write to internal systems?

Treat writes as a separate, later decision. Read-only tools deliver most of the value at a fraction of the risk, and role-scoped reads limit even that. When writes become necessary, put them behind narrow schemas, server-filled identity fields and human approval steps, and log everything. A wrong read costs a correction; a wrong write costs an incident.

Does an internal MCP server need a masking layer?

Yes, whenever the assistant behind it talks to an external model. Tool results are exactly where sensitive data enters the conversation, so they pass through masking at the server boundary before leaving the infrastructure. Grounding an assistant in real data and protecting that data are the same project, solved at the same choke point.

References

Roberto dos Reis, Vinayak P. Dravid. Schema-Bound LLM Control of Scientific Instrumentation through Model Context Protocol Skills, 2026. arxiv.org/abs/2607.17012
Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang. Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation, 2026. arxiv.org/abs/2606.10749

The real trade-off between an API model and a local one

Ivan Zdravkov — Wed, 28 Oct 2026 06:46:18 +0000

An API model costs nothing until the bill arrives, and a local model costs everything up front: hardware, quantisation loss, engineering time, before it answers a single question. Which one is cheaper depends on volume and constraints, not on which sounds more modern.

Key takeaways

The break-even point between API and local hosting depends on request volume and GPU utilisation, not a fixed rule of thumb. A card that sits idle most of the day never pays for itself, however cheap its per-token math looks on a spreadsheet.
Real numbers from a registered quantisation experiment on an RTX 5070 Ti (16GB): Q4_K_M quantisation delivered 2.22x the throughput of fp16 with 0.9835 average embedding similarity, a genuine trade, not a free lunch.
API providers get batching economies of scale that a single-GPU local deployment cannot replicate at low request volume, so a naive per-token price comparison is not comparing like with like.
Privacy and data residency requirements can override the entire cost calculation. Some data cannot leave the building at any price, and no API pricing tier changes that.

What does an API call actually cost?

The advertised per-token price is the smallest part of the real cost. It buys elastic capacity: no idle hardware, no upgrade cycle, and access to whatever batching efficiency the provider has built into their serving stack. Yao and Zhou (2026) study exactly this kind of efficiency, showing that batched serving performance depends heavily on how fairly a scheduler shares GPU resources across requests of different sizes, an engineering problem a provider solves once across millions of customers and a single company solves, if at all, alone. That difference in scale is why per-token API pricing can undercut a naive local cost estimate even when the underlying hardware is comparable: the provider's utilisation is close to constant, while a single company's local GPU sits idle outside business hours or peak load.

The real cost that shows up later is variance. A quiet month costs little. A viral feature, a marketing push, or a seasonal spike costs whatever the new volume multiplies out to, with no ceiling unless one is deliberately engineered in. That unpredictability is the actual price of "nothing until the bill arrives": not the rate itself, but the absence of a cap on the total.

What does a local model actually cost?

Hardware first, and it is paid whether or not the card is used. A consumer GPU capable of running a useful open model, an RTX 5070 Ti with 16GB of VRAM in this case, is a fixed cost that starts amortising from day one, independent of how many requests actually arrive. A registered experiment on exactly this card measured what changes when a model is quantised to fit that budget: Q4_K_M quantisation reached 2.22x the throughput of fp16 inference, comfortably past a pre-registered 1.3x bar, while average embedding similarity between quantised and full-precision outputs held at 0.9835, against a 0.90 bar. Knoop and Holtmann (2026) describe the same trade-off across a range of consumer Blackwell cards: real cost-effectiveness for SMEs, bounded by exactly how much precision a business is willing to trade for how much throughput.

The trade is real, not free. Quantisation loss is small in this measurement but not zero, and 16GB sets a hard ceiling on model size regardless of how well quantisation works. A model that needs more context or more parameters than that budget allows is not a local option at any quantisation level, only a different, more expensive hardware conversation.

Where does privacy override the price question?

Some data cannot leave the premises under any pricing model, and that constraint decides the architecture before cost enters the conversation at all. Healthcare records, certain financial data, and anything under a contractual data-residency clause fall into this category regardless of how favourable an API's per-token rate looks. In these cases the honest framing is not "which is cheaper" but "which options are even legally or contractually available," and a local deployment, whatever its hardware ceiling, is sometimes the only one left standing once that filter is applied.

This is also the case where a masking layer changes the calculation, not the venue. Stripping identifying data before it reaches any model, local or API, can turn an otherwise-forbidden API call into a legitimate option, because what leaves the building is no longer the sensitive part. That reframing is worth doing before assuming a privacy requirement forces local hosting by default.

So which one should you pick?

There is no single answer, only a smaller set of questions worth answering in order. What is the realistic request volume, and how steady is it across the day? Does the data need to leave the building at all, with or without a masking layer in front of it? Does the task fit inside what a quantised model on the hardware actually available can deliver, measured, not assumed? A low, steady volume with no privacy constraint usually favours the API. A high, steady volume with spare GPU capacity to amortise usually favours local, once the quantisation trade-off has been measured rather than guessed. Anything in between, most real cases, tends to end up as both: API for the unpredictable long tail, local for the predictable core that justifies the hardware.

A concrete shape this takes in practice: an internal document-tagging pipeline that runs continuously against a steady inbound queue is exactly the profile that justifies the RTX 5070 Ti and the quantisation work above, because the GPU is busy most of the day and the throughput measured on it directly offsets its cost. A marketing-copy generator that fires only around occasional campaign launches, in bursts nobody can predict a month ahead, is the opposite profile: the hardware would sit idle between launches, and the API's elastic capacity is worth its per-token premium precisely because nobody is paying for idle time. The same organisation can reasonably run both at once, on the same hardware budget, once the two workloads are told apart rather than treated as one decision.

FAQ

Does buying a GPU always pay for itself eventually?

No. It pays for itself only if utilisation is high enough, often meaning steady daily volume, not occasional bursts. A card that runs one hour a day is amortising over a tiny fraction of its available capacity, and the API alternative for that same hour may already be cheaper than the electricity.

Does quantisation quality loss actually matter in production?

It depends on the task. The 0.9835 embedding similarity measured on one card and one model is a real number, not a universal one, and a task more sensitive to small output differences than embedding similarity captures could show a larger practical gap. Measure the specific task, do not assume the number transfers.

Is running some models locally and others through an API a realistic setup?

Yes, and it is common in practice once volume is uneven across use cases. A steady, high-volume internal tool justifies local hardware; an occasional, unpredictable one rarely does. Splitting by workload rather than picking one venue for everything usually reflects the actual cost structure better.

What is the simplest way to decide without building a full cost model?

Estimate daily request volume and multiply by the API's per-token price for a rough monthly bill, then compare that to a GPU's amortised monthly cost at realistic utilisation, not full-time. If the API estimate is comfortably lower, that is usually enough to decide without further modelling; if the two are close, only then does a fuller analysis, including the privacy and volatility questions above, earn its time.

References

Jonathan Knoop, Hendrik Holtmann. Private LLM Inference on Consumer Blackwell GPUs: A Practical Guide for Cost-Effective Local Deployment in SMEs, 2026. arxiv.org/abs/2601.09527
Dayi Yao, Zijie Zhou. Efficiency and Cost Alignment in Batched LLM Serving via Resource-Fair Scheduling, 2026. arxiv.org/abs/2608.02244

The demo is the easy part

Ivan Zdravkov — Wed, 28 Oct 2026 06:45:41 +0000

A working demo is the cheapest part of an AI system and the least informative. It proves the happy path exists. Everything that makes the system usable every day, on inputs nobody curated, still remains to be built. I have watched that remainder quietly swallow projects that looked finished on the projector.

Key takeaways

A demo proves feasibility on inputs chosen by the person presenting it; it says nothing about the input distribution of a normal working day.
A synthesis of 27 benchmark, taxonomy and audit papers on LLM agents finds performance degrades nonlinearly with task complexity, and component success does not guarantee end-to-end completion.
Additional scaffolding does not consistently improve reliability; narrowing the scope works better than adding machinery.
The work after the demo is a lifecycle: monitoring, drift detection and feedback loops, which current MLOps research treats as first-class components.
Judge any demo by asking what happens on malformed input and who notices when quality drops.

What does a demo actually prove?

A demo is an existence proof. It shows that for some input, under some conditions, with a person steering, the system produces the intended result. All three qualifiers matter. The inputs were selected, consciously or not, by someone who knows where the potholes are. The person driving retries silently when something goes wrong. The audience sees a selection, not a sample.

None of this makes demos dishonest. A demo answers a real question: is this feasible, and roughly what would it look like. That is worth knowing, and cheap to find out. The mistake is reading the demo as evidence about reliability, when reliability was never on trial. The demo was built to win a meeting, and everything in it serves that goal.

My rule is to treat the demo as the start of the estimate, not the end. When a demo works, the honest statement is that the cheap part is done and the expensive part is now better specified.

What changes when the system runs every day?

Production removes all three qualifiers at once. Inputs arrive uncurated, nobody steers, and errors compound across steps instead of being quietly retried. The gap this opens is documented systematically. A 2026 synthesis of 27 benchmark, taxonomy and audit papers, spanning 19 distinct benchmarks, assembled a taxonomy of LLM agent failures: tool invocation and parameter errors, planning and constraint-satisfaction failures, long-horizon degradation as context accumulates, multi-agent coordination failures, safety failures under adversarial input, and measurement validity problems.

Two findings in that synthesis match what I see in practice. First, performance degrades nonlinearly with task complexity: a system that handles each step at ninety percent does not handle a ten-step chain at anything close to that. Second, additional scaffolding does not consistently improve reliability. Wrapping a fragile agent in more orchestration mostly relocates the failure. The same review notes where genuine progress holds: narrow domains, single-turn tool use, short-horizon tasks. Constrained scope is not a compromise; it is the condition under which these systems currently work.

Why do benchmark scores not settle the argument?

Benchmark results are demos at scale, and they inherit the same selection problem. The tasks are fixed, the distribution is known, and the score summarises performance on that distribution rather than yours. The synthesis cited above puts it bluntly: reported benchmark gains often obscure recurring failure modes that keep being documented across otherwise unrelated evaluations.

A leaderboard position answers the question the leaderboard asks. Whether a system survives your Tuesday afternoon traffic is a different question, and no public number answers it. The only score that predicts production behaviour is one computed on your own data, which is why the first piece of engineering after any demo should be measurement: log every model call, label a sample of real traffic, and watch the number move over time. I made the same argument about masking layers in why the masking layer matters more than the model.

The incentive problem has no methodological fix. Demos and leaderboards exist to persuade. Production metrics exist to disappoint you early, and a team that has only ever built the persuasive kind of evaluation has never been disappointed at the right time.

What does it take to close the gap?

The distance between demo and daily operation is filled with unglamorous components: monitoring on real traffic, drift detection when the input distribution shifts, feedback loops for correcting the system, access control, data masking, audit trails, and a plan for the day your model version is retired. None of these appear in a demo, and each takes longer than the demo did.

Research infrastructure is catching up to this reality. A recent framework for LLM-orchestrated data pipelines dedicates specialised agents to ingestion, monitoring and drift detection, includes human-in-the-loop checkpoints, and reports drift recovery among its lifecycle-level reliability improvements. Whatever one thinks of the architecture, the allocation of effort is telling: the post-deployment half of the lifecycle now gets as much machinery as training, because systems fail there, not on the projector.

The practical conclusion is budgetary. If the demo took two weeks, the production system is a matter of months, and most of those months go into components the demo audience will never see. A team that plans for this is slower to celebrate and far more likely to still be running a year later. This has not made me popular in demo meetings.

FAQ

How can I tell whether an AI demo will survive production?

Ask three questions: what happens on malformed input, who notices when output quality drops, and what is explicitly out of scope. A team that answers concretely has engineered for the boring days. A team that answers with another demo has not, and the gap will surface after the contract is signed.

Why do agent demos look so much better than agent products?

Agent demos are short-horizon and steered, precisely the conditions where the research literature finds agents reliable. Documented failure modes, degradation over long contexts, planning errors and compounding tool mistakes need time and complexity to appear, and a five-minute demo offers neither.

Should teams stop building demos?

No. A demo is the cheapest way to test feasibility, negotiate scope and expose hidden assumptions with stakeholders. The failure mode is treating it as evidence of production readiness. Present a demo as an existence proof with a price tag attached, and it does honest work.

What should be built first once the demo convinces everyone?

Measurement. Log every model call, hand-label a few hundred real examples, and compute a quality number you can watch weekly. Every later decision, about scope, retraining or cancellation, depends on that number existing. Register the thresholds before the numbers arrive; the practice is described in why I pre-register a kill-gate for every experiment. Features can wait; the ability to notice degradation cannot.

References

Wael Albayaydh, Rui Zhao, Ivan Flechais. Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents, 2026. arxiv.org/abs/2607.05775
Aueaphum Aueawatthanaphisut, Badri Raj Lamichhane. Trustworthy Self-Composable Big-Data-as-a-Service: An LLM-Orchestrated Multi-Agent Framework for Automated Data Engineering, AutoML, MLOps Deployment, and Drift-Aware Lifecycle Optimization, 2026. arxiv.org/abs/2606.17915

Permissions that travel with the person

Ivan Zdravkov — Wed, 28 Oct 2026 06:39:48 +0000

An AI assistant should own no permissions. It borrows the rights of the person using it, for the duration of the request, and nothing more. The shortcut that replaces it, one shared service account for everyone, eventually hands a language model more access than any single employee has, which is exactly backwards.

Key takeaways

A shared service account accumulates the union of every user's needs, so the assistant behind it ends up with more access than any individual person.
Prompt injection is a matter of when, not if: ARGUS, one of the strongest published defences, cuts attack success from 28.8% to 3.8% on its benchmark, which is impressive and still not zero.
Because no defence reaches zero, the permission model decides what a successful attack is worth, and per-person scope makes that price small.
Tool lists derived from the user's role make out-of-scope actions invisible rather than forbidden: a tool that is not listed cannot be called.
Identity, role and scope belong in fields the server fills in; nothing the model writes should decide who is asking.

Why is a shared service account the default failure?

Convenience produces the same architecture in every company: one service account for the assistant, wired once, working for everyone. The account needs to serve the support team, the finance team and management, so its permissions grow into the union of all their needs. The result is a single identity with more access than any employee in the building, standing behind a component that takes instructions in natural language from anyone who can type at it.

Every request then looks the same to the downstream systems. The log says the assistant read a contract; it cannot say for whom, because the identity that reached the database was the assistant's own. Access reviews stop working too: nobody can answer which people would lose reach if a permission were removed from the shared account, so permissions are only ever added.

None of this involves an attacker yet. It is just entropy: shared identities absorb scope, and scope, once absorbed, has no owner who could give it back. The failure is structural, and it is in place before the first malicious prompt arrives.

What does it mean for permissions to travel with the person?

Inversion is the whole idea: the assistant is not a user, it is a proxy. When a person asks their assistant a question, the request that reaches internal systems carries that person's identity, their role and their scope, and the systems answer exactly what they would answer the person directly. The assistant adds convenience, not reach.

Mechanically that means per-person credentials instead of one shared key, so every downstream call is attributable to a human. It means scope is evaluated at request time against the person's current rights, not against a snapshot: a permission revoked at noon is gone from the assistant at noon. And it means offboarding needs no special case, because revoking the person's access revokes their assistant's access in the same motion; there is no separate assistant account to remember and forget.

The attribution alone changes behaviour. A log line that reads "the assistant queried salaries" starts an investigation with no suspect. A log line that reads "the assistant queried salaries for this authenticated person" is an answer, and everyone knows their questions arrive under their own name.

How do roles decide what an assistant can even see?

Filtering the tool list is the cleanest enforcement point, and it is the reason I put role logic in the MCP server I described in what an internal MCP server actually solves. When an assistant connects, the server derives the visible tool list from the authenticated person's role. A tool outside that role is not rejected when called; it is never listed, so the model does not know it exists. Invisible is a stronger property than forbidden: a forbidden action can be requested in a hundred phrasings, an invisible one cannot be requested at all.

The parameters that decide scope get the same treatment. Nothing the model writes should determine who is asking: identity fields are filled server-side from the authenticated session, and a schema whose region or owner parameter is server-filled cannot be talked into someone else's data. The model chooses what to ask for; the server decides, from information the model never touches, whom it is asking as. Prompts remain what they are, requests, and the contract lives where contracts belong, in the code that validates every call.

What do permissions buy when an injection succeeds?

Assume the attack works, because sometimes it will. The research on this is converging: a 2026 survey of 247 papers on agent security finds prompt injection and tool-mediated control-flow hijacking still dominate the field, and judges current defences useful but weakly compositional. The most encouraging result I have seen this year is ARGUS, a defence against context-aware injection, where a malicious instruction hides inside the very evidence the agent must read to decide what to do. On the AgentLure benchmark, ARGUS cuts attack success from 28.8% to 3.8% while keeping 87.5% of clean-task utility.

A drop by a factor of seven is real progress, and 3.8% is still not zero. At production volume, a few percent means a matter of time. So the practical question is not whether an injection will ever land but what it can reach when it does, and permissions are that answer. Under a shared account, a landed injection speaks with the union of everyone's rights. Under per-person scope, it speaks with the rights of one person, on whose behalf every action was logged, in a tool list that hides everything outside their role. Detection tells you how often things go wrong; the permission ceiling decides how expensive wrong ever gets.

What does this look like in practice?

In the internal MCP server I am building, currently in testing, permissions travel exactly this way by design: each colleague authenticates with a personal key, the tool list is derived from their role, identity fields are server-filled, and read-only tools come before anything that writes. None of that required inventing new security machinery; it required refusing the shortcut of a shared account on day one, when the shortcut was tempting and the user count was small.

Open questions remain, stated honestly. Delegation is unsolved in the design: when a manager's assistant prepares something using an analyst's data access, whose permissions apply and how is that recorded. Long-running background tasks are a second gap, because rights valid when a task starts may be revoked before it finishes, and re-checking at every step has a cost I have not measured yet. Those are the next design decisions, and I would rather publish them as questions than pretend the model is complete.

FAQ

Why not one service account with careful prompt instructions?

Prompt instructions are requests to a model, not constraints on it, and the survey cited above finds injection attacks remain the dominant, unsolved threat. A shared account puts the union of everyone's permissions behind those breakable instructions. Per-person scope moves the constraint into infrastructure, where phrasing cannot negotiate with it.

What happens to the assistant when someone leaves the company?

Nothing special, which is the point. Offboarding revokes the person's credentials, and their assistant loses every capability in the same motion, because it never had rights of its own. Under a shared-account design the leaver's reach survives inside the service account, invisibly, until someone remembers to audit it.

Do defences like ARGUS make strict permissions unnecessary?

No, they stack. Detection and provenance decide how often an attack lands, and the residual few percent that ARGUS still leaves shows detection alone is not enough; the permission model decides what a landed attack is worth. Run both, and size the ceiling as if detection will occasionally fail, because it will.

How granular should assistant roles be?

Start from the roles your organisation already has, and map each to a tool list; resist inventing a per-person permission matrix on day one. Coarse roles with server-filled identity fields already bound the damage to one person's scope. Refine where the audit log shows real need, not where the org chart suggests theoretical elegance.

References

Shihao Weng, Yang Feng, Jinrui Zhang, Xiaofei Xie, Jiongchi Yu, Jia Liu. ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection, 2026. arxiv.org/abs/2605.03378
Yuchen Ling, Shengcheng Yu, Zhenyu Chen, Chunrong Fang. Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation, 2026. arxiv.org/abs/2606.10749

The masking layer matters more than the model

Ivan Zdravkov — Wed, 28 Oct 2026 06:39:12 +0000

Once sensitive data reaches a model provider, no later decision can bring it back. The layer that detects and masks that data before any model call deserves more design time than model selection. In the AI systems I build, designing the masking layer was among the hardest parts of the work, harder than choosing the models behind it.

Key takeaways

A masking layer detects and replaces sensitive data before any text reaches a language model, local or external.
Masking belongs at the boundary of the pipeline, as one service in front of every model call, not code repeated inside each feature.
Detection does not need a large model. GLiNER2-PII covers 42 PII entity types with 0.3 billion parameters.
Detection is never complete: the strongest configuration compared in the PIIBench paper reaches an F1 of 0.6455 across 82 entity types, so design for leakage.
A few hundred labelled examples from your own traffic tell you more than any vendor benchmark.

Why does the masking layer matter more than the model?

Model selection is a reversible decision. A pipeline built behind an abstraction can swap one provider for another or move to a local model without touching business logic. Sending a customer's name, address and order history to the wrong place is not reversible. The data has left the building, it may be logged, cached or used for training, and no later architecture change undoes that.

The asymmetry is why I treat masking as the core of any AI system that touches operational data. The model decides how good the answers are. The masking layer decides whether the system may run inside a real company at all. GDPR obligations follow the data, not the model, so the question a data protection officer will ask is not which model I chose but what exactly left the infrastructure.

The other reason is practical. Models improve every quarter with no effort on my side, because vendors and open-source teams keep releasing better ones. The masking layer improves only when I work on it. Nobody will ship me a better understanding of which fields in my own systems are sensitive or which partners put phone numbers in comment fields.

What does a masking layer actually do?

A masking layer performs three steps: it detects spans of sensitive data in the text about to be sent to a model, replaces them with placeholders, and restores the original values in the response before a human sees it.

Detection covers two different problems. Structured fields are easy: the schema already says which column holds a name or a bank account, so masking is a lookup, not a guess. Free text is the hard part. A support ticket or an invoice line can contain a phone number, an identification number or an address in any format and any language, and finding those spans is a named entity recognition task, not a string match.

Replacement, in the design I would recommend, uses deterministic placeholders within a request, so the same person resolves to the same token everywhere. A model that sees CUSTOMER_1 ordering and CUSTOMER_1 complaining can reason about the connection; one that sees two unrelated tokens cannot.

A consistent token is also a pseudo-identifier. With enough surrounding context, a stable CUSTOMER_1 can be linked back to a person, so the mapping should be scoped as narrowly as the task allows and long-lived tokens should be treated as data that still needs protection.

Restoration means the mapping between placeholders and real values never leaves the infrastructure: the model only ever sees tokens, and unmasking happens on the way back, under the same access rules as the rest of the system.

Where does the masking layer sit in the pipeline?

One service in front of every model call is the placement that works. Every feature that talks to a model, whether it classifies tickets, checks invoices or answers a colleague's question, routes its request through the same masking gateway, and nothing else in the codebase holds an API key to a model provider.

Centralising the layer buys three things. First, one place to test: a single suite and one labelled dataset cover every feature at once. Second, one place to improve: when a new leak pattern shows up, the fix lands in one service instead of five prompt builders. Third, one audit log: the gateway records exactly what left the infrastructure and in what form, the record a GDPR conversation needs.

The alternative, masking logic living inside each feature, decays fast. Different features drift to different rules, a new feature ships in a hurry without any masking, and no one can answer what the system as a whole sends out.

Can a small model handle the detection?

Detection does not require a large generative model. GLiNER2-PII, published in May 2026, is a 0.3 billion parameter model extracting 42 types of personally identifiable information across languages, trained on a constraint-generated synthetic corpus of 4,910 annotated texts rather than real people's text. On the SPY benchmark it reports higher span-level F1 than the systems it was compared against, the OpenAI Privacy Filter and three GLiNER-based detectors. A model of that size runs on modest hardware inside your own network, where a masking component should run.

A second result points at where the effort should go. On PIIBench, a corrected multi-source dataset with 82 entity types, direct token-classification fine-tuning of DeBERTa reached an F1 of 0.6455 on a test set of 100,002 records, beating a source-conditioned hierarchical model and a curriculum-based variant from the same author. The conclusion of that paper matches my experience: diverse task-specific training data moves detection quality more than architectural sophistication does.

Using a large hosted model to detect PII defeats the purpose. The detector would receive the raw sensitive text, precisely the exposure the masking layer exists to prevent. The detector has to be the one component you can run entirely inside your own boundary. My own later measurement of GLiNER2-PII on public data confirmed the strengths and found the weak spot, surnames; the numbers are in why I pre-register a kill-gate for every experiment.

What slips through, and how do you measure it?

An F1 of 0.6455 reported in that paper across 82 entity types means a substantial share of entity mentions are missed or mislabelled. Narrower, well-represented types score much higher, but the honest reading is that free-text detection leaks, and the design has to survive that.

Leakage is not only a missed entity. Three unmasked details that look harmless on their own, a town, a delivery window and an unusual surname, identify a person more reliably than any single field does.

Benchmark scores are measured on someone else's data. Mine has Bulgarian names in Cyrillic and in several Latin transliterations, national identifiers whose format collides with other numbers, and addresses buried in free-text comments. No public benchmark contains that mix, so no public score predicts my leakage. The same gap between benchmark scores and production behaviour shows up everywhere in AI systems, not only in PII detection: see the demo is the easy part.

The measurement is small and boring. Take a few hundred real examples from your own traffic, hand-label the sensitive spans, and compute recall per entity type. Recall matters more than F1 here: a false positive masks too much and costs a little answer quality, a false negative sends personal data to a provider. The errors are not symmetric, so tune thresholds toward over-masking.

For categories where a miss is unacceptable, the fallback is structural, not statistical: route those requests only to a locally hosted model, or keep those fields out of model input entirely. I do not have a method that reaches zero leakage, and I distrust claims of one.

What does this look like in practice?

In the systems I work on, several features share one masking layer: text-heavy request handling, document reconciliation, and an internal service that lets colleagues' assistants query company data. Each of them sends text to a language model, and none of them is allowed to do so directly.

Designing that layer was among the hardest parts of the whole effort. I consider the time well spent, because it is the reason the features may run in production at all. The building blocks are ordinary: schema-driven masking for structured fields, a named entity recognition model for free text, one gateway in front of every model call, and tracing on each request. The mapping between placeholders and real values stays inside the infrastructure, and what any given assistant may see is decided by the role of the person using it, not by the model.

Not everything worked on the first attempt. The structured-field side was straightforward, because the schemas were known. The free-text side was not. The spans I trust least are those where a harmless-looking string carries identity: organisation names shaped like personal names, addresses written into comment fields, and numbers whose meaning depends entirely on context. The per-entity recall measurement described above now exists, and its first run turned that impression into numbers.

FAQ

Do I still need a masking layer if the model runs on my own hardware?

Yes. A local model removes the external provider, but prompts still end up in logs, traces and evaluation datasets. Masking limits what spreads into those secondary stores. It also keeps the pipeline portable if part of the workload later moves to an API model.

Is regex enough for PII detection?

For structured fields and strictly formatted identifiers, yes: regex is fast and auditable. For free text it is not enough: names, addresses and mixed-language content do not follow patterns. The practical setup is layered: schema-based masking first, regex for formatted identifiers, an NER model for everything else.

Does masking hurt the quality of the model's answers?

Little, provided placeholders are deterministic: the model reasons about CUSTOMER_1 and ORDER_17 as consistent entities, and real values return during unmasking. Quality drops when the masked value carried meaning the task needed, such as a city name in a routing question, so masking rules must be scoped per task.

What is the difference between masking and anonymisation?

Anonymisation is irreversible: once applied, nobody can recover the original values, which suits published datasets. Masking is reversible inside a trusted boundary: the mapping to real values is kept, and authorised users see restored output. Assistants that act on real operations need masking: an answer about ORDER_17 is useless if nothing can say which order that is.

References

Urchade Zaratiana, Ash Lewis, George Hurn-Maloney. GLiNER2-PII: A Multilingual Model for Personally Identifiable Information Extraction, 2026. arxiv.org/abs/2605.09973
Pritesh Jha. Fine-Tuning Over Architectural Complexity: Broad-Coverage PII Detection on PIIBench with DeBERTa, 2026. arxiv.org/abs/2605.25816

LoRA on a 16 GB card: what fits and what does not

Ivan Zdravkov — Wed, 28 Oct 2026 06:33:20 +0000

A 7B model is the honest ceiling for LoRA fine-tuning on a 16 GB consumer card, not because anything bigger throws an error, but because a 14B model silently spills into system memory and runs 50 times slower without ever reporting a failure.

Key takeaways

7B is the real ceiling on an RTX 5070 Ti at LoRA rank 16, sequence length 512, batch size 1: 11,897.9 MiB peak, well inside the card's 16,303 MiB.
14B completed without a raised exception in both tested conditions, but peak VRAM read 22,189.8 MiB (136% of the card) and 17,470.9 MiB (107%) - both over the physical limit, through Windows' silent system-memory fallback.
The cost of that fallback is severe: 21.7 seconds per training step at 14B against 0.4 seconds at 7B, roughly 51 times slower, with no error message anywhere.
Gradient checkpointing is close to free at 7B and below - same or less time, meaningfully less memory - but does not rescue 14B: it leaves the model still over the limit and makes the step 59% slower, not faster.
The registered hypothesis that checkpointing moves the ceiling up a tier is rejected; the wording of the first kill-gate also needed correcting, because "did not throw an out-of-memory error" turned out to be the wrong proxy for "fits".

Why doesn't "no error" mean "it fits"?

I pre-registered two kill-gates on an RTX 5070 Ti, 16,303 MiB of physical VRAM: whether a 14B Qwen2.5-Instruct model, loaded in 4-bit with a LoRA adapter, completes one real training step without a raised CUDA out-of-memory exception, and whether gradient checkpointing moves that ceiling up by at least one tested size, following the same discipline described in why I pre-register a kill-gate for every experiment. Neither 14B run threw an exception. By the literal wording of the first gate, it fired - 14B "fits". But torch.cuda.max_memory_allocated() reported 22,189.8 MiB without checkpointing, 5,887 MiB above the card's physical limit, and the step took 21.7 seconds instead of the sub-second range every smaller model showed. Windows ships a driver-level system-memory fallback policy that lets CUDA allocations succeed past a card's physical limit by quietly paging into ordinary RAM instead of raising an error, and nothing in the console said so. The gate measured the wrong thing: "did not throw an exception" is not the same claim as "fits in 16 GB", and only comparing the allocator figure against the card's actual VRAM, plus noticing the 50x time cliff, exposed the gap.

What did the clean numbers actually look like?

Three of the four sizes behaved exactly as expected. Without checkpointing, 1.5B peaked at 4,000.7 MiB, 3B at 6,749.0 MiB, 7B at 11,897.9 MiB - all comfortably under 16,303 MiB, all completing a forward-backward-optimizer step in under half a second. 7B used 73% of the card's VRAM at rank 16, sequence length 512, batch size 1, with real room to spare. Full numbers for all eight configurations sit in the frozen plan and results in the public experiment repository. Enabling gradient checkpointing lowered every one of those three figures - 2,855.5 MiB, 4,490.8 MiB and 9,709.9 MiB respectively - at essentially no time cost: 7B ran in the same 0.425 seconds either way, 3B cost 25% more time for a third less memory. The 1.5B timing is excluded from this comparison because it ran first in the script and likely carries a one-time CUDA initialisation cost. The honest ceiling for this exact setup, Qwen2.5-Instruct, 4-bit base, LoRA rank 16, one sequence of 512 tokens, batch size 1, is 7B. Push any of those variables up and the real number moves down, not up.

Does gradient checkpointing rescue the 14B case?

No. With checkpointing enabled, 14B still reported 17,470.9 MiB, 1,168 MiB over the card's physical limit - smaller than the 5,887 MiB overflow without checkpointing, but still spilling into system memory, not fitting inside it. The step time went the wrong direction: 34.5 seconds, 59% slower than the 21.7 seconds without checkpointing. Checkpointing buys memory by deleting intermediate activations during the forward pass and recomputing them during backward, and that recompute cost is normally hidden by the time saved from avoiding memory pressure, which is exactly why it helped 7B and below for free. Once the card is already paging to system RAM, the saved memory only shrinks an already-severe penalty instead of removing it, and the extra computation adds straight on top. The registered hypothesis, that checkpointing moves the ceiling up by at least one tested tier, is rejected. On this card, for this model family, checkpointing is a real, close-to-free lever below the ceiling, not a way past it.

FAQ

Does this mean a 16 GB card can never run a 14B model?

Not for training, no, not with this exact setup. A smaller rank, a shorter sequence, a smaller LoRA target set, or a more aggressive quantisation scheme could plausibly bring a 14B model under the limit; none of those were tested here. For inference rather than training, the memory math is different again and was not measured in this experiment.

Why trust torch's reported VRAM figure at all, if it can be wrong?

It is not wrong about what it allocates - torch.cuda.max_memory_allocated() reports real tensor memory the CUDA allocator granted, honestly. What it does not tell you is whether that memory was physically on the card or transparently paged to system RAM by the driver. Comparing the figure against the card's known physical limit, rather than trusting the absence of an exception, is what catches the difference.

Why only test rank 16 and batch size 1?

Those are close to the smallest realistic defaults most people reach for first. Testing a single point cleanly, rather than sweeping many configurations in one pass, keeps the registered claim specific and checkable. A larger rank or batch size sweep is a legitimate follow-up, not something this experiment's kill-gates covered.

Where do the numbers come from?

Every figure in this piece is read directly from results.json in the registered experiment, not estimated or rounded from memory. The frozen plan, the full experiment code, and the raw per-run output all sit in the public repository linked above, so the eight numbers in the table can be checked against the source instead of taken on trust.

References

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, 2021. arxiv.org/abs/2106.09685

Why I pre-register a kill-gate for every experiment

Ivan Zdravkov — Wed, 28 Oct 2026 06:32:38 +0000

A kill-gate is a threshold written down before an experiment runs: if the result crosses it, the idea dies. The writing down happens first, because after the numbers arrive, every threshold becomes negotiable. Last week two of my gates fired and one refused to answer, which taught me more than a success would have.

Key takeaways

A kill-gate names, in advance, the number at which I abandon my own hypothesis; the plan file is frozen before the first run and never rewritten afterwards.
Gates exist because results negotiate: search-time contamination alone can inflate agent benchmark scores by up to 4%, and LLM-judge scores now come with conformal prediction intervals because point values wobble.
Of the three verdicts a gate can return, "cannot be pronounced" is the most valuable one: it indicts the instrument instead of the hypothesis.
The axis you are tempted to leave out is the one to register: my phone-detection gate only told the truth because it measured precision as well as recall.

What is a kill-gate?

A kill-gate is one sentence in a plan file: a hypothesis, a metric, a number, and the action taken if the number is crossed. The file is dated and frozen before any code runs; corrections are appended as a dated journal, never edited in place. Everything else about the experiment may be humble, but this part is a contract.

The contract matters because evaluation without pre-set rules drifts toward the answer its author wants. The drift is documented at every scale. A 2026 study of deep research agents found search-time contamination, where the agent's own web searches touch benchmark material, across all six public benchmarks tested, inflating scores by up to 4% without anyone cheating deliberately. Evaluations that use a model as judge carry enough score variance that researchers now wrap them in conformal prediction intervals rather than trust a single number. If public leaderboards with reviewers need this much guarding, a solo practitioner grading his own idea needs more. The gate does not make me objective; it makes my bias arrive too early to help me.

What did the gates do in a real experiment?

Registration came first: before evaluating a small PII detection model (GLiNER2-PII, 0.3B parameters) on 4,000 public examples from the ai4privacy/pii-masking-openpii-1m dataset on Hugging Face, I froze a gate saying that name recall below 0.90 on English kills the claim "a small open model suffices out of the box". The result was 0.850 for given names and 0.291 for surnames. The gate fired, the claim is dead, and I am not allowed to describe that model as sufficient in any future article.

The surname failure dictated a second experiment: whether an ensemble of standard open components, pattern recognisers orchestrated with a multilingual NER model, recovers what the single model misses. On the same frozen sample, surname recall rose to 0.970, so the ensemble genuinely fixes the single model's blind spot. Its phone detection then returned recall of 0.400 against the small model's 0.995, and the registered precision axis came back at 0.36 against a 0.99 bar; both numbers describe this dataset's synthetic formats, not any live traffic. Two experiments, two dead claims, and a conclusion neither run would have volunteered on its own: the two shapes of detector are blind to different things. Numbers I registered thresholds for became findings; numbers I merely observed stayed observations.

Why is the gate that refused to answer the most useful?

One registered gate targeted a hypothesis I raised in why the masking layer matters more than the model: that Cyrillic text weakens detection. If English and Bulgarian name recall differ by under 5 points, it dies. Before pronouncing it, I ran a validity check on the data, and the check ended the question: 100% of the 3,410 name values in the Bulgarian subset turned out to be Latin-script. The dataset's language tag promises Bulgarian, and the surrounding text is Bulgarian, but every person in it has a Latin-alphabet name. The gate cannot be pronounced, because the instrument does not contain the thing the hypothesis is about.

Without a registered gate I would have computed the difference, found it under 2 points, and written that Cyrillic is not a problem: numeric, plausible and wrong. The registered wording forced the prior question of whether this data can answer me at all. An unpronounceable gate returns "unmeasured", which is an honest state that a spreadsheet will never volunteer on its own.

How do I write one?

Four rules cover what I currently know. First, one sentence per gate: hypothesis, metric, number, action; a gate that needs a paragraph is hiding a decision. Second, choose the metric by asymmetry of cost: in PII masking a false negative leaks personal data while a false positive slightly degrades an answer, so recall guards the gates for leakage. Third, register the axis you are tempted to omit. My phone gate began as recall-only; the operational concern is really false positives, an order reference flagged as a phone number, so a precision axis with a 0.99 bar went in before the run. That axis returned 0.36 and caught the actual failure. Recall alone would have shown a softer problem and let the dangerous one pass as a green light. Fourth, keep a dated journal: amendments before results are method, amendments after results are negotiation, and the file format should make the difference visible to any reader.

FAQ

Is a kill-gate the same as a statistical hypothesis test?

No, and it does not need to be. A kill-gate is a commitment device about timing: the threshold exists before the data does. A significance test can serve as the metric inside a gate when samples are small, but the protection comes from the pre-registration, not from the formula.

What happens when a result lands just under the gate?

The idea dies, and that is the entire point. A new gate may be registered for the next experiment, with a written reason; this one does not move. The sentence "0.88 is basically 0.90" is precisely the sentence kill-gates exist to kill, and it always arrives dressed as reasonableness.

Do experiments whose gates fire get published?

Yes, with the same weight as successes. A fired gate is a result: it names a claim nobody should repeat and the number that killed it. The frozen plans live in a public repository, so a reader can check what was promised before the numbers existed.

References

Yongjie Wang, Xinyue Zhang, Kunhong Yao, Zhiwei Zeng, Kaisong Song, Jun Lin, Zhiqi Shen. Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation, 2026. arxiv.org/abs/2606.05241
Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, Jian Kang. Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction, 2025. arxiv.org/abs/2509.18658

How wrong is a model's confidence?

Ivan Zdravkov — Wed, 28 Oct 2026 06:26:47 +0000

A ResNet-34 trained on CIFAR-100 is measurably overconfident: its top prediction bin claims 99% certainty while landing correctly only 94% of the time. Temperature scaling, a single learned number, cuts that gap by nearly two-thirds without changing a single prediction.

Key takeaways

A registered gate said: if Expected Calibration Error (ECE) on the held-out half of the test set came in under 3%, the claim "this model is overconfident" would be dead. The real number is 7.87%, more than double the threshold.
Temperature scaling cut ECE from 7.87% to 2.85%, a 63.7% relative reduction, well past the 30% bar registered to count as a fix.
The gap concentrates where most predictions live: 70.1% of test examples fell in the top confidence bin, claiming 99.11% certainty on average while landing correctly 94.27% of the time, before any correction.
A single scalar, fitted on one held-out half of the data and applied to the other, does this without moving accuracy at all: 80.24% before and after.

What is Expected Calibration Error, and why should it matter?

A model's softmax output is often read as a probability: "94% confident" should mean the model is right about 94% of the time it says that. Expected Calibration Error (ECE) checks whether this holds. Predictions are grouped into confidence bins (15 bins here, 0-100%), and for each bin the gap between average stated confidence and actual accuracy is measured, then averaged across bins weighted by how many predictions fall in each one.

Guo et al. (2017) showed this gap is not a rounding error in modern networks. Depth, width and batch normalisation, the same ingredients that improved accuracy through the 2010s, tend to push confidence upward faster than accuracy actually improves. A network can get better at the task while getting worse at knowing how sure it should be. ECE is the number that catches this, because accuracy alone cannot: a model can be 80% accurate and either well calibrated or badly overconfident, and only the confidence values reveal which.

How overconfident was this model, in real numbers?

The experiment used edadaltocg/resnet34_cifar100, a public ResNet-34 checkpoint (MIT licence, reported 79.78% test accuracy, trained 300 epochs with SGD and cosine annealing) evaluated on half of the official CIFAR-100 test set: 5,000 images, held out purely for this measurement and never used to fit anything. Top-1 accuracy on that half came in at 80.24%, close enough to the reported figure to trust the checkpoint loaded correctly. ECE came in at 7.87%, against a registered kill-gate of 3%: the gate did not fire, so the claim that this model is overconfident survives.

The clearest view is the top confidence bin, 93.3% to 100%, where 3,507 of 5,000 predictions landed, 70.1% of the entire test half. Average stated confidence there was 99.11%. Actual accuracy in that same bin was 94.27%, a 4.84 percentage point gap sitting inside the bin most predictions occupy. This is not noise from a handful of edge cases. It is the model's default operating point.

Does temperature scaling actually fix it?

Temperature scaling divides every logit by a single learned constant T before the softmax, which softens or sharpens the whole probability distribution without changing which class wins. That last part matters: because the operation is monotonic, the argmax, and therefore accuracy, cannot change. It is a diagnostic gate in this experiment (KG3), not a hypothesis, and it passed exactly as the maths predicts: 80.24% before, 80.24% after.

T was fitted on the other half of the test set (5,000 images, never touched during evaluation) by minimising cross-entropy loss with the rest of the network frozen. The fitted value was T = 1.2508. Applied to the evaluation half, ECE dropped from 7.87% to 2.85%, a 63.7% relative reduction against a registered 30% bar. The top confidence bin shrank to 56.9% of the data, with average confidence easing to 97.93% against 97.05% actual accuracy, a 0.88 point gap where there was 4.84. One number, fitted once, closed most of the distance.

What does this not tell us?

The experiment covers one checkpoint, one architecture family, one dataset. It says nothing about whether a different ResNet, a transformer, or the same model on a harder task would show the same overconfidence or the same fix. The dev and eval splits here both came from the same official test set, split in half, because the checkpoint was already trained on the full training set and no independent held-out validation data existed. A researcher training the model from scratch could hold out a proper validation split before ever touching the test set; that would be the cleaner version of this experiment.

It also says nothing about large language models. Confidence calibration in text generation is a substantially harder, more open problem, covered by a growing body of work rather than one fix (Liu et al., 2025). Temperature scaling on a classifier's softmax and calibrating a generative model's stated or implied confidence are related in name only.

FAQ

Is Expected Calibration Error the same thing as accuracy?

No. Accuracy measures how often the model is right. ECE measures how well its stated confidence matches how often it is right. A model can be accurate and badly calibrated, or less accurate and well calibrated; the two numbers answer different questions and neither substitutes for the other.

Why does temperature scaling leave accuracy unchanged?

Dividing all logits by the same constant before softmax rescales the probabilities but preserves their order, so the highest-scoring class stays the highest-scoring class. Accuracy depends only on that ordering (the argmax), never on the actual probability values, which is exactly what temperature scaling touches.

Could a well-trained model already be calibrated out of the box?

Yes, in principle; this experiment measured one checkpoint, not a law of nature. Guo et al. (2017) found the overconfidence pattern across several architectures and datasets, which is why the kill-gate here was registered at 3%, not zero, but a different model could plausibly clear that gate.

Does this result say anything about how confident an LLM's answers are?

No. This measures a classifier's softmax output over a fixed set of labels, which is a much narrower object than a language model's output over an open-ended vocabulary. Whether and how a generative model's stated or implied certainty should be trusted is a separate, less settled question, covered by a growing survey literature rather than one fix (Liu et al., 2025).

The registered plan, the raw numbers and the reliability diagram data behind these figures sit in the public experiment repository, alongside the other experiments this site has published.

References

Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. On Calibration of Modern Neural Networks, 2017. arxiv.org/abs/1706.04599
Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, Hua Wei. Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey, 2025. arxiv.org/abs/2503.15850

Fluency is not understanding

Ivan Zdravkov — Wed, 28 Oct 2026 06:26:11 +0000

A model that sounds equally confident whether it is right or wrong is not a rare failure mode. It is the default: a ResNet-34 I tested claimed 99% certainty on predictions that were only 94% accurate, and nothing in its tone gave that away.

Key takeaways

Fluency and correctness are produced by different parts of a model's behaviour, and nothing forces them to move together. A confident-sounding answer is evidence of confident-sounding training, not of accuracy.
In a registered experiment on a public CIFAR-100 checkpoint, the model's highest-confidence bin held 70% of all predictions, claimed 99.11% certainty, and delivered 94.27% accuracy. The gap did not announce itself.
Guo et al. (2017) found this overconfidence pattern across multiple architectures, not one unlucky model - closer to a property of how modern networks train than a bug in one of them.
Large language models make this harder to catch: there is no clean softmax over a fixed label set to audit, only fluent text that can sound equally sure when right and when fabricating.

Why does a model's tone not track its accuracy?

A classifier's stated confidence and its actual accuracy are two different outputs of the same training process, and nothing in standard training forces them to agree. A network is optimised to get the label right, not to know how sure it should sound doing it. Guo et al. (2017) showed that as networks grew deeper and more capable through the 2010s, their confidence tended to outpace their accuracy. The pattern showed up across architectures and datasets, which is the detail that matters: not one poorly trained model having a bad day, but a systematic side effect of how the field made models more accurate.

Fluency, in the language model sense, is the same phenomenon wearing different clothes. A model trained to produce coherent, well-formed text learns to sound coherent and well-formed regardless of whether the content is correct. Sounding sure and being right are correlated often enough that it feels safe to conflate them, and decoupled often enough that the conflation eventually costs something.

What does this look like in a real measurement?

Abstract claims about overconfidence are easy to nod along to and easy to forget. A number is harder to wave away. In a registered calibration experiment on a public ResNet-34 checkpoint, I measured Expected Calibration Error (ECE) on half of the official CIFAR-100 test set before touching anything: 7.87%, more than double the 3% threshold that had been set in advance to kill the claim that the model was overconfident.

The gap concentrated, not scattered: 70.1% of all predictions landed in the single highest-confidence bin, claiming 99.11% average certainty and delivering 94.27% actual accuracy. That 4.84 point gap sits inside the bin where most of the model's output lives, not in some rare tail case a user would already distrust. Temperature scaling, one learned number, closed most of it without changing a single prediction, which says the overconfidence was a correctable distortion, not evidence the model secretly knew less than it claimed.

Why does this matter more for language models than classifiers?

A classifier's confidence is at least auditable in principle: a number between 0 and 1, attached to a fixed set of labels, sitting there waiting for someone to compute an ECE against it. A language model's fluency offers no equivalent handle. There is no small label set to bin against, no single softmax value standing in for "how sure was I," just text that can hedge, assert, or fabricate in the same even, competent register.

Liu et al. (2025) survey a large, growing body of work trying to build something like a confidence signal for language model output, precisely because none exists by default. Token-level probabilities measure how predictable the next token was, not how well-founded the claim in it is; a model can be entirely unsurprised by its own fabrication. The classifier case here is the easy version of the problem, with a clean number to measure and a known fix. The language model version has neither yet.

What should change in how we read model output?

The practical shift is not learning to detect uncertainty by ear. That skill does not exist, because the whole finding is that tone does not carry the information. What holds up instead is refusing to treat fluency as evidence and asking, for any claim that matters, what would actually confirm or kill it: a benchmark with a pre-registered threshold, a citation checked rather than trusted, a kill-gate written down before the number exists. None of that is available mid-conversation, which is why it has to be built into the process around the model, not read off its tone in the moment.

This is a workflow change, not a model change. It moves the question from "does this answer sound right" to "what would have to be true for it to be right, and has anyone checked." Fluency answers the first question for free, every time, whether it deserves to be trusted or not. The second question needs a step outside the conversation: a test run before the claim is trusted, a source opened and read, a threshold set before the output existed. Neither classifiers nor language models announce which outputs need that step, so the safer default treats any output a real decision rests on as one that does.

FAQ

Isn't a well-written, coherent answer at least some evidence the model understood the question?

Some, but weaker than it feels. Coherence shows the model produced text consistent with learned patterns, which correlates with correctness often enough to be useful and fails often enough to be dangerous. It is a prior, not a verification, and the two get conflated exactly when it matters most.

Can people just get better at detecting an uncertain model by how it phrases things?

Not reliably. The CIFAR-100 experiment measured a classifier with no phrasing at all, just a number, and the overconfidence was still large. Hedging language in a text model is a stylistic choice the model can produce or omit independent of whether the underlying claim is true.

Does temperature scaling or similar fixes solve this for language models too?

Not directly. Temperature scaling works cleanly on a softmax over fixed labels, which is what a classifier has and a language model's free-form output does not. Liu et al. (2025) cover the wider set of methods attempting something equivalent for generative models, none of them as settled as temperature scaling is for classifiers.

Is overconfidence only a problem for smaller or older models?

No. Guo et al. (2017) found the pattern in the architectures that were state of the art at the time specifically because they were more capable, not despite it. There is no basis in that finding for assuming a newer or larger model is exempt by default.

References

Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger. On Calibration of Modern Neural Networks, 2017. arxiv.org/abs/1706.04599
Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, Hua Wei. Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey, 2025. arxiv.org/abs/2503.15850