DEV Community: Dmytro Nasyrov

Top 5 LLM Evaluation Frameworks for Release Engineering in 2026

Dmytro Nasyrov — Thu, 27 Aug 2026 16:32:58 +0000

Choosing an LLM evaluation framework for release engineering is not a contest for the longest metrics catalog. The practical question is whether a tool can bind results to an exact model, prompt, dataset and application revision, then turn a failed requirement into a blocked release.

Under that test, Promptfoo is the strongest CI-native option, DeepEval fits Python test suites, LangSmith leads when managed traces and experiment history matter, OpenAI Evals is useful for reusable eval specifications, and Ragas is the specialist for RAG quality. None replaces deployment controls. The ranking below uses official documentation verified on August 27, 2026.

The rubric comes from the release criteria we use in production AI engineering at Pharos Production, so Pharos is the evaluator here, not a sixth framework in the ranking.

How the five frameworks were ranked

Every alternative had to support repeatable evaluation against a versioned dataset and some form of custom evaluator. I then compared the five on the same release-engineering questions:

What does a regression run compare?
Can a team encode deterministic checks and LLM-as-a-judge criteria?
How does a failure reach CI/CD?
Can the result be traced to a pull request or candidate build?
What operating constraint makes the tool a poor fit?

Release-gate readiness has three meanings in this article. Native means the official workflow documents a failing test or build path. Composable means test hooks exist but the team writes the blocking policy. External means the framework creates evidence while another system must own the block.

This is a release-engineering ranking, not a popularity ranking. A team optimizing a research benchmark, a RAG pipeline or a Python library can rationally choose a different order.

Decision matrix

Framework	Best fit	Regression unit	Custom evaluators	CI/blocking path	Traceability evidence	Main constraint	Disqualifier
1. Promptfoo	CI-native prompt, model and security regression	baseline versus candidate configuration and test cases	deterministic assertions, JavaScript/Python hooks, semantic and model-graded checks	Native: failed tests or pass-rate threshold can return a failing process status	tags for Git SHA or CI run; JSON, HTML and JUnit output	Node-based CLI and provider credentials	reject when the team will not maintain declarative eval configuration in CI
2. DeepEval	Python and pytest-based LLM regression testing	test case plus metric threshold, run as a pytest-style suite	`BaseMetric`, GEval and built-in task metrics	Native: `assert_test` and `deepeval test run` can fail a build	local test output; shared reports and official baselines through Confident AI	judge credentials; hosted history needs an additional service account	reject when the release stack is not Python-oriented and cannot host pytest-style tests
3. LangSmith	managed experiments tied to traces and application components	dataset experiment, including intermediate trace steps	custom code evaluators and LLM evaluators	Composable: pytest/Jest hooks exist; the team owns the release policy	datasets, experiment metadata, traces, comparisons and exports	managed platform dependency and explicit policy glue	reject when evaluation data or traces cannot be sent to the managed service
4. OpenAI Evals	reusable eval specifications and benchmark-registry workflows	JSONL samples plus an eval definition and completion function	templates and custom eval classes	External: run the eval in CI, then add your own threshold and evidence binding	run artifacts depend on the wrapper and logging setup	OpenAI API cost/key; some solver interfaces are beta	reject when a turnkey pull-request gate and release ledger are required
5. Ragas	RAG evaluation metrics and experiment comparison	RAG or agent dataset run, optionally compared with a baseline experiment	discrete and numerical metrics plus specialized RAG metrics	External: CLI and experiment comparison provide results; CI must interpret them	timestamped experiment results and CSV-backed comparisons	strongest vocabulary is RAG-specific; gate policy is team-owned	reject when tool-use safety, permissions and broad system regression dominate the release

1. Promptfoo: strongest CI-native release gate

Promptfoo puts the shortest distance between an evaluation failure and a blocked build. Its CI/CD integration shows evaluations running in standard pipelines, while the CLI can fail on errors or enforce a pass-rate threshold. That makes it practical for pull request evaluation: compare a candidate prompt or model with a baseline, test protected scenarios, then stop the merge when the policy fails.

The custom assertion surface is broad enough to mix exact checks, JavaScript or Python logic, semantic similarity and model-graded rubrics. Its expected-output configuration can express assertions beside test cases instead of hiding the release rule in a dashboard.

For traceability, tags can carry a Git SHA or CI run ID, and the command-line interface can emit JSON, HTML or JUnit evidence. The trade-off is operational: the team must maintain Node-compatible tooling, provider credentials and a disciplined configuration repository. Promptfoo is the best choice here when evaluation is expected to behave like a software quality gate rather than an analyst workflow.

2. DeepEval: best fit for Python test suites

DeepEval treats LLM regression testing as unit testing. A developer creates an LLM test case, attaches metrics and thresholds and calls assert_test inside pytest. Its CI/CD guide documents deepeval test run as a build step where failed metrics fail the job.

That model works well for Python services because an eval suite can live beside application tests. Teams can extend BaseMetric for deterministic logic or use GEval when a rubric needs an LLM judge. Local execution does not require a Confident AI account, although the judge provider still needs credentials. Shared reports, trends and an official baseline add the Confident AI service.

The disqualifier is not evaluation quality; it is stack fit. A TypeScript-first release system may not want Python and pytest as the control plane. If the application is already Python-based, however, DeepEval offers a clear test-to-build failure path without forcing all result review into a separate UI.

3. LangSmith: best managed traceability

LangSmith is strongest when an evaluation result must be inspected with the execution that produced it. Its evaluation workflow covers datasets, offline experiments, code evaluators and LLM evaluators. Teams can attach experiment metadata for model, prompt and tool versions, compare runs and export results.

The differentiator is component-level evidence. LangSmith can evaluate intermediate trace steps, which helps diagnose whether retrieval, tool selection or another stage caused the regression. Pytest and Jest integrations make the results available during testing, but release authority is still composable: the team must define which metric, slice or critical case blocks the candidate.

In the MLOps release and monitoring work, that boundary matters because offline evaluation still has to hand evidence to canary deployment, observability, rollback and an accountable release owner.

Choose LangSmith when managed experiment history and traces justify the platform dependency. Reject it when policies prohibit sending evaluation data or traces to the service, or when the team wants a completely local gate with no managed control plane.

4. OpenAI Evals: best for reusable eval specifications

OpenAI Evals is an open-source framework and registry for evaluating language models and model-powered systems. Its useful release-engineering contribution is the specification: private JSONL samples, reusable templates, completion functions and custom eval classes can turn an evaluation method into versioned code.

That flexibility is also why the framework ranks below the first three for release gating. The official repository explains how to build and run evals, but it does not provide the same documented pull-request policy, candidate binding and build-fail workflow as Promptfoo or DeepEval. Teams need a wrapper that translates scores and critical failures into process status, stores artifacts and binds the run to the candidate commit.

The framework also requires an OpenAI API key for relevant runs and incurs model-call cost. Its newer solver interface is documented as beta, which matters if a release process depends on a stable extension boundary. Choose it when the eval definition and benchmark corpus are the primary assets. Do not choose it expecting a turnkey release ledger.

5. Ragas: best for RAG-specific evaluation

Ragas earns the fifth place because retrieval-augmented generation has failure modes that generic answer scoring often misses. Its RAG evaluation workflow starts from a dataset and applies metrics suited to retrieved context and generated answers. The framework also supports custom discrete and numerical metrics.

The experimentation layer stores runs and supports comparison with a baseline experiment. That is useful for testing a new embedding model, chunking strategy, reranker or prompt against a known RAG configuration.

Ragas becomes weaker when the release target is a broad agentic system. RAG metrics do not prove that tool permissions, side effects, schema contracts or rollback controls are correct. Its CLI can produce evaluation results, but the official workflow leaves the hard CI policy to the team. Use Ragas as the evaluation specialist inside a wider release contract; reject it as the sole gate when non-RAG system behavior carries most of the risk.

The framework is only one part of the release gate

Whichever tool you select, bind every run to an immutable candidate fingerprint:

{
  "model": "provider/model@version",
  "prompt_sha256": "...",
  "dataset": "golden-set-v12",
  "evaluator": "release-rubric-v4",
  "app_commit": "9f4c2d..."
}

Store per-slice results, not only an average. A critical safety case, an unauthorized tool call or a broken schema can be a hard failure even when the aggregate score improves. Save the failing examples, evaluator version, CI run ID and threshold policy with the verdict.

Finally, keep the controls that evaluation frameworks do not own: canary exposure, production monitoring, side-effect receipts, rollback and the person authorized to accept residual risk. The evidence bundle should answer three questions without reopening the dashboard: what exact candidate was tested, which rule passed or failed, and who owns the release decision.

Choose Promptfoo for a CI-native gate, DeepEval for Python-native regression tests, LangSmith for managed trace evidence, OpenAI Evals for reusable eval specifications, or Ragas for RAG-specific measurement. Then make the blocking rule explicit. A framework can calculate evidence; release engineering decides whether that evidence is sufficient to ship.

Smart Contract Release Audit: 8 Repository Artifacts

Dmytro Nasyrov — Thu, 27 Aug 2026 10:45:19 +0000

An auditable smart contract release is a chain of evidence, not a green CI badge. The repository should bind the build, threat model, invariant tests, privileged roles, deployment rehearsal, audit scope and handover runbook to one commit.

This guide turns that chain into eight repository checks and a 60-minute technical review. Use it to determine whether a delivery process can reproduce and explain one EVM release. A failed build, unmapped privileged authority or audit report without a scope commit blocks the review.

This is a release-audit framework, not a security certification.

A release claim needs a repository artifact

"Security-first" is not evidence. A threat model and a failing test are evidence. So are a CI result and a deployment manifest.

The Solidity security guidance recommends code review and testing. It also discusses audits, small modular contracts and fail-safe design. A release review should therefore inspect how those practices appear in the repository, not merely ask whether the team uses them.

A neutral rule matters here: Pharos Production's company profile should receive no credit for a claim that cannot be mapped to an inspectable artifact either. Apply the same test to any team presenting the release.

Confidential client work does not invalidate this approach. The delivery team can provide a sanitized repository, a public project, an internal reference implementation or a live screen-share with sensitive names removed. You do not need customer source code. You need proof that the release process exists and can be reproduced.

Freeze one release as the review boundary

Select one recent project close to the target chain and risk model. The review package should identify:

the exact commit being demonstrated.
compiler and framework versions, plus the dependency lock.
reproducible build commands and CI entry points.
the threat model with declared system invariants.
every role that can upgrade, pause, mint or administer the system.
deployment scripts plus a non-production rehearsal record.
audit scope, remediation changes and unresolved risks.
the handover runbook with named owners.

Do not request seed phrases, private keys, customer data or proprietary business logic. A good evidence room proves the process without exposing secrets.

An eight-check repository scorecard

Score each check from 0 to 2:

0: absent, verbal only or not reproducible.
1: an artifact exists but is incomplete, stale or not bound to the demonstrated release.
2: a reviewer can reproduce it and trace it to the same commit, configuration and release scope.

Check	Evidence to inspect	`0` looks like	`2` looks like
1. Build provenance	pinned compiler, lockfile, documented commands and CI workflow	"It builds on our lead developer's laptop"	a clean checkout builds with the documented toolchain and no hidden local step
2. Threat model	assets, actors, trust boundaries, external dependencies, abuse cases and invariants	a generic security checklist	named failure scenarios are mapped to controls and tests
3. Test design	unit, fuzz, invariant, integration or fork tests selected for the system's risks	a coverage percentage with no risk mapping	the team can reintroduce a bounded defect and show the relevant test fail
4. Static analysis and review debt	tool configuration, CI output, suppressions, manual-review notes and finding owners	a green badge or tool logo	every material finding is fixed, accepted with rationale or assigned for action
5. Upgrade and admin authority	immutable or proxy decision, role map, current controllers, emergency powers and storage-layout checks	"We use a multisig" with no address or authority map	every privileged action has a controller, change path and observable event
6. Deployment rehearsal	versioned scripts, network parameters, address manifest, verification steps and dry-run receipt	manual console commands reconstructed from memory	the team can replay a non-production deployment from the reviewed commit
7. Audit handoff	scope commit, exclusions, report, remediation pull requests, retest and accepted residual risk	an audit PDF with no repository reference	each finding traces from the reviewed commit to a change and final disposition
8. Operational handover	monitoring, incident roles, pause or containment procedure, key rotation and upgrade runbook	support described only in the proposal	the operator can run the system without an undocumented team-only action

Across the 8 checks, the maximum is 16, but the total is not the decision. A hard stop overrides the score. Adjust the depth of each check to value at risk. Privilege concentration and external dependencies also matter.

A claim earns points only after it is bound to one release, reproduced and assigned a disposition. Any failed hard gate routes to 0 or BLOCK.

Broader procurement questions such as team scope, regulatory fit, multi-chain experience and engagement evidence belong in a separate blockchain development company evaluation framework. This repository review begins after one delivery system and one release boundary have been selected.

Check the test model, not the test count

Line coverage says which lines executed. It does not say that the important property was asserted.

Ask the team to name the properties that must survive arbitrary user actions. Examples include conservation of assets, bounded issuance, withdrawal availability and authorization of privileged state changes. Then locate those properties in the test suite.

Foundry's invariant-testing documentation explains that invariant campaigns run randomized sequences of calls and assert the declared invariants after each call. The repository should also expose what the fuzzer actually explored. Inspect its targets and run depth, then examine reverts and handler behavior. A passing campaign that never reaches a meaningful state is weak evidence.

One demonstration is especially useful: ask the team to make a reversible local change that violates a declared invariant, then run the relevant test. The point is not theater. It verifies that the test can detect the failure it claims to control.

Read static-analysis suppressions as engineering debt

Static analysis is a gate only when its configuration and output are reviewable. Slither can analyze Solidity and Vyper projects, integrate with CI and emit machine-readable results, as described in the project documentation.

Do not award points for running slither . alone. Inspect excluded detectors and path filters. Then review inline suppressions and the treatment of each material finding. A suppression needs a technical reason and a defined scope. It also needs an owner. Otherwise the tool may be green because the repository taught it not to look.

Required status checks can prevent a protected GitHub branch from accepting changes until configured checks pass. That mechanism is documented in GitHub's protected-branch guidance. During due diligence, confirm that the security job is actually required on the release branch and that its expected source cannot be replaced casually.

Make upgrade authority explicit

"Upgradeable" is not a feature checkbox. It changes the trust model.

For proxy-based systems, request the proxy pattern, initializer logic, storage-layout comparison and the current authority that can change implementation code. OpenZeppelin's upgradeable-contract guidance documents initializer and storage-layout constraints. ERC-1967 defines standard implementation, beacon and optional admin storage slots for common proxy designs.

The evidence room should answer four questions without a sales call:

Which address can upgrade, pause, mint, seize, recover or change critical parameters?
What contract, multisig, timelock or governance path controls that address?
Which event or monitor reveals a change?
What happens if the controller is compromised or unavailable?

An immutable contract still needs an authority review. The answer may be "no upgrade path," but ownership, external dependencies and emergency behavior must remain visible.

Bind the audit to the release

An audit report describes a scope at a point in time. Record the repository URL and commit hash beside it. Preserve the compiler settings, excluded components and deployed addresses in the same record. Then trace every finding to a remediation pull request, retest result or explicit risk acceptance.

This avoids a common evidence break: the audit covers one commit while the deployment comes from a later branch with unreviewed changes. The team does not need to claim that an audit guarantees safety. It needs to show which code was reviewed and what changed afterward.

The OWASP Smart Contract Security Verification Standard can help structure the threat model and review scope. Its control groups cover architecture and code as well as governance, authorization and external interactions. Treat it as a control reference, not proof that the repository satisfies those controls.

Run a 60-minute technical review

A bounded live session is more discriminating than another capability deck:

Time	Reviewer asks the delivery team to do	Evidence produced
0-10 minutes	check out the named commit and build it with documented commands	reproducible build or a concrete blocker
10-25 minutes	run one unit path and one risk-based fuzz or invariant test	test output plus an explanation of the property being checked
25-40 minutes	walk through roles, upgrade paths and one compromised-admin scenario	privilege map and containment decision
40-50 minutes	trace one audit finding from scope commit to remediation	commit-bound audit trail
50-60 minutes	replay a non-production deployment and inspect the handover package	deployment receipt, address manifest and runbook

The session is not a coding contest. It tests whether the release evidence remains coherent under inspection.

Hard stops that override the score

Block or pause the release review when any of these remains unresolved:

a clean checkout cannot build without an undocumented machine or manual step.
the audit report cannot be tied to a commit and declared scope.
upgrade, pause, mint or deployment authority cannot be mapped to current controllers.
a material static-analysis or audit finding is suppressed without rationale and ownership.
production deployment depends on unrecorded console actions.
a critical invariant has no test or explicit compensating control.
the operator cannot monitor or contain the system without undocumented team-only knowledge.

A release can score well on six checks and still fail on unknown upgrade authority. Do not let arithmetic hide a control that can replace the code after review.

Adapt the evidence request to the engagement

During early discovery, finished code may not exist. Review the team's repository template and threat-model method instead. Inspect its CI gates and sample handover package separately. Mark future evidence as a delivery requirement.

For non-EVM work, replace Solidity and Foundry with the target ecosystem's compiler and test framework. Substitute its analyzer and authority model for Slither and ERC-1967. The evidence jobs stay the same.

A small immutable utility contract needs less ceremony than a protocol controlling treasury assets. It does not get a free pass on reproducible builds, authority mapping or release provenance. Scale the depth, not the existence, of the controls.

Send this request before the sales call:

Please prepare a 60-minute technical review tied to one recent, representative release. First, show a sanitized repository or screen-share with a clean build, risk-mapped tests and static-analysis disposition. Then show the privileged-role map, non-production deployment rehearsal, commit-bound audit handoff and operational runbook. Do not share private keys or customer data. Omit confidential business logic as well.

The response shows whether another operator can reproduce the release without undocumented help.

LLM Evaluation Scores Are Not Release Gates

Dmytro Nasyrov — Thu, 27 Aug 2026 07:35:49 +0000

Your new prompt scores 94% on the golden dataset. The current version scores 91%. That result supports a change, but it does not authorize a production release.

An LLM evaluation score answers a bounded question about a dataset, a grader and a run configuration. A release gate answers a wider question: can this exact system version enter production without an unacceptable regression, uncontrolled side effect, policy violation or unrecoverable failure? A practical release contract uses five independent gates. Model quality is only one of them.

Scores do not carry release authority.

An eval score is a measurement with a scope

An LLM evaluation is useful because generative AI is variable. OpenAI's evaluation guidance describes evals as structured tests for measuring model performance and distinguishes application-specific tests from industry benchmarks and standard numerical scores.

The scope is carried by the test dataset, evaluation criteria and grader. Change the golden dataset, ground-truth labels, evaluation rubric, model version, prompt version, scorer version or dataset version and the meaning of the number changes. A useful LLM evaluation framework keeps those bindings beside every run. A 94% pass rate without them is not reproducible release evidence.

That distinction matters in production AI systems engineering because the deployed product includes model behavior, tools, retrieval, data policies and operators. A model can improve on an aggregate score while the application becomes less safe or more expensive to run.

Why one LLM evaluation metric hides blockers

An aggregate score compresses failures that do not have equal consequences. Ten awkward answers and one unauthorized refund may produce the same pass rate as eleven awkward answers. A confidence interval can describe uncertainty in the estimate, but it cannot repair a test set that omitted the dangerous behavior. The release decision should not treat those outcomes as equivalent.

Four blind spots appear repeatedly:

Slice loss: the overall LLM evaluation score rises while a high-risk language, tool or customer segment regresses.
Dataset drift: the benchmark dataset no longer represents current production traffic.
Judge drift: an LLM-as-a-judge changes behavior after a model update, or a revised rubric changes what counts as correct.
System omission: the eval grades the final text but never checks whether the agent called the wrong tool, leaked data or created an irreversible side effect.

LLM regression testing must therefore preserve the full LLM evaluation pipeline: fixtures, expected outputs, grader configuration and failure taxonomy. It should also report slice-level metrics beside the aggregate. The question is not merely "Did the average improve?" It is "Which behaviors moved, which failures became possible and which risks cross the release threshold?"

The five-gate LLM release contract

Treat the release decision as a conjunction. Every required gate must pass for the same immutable candidate.

Gate	Required evidence	Blocks release when	Owner
1. Quality regression	versioned test dataset, baseline run, candidate run, slice-level metrics and reviewed failures	a protected slice crosses its regression threshold or a critical case fails	product or domain owner
2. Deterministic system behavior	schema tests, tool-call assertions, permission checks, idempotency tests and side-effect receipts	the system can call an unauthorized tool, repeat a non-idempotent action or violate an output contract	application engineering
3. Safety and policy	safety evaluation, LLM guardrails, prompt-injection tests, data-leakage checks and red-team findings	a zero-tolerance policy case fails or residual risk exceeds the declared tolerance	security or risk owner
4. Operational recovery	canary plan, production monitoring, rollback plan, pinned prior version and incident runbook	the team cannot detect harm, stop traffic or restore a known version inside the required window	platform or SRE owner
5. Decision authority	signed release criteria, evidence manifest, known exceptions, expiry and named approver	evidence is stale, mismatched to the candidate or no accountable owner accepts the residual risk	release owner

This is stricter than a single regression gate, but it is not a demand for perfect model behavior. Each team still chooses thresholds based on use case and risk tolerance. The contract only prevents a quality metric from silently carrying authority it was never designed to hold.

The candidate fingerprint binds every gate to the same model, prompt, dataset and scorer. One failure routes the candidate to BLOCK.

Think of the table as an LLM production readiness checklist with explicit failure authority. It is broader than an LLM eval framework because it binds quality evidence to software checks, AI governance, recovery and an approval owner. The resulting audit trail explains not only what scored well, but why this candidate was allowed to ship.

Gate 1: LLM regression testing protects behavior

The first gate compares the candidate with the current production baseline on representative cases. OpenAI's eval workflow binds test inputs to explicit testing criteria and, where appropriate, human-labeled ground truth. That is the right foundation for model regression and prompt regression checks after a model upgrade, prompt edit or retrieval change.

The gate should keep critical failures separate from averages. A medical contraindication, financial action or access-control decision may be a zero-tolerance case even when ordinary response quality uses a statistical threshold. Human evaluation remains necessary for disputed semantic cases, especially when an LLM-as-a-judge sees the same limited channel as the system it grades.

Gate 2: check evidence outside the model answer

Text quality does not prove that the application behaved correctly. An agent can write a persuasive summary after calling six tools instead of two. It can produce valid JSON after querying data it should not access. It can report success after a write failed.

Deterministic checks should inspect what code and infrastructure can prove: tool allowlists, argument schemas, authorization decisions, transaction identifiers, idempotency keys, file hashes and command exit status. These checks do not replace semantic evaluation. They cover a different failure channel.

That wider contract is where MLOps release and monitoring design enters the decision: the team must be able to deploy gradually, observe impact, stop traffic, restore a known version and assign a human owner.

Gate 3: LLM guardrails are release criteria

Safety evaluation should be risk-specific. Prompt injection, data leakage, harmful instructions, policy evasion and excessive tool permissions need separate policy checks because an average helpfulness score can hide every one of them.

Red teaming finds candidate failures. The release gate turns selected findings into repeatable abuse testing and declares which classes are zero tolerance. LLM guardrails then enforce runtime constraints, but their presence alone is not evidence that they work. The gate must test the guardrail response and the failure path around it.

Gate 4: production readiness includes rollback

Offline evaluation happens before exposure. Online evaluation and LLM production monitoring observe behavior under real traffic. Google Cloud's operations guidance lists continuous evaluation as one approach to monitoring generative AI applications.

Neither makes rollback optional. A canary deployment needs a traffic boundary, stop condition and known-good target. LLM observability should bind traces to the model, prompt, retrieval index, tool set and policy version that produced them. Otherwise model drift and prompt regressions arrive as anonymous dashboard changes.

The gate fails when alerts have no response action, when the prior version cannot be restored or when an external provider update makes rollback impossible and no containment plan exists. Production monitoring, incident response and rollback must refer to the same candidate identity.

Gate 5: a person owns the decision

NIST's AI Risk Management Framework Core separates Govern, Map, Measure and Manage. Measurement informs the decision to proceed, while management covers response, recovery, monitoring and assigned responsibilities. That separation is a useful model for an LLM release process.

The approver is not signing a score. The approver accepts a release contract with defined evidence, thresholds, exceptions and rollback conditions. Approval should expire when any bound component changes.

{
  "candidate": {
    "model": "provider/model@version",
    "prompt": "sha256:...",
    "dataset": "eval-set-v17",
    "scorer": "rubric-v6"
  },
  "gates": {
    "quality_regression": "PASS",
    "deterministic_side_effects": "PASS",
    "safety_policy": "PASS",
    "rollback_readiness": "PASS",
    "decision_authority": "PASS"
  },
  "owner": "release-owner@example.com",
  "expires_at": "2026-08-28T12:00:00Z"
}

The record ties every verdict to one candidate. If the prompt, model, dataset, scorer, tool permissions or retrieval index changes, the release evidence no longer matches.

What should block an LLM release?

Use a hard block when at least one of these conditions is true:

a critical scenario fails, even if the aggregate LLM evaluation metric improves.
a protected slice crosses its regression threshold.
a tool call, permission or side effect cannot be verified deterministically.
a safety or policy control fails its declared test.
monitoring cannot identify the deployed candidate.
rollback or containment cannot meet the required response window.
the evidence manifest is stale or no release owner accepts the remaining risk.

The practical rule is compact: evals measure behavior, gates authorize change. Keep the score inside the first gate and require the other four to speak before production traffic moves.

One Payment, Three Ledgers: Where Reconciliation Actually Breaks

Dmytro Nasyrov — Thu, 20 Aug 2026 19:55:31 +0000

A processor says a payment settled. Your product ledger shows the customer balance and fee entries. The bank account still has no matching credit. None of those records is necessarily wrong, yet the payment is not reconciled.

The mistake is asking which system has the true status. Each system owns a different fact. Reconciliation belongs to the process that can compare those facts, preserve the mismatch and prove what happened at every boundary. A useful design sets invariants across the product ledger, the processor settlement record and the bank statement.

One payment creates three different facts

Your product ledger records the economic event the application accepted: a capture, refund, fee, reserve movement, chargeback or correction. It should preserve the customer-facing and accounting consequences even when an external provider changes later.

A processor owns another view. Its settlement data determines which transactions and adjustments entered a payout. Stripe, for instance, exposes immutable balance transactions and lets automatic payouts retain their association with the transactions they contain. Adyen's settlement details report includes settled payments, fees, corrections, payouts and a payout reference that can appear on the bank statement.

Only the bank statement proves the cash movement. It cannot explain the composition of a processor batch, and the processor cannot prove that a credit reached the bank merely by marking a payment as settled.

This separation is part of payment systems engineering because the integration call that moves money and the evidence that closes the books have different owners. Treating a provider response as all three facts makes the happy path simple and every exception ambiguous.

I use "ledger" here as shorthand for a durable reconciliation surface. A processor report or bank statement may not be a double-entry ledger. The design still needs to preserve their facts as immutable evidence rather than flattening them into one mutable payment status.

The invariant map

Start with a currency and a reconciliation window. Then define the equations that must hold after accounting for cutoffs, fees, reserves, refunds, chargebacks, corrections and foreign-exchange effects.

Boundary	Invariant	Evidence that closes it	What a mismatch means
Product ledger to processor	Expected provider payable from internal entries equals the processor's net settlement components	internal journal entries plus transaction-level processor report	missing event, duplicate event, amount mapping error, fee rule drift, or timing classification error
Processor batch to bank	Processor payout amount and currency equal the bank credit after declared bank or FX adjustments	payout record, batch reference, value date, and bank statement line	payout still in transit, wrong destination, bank fee, FX difference, rejection, or missing cash movement
Period continuity	Opening unsettled balance plus new net activity minus payouts equals closing unsettled balance	consecutive settlement periods and retained adjustments	orphaned balance, late event, duplicated payout application, or a broken cutoff rule
Reference completeness	Every bank credit maps to one processor payout and every completed payout maps to one bank credit	stable payout reference and a controlled fallback match	unlinked cash, one-to-many aggregation, reused reference, or incomplete ingestion

The point is not to force all systems to agree at every moment. They operate on different clocks. The point is to know which differences are expected, which deadline makes them actionable, and what evidence will eventually close them.

A modeled break: settled, posted, but not received

Assume the processor marks a card payment as settled. The product ledger has already posted the customer balance and expected fee. The safeguarding or operating bank account has no corresponding credit.

Do not reverse the customer entry just because the bank credit is absent. "Payment settled" and "payout reached the bank" describe different boundaries. First ask whether the payment was included in a closed settlement batch. If it was not, the first invariant may still be open because of the provider's cutoff, reserve policy or status mapping. If it was included, locate the payout reference and compare the batch net amount with the expected bank credit.

The absence becomes a bank-boundary exception only after the payout has a provider-side completion fact and the agreed arrival window has expired. Until then, the discrepancy needs a state such as awaiting_bank_evidence, not a generic failed flag.

That distinction changes the recovery action. A ledger mapping defect can require a replay from immutable source events. A missing bank credit needs provider or bank investigation. Re-running the original payment cannot repair either one and may create a second economic event.

Build a discrepancy record, not an alert string

A reconciliation job should emit a durable object that another worker or analyst can resolve without reconstructing the original comparison from logs.

{
  "discrepancy_id": "rec_01J...",
  "invariant": "processor_payout_equals_bank_credit",
  "currency": "EUR",
  "window": "2026-08-20",
  "internal_evidence": ["journal_batch_..."],
  "processor_evidence": ["payout_...", "settlement_batch_..."],
  "bank_evidence": [],
  "status": "awaiting_bank_evidence",
  "owner": "treasury_operations",
  "next_check_at": "..."
}

The record names the failed invariant, the evidence already present, the evidence still missing, the current owner and the next permitted action. Its identity should remain stable across retries. Otherwise every polling cycle creates a new exception and hides the age of the original break.

This is where reconciliation design for payment platforms becomes an architectural concern rather than a reporting feature: the discrepancy must survive provider outages, late files, replayed webhooks and human investigation without losing its lineage.

Detect the break at the boundary that owns it

One reconciliation worker can coordinate the process, but it should not become an unquestioned source of truth. Authority remains fact-specific.

Invariant	Required evidence	Owner	Detection time	Recovery action
Internal entries equal processor components	immutable internal journal and transaction-level settlement rows	payments accounting	after the processor report closes for the window	replay ingestion, repair mapping, or open a controlled accounting adjustment
Processor payout equals bank credit	payout ID, batch reference, expected arrival window, and statement line	treasury operations	when the arrival window expires	query payout status, trace the transfer, then escalate to provider or bank
Opening plus activity minus payouts equals closing	consecutive balances and every intervening adjustment	financial control	at period close	freeze close, locate orphaned movement, and post only an approved correction
Every external line has one controlled match	provider and bank references plus fallback match rationale	reconciliation service	during ingestion and rematching	quarantine ambiguity for review instead of choosing the nearest amount

The detection time matters as much as the equation. Flagging a payout before its contractual arrival window produces noise. Waiting until month-end to detect a duplicated settlement row turns a local ingestion bug into an accounting close problem.

Matching rules also need an order. Prefer stable provider and bank references. Use amount, currency, value date, merchant account and batch metadata only as controlled secondary evidence. A fuzzy match may propose a candidate, but it should not silently convert ambiguous cash into a reconciled state.

The system of record depends on the question

Teams often try to settle the argument by declaring one database the system of record. That is useful for one class of fact and dangerous when applied to all of them.

For customer obligations and internal accounting, the product ledger can be authoritative. A processor remains authoritative for the composition and status of its settlement batch. Cash received belongs to the bank's evidence. The reconciliation record proves that those sources were compared under a named rule and either matched or produced a controlled exception.

This model also makes observability concrete. The useful metrics are not just payment success rate. Track the age of open discrepancies, unmatched value by currency, breaks by invariant, time to first owner, repeated ingestion events and corrections posted without complete evidence.

When the three-ledger model changes

Some architectures collapse or add boundaries.

If the processor is also the contractual ledger of record, the product may not maintain a separate customer-money ledger. The application still needs evidence that processor balances reconcile with bank cash, but it should not invent a duplicate accounting authority.

Prefunded settlement changes the direction of the cash invariant. You reconcile funding movements and consumption of the prefunded balance rather than waiting for a payout after each sales batch. Split settlements, marketplace subaccounts, reserves and multi-currency conversion can add more than three reconciliation surfaces.

The decision rule stays the same: model one invariant for every boundary where economic meaning, provider settlement or cash possession can diverge. Assign an evidence source, deadline, owner and recovery action before calling the payment reconciled.

Which system in your stack can prove a payment is settled when the processor and bank disagree?

MCP Went Stateless. Your Recovery Logic Didn't.

Dmytro Nasyrov — Thu, 13 Aug 2026 11:35:31 +0000

The 2026-07-28 MCP revision removed the initialization handshake and protocol-level sessions. That makes a remote server easier to route and scale, but it does not make tool execution stateless. If a call can charge a card, create a ticket, send a message, or mutate a repository, retries still need durable facts. The practical conclusion is simple: move state out of the transport, then make execution, side effects, results, and delivery explicit. Otherwise a stateless server will recover cleanly at the HTTP layer while duplicating work in the system that actually matters.

Stateless transport removed one state owner, not state

The new MCP version retires initialize/initialized and the Mcp-Session-Id header. Each request carries what the server needs to interpret it, and any request can land on any compatible instance behind a normal load balancer. The official 2026-07-28 specification release describes the result as a stateless protocol core.

That is an important infrastructure simplification. It removes sticky routing and protocol-owned session storage from the default path. It does not answer these application questions:

Did this logical operation already start?
Did an external side effect happen?
Was the result persisted?
Did the caller receive that result?
Who is allowed to retry each step?

The discipline of production AI infrastructure engineering begins by naming those owners before choosing queues, databases, or worker frameworks. A session used to hide some of those decisions by giving related calls a convenient container. Once the container disappears, an implementation either models the decisions directly or leaves them implicit in process memory.

Implicit state is the dangerous option. Process memory can tell the current worker what it has seen, but it cannot tell the next worker what already happened after a crash, timeout, reschedule, or deployment.

The four crash windows that matter

Consider a tool called issue_refund. The client sends one request, but the business operation crosses several boundaries:

The server accepts the intent.
A worker starts the operation.
A payment provider applies the refund.
The server records the provider result.
The result reaches the client.

“The request failed” is not one state. It can mean the process died before work began, during local work, after the provider committed the refund, or after the result was stored but before the response arrived. Those states require different recovery behavior.

If the server blindly replays the whole handler after every timeout, the third window can issue the refund twice. If it never retries once a worker marked the operation started, the first or second window can strand legitimate work forever. If it treats a stored result as proof that the caller received it, the fifth step can silently lose delivery.

The protocol cannot resolve this ambiguity because the ambiguity belongs to the application. Stateless MCP makes that boundary visible; it does not remove it.

A recovery-owner matrix

Before writing retry code, define one logical operation_id and map every crash window to durable evidence and one owner:

Crash window	Durable fact required	Safe recovery action	Recovery owner
Before execution starts	operation ID, input hash, accepted status	claim the operation and execute once	task service or queue
During local execution, before any external effect	attempt ID and lease expiry	let the lease expire, then run a new attempt	worker coordinator
After an external effect, before its result is recorded	stable idempotency key and side-effect intent	query or replay through the same idempotency key; never issue a fresh command	external adapter
After the result is stored, before the client receives it	immutable result plus delivery status	return the stored result and retry delivery separately	API or delivery worker

The table separates three ideas that are often collapsed into status = done:

Execution state says whether the logical operation produced a result.
Side-effect state says what changed outside the worker's database.
Delivery state says whether the result crossed the final boundary to its consumer.

A workable stateless MCP server architecture therefore needs a task record whose lifetime is independent of any one request or server instance. It can be compact, but it must survive the exact failures that are safe to retry.

This also clarifies what an explicit MCP state handle is for. SEP-2567 replaces implicit session-scoped application state with ordinary server-minted handles that a model can carry between calls. A handle identifies a basket, browser, workflow, or operation. It is not evidence that an effect did or did not happen. The resource behind the handle still needs its own concurrency and recovery contract.

Make the operation record the source of truth

A minimal operation record might look like this:

{
  "operation_id": "refund_01J...",
  "input_hash": "sha256:...",
  "status": "effect_pending",
  "attempt": 2,
  "effect_key": "refund_01J...",
  "effect_receipt": null,
  "result": null,
  "delivery_status": "not_ready"
}

The fields are less important than the invariants around them:

Reusing an operation_id with different inputs must fail closed.
Only one live attempt may own the current lease.
Every non-idempotent adapter receives a stable effect key.
A provider receipt is persisted before execution is marked complete.
Delivery can be retried without rerunning execution.

This design changes a retry from “call the handler again” into “advance the known operation from its last durable state.” That is the difference between resilience and duplicate execution.

It also gives observability a useful unit. Logs from server instance A and worker instance B are no longer unrelated request traces; they are attempts attached to one operation. Alerts can distinguish a stuck lease, an uncertain side effect, a completed result awaiting delivery, and a client that simply stopped polling.

Tasks help with lifecycle, not business idempotency

The 2026-07-28 release also moves long-running work into the Tasks extension. A server can return a task handle, and the client can poll or update that task. That is a better fit for work that outlives one request, but a task handle alone does not make a payment, email, deployment, or repository mutation idempotent.

The task service can own lifecycle transitions such as accepted, working, input required, completed, failed, or cancelled. The adapter that talks to an external system must still own the effect key and reconciliation behavior. The delivery layer must still own the distinction between “result exists” and “consumer received it.”

This division prevents a common architectural mistake: asking one status field to describe three independent systems. A queue can truthfully say that a job completed while the email provider timed out after accepting the message. An MCP task can truthfully expose a result while the client disconnects before seeing it. Both statements can be true at the same time.

When this machinery is unnecessary

Not every tool needs an operation ledger. A read-only search can usually be retried from scratch. A deterministic transformation with no external effects may only need request-level timeout handling. A handler whose downstream system provides strong idempotency and queryable receipts can delegate much of the effect recovery to that system.

There is also a cost to durable recovery: more states, retention rules, reconciliation paths, access control, and tests. Explicit handles can appear in chat logs or subagent context, so authenticated servers should bind a handle to the request's authorization context. For unauthenticated servers, SEP-2567 recommends treating a handle as a short-lived capability token with high entropy.

The goal is not to make every tool a workflow engine. It is to match the recovery model to the irreversible boundary.

The decision rule

Use one test: if retrying the operation can make the outside world different twice, persist the operation and a stable idempotency key before the first side effect. If execution and delivery can fail independently, persist them independently. If neither condition is true, keep the design smaller.

MCP's stateless core is a real improvement because it stops transport sessions from pretending to be application state. The migration is complete only when every durable fact has an explicit home and every retry has one owner.

Which failure does your implementation still treat as session-owned after the stateless change?

LLM Observability Cost Starts With a Trace Budget

Dmytro Nasyrov — Sun, 09 Aug 2026 17:01:44 +0000

LLM observability cost is not a single price per trace. It is the sum of several workload multipliers: requests become traces, traces contain spans, selected spans become evaluation jobs, judges consume tokens, payloads occupy storage, and retention keeps those bytes billable over time.

The practical fix is to build a trace budget before comparing vendor plans. Start with six variables you can measure in your own application: monthly traces, spans per trace, bytes per span, evaluation sampling rate, judge tokens per score, and retention days. A vendor tier is affordable only after those units have been converted into your workload.

TL;DR: Forecast events, bytes, scores, judge tokens, and retained data separately. Then apply each vendor's meter to the same trace budget. Never compare tier names as if they represented the same physical quantity.

This article is for engineers and technical leads operating production LLM features. It provides a worked example, a copyable budget receipt, and guardrails that prevent silent cost growth.

Why one trace is not one billing unit

A trace represents one end-to-end interaction, but it can contain a changing number of observations. A simple retrieval request might create spans for orchestration, embedding, vector search, reranking, generation, and a tool call. Add retries or parallel tools and the same user request can create more events without increasing request count.

That is the first trap in AI observability pricing: two systems with 100,000 monthly traces can produce very different ingestion volumes. One may average three spans per trace; the other may average 18. If a platform meters observations, processed bytes, or both, “trace count” is only the first input.

The budget therefore belongs inside a production AI engineering workflow, where feature ownership, instrumentation, evaluation, and spend share the same boundary. Otherwise the application team controls trace shape while a separate platform team receives the bill after the multiplier has already changed.

The four cost layers inside LLM observability

Treat the bill as four independent layers. A platform may bundle some of them, but your model should keep them separate.

1. Ingestion

Ingestion is driven by events or bytes. Its useful variables are monthly traces, observations per trace, and serialized bytes per observation. Recording complete prompts, tool arguments, retrieved documents, and outputs can make byte volume grow much faster than the event count.

2. Retention

Retention is a volume-time problem. Storing 1 GB for 90 days consumes roughly three times the steady-state capacity of storing the same monthly inflow for 30 days. Indexes, replicas, compression, and vendor-specific accounting can change the invoice, but they do not remove the underlying time multiplier.

3. Evaluation

Evaluation adds scores. A trace can receive zero, one, or several scores from deterministic checks, human review, or LLM-as-a-judge workflows. The Braintrust pricing model, for example, exposes processed data, scores, and retention as distinct meters. That separation is a useful reminder that capturing a trace and judging it are different operations.

4. Judge-model inference

An LLM judge creates another model call. Its input may include the original prompt, response, retrieved context, rubric, and reference answer. The observability platform's score meter and the model provider's token charge are therefore two different cost lines for the same evaluation job.

Build the LLM observability cost equation

Use a vendor-neutral equation first:

monthly_observability_cost =
    ingestion_cost(events, bytes)
  + retention_cost(bytes, days)
  + evaluation_platform_cost(scores)
  + judge_model_cost(input_tokens, output_tokens)

The variables become easier to audit when they are arranged as an LLM observability cost model by trace volume, because every conversion assumption sits beside the meter it converts. For each feature, record these six inputs:

Variable	How to measure it
Monthly traces	Count root interactions by feature and environment
Spans per trace	Use p50 and p95, not only an average
Bytes per span	Measure serialized payload after redaction
Evaluation rate	Divide scored production traces by eligible traces
Judge tokens	Capture input and output tokens per evaluation
Retention days	Record the policy for raw, redacted, and aggregated data

Do not replace unknown values with zero. Mark them unknown, add an instrumentation task, and calculate a range. An explicit range is more useful than a precise total built on missing multipliers.

A worked budget for 100,000 monthly traces

Consider an application with these stated planning assumptions:

100,000 traces per month
6 spans per trace on average
2 KB per span after redaction and serialization
10% evaluation sampling, with one score per selected trace
1,200 judge input tokens and 120 judge output tokens per score
30 days of raw-trace retention

The conversions are straightforward:

observations = 100,000 × 6 = 600,000
raw_ingestion = 600,000 × 2 KB = 1.2 GB/month
scores = 100,000 × 10% = 10,000/month
judge_input = 10,000 × 1,200 = 12,000,000 tokens/month
judge_output = 10,000 × 120 = 1,200,000 tokens/month

At a steady ingestion rate, increasing raw retention from 30 to 90 days raises the retained raw-data estimate from about 1.2 GB to 3.6 GB before vendor-specific compression, indexing, and replication. Increasing evaluation coverage from 10% to 25% raises the monthly score count from 10,000 to 25,000 and multiplies both judge-token lines by 2.5.

These are not market prices. They are workload units. Apply current vendor rates only after this conversion, and keep the rates in a separate configuration so a pricing change does not rewrite the workload model.

Instrument the inputs, not just the total

A dashboard that shows only total dollars cannot explain a change. Record the dimensions that caused it: feature, environment, model, trace name, observation type, score name, and retention class.

Langfuse's trace-structure guidance describes observations grouped into traces and recommends carrying model, usage, and cost details on generation observations. Stable names matter because a renamed generation can break a cost series even when the application still works.

Token coverage also needs its own reliability metric. The OpenTelemetry GenAI attribute registry says instrumentation should make a best effort to populate input-token usage. In practice, that means tracking the percentage of generation spans with usable token counts. A total derived from 62% coverage should not be presented as the full bill.

Four guardrails that keep the trace budget stable

Set an event budget per feature. Alert when p95 spans per trace crosses the expected boundary. This catches retry loops and accidental nested instrumentation before the monthly invoice does.
Sample evaluations by decision value. Score all release candidates and high-risk paths, then sample routine production traffic. A flat 100% judge rate is rarely the only useful policy.
Tier retention. Keep raw payloads only as long as debugging or audit needs require; retain redacted summaries and aggregate metrics longer when they carry the necessary signal.
Fail visibly on missing usage. Report token-coverage and byte-coverage percentages beside every cost estimate. Missing telemetry is uncertainty, not free usage.

Review those guardrails whenever a prompt, tool graph, model, evaluator, or retention policy changes. Each can change the bill without changing request volume.

A copyable trace-budget receipt

Keep the assumptions next to the result:

period: 2026-08
feature: support-agent
monthly_traces: 100000
spans_per_trace:
  p50: 5
  p95: 9
bytes_per_span: 2048
evaluation_sample_rate: 0.10
scores_per_sampled_trace: 1
judge_tokens_per_score:
  input: 1200
  output: 120
retention_days:
  raw: 30
  aggregate: 365
coverage:
  token_usage: 0.98
  serialized_bytes: 1.00

Version this receipt with the instrumentation configuration. When the estimate changes, the diff should reveal whether traffic, trace shape, evaluation policy, token usage, or retention caused it.

Frequently asked questions

What is the biggest driver of LLM observability cost?

There is no universal biggest driver. Event-heavy agent workflows amplify span counts, verbose payloads amplify processed bytes, broad judge coverage amplifies scores and model tokens, and long retention amplifies storage. Measure all four layers for your workload before optimizing one of them.

Should every production trace be evaluated?

Usually not by an expensive LLM judge. Evaluate high-risk paths and release candidates deliberately, then sample routine traffic at a rate that can detect meaningful regressions. Deterministic checks and user feedback can cover additional traces without paying judge-model inference for every request.

How often should a trace budget be recalculated?

Recalculate monthly and whenever a model, prompt, tool graph, evaluator, sampling rule, payload policy, or retention period changes. Those changes can alter observations, bytes, scores, or tokens even when user traffic remains flat, so request-count monitoring alone will miss the shift.

Start with units, then compare plans

An LLM observability cost estimate becomes defensible when every price is attached to a measured unit and every unknown remains visible. Count traces, expand them into observations and bytes, apply evaluation coverage, calculate judge tokens, and extend storage across the real retention window.

Then compare vendors against the same receipt. The cheapest tier name is not the answer. The answer is the plan that fits your measured trace budget without hiding the multiplier most likely to grow next.

AI Coding Agents Need a Handoff Contract, Not More Prompts

Dmytro Nasyrov — Sun, 09 Aug 2026 06:47:56 +0000

An AI coding agent is not done when it says “done.” It is done when the next person or agent can accept its work without reconstructing what changed, what was assumed, what was tested, and what remains uncertain.

That distinction calls for a handoff contract: a small, structured artifact that travels with every patch. It records scope, assumptions, evidence, integration state, residual risk, and the owner of the next acceptance decision. Better prompts can improve code generation. They cannot replace this transfer of responsibility.

Prompts solve generation, not acceptance

Teams often react to a weak agent result by adding another instruction: run tests, check edge cases, avoid unrelated files, explain the diff. These instructions can help. But they still describe what the sender should do; they do not define what the receiver must be able to verify.

That is why a handoff belongs inside a production software delivery workflow, where code, evidence, integration state, and acceptance travel as one object. A patch without that context may be locally correct and still be expensive to integrate.

DORA's analysis of 1,110 open-ended responses from Google software engineers found that time saved during initial creation was frequently reallocated to auditing, verification, and iteration. The study is qualitative evidence from a specific population, not a universal productivity benchmark. Still, it names a pattern many teams recognize: faster output can move work downstream instead of removing it.

The same DORA analysis reports a tension between higher AI adoption, greater throughput, and greater delivery instability. Its broader software-delivery metrics framework deliberately measures both throughput and instability. “The agent produced more code” therefore tells us very little about whether the system can safely absorb that code.

A useful definition of a valid handoff

A 2026 survey of agentic electronic design automation defines a valid handoff in terms of the recipient: the transferred artifact must satisfy the next stage's acceptance conditions and carry enough context, evidence, and provenance for that stage to proceed. The paper is about EDA, not general application development, but its handoff perspective gives software teams a useful design rule:

Validate the transfer from the receiver's side, not only the work from the sender's side.

That rule changes the completion question from “Did the agent follow the prompt?” to “Can the next consumer make an informed acceptance decision?”

The five parts of an agent handoff contract

1. Scope

Record the exact requirement, owned files, and forbidden files. “Implement authentication” is not a scope. “Add refresh-token rotation in these three modules without changing the public session schema” is.

Scope prevents two opposite failures: incomplete work hidden behind a broad completion claim, and unrequested changes that make review harder.

2. Assumptions

List the conditions the patch relies on: base commit, interface versions, environment, fixtures, feature flags, data shape, or ordering guarantees. An assumption is not an embarrassment; an invisible assumption is a defect waiting for integration.

3. Evidence

Include the exact checks that ran and the results that matter. “Tests pass” is too lossy. A useful receipt names the command, exit status, relevant case count, and any validation that was intentionally not run.

Evidence should be reproducible and proportional. A one-line configuration change rarely needs a full-system stress run. A persistence or concurrency change needs more than a type checker.

4. Integration state

Say whether the patch was tested alone, applied to the intended base, combined with adjacent work, and validated through the shared seam it changes. Local green and integrated green are different states.

This matters even more in production AI evaluation and monitoring systems, because prompt versions, evaluation results, drift signals, and rollback state must survive every model or workflow change. A locally successful prompt or evaluator is not yet an accepted production change.

5. Residual risk and acceptance owner

Name what remains unknown, the next consumer, and the decision that consumer owns. “No known risks” is credible only when the evidence supports it. Otherwise, state the boundary plainly: migration behavior was not exercised, browser compatibility remains unverified, or the adjacent branch has not landed.

The acceptance owner might be a human reviewer, an integration agent, a release gate, or an on-call engineer. The important part is that ownership is explicit.

The five fields that turn a completion message into an acceptance-ready handoff.

A copyable handoff format

The contract does not need a new platform. Start with a versioned YAML block in the pull request, task artifact, or agent output:

handoff_version: 1
work_item: AUTH-217
producer: coding-agent-session-42
base_commit: 8d31c2a

scope:
  requirement: Rotate refresh tokens after every successful use
  owned_paths:
    - src/auth/refresh.ts
    - tests/auth/refresh.test.ts
  forbidden_paths:
    - src/auth/session-schema.ts

assumptions:
  - The token store provides compare-and-swap semantics
  - Existing session payload fields remain stable

evidence:
  - command: npm test -- tests/auth/refresh.test.ts
    exit_status: 0
    result: 12 passed
  - command: npm run typecheck
    exit_status: 0

integration_state:
  applied_to_base: true
  adjacent_changes_combined: false
  shared_seam_verified: false

residual_risk:
  - Concurrent refresh from two browser tabs was not exercised

acceptance:
  owner: integration-agent
  next_gate: Combine the session-store patch and run the auth integration suite

This format makes absence visible. If shared_seam_verified is false, the workflow does not have to guess whether anyone checked it. If a command is missing, the receiver can request the smallest useful check instead of rerunning everything.

“Agent done” and “handoff valid” are different states

Imagine two agents working in parallel. One changes a serializer. The other adds validation to its consumer. Both run focused tests, both report success, and both are truthful. When combined, the serializer emits a value that the new validator rejects.

Neither local report captures the failure because the failure exists in the transfer between the two pieces of work. A handoff contract exposes the missing state:

Sender report	Receiver question
“My tests pass.”	Were adjacent patches combined?
“The interface is unchanged.”	Which interface version and invariants were checked?
“No conflicts.”	Was only textual conflict checked, or behavioral compatibility too?
“Ready to merge.”	Who owns acceptance, and which gate makes it true?

The contract does not magically prevent the incompatibility. It prevents the workflow from mislabeling unintegrated work as accepted work.

Validate the contract from the next consumer's viewpoint

A useful receiver loop has four steps:

Check completeness. Are all five parts present, or is an omission explicitly marked?
Check claims against artifacts. Do the diff, base commit, and test receipts support the summary?
Run the cheapest discriminating gate. Test the shared seam before rerunning an entire repository.
Accept, return, or escalate. Record the decision and its reason instead of silently repairing the patch downstream.

That last step matters. If integrators repeatedly fix incomplete handoffs without returning the signal, the producing agent never receives a useful boundary and the organization hides the true cost of the workflow.

Start small

Do not begin by building a handoff service. Add the YAML block to one workflow where parallel agents touch a shared interface. Require only fields that change an acceptance decision. Track three outcomes for a few real tasks:

how often the receiver had to reconstruct missing context;
how often local green became integrated red;
how long it took to reach an acceptance decision.

Then revise the contract when a missing field causes a real ambiguity. A handoff contract should become smaller and sharper over time, not grow into ceremonial paperwork.

Frequently asked questions

Is this just a pull request template?

It can live in a pull request, but it serves a broader role. A PR template is usually organized around human review. A handoff contract is a machine-readable transfer between any producer and the next acceptance stage, including agent-to-agent workflows before a PR exists.

Should the agent decide whether its own handoff is valid?

The agent can validate completeness and attach evidence. Final validity belongs to the receiver because only the receiver owns the next stage's acceptance conditions.

Does every task need all five parts?

Yes, but a field can be short. For a tiny documentation edit, integration state may be one sentence and residual risk may be “none identified.” Keeping the schema stable is useful; scaling the evidence to the risk keeps it practical.

The real unit of progress

Code generation is an activity. Accepted change is progress.

Once teams measure the transfer between those states, the next improvement is often not a longer prompt. It is a clearer contract at the boundary: what moved, what it depends on, what proves it, what remains unintegrated, and who can accept it next.

The AI Agents Were Done. The Delivery Clock Wasn’t.

Dmytro Nasyrov — Sat, 08 Aug 2026 06:31:14 +0000

The metric that stops at generated code hides review, rework, and acceptance — the work that decides whether the change can ship.

Cover illustration: the metric stops at generated code while delivery continues through review, rework, validation, and acceptance.

Both AI agents had finished. Their assigned checks passed. Then their patches were combined, and eight of the 24 acceptance methods turned red.

On sequence 21 of a paused coding study, the integrator needed 19.516 seconds to reconstruct shared branches before all 24 methods passed. If the timer had stopped when the agents reported done, the run would have looked successful. The delivery clock told a different story.

AI coding productivity has a finish-line problem. Generated code, a locally green test, and an opened pull request are useful events. None of them proves that a change survived the handoff into the system that must accept it.

AI does not become productive when output appears. It becomes productive when accepted work moves through the system without exporting more verification than it saves.

AI coding productivity and the seductive finish line

AI makes activity visible almost immediately. A terminal fills. A diff appears. A worker says the task is complete. Those signals feel like speed because the old blank space has disappeared.

The study used a production software delivery boundary: the timer stopped only after the integrated tree passed all 24 frozen methods and the evidence receipt was sealed.

There are at least four plausible places to stop a coding clock:

the first code is generated;
the assigned local checks pass;
the pull request is opened;
the integrated change reaches its agreed acceptance state.

The first three are useful telemetry. The fourth is delivery.

When a team reports only time-to-first-output, it can improve the reported metric while moving work into code review, integration, security checks, rollback preparation, or another person’s queue. The work did not disappear. It changed owners.

What one failed merge revealed

Sequence 21 was small enough to inspect. Two agents worked on separate requirement partitions, but both patches touched a shared module. Each patch was locally valid against its assigned checks.

The seam failed only after integration. Applying the second patch removed branches required by the first combined tree. The first full validation exposed eight failures. The integrator rebuilt the missing paths, reran the suite, and reached 24 of 24.

No single worker had an obvious local failure. The workflow had a delivery failure.

That distinction matters because dashboards often reward the event nearest to generation. A faster PR can coexist with a longer review. More accepted suggestions can coexist with larger batches. More completed agent tasks can coexist with a slower path to deployable software.

The research is messier — and more useful — than a slogan

In a 2025 randomized study, METR observed 16 experienced open-source developers completing 246 real issues in repositories they knew well. In that setting, developers took 19% longer when AI tools were allowed. Before the study, they expected a 24% speedup; afterward, they still believed AI had made them 20% faster.

That result does not prove that AI slows most developers. METR explicitly limits the claim to the studied people, tools, repositories, and period. It does show that perceived acceleration and measured task completion can point in opposite directions.

The follow-up became even more revealing. In February 2026, METR said it was changing the experiment design because adoption had altered who would participate and which tasks they would submit. Some developers would not work without AI. Others withheld tasks they did not want randomized into a no-AI condition. Multi-agent use also made time reporting unreliable for part of the sample.

The honest conclusion was not “AI won” or “AI lost.” It was that the measurement boundary no longer captured the work cleanly.

The same shift appears outside coding. A Microsoft Research study of 319 knowledge workers found that AI-assisted work changes where critical effort is applied: away from direct production and toward verification and integration. Generation becomes cheaper while judgment remains attached to the human accepting the result.

Measure the acceptance gap

The simplest correction is to add one interval:

Acceptance gap = accepted delivery timestamp − first generated completion timestamp

The gap is not automatically waste. Review, validation, and release decisions can be the work that makes an output safe. The purpose of measuring it is not to force it to zero. The purpose is to stop pretending it is free.

The same logic appears in production AI evaluation, quality gates, and monitoring: a plausible output is an intermediate state, while prompt versioning, shadow-mode validation, drift detection, and rollback procedures determine whether it can move forward.

The acceptance gap begins when generated work first looks complete and ends only at the agreed delivery boundary.

This is compatible with established delivery measurement. DORA’s current framework carries change lead time to deployment and pairs throughput with instability measures. It also warns against choosing one metric to rule them all.

An AI-specific measurement should preserve the same tension: speed matters, and the accepted outcome matters.

A five-event measurement card

You do not need a new analytics platform. Capture five timestamps for one recurring task type:

Generation complete: the agent first presents a supposedly complete change.
Pull request opened: the change enters the shared review system.
First full validation: the complete acceptance suite runs against the integrated state.
Last rework complete: required corrections stop changing the result.
Accepted or deployed: the team’s real delivery boundary is crossed.

Then record three companion values: the number of rework cycles, the number of acceptance failures, and whether the change caused rollback or unplanned follow-up.

Compare the same task class over time. Do not compare a documentation edit with a payment-system migration and call the difference an AI effect.

Three questions before calling AI productive

Where does the clock stop? If it stops at generation, local green, or PR creation, label the metric accordingly. Do not present an intermediate event as delivery time. The name of the metric should reveal its boundary.

Who absorbs verification? An agent can reduce the author’s typing while increasing a reviewer’s reconstruction work. Count the human and machine effort needed to reach acceptance, even when it occurs in another queue.

What counts as failure? A rejected patch, rollback, reopened issue, or second validation cycle must remain in the record. Removing inconvenient runs makes the tool look faster by redefining the work after it happens.

When the shorter clock is honest

Sometimes AI really does shorten delivery.

A narrow task with a frozen interface, fast tests, low shared-state risk, and a clear reviewer can move from request to acceptance faster because generation was the actual constraint. If accepted lead time falls while rework and failure do not rise, call it a productivity gain.

The acceptance gap is not an anti-AI metric. It is what allows a positive result to survive scrutiny.

Keep the finish line after the handoff

The useful question is not whether AI writes code faster. It often does. The useful question is whether the system accepts valuable changes faster after review, integration, and failure recovery are included.

For one recurring workflow, capture the five events. Keep the task class stable. Preserve failed runs. Then look at both generated speed and accepted delivery.

That is a narrower claim than “AI transformed productivity.” It is also a definition of AI coding productivity an engineering team can verify.

The delivery clock keeps running after the AI says done.

How We Built Kimlic: Reusable Blockchain-Based KYC with Elixir and Kubernetes

Dmytro Nasyrov — Tue, 04 Aug 2026 11:05:13 +0000

Most KYC systems are built as isolated checkpoints.

A user uploads an ID document to one service, waits for approval and repeats the same process when joining another platform. The verification may already have been completed successfully, but the result usually remains locked inside the first provider's database.

For users, each repeated check can take another 15 to 20 minutes. For businesses, a single verification may cost between $5 and $15 while adding friction to the onboarding funnel.

It also creates a data protection problem. The same sensitive documents can end up stored by many separate companies.

Kimlic started with a different model:

Verify the user once, create a reusable credential and allow other services to validate the required proof without asking for the same documents again.

I am the founder and CTO of Pharos Production. Our partnership with Kimlic began in 2018. We worked with the client to design and build a blockchain-based KYC and digital identity platform for Web3 and FinTech products.

The production system now processes more than 10,000 identity verifications per day with 99.8% verification accuracy.

This article explains the public architecture behind the platform, the decisions that shaped it and the lessons we learned while building reusable identity infrastructure.

The product challenge

Kimlic needed to connect three requirements that often pull an identity platform in different directions.

The first was reuse. Users should not have to complete the same verification process for every service.

The second was privacy. Reuse should not require sensitive personal information to be copied across multiple databases or published on a blockchain.

The third was compliance. Businesses still needed a reliable way to request a verification, validate its current status and preserve an auditable history of the process.

The platform also had to operate under real production load:

KYC and AML requirements across at least four jurisdictions
More than 10,000 verification flows per day
More than 3,000 concurrent verification sessions
Response times below three seconds
99.9% platform uptime
99.8% verification accuracy
Secure APIs for external FinTech and Web3 products

This was not only a smart contract project.

It required blockchain engineering, identity workflow design, high-concurrency backend development, cloud infrastructure, auditability and integration APIs to function as one system.

Blockchain as a proof layer

Using blockchain for KYC does not mean placing passports, addresses or other personal documents on a public ledger.

That would create an obvious privacy problem. Blockchain data is difficult to remove and can remain visible long after the original business purpose has ended.

Kimlic uses the blockchain as a verification and proof layer.

Users complete the identity process through the platform. The system creates cryptographic evidence of the verification while limiting the disclosure of sensitive personal data.

The reusable credential can then be presented to another service. That service validates the required proof instead of asking the user to upload the same identity documents again.

At a simplified level, the flow looks like this:

User starts identity verification
                |
                v
Kimlic verification workflow
                |
                v
Identity status and evidence are created
                |
        +-------+--------+
        |                |
        v                v
Metadata and         Cryptographic
audit records        blockchain proof
        |                |
        +-------+--------+
                |
                v
User grants access to a service
                |
                v
Service validates the KYC proof through an API

This division is important.

The blockchain provides tamper-resistant evidence while the application layer manages identity workflows, consent, access and compliance-related state.

The result is a portable credential without turning the blockchain into a public personal-data database.

A reusable credential is more than a boolean value

It is tempting to represent KYC as one field:

verified = true

A production identity platform needs a much richer model.

The meaning of a verification can depend on when it was completed, which checks were performed and which jurisdictional requirements applied. Its status may also change after the initial decision.

A reusable credential therefore needs a lifecycle.

The platform must track the current verification state, the evidence supporting it and the consent under which another business can access it. It also needs an audit trail of important state changes.

This is one reason we separated the user-facing proof from the complete internal workflow.

A partner may only need to know whether an acceptable verification exists. The platform still needs the underlying metadata and audit history required to support that answer.

The score, status or proof presented to an external product is only the visible surface of a larger identity system.

The public architecture

The main public technology stack behind Kimlic includes:

Elixir and Erlang
PostgreSQL
React and Next.js
AWS
Kubernetes
Terraform
Blockchain-based cryptographic proofs

A simplified architecture looks like this:

       User identity portal
          React / Next.js
                 |
                 |
      Business dashboard and APIs
          React / Next.js
                 |
                 v
      Elixir and Erlang services
   Identity workflows and verification logic
        /             |              \
       /              |               \
      v               v                v
PostgreSQL      Blockchain proof    Integration APIs
metadata,       and credential      for FinTech and
states and      validation          Web3 products
audit logs
      \               |                /
       \              |               /
        +-------------+--------------+
                      |
                      v
            AWS and Kubernetes
                      |
                      v
           Infrastructure as code
                 Terraform

This is intentionally a high-level view. It excludes proprietary verification logic, security controls and implementation details that cannot be published.

Why we used Elixir and Erlang

Identity verification involves many concurrent workflows.

Different users can be submitting data, waiting for checks, receiving updated statuses or granting access to business clients at the same time. Each workflow can move independently and may depend on external operations.

The backend also needs to remain stable when an individual process or integration fails.

Elixir and Erlang provide a strong foundation for this model. They are well suited to systems with high concurrency, isolated processes and long-running state transitions.

For Kimlic, the backend is responsible for:

Identity workflow orchestration
Verification logic
API services
Real-time identity status updates
Event-driven communication
Integration with the blockchain proof layer
Communication with user and business interfaces

The key benefit was not language syntax.

The important factor was the ability to isolate work, process many verification sessions concurrently and recover from failures without treating every exception as a platform-wide incident.

That became particularly relevant as the system scaled beyond 3,000 concurrent verification sessions.

Event-driven identity workflows

A verification process contains a sequence of state changes.

A user initiates the process. The platform receives the required information. Verification logic evaluates it. The identity status changes and the result becomes available to the user or an authorized business.

Processing all of this as one large synchronous request would create several problems.

The request could remain open while the platform waits for another component. A temporary failure could force the complete workflow to restart. Higher traffic could also make the API layer responsible for too much long-running work.

Kimlic uses event-driven communication and scalable messaging patterns for workflow and state changes.

This lets different parts of the platform progress independently.

A request can be accepted, processed and updated without forcing the user interface to hold one connection open for the complete verification lifecycle.

The architecture also makes it easier to isolate failures. One delayed operation does not need to stop unrelated verification sessions.

The broader lesson is useful for any regulated workflow:

Model the business process as a sequence of durable state transitions instead of one oversized request-response operation.

PostgreSQL as the operational source of identity state

The blockchain proof is only one part of the system.

Kimlic also needs a reliable operational record of identity metadata, current verification states and audit events.

PostgreSQL serves as the primary data store for that information.

This separation gives each layer a clear responsibility:

Layer	Responsibility
PostgreSQL	Identity metadata, workflow states and audit records
Blockchain	Cryptographic proof and tamper-resistant verification evidence
Elixir and Erlang services	Workflow orchestration, verification logic and APIs
Next.js interfaces	User consent, identity management and business access
AWS and Kubernetes	Availability, deployment and horizontal scaling
Terraform	Repeatable infrastructure provisioning

Keeping operational data outside the blockchain also gives the application more control over access patterns and internal workflow changes.

The blockchain does not need to become the primary database for every identity operation. It is used where its properties add value.

Privacy-preserving data sharing

Reusable KYC creates a difficult question:

How can another service trust the verification without receiving every piece of personal data collected during the original process?

Kimlic addresses this through selective proof sharing and user consent.

The user manages the verified identity through a secure interface. When another service requires KYC, the platform shares the necessary proof rather than broadly exposing the complete identity record.

This model reduces unnecessary document duplication.

It also creates a clearer access boundary. The consuming service requests a specific verification result while the user retains control over the sharing process.

From an architecture perspective, privacy depends on more than encryption.

A privacy-preserving identity platform also needs to control:

Which data is collected
Where each category of data is stored
Which service can request a proof
Which user action grants access
Which events must be recorded for an audit
Which information the receiving business actually needs

The safest sensitive record is often the one that never needs to be copied into another system.

APIs are part of the compliance product

Kimlic was designed for use by FinTech companies, exchanges and Web3 applications.

That meant the integration layer could not be treated as a secondary feature.

External businesses needed secure APIs through which they could:

Request a KYC verification
Check the current identity status
Validate an existing proof
Integrate the result into onboarding
Maintain a traceable verification workflow

The API is where the reusable identity model becomes commercially useful.

Without a stable integration layer, the credential would remain limited to the Kimlic interface. Each new partner would require custom development and the platform would struggle to grow beyond its original use case.

We therefore treated API design as part of the core architecture.

The business application should not need to understand the complete internal verification pipeline. It needs a predictable contract for requesting a check and receiving a result.

That contract also needs to remain stable as internal verification logic evolves.

Separate interfaces for users and businesses

The platform serves two different audiences.

Users need to complete verification, view their identity status and manage consent.

Business clients need to request proofs, review verification results and integrate KYC status into their own systems.

Kimlic provides React and Next.js interfaces for both groups.

The user experience focuses on identity management and consent. The business experience provides access to dashboards and APIs for requesting and validating proofs.

Keeping these concerns separate helped us avoid exposing the full internal workflow to every participant.

A user does not need a compliance operations dashboard. A business client does not need access to the user's complete identity interface.

Each audience receives the surface required for its role.

Compliance had to influence the architecture

Compliance cannot be added to an identity platform after the technical design is complete.

The verification model affects data collection. Data retention affects storage. Audit requirements affect event logging. Jurisdictional rules affect workflow behavior and the information a business needs to validate.

Our engineers worked closely with Kimlic's compliance team while designing the platform.

The objective was to ensure that technical decisions reflected the KYC and AML requirements of the target jurisdictions from the beginning.

This included the relationship between:

Verification evidence
Identity status
Consent
Auditability
Blockchain proofs
Business access

The platform achieved a 100% compliance audit pass rate after deployment while operating across four jurisdictions.

That outcome came from treating compliance as an architectural input rather than a final checklist.

Scaling the platform on AWS and Kubernetes

Identity onboarding traffic is not always consistent.

A new partner launch, marketing campaign or customer migration can create a sudden increase in verification sessions. The infrastructure needs to absorb that increase without turning KYC into an onboarding bottleneck.

Kimlic runs on AWS and Kubernetes.

The containerized architecture allows services to scale as load increases. Workloads can be deployed and updated independently while Kubernetes handles orchestration and service recovery.

Terraform manages infrastructure as code.

This gives the team a repeatable way to provision and change infrastructure while reducing reliance on manual cloud configuration.

The production platform supports:

More than 3,000 concurrent verification sessions
Response times below three seconds
99.9% uptime
Automatic scaling during peak onboarding periods

The value of this infrastructure is visible in the user funnel.

A reliable verification system keeps onboarding moving during high demand. An unreliable system converts infrastructure pressure into abandoned registrations.

What Pharos Production delivered

Pharos Production worked with Kimlic across the complete technical platform rather than delivering one isolated component.

Our contribution included:

Blockchain-based credential and proof architecture
Smart contract engineering
Identity verification backend services
Event-driven workflow processing
KYC and AML integration logic
PostgreSQL data and audit architecture
APIs for external FinTech and Web3 products
User and business web interfaces
AWS and Kubernetes infrastructure
Terraform-based infrastructure automation
Performance and production scaling

The work required coordination between blockchain engineers, backend developers, frontend developers, DevOps specialists and compliance stakeholders.

That cross-functional delivery model was necessary.

A smart contract alone cannot run a KYC platform. A verification backend alone cannot create reusable credentials. A compliant workflow still needs reliable infrastructure and a usable integration layer.

The platform only works when all of those components operate as one product.

Production results

The public results from the project include:

Metric	Result
Daily identity verifications	10,000+
Verification accuracy	99.8%
Concurrent verification sessions	3,000+
Response time	Below 3 seconds
Platform uptime	99.9%
Target jurisdictions	4
Compliance audit pass rate	100%
Reduction in onboarding time	60%

The measured onboarding completion rate increased from 62% to 89%.

Verification volume on the same measured workload grew from 20,000 to 140,000 flows per month compared with the pre-optimization baseline.

These results show the connection between architecture and business performance.

Reducing duplicate checks shortens onboarding. Reliable processing prevents traffic spikes from blocking users. Reusable proofs lower the amount of work required when the same identity moves between services.

What we learned

1. Do not use blockchain as a personal-data database

The blockchain should store the proof required for verification rather than a public copy of the underlying identity documents.

Immutability is useful for evidence. It becomes a liability when applied to sensitive data that should not remain permanently exposed.

2. Design the credential lifecycle before choosing the stack

A reusable identity is not a static record.

Define how it is created, updated, validated and shared. Also define which state changes must remain auditable.

Those decisions shape the backend, data model, APIs and blockchain layer.

3. Separate proof from operational state

The blockchain proof and the operational identity workflow serve different purposes.

Keeping them separate allows each layer to use the storage and access model appropriate to its role.

4. Treat consent as a system capability

Consent cannot exist only as text next to a checkbox.

The platform needs to know which user authorized which proof to be shared with which service. That decision must be reflected in the product flow and audit history.

5. Keep long-running work outside synchronous API requests

Identity checks may depend on several processing steps.

An event-driven workflow prevents the public API from becoming responsible for waiting on the complete lifecycle.

6. Build integration contracts early

Reusable credentials only create network value when other products can consume them.

Stable APIs and clear business-facing workflows should be designed alongside the identity engine.

7. Bring compliance into architecture discussions

Compliance requirements influence data storage, logging, workflow states and access control.

Involving compliance specialists after implementation usually means redesigning decisions that were already embedded throughout the platform.

Client feedback

Kimlic's CEO highlighted both the production results and the contribution of our engineering team:

“Their smart contract expertise and understanding of KYC compliance were critical to our success.”

For teams evaluating Pharos Production as a development partner, the complete Kimlic case study includes the public technology stack, project results and client feedback.

Final thought

The main value of blockchain-based KYC is not putting identity data on-chain.

It is making a verified result portable and independently provable while keeping sensitive information under controlled access.

Kimlic turned that model into a production platform.

Users can verify once and reuse the result. Businesses receive a compliant API instead of another manual document flow. The infrastructure supports thousands of concurrent sessions while preserving the auditability required for regulated identity workflows.

For Pharos Production, Kimlic demonstrates how blockchain engineering, high-concurrency backend development and compliance-driven architecture can be combined within one real product.

Teams building a similar KYC, digital identity or regulated onboarding platform can review the full technical case and contact us through the project page.

Which part of reusable KYC creates the harder engineering problem in your experience: privacy, credential portability or regulatory consistency?

How We Built Ludo's Real-Time Cross-Chain Reputation Platform

Dmytro Nasyrov — Sat, 25 Jul 2026 21:19:23 +0000

Public blockchains make transactions visible.

They do not automatically answer a much more useful product question:

Can we trust the wallet behind those transactions?

I am the founder and CTO of Pharos Production - software development company. Since 2021, our team has worked with Ludo to build and scale the platform's blockchain indexing, real-time data processing, smart contracts, backend infrastructure and external APIs.

A wallet may have years of activity across several networks. That history can include transfers, smart contract calls and interactions with hundreds of protocols. The data is public, but turning it into a current and usable trust signal is a separate engineering problem.

And that is the problem behind Ludo - Reputation platform of Web3

Ludo is a live cross-chain reputation platform that converts onchain behavior into reputation data that users and other Web3 products can consume.

This article covers the architecture we can discuss publicly, the constraints that shaped it and the engineering lessons we took from the project.

The production requirements

This was not a prototype built around a small sample of wallets.

The platform needed to operate across more than 15 blockchain networks while supporting a growing number of users and external integrations.

The production targets included:

More than 2 million scored wallet addresses
More than 10,000 blockchain events processed per second
API response times below 200 milliseconds
99.9% platform uptime
More than 50 partner integrations
Real-time reputation updates

The system also needed to support additional networks without requiring a redesign of the entire processing pipeline.

Those requirements immediately ruled out an architecture based only on scheduled batch jobs or heavy calculations inside the API request path.

Reputation is a streaming data problem

A reputation score is useful only while it reflects the current state of a wallet.

Suppose a score is recalculated once every 24 hours. A wallet can change its behavior shortly after the calculation, while connected products continue making decisions with stale information.

Calculating the complete score every time an API consumer requests it creates a different problem. The request path becomes responsible for reading and aggregating large amounts of historical data. Latency grows with the amount of activity associated with the wallet.

For Ludo, we treated reputation as a continuously updated data product.

Blockchain activity enters the system as a stream. New events are processed as they arrive and the relevant reputation state is updated before an external product requests it.

That keeps expensive event processing outside the latency-sensitive API path.

At a simplified level, the architecture looks like this:

Blockchain networks
        |
        v
Chain-specific indexing
        |
        v
Apache Kafka
        |
        v
Apache Flink
        |
        v
Reputation processing
        |
        +--------------------+
        |                    |
        v                    v
Apache Cassandra       Apache Pinot
        |                    |
        +----------+---------+
                   |
                   v
        Spring Boot API services
                   |
        +----------+-----------+----------------+
        |                      |                |
        v                      v                v
   Web platform       Browser extension   Partner APIs
                                                |
                                                v
                                      External Web3 products

This is a public, simplified view. It intentionally leaves out proprietary scoring rules and internal data models.

Separating blockchain ingestion from reputation logic

Supporting more than 15 networks creates a problem that has little to do with the scoring formula itself.

Different chains expose different data structures, finality models and indexing behavior. Even similar actions can be represented differently from one network to another.

If every downstream service has to understand the native format of every blockchain, adding a new network becomes progressively more expensive. The scoring layer also becomes tightly coupled to chain-specific implementation details.

The architecture therefore needs a clear boundary between blockchain ingestion and reputation processing.

Chain-specific indexing is responsible for reading the network and translating relevant activity into a consistent internal representation. Downstream processing can then operate on normalized events instead of repeatedly interpreting raw chain data.

This separation helped our team deliver production-grade indexing for additional blockchain networks without rebuilding the rest of the platform each time.

The broader lesson is straightforward:

Normalize chain-specific behavior at the system boundary. Do not allow native blockchain formats to spread through the complete backend.

Kafka as the event backbone

Apache Kafka handles the continuous stream of blockchain activity.

Its role is larger than moving data between two services. Kafka provides a durable boundary between ingestion and processing.

Blockchain indexers can publish events without being directly coupled to every consumer. Processing services can evolve independently and additional consumers can be introduced without modifying the indexing layer.

This matters in a reputation platform because the same source activity may eventually support several workflows:

Reputation state updates
Historical analysis
Product dashboards
Audit and traceability
New scoring models

Kafka also makes it possible to absorb short-term differences between the rate at which events arrive and the rate at which downstream services process them.

For this type of system, partitioning requires particular care. Events that update the same logical entity need predictable processing order while the workload still needs to be distributed across the cluster.

The exact partitioning model depends on the product's scoring rules, but the general requirement is consistent: ordering guarantees and horizontal scalability must be considered together.

Flink for incremental processing

Apache Flink processes blockchain events as they move through the system.

The important architectural decision was to update reputation state incrementally rather than repeatedly rebuilding it from the wallet's entire history.

When a relevant event arrives, the processing layer determines how that event affects the existing reputation state. The updated result can then be persisted for fast retrieval.

This approach provides two advantages.

First, the amount of processing required for a new event does not need to grow with the complete age of the wallet.

Second, the score can remain current without forcing API consumers to wait for historical aggregation.

Real-time processing also introduces operational questions:

How should duplicate events be handled?
What happens when events arrive late?
How should a chain reorganization affect derived state?
How can a calculation be replayed after scoring logic changes?
Which state must remain auditable?

These questions cannot be solved by selecting a stream-processing framework alone. They need to be part of the data model and operating procedures from the beginning.

Using different storage systems for different workloads

A platform like Ludo does not have one universal database workload.

Operational reputation state, historical data and real-time analytical queries have different access patterns. Forcing all of them into one database would simplify the technology list while making the system harder to scale.

The public architecture uses several specialized components.

Component	Primary role
Apache Cassandra	Scalable storage for user and reputation data
Apache Pinot	Low-latency real-time analytics and query workloads
Delta Lake	Organized historical data for analytics
Apache Kafka	Durable blockchain event streaming
Apache Flink	Stateful real-time event processing
Spring Boot	Backend services, APIs and reputation logic

Cassandra supports distributed operational storage where the access patterns are known and horizontal scale matters.

Pinot serves analytical workloads that need fresh data and low query latency. This is useful for dashboards, curation tools and API consumers that need to filter or analyze reputation information.

Delta Lake retains organized historical data for deeper analysis and long-term processing.

This is a deliberate use of polyglot persistence. Each system has an explicit responsibility instead of becoming another item in the stack.

Keeping the API path fast

The platform needed to keep API response times below 200 milliseconds.

The main architectural principle behind that target was to avoid performing full reputation calculations during an external request.

By the time an API call arrives, the stream-processing layer has already processed relevant blockchain activity and updated the materialized reputation state.

The API layer can focus on controlled retrieval, authorization and response construction.

Spring Boot provides the backend foundation for these services. It also gives the platform a consistent way to expose reputation data to different consumers.

Those consumers include:

The Ludo web platform
A browser extension
A Telegram Mini App
Curator tools
External Web3 products
Mobile integrations

The platform also represents reputation through soulbound NFTs. Since these assets cannot be transferred or sold, the representation stays connected to the wallet activity that produced it.

The NFT is the verifiable product surface. The streaming and scoring infrastructure behind it is what keeps the representation current.

Integration is part of the product

It is easy to treat APIs and documentation as work that happens after the main platform is complete.

That approach would have limited Ludo's value.

A cross-chain reputation system becomes more useful when other products can place its trust signal inside their own workflows. That makes developer experience part of the architecture rather than a separate documentation task.

Pharos Production developed and documented APIs for external platforms. We also created SDKs and integration materials intended to reduce the amount of custom work required from each partner.

The result was a reduction of more than 60% in partner onboarding time.

More than 50 partners now use the platform across Web3 use cases including DeFi, gaming and social products.

The lesson applies beyond blockchain:

An infrastructure product is not complete when its internal services work. It is complete when another team can integrate it safely without depending on its original developers.

Infrastructure and reliability

The platform runs on AWS with Kubernetes and Istio.

Kubernetes provides the orchestration layer needed to deploy and scale independently evolving services. Istio handles service traffic management across the cluster.

Terraform supports repeatable infrastructure management while continuous load testing verifies that performance remains within the required range as the platform changes.

This operating model was designed around two practical requirements.

The first was traffic variability. Blockchain activity and partner demand do not always grow at a predictable rate.

The second was continuous expansion. Adding networks and integrations introduces new services, data flows and deployment requirements.

The infrastructure therefore needed to scale during traffic spikes without interrupting the products that rely on the reputation API.

The resulting platform maintains 99.9% uptime while processing more than 10,000 blockchain events per second without long queuing delays or data loss.

At this scale, uptime and latency are not only infrastructure metrics. They are part of the trust model.

A reputation signal is difficult to depend on when the API becomes unavailable during the exact periods when activity increases.

What Pharos Production delivered

Our work on Ludo covered more than one isolated service.

Pharos Production contributed to:

Smart contracts deployed across multiple blockchain networks
Cross-chain indexing and data aggregation
Real-time event-processing infrastructure
Reputation backend services
Scalable server architecture
Web and mobile integration APIs
API documentation and SDKs
Web, browser and Telegram product interfaces
AWS and Kubernetes infrastructure
Performance testing and production scaling

The project required blockchain engineering, backend architecture, data engineering and DevOps work to function as one delivery stream.

That combination is important.

A scoring model cannot become a reliable product without current source data. A data pipeline cannot create value without an accessible API. An API cannot become infrastructure for other businesses without predictable performance and operational reliability.

Production results

The current public project results are:

Metric	Result
Supported blockchain networks	15+
Scored wallet addresses	2M+
Processing throughput	10K+ events per second
API response time	Below 200 ms
Platform uptime	99.9%
Partner integrations	50+
Reduction in partner onboarding time	More than 60%

These figures are useful as proof of scale, but the more important result is architectural.

Ludo moved from a Web3 reputation concept to production infrastructure that other products can use inside live workflows.

Five lessons I would carry into another real-time Web3 platform

1. Start with the product decision

A reputation score has no meaning without the decision it is intended to support.

Before choosing a database or writing a scoring formula, define what another person or system will do with the result.

2. Make freshness an explicit requirement

"Real time" should not be a vague product claim.

Define how quickly a blockchain event must affect the result. That decision shapes ingestion, processing, storage and API design.

3. Isolate chain-specific complexity

Each new blockchain will introduce its own implementation details.

Keep those details at the indexing boundary and give downstream systems a stable internal event model.

4. Keep heavy computation outside the request path

An API with a strict latency target should retrieve prepared state rather than rebuild it from complete historical data.

Streaming and materialized state make that possible.

5. Design external integration from the beginning

Documentation, SDKs, versioning and predictable API behavior are core parts of an infrastructure product.

They should not be postponed until after the internal platform is considered finished.

Client perspective

Sergiu Draganus, co-founder and CEO of Ludo, has publicly confirmed that Pharos Production delivered the project's smart contracts, cross-chain data aggregation, scalable server infrastructure and documented integration APIs.

For teams carrying out technical vendor due diligence, Sergiu can provide a first-hand client reference about the collaboration and the process of building the platform with Pharos Production.

You can also review the complete Ludo case study with the project stack, results and client feedback.

Final thought

Public blockchain data does not create trust by itself.

It provides evidence.

The engineering challenge is to collect that evidence across networks, process it while it is still current and deliver it through infrastructure that other products can reliably use.

That is what our teams built with Ludo.

Where would you draw the boundary between a universal cross-chain reputation score and a score designed for one specific product?

What Is MiCA? The EU Crypto-Assets Regulation Explained

Dmytro Nasyrov — Sun, 28 Jun 2026 22:08:15 +0000

One rulebook for crypto across 27 EU states: who it applies to, the token categories, the timeline and what compliance actually requires.

TL;DR

MiCA (Markets in Crypto-Assets Regulation) is the EU's single framework for crypto-assets, applying across all 27 member states with one passport for authorized firms.
It covers crypto-asset service providers (CASPs) and stablecoin issuers, which fall into three token categories: asset-referenced tokens (ART), e-money tokens (EMT) and other crypto-assets.
Rules applied in phases: stablecoins from June 2024, CASPs from December 2024, with national transition into 2025-2026.
Compliance is architectural, not a bolt-on: KYC/AML, transaction monitoring, market-abuse surveillance and proof of reserves must be built into the platform.

MiCA (Markets in Crypto-Assets Regulation) is the European Union's comprehensive framework for regulating crypto-assets and the firms that issue or service them. It applies across all 27 EU member states, replacing a patchwork of national rules with a single regime and a passport that lets an authorized firm operate EU-wide. MiCA covers crypto-asset service providers (CASPs), stablecoin issuers and crypto-asset white papers, and it became fully applicable through 2024 and 2025.

What is MiCA?

MiCA stands for Markets in Crypto-Assets. It is an EU regulation (Regulation 2023/1114) that creates harmonized rules for the issuance, offer and trading of crypto-assets and for the provision of crypto-asset services. Before MiCA, crypto firms faced 27 different national approaches. MiCA replaces that with one rulebook: a firm authorized in one member state can passport its services across the entire EU and EEA, the same single-market model that governs traditional finance.

Who does MiCA apply to?

MiCA applies to two broad groups: issuers of crypto-assets and crypto-asset service providers (CASPs). CASPs include exchanges, custodians, brokers, trading platforms, portfolio managers and advisers. Issuers of asset-referenced tokens and e-money tokens face additional requirements around reserves and authorization. If you operate any crypto business serving EU customers you are likely in scope. See the 10 CASP services under MiCA for the full breakdown of regulated activities.

The three categories of crypto-assets under MiCA

MiCA classifies crypto-assets into three types, each with different rules:

Asset-referenced tokens (ART): tokens that reference multiple currencies, commodities or crypto-assets to maintain stable value.
E-money tokens (EMT): tokens that reference a single official currency, functioning like electronic money.
Other crypto-assets: utility tokens and most other tokens not covered by existing financial rules.

Getting this classification right determines which obligations apply. Our guide to ART vs EMT classification explains the boundaries and the software each token type needs. Where a token behaves like a financial instrument, MiCA may not apply at all - MiCA vs MiFID II covers that line.

MiCA timeline: when did it take effect?

MiCA entered into force in 2023 and became applicable in phases. The rules for stablecoins (asset-referenced and e-money tokens) applied from June 2024, and the rules for crypto-asset service providers applied from December 2024, with national transitional periods running into 2025 and 2026. Firms operating in the EU need authorization or must wind down activities once their transitional window closes.

Key MiCA requirements

MiCA obligations span the full lifecycle of a crypto business:

CASP authorization: a licensing process with capital, governance and custody requirements.
Crypto-asset white papers: mandatory disclosure documents for token offerings.
Market abuse rules (Title VI): prohibitions on insider dealing, wash trading and spoofing, with surveillance obligations - see crypto market abuse under MiCA Title VI.
Reserves and safeguarding: stablecoin issuers must hold and prove adequate reserves.
KYC, AML and the Travel Rule: onboarding and transaction-monitoring duties - see MiCA KYC requirements.

The full operational picture is in our MiCA compliance checklist, and the spend involved is broken down in MiCA compliance cost in 2026.

MiCA vs other crypto regulations

MiCA is the most comprehensive crypto framework globally, but it is not the only one. The UK, US and Dubai take different approaches, which matters for firms operating across borders. Our MiCA vs UK, US and Dubai compliance map compares the regimes. Within the EU, the key boundary question is whether a token is a financial instrument under MiFID II rather than MiCA.

What MiCA means for crypto software

MiCA compliance is architectural, not a bolt-on. CASP authorization, transaction monitoring, market-abuse surveillance, proof of reserves and Travel Rule data exchange all have to be built into the platform. Retrofitting them after launch is expensive and risky. Pharos Production builds MiCA compliance software - KYC/AML platforms, transaction monitoring and regulatory reporting - designed for CASPs and stablecoin issuers operating under the regulation.

Reviewed by Dmytro Nasyrov, Founder and CTO, Pharos Production.

Originally published at pharosproduction.com/insights/engineering/what-is-mica/. Written by Dmytro Nasyrov, Founder and CTO at Pharos Production, a MiCA compliance software company.

State of Smart Contract Audits 2026: What 30+ Engagements Tell Us About Cost, Quality and Coverage

Dmytro Nasyrov — Sun, 28 Jun 2026 22:07:34 +0000

Real audit pricing tiers, critical-bug density and the findings that actually dominate in 2026 - from a 30+ engagement archive.

TL;DR

Top-tier smart contract audits in 2026 cost 80,000-350,000 USD per scope. Mid-tier 25,000-80,000 USD. Boutique 8,000-25,000 USD. Source: Pharos engagement archive 2024-2026 cross-checked against public OpenZeppelin and Trail of Bits engagement disclosures.
Critical bug density per 1,000 lines of Solidity averaged 0.4-0.7 across our 2023-2025 engagements before remediation. Industry public reports cluster between 0.3 and 1.1 (Pharos internal data, Halborn 2025 Web3 Threat Report).
Reentrancy is no longer the top finding. Oracle manipulation, access-control drift and cross-chain message replay now dominate critical findings (Chainalysis 2025, CertiK Hack3d 2024).
Multi-firm audit cycles - two independent firms in shadow mode - are now standard for any TVL above 50M USD. Single-firm audits correlate with higher post-launch incident rates in our sample.
Formal verification adoption crossed an inflection point in 2025. Roughly one third of our high-value engagements now ship with at least one Certora or Halmos invariant suite alongside the human review.

Pharos Production runs smart contract security audits and smart contract development for DeFi and Web3 teams. The data below comes from that engagement archive.

Method

This piece combines two data sources. First, the Pharos engagement archive 2018-2026, covering more than 30 smart contract audit and audit-adjacent projects across Ethereum, Polygon, BNB Chain, Solana and several L2 rollups. Engagements span DeFi protocols, NFT systems, cross-chain bridges, RWA platforms and FinTech custody backends. Names are withheld under NDA. Numbers are reported as ranges, not per-client identifiers.

Second, public data from tier-1 audit firms and incident trackers: Trail of Bits publication archive, OpenZeppelin audit reports, ConsenSys Diligence audit archive, Halborn research blog, CertiK Hack3d annual reports, Chainalysis Crypto Crime Report 2025 and DeFiLlama exploit data. Where Pharos internal numbers and industry data agree we treat the claim as well supported. Where they disagree we flag it.

All figures are advisory not financial advice. Sample bias is discussed in section 10.

Audit Cost Trends 2024-2026

Audit cost is a function of scope complexity, code novelty, deadline and firm reputation - not lines of code alone. Across our 2024-2026 engagements pricing settled into three tiers.

Boutique tier - small specialist teams, 8,000-25,000 USD per scope. Useful for narrow contracts, library forks or pre-launch sanity checks. Boutique findings are typically high signal but low coverage.

Mid tier - established regional firms with 5-15 auditors, 25,000-80,000 USD. This is where most production DeFi protocols below 50M USD TVL get their first audit. Reports are formatted, fix-cycle is included, response time is days not weeks.

Top tier - Trail of Bits, OpenZeppelin, ConsenSys Diligence, Halborn, Spearbit, Cantina, Sigma Prime - 80,000-350,000 USD and up. Engagements at the high end include formal specification review, fuzzing harness construction and post-deploy retainer time. Booking lead time was 4-8 weeks in 2024 and has since compressed to 2-4 weeks for most firms (Pharos internal observation, cross-checked against OpenZeppelin public scheduling data).

Regional variation matters. EU and US firms charge a 30-60 percent premium over equally credentialed Asia-Pacific and Eastern European firms for comparable scopes. We see no quality delta in the report quality of mid-tier non-US firms in our sample.

Trend to watch - multi-firm audits. For any deployment with TVL projection above 50M USD a two-firm shadow audit is now table stakes. a16z crypto and Paradigm portfolio guidance both reflect this. Cost goes up roughly 1.6-1.8x not 2x, since the second firm often runs in parallel with a narrower invariant focus.

Bug Density per 1k LOC

Bug density is the most useful single number for engineering managers planning remediation budget. Across 30+ Pharos engagements, pre-remediation findings broke down approximately as follows per 1,000 lines of Solidity (excluding test code, comments and OpenZeppelin imports). These are Pharos internal observations.

SeverityPer 1k LOC, our sampleNotes

Critical0.4-0.7Direct loss-of-funds or admin takeover paths
High1.1-1.8Logic flaws requiring privileged or unlikely conditions
Medium2.5-4.0DoS, griefing, accounting drift
Low4-8Style, gas inefficiency, minor edge cases
Informational6-15Documentation, naming, missing events

Public Halborn and CertiK reports cluster critical density between 0.3 and 1.1 per 1k LOC depending on protocol category. Bridges and cross-chain messaging consistently show the highest density, simple ERC-20 forks the lowest. Our numbers sit inside that band, weighted toward DeFi which is most of our engagement mix.

A useful planning heuristic - budget at least 1 engineering week per critical and high finding for fix and re-test. For a 5,000 LOC codebase that is typically 8-12 engineer-weeks of remediation before re-audit.

Most Common Vulnerability Classes 2024-2026

Reentrancy taught a generation of Solidity developers and is now commodity-defended. The dominant classes in 2024-2026 are different.

Oracle manipulation - low-liquidity TWAP windows, spot-price reads, unverified Chainlink fallback paths. This is the single largest exploit value category in Chainalysis 2025 data. We flagged at least one oracle issue in roughly 70 percent of DeFi engagements (Pharos internal observation).
Access control drift - upgradeable proxies with under-scoped role hierarchies, EIP-2535 diamond facets shipped without role audits, governance time-locks bypassed via emergency multisigs. CertiK Hack3d 2024 lists access control as the largest dollar-loss category for the year.
MEV and sandwich-resistant ordering - finding subtle, exploit subtler. Most reports surface MEV exposure as informational, but the actual dollar drain accumulates silently. EIP-7702 and EIP-4844 reshape this surface in 2025-2026.
Flash-loan composability - the 2020-2022 flash-loan era never ended, it refactored. The new shape is multi-protocol price feedback loops where each protocol passes its own assertions but the composed flow is exploitable.
Cross-chain message replay - LayerZero, Wormhole, CCIP and IBC patterns. Bridges remain the highest dollar-loss category per incident in Chainalysis 2025.

Reentrancy still appears - mostly in lower-severity findings around ERC-777 and ERC-1155 hooks, or in non-standard tokens that pass control mid-transfer.

Time-to-Audit and Audit-to-Fix Cycles

Across our 2024-2026 engagements typical timelines settled at:

Booking to kickoff: 2-6 weeks for top-tier, 1-3 weeks for mid-tier
Initial review: 5-15 working days for a single contract suite under 5,000 LOC, 3-6 weeks for a full protocol of 15,000+ LOC
Fix cycle: 1-3 weeks for the team to remediate
Re-audit: 3-7 working days
Final report and public disclosure: 1-2 weeks after re-audit signoff

Pharos shadow-mode pattern - we deliberately overlap our internal review with the external firm's review for the first 5 working days. This catches the cheapest 30-50 percent of findings before they consume external auditor time, and gives the external firm a head-start on the deeper invariant work. OpenZeppelin and Trail of Bits engagement notes describe similar overlap patterns in their public retros.

Net calendar - plan for 8-14 weeks from booking to public final report on a non-trivial protocol. Compress this at your peril.

The False-Positive Tax

Static analysis tools - Slither, Mythril, Aderyn, Wake, Semgrep rules - are essential and overrated. Across our 2024-2025 engagements, automated tooling produced an average of 40-90 raw findings per 1,000 LOC. After human triage, less than 10 percent typically survive as real high or medium issues (Pharos internal observation).

The other 90 percent is the false-positive tax. It is paid by engineers who chase every red badge, by junior auditors who pad reports with noise and by clients who think a clean Slither run means a clean codebase.

Our position: tooling is necessary as a coverage floor and catastrophic when treated as a coverage ceiling. The real audit happens in invariant identification, manual flow tracing and adversarial scenario construction. Trail of Bits has argued this in public repeatedly. Our own data agrees.

Practical rule - measure auditor hours against findings-per-hour after triage, never against raw scanner output. The latter rewards noise.

What Audit Quality Actually Means

The term audit collapses three distinct activities. Quality requires all three.

Surface scanning - automated tools, syntax-level checks, dependency hygiene. Necessary, not sufficient. Cost-of-execution is cheap.
Invariant testing - statements that must hold for all states, asserted via fuzzers like Echidna, Foundry invariants, Medusa or formal tools like Halmos and Certora. Cost-of-execution is moderate. Catches whole classes of bugs that surface scanning cannot.
Adversarial reasoning - human auditors constructing exploit chains across functions, contracts, protocols and time. Cost-of-execution is high. Catches the bugs that ship to mainnet.

Formal verification adoption crossed an inflection point in 2025. Roughly one third of our high-value engagements now ship with at least one Certora or Halmos invariant suite. a16z crypto guidance and EF research grants have both pushed in this direction. The remaining two thirds rely on Foundry invariant fuzzing as a cheaper proof-carrying baseline. NIST IR 8408 references invariant assurance as a stablecoin technical hygiene baseline - a useful external anchor for non-blockchain stakeholders evaluating audit reports.

Proof-carrying patterns - shipping a contract alongside an invariant suite that re-runs in CI for every PR - are the single largest leap in audit quality we have seen this cycle. They convert audit findings from one-off events into continuous regression checks.

Cost-vs-Quality Decision Matrix

Project typeRecommended tierWhy

ERC-20 fork, no novel logicBoutiqueDiminishing returns above 25k USD
DeFi primitive, under 10M USD TVLMid + invariant suiteCatch invariant violations cheaply
DeFi primitive, 10-50M USD TVLTop tier single-firmReputation matters for LP trust
DeFi primitive, above 50M USD TVLTop tier dual-firm shadowInsurance-grade assurance
Cross-chain bridge, any TVLTop tier dual-firm + formal verificationHighest dollar-loss category in incidents
RWA or FinTech custodyTop tier + legal review + SOC 2 alignmentRegulatory exposure compounds technical risk
NFT mint, no royalties or feesBoutiqueSurface area is small
Governance systemTop tier with timelock specialistAccess control drift is a top-three loss category

Methodology Caveats and Limitations

Sample bias - the Pharos engagement archive over-represents DeFi, FinTech adjacent custody backends and cross-chain projects. ERC-20 fork audits and pure NFT mint audits are under-represented in our numbers. Critical density figures for bridges and DeFi protocols should not be extrapolated to simpler categories.

NDA constraints - we cannot publish per-client breakdowns. All numbers are reported as ranges across the sample, never as point estimates tied to identifiable engagements. Where ranges feel wide that is the cost of confidentiality.

Time bias - our 2018-2022 engagements skew the historic comparison toward earlier vulnerability classes such as reentrancy. Trend statements about 2024-2026 prevalence are based on the 2023-2026 subset.

External data - tier-1 audit firms publish report archives but not raw finding-density data. Cross-checks against Halborn, CertiK and Chainalysis are at the category level, not contract level. We treat agreement at the category level as a confirmation signal, not a numeric calibration.

Numbers in this report should be read as well grounded order-of-magnitude estimates, not engineering precision. Where you need precision for a procurement decision, talk to us directly or to any of the firms we cite.

Originally published at pharosproduction.com/insights/engineering/state-of-smart-contract-audits-2026/. Written by Dmytro Nasyrov, Founder and CTO at Pharos Production.