DEV Community: Ariftly

AI Agent Autonomy: A Measurable Enterprise Framework

Ariftly — Wed, 21 Oct 2026 04:40:15 +0000

Autonomy is an authorization decision

A useful enterprise definition of AI autonomy is the amount of operational authority a system may exercise without a human approving each action.

That makes autonomy distinct from capability. Benchmark improvement can expand what a model could do; it does not determine what an enterprise should permit it to do.

This distinction leads to a practical rule: assign autonomy to workflow steps or action classes, not to an entire agent.

A five-stage workflow model

The Ariftly Workflow Autonomy Ladder uses five operating modes:

L0 Human-operated. A person performs the consequential action.

L1 AI-assisted. AI retrieves, interprets or recommends, but the person decides and acts.

L2 AI-prepared. AI prepares an executable artifact or action payload; a person reviews and commits it.

L3 Bounded AI execution. AI can execute pre-authorized actions inside an enforced envelope. Independent completion checks, exception routing and recovery controls are required.

L4 AI-operated by exception. AI runs a defined multi-step workflow segment. Humans own policy, exceptions, incidents and aggregate outcomes rather than approving every routine action.

L4 is not “no human.” Accountability changes location.

The promotion gate matters more than the labels

A step should gain authority only when evidence meets pre-specified thresholds across:

intended outcome;
false-success rate;
evidence completeness;
exception recall and precision;
policy boundaries;
adversarial security;
rollback/recovery;
human intervention;
drift/regression;
operational value.

NIST’s agentic evaluation work provides useful grounding for this approach. Its 2026 evaluation-probes project is building automated verifiers that can run inside agentic workflows and create machine-readable audit trails. The initial work checks factual grounding against trusted corpora.

NIST’s ARIA approach also combines model testing, red teaming and user testing. That matters because an autonomy decision based on one benchmark is structurally weak.

Static security testing can understate risk

NIST/CAISI’s AgentDojo work illustrates the problem. In one tested setup, the strongest baseline attack achieved 11% success; a newly developed attack tailored through red teaming reached 81%. Across five injection tasks, average attack success rose from 57% to 80% when each task was attempted 25 times.

A later public red-team competition analyzed by NIST/CAISI involved 13 frontier models, more than 250,000 attempts and more than 400 participants. At least one successful hijacking attack was found against every target model.

Do not read these as production breach probabilities. Read them as evidence that the evaluation should match the adversary and retry conditions a deployed workflow can face.

Automated success can miss the acceptance criterion

METR’s 2026 SWE-bench maintainer study is another warning. Four active maintainers across three repositories reviewed 296 AI-generated PRs. In the main normalized analysis, automated grader results averaged 24.2 percentage points higher than maintainer merge decisions.

This is not an argument to remove deterministic tests. It is an argument to make the deterministic acceptance suite match the real target state.

A workflow needs a definition of verified completion: the action both occurred and passed the independent checks that represent business acceptance.

It should also track false success: cases treated as complete where the intended state was not achieved.

Add a reliability budget

If a workflow has many dependent steps, aggregate reliability can fall quickly.

Under a deliberately simplified model where all n steps succeed independently with probability p:

R = p^n

At 99% per-step reliability:

10 steps → ~90.4%
20 steps → ~81.8%
50 steps → ~60.5%

To reach 99% end-to-end across 50 equal independent steps:

p = 0.99^(1/50) ≈ 0.999799

or about 99.98% per step.

Real workflows may have correlated errors, retries, compensating controls and branching. Those need to be modeled. The simplified result is useful because it forces teams to ask for an end-to-end reliability budget instead of assuming the component pass rate survives a long chain.

Clean tests need uncertainty

Even “zero failures observed” needs a denominator. Under the statistical rule of three, zero events across n comparable independent trials gives an approximate 95% upper bound near 3/n.

That means zero observed critical failures in:

100 trials → upper bound ~3%
300 → ~1%
1,000 → ~0.3%
3,000 → ~0.1%

Agent evaluation frequently violates the simple binomial assumptions, so this is not a universal assurance method. It is a discipline: report the sample and uncertainty instead of translating “zero observed” into “safe.”

Autonomy should be reversible

Model, tool, prompt, policy and environment changes can invalidate previous evidence. A production autonomy policy therefore needs demotion triggers.

Critical false-success events, missed exceptions, policy violations, security regressions, failed recovery, drift or material system changes should move the action back to a lower level or review state until evidence is re-established.

The result is not a straight maturity staircase. It is a controlled operating system for authority.

Source and full research: Ariftly’s canonical article. This framework is an editorial engineering proposal, not a NIST standard or a claim of measured Ariftly production performance.

Why 20% vs 18% Is Not a US–EU AI Adoption Ranking

Ariftly — Tue, 20 Oct 2026 20:03:54 +0000

What the evidence says: Official statistics show growing business AI use, but Eurostat's 20.0% and the U.S. Census's 18% do not establish a transatlantic ranking. Their populations, periods, questions, and weighting differ.

A comparability contract for enterprise AI adoption metrics

An analytics team can make an official adoption statistic misleading without changing a single digit. It is enough to drop the population and question wording from the visualization. That is particularly likely when comparing different official surveys.

For example, Eurostat's enterprise ICT survey reports 20.0% AI use in 2025 among enterprises with at least ten employed people in specified economic activities. The measure counts use of one or more technologies in a defined list. A 2026 U.S. Census research paper reports that 18% of employer firms used AI in a business function during its November 2025–January 2026 supplement reference period. The Census data cover employer businesses excluding farms and do not apply Eurostat's minimum size threshold.

The analytical question is whether these observations estimate the same target quantity. In this case, they do not. A responsible monitoring system must encode the estimand, not merely the percent field.

A useful observation record contains an immutable source ID, URL, original question or indicator ID, release version, reference date, geography, unit of analysis, eligible workforce size, included industries, metric numerator, denominator, weighting method, standard errors if published, data flags and an interpretation label. Any difference computation should verify a compatibility contract first.

The check can be hierarchical. Compare exact matches within the same survey release and indicator first. Compare older and newer releases only after reviewing changes to questionnaires, classification systems or sampling design. Compare different countries only when the population and concept are sufficiently aligned, and explain any remaining residual differences. If they are not, display the values side by side for context but disable subtraction and ranking.

Three within-source views illustrate the payoff. In Eurostat's series, covered AI use increased from 8.1% in 2023 to 20.0% in 2025. Arithmetic on rounded values gives +11.9 percentage points and approximately 2.47 times the 2023 level. That does not remove a 2025 questionnaire change introducing a visual/audio generation category. A trend annotation matters even when values remain useful.

The EU size comparison is similarly informative: 17.0% of enterprises with 10–49 employed people, 30.4% of those with 50–249 and 55.0% of those with 250+ reported AI use in 2025. The 55.0 / 17.0 ratio is about 3.24. It is a descriptive prevalence ratio, not a causal treatment effect and not adjusted for industry composition. An analyst should avoid stating that firm size explains the difference without further modeling.

The U.S. study offers a different view. The firm-weighted adoption estimate was 18%; the employment-weighted estimate was 32%. This 14-percentage-point contrast is not independent evidence from another country or measure of uncertainty. It represents an alternative weighting of the same underlying adoption question. Its interpretation is linked to where workers are employed rather than an equal vote per firm.

Sector breakdowns require caution, too. Census found that, for the wave ending 3 May 2026, Information firms reported 39.7% AI use and Finance and Insurance firms 33.9%, compared with a national figure of 19.8%. These figures should be dated on the chart. U.S. NAICS and EU NACE classifications are not a drop-in equivalent, and coverage differences must be considered before cross-region comparisons.

The last validation is semantic: what does an affirmative AI-use answer establish? It can establish that a respondent met a survey definition. It does not, by itself, document approved machine actions, deterministic acceptance checks, exception handling or end-to-end task completion. The 2026 Census supplement adds valuable business-function and worker-task information; for example, among adopting firms, 57% used AI in three or fewer business functions. Nevertheless, production assurance would need an additional operational evidence chain.

For workflow engineering, capture each input, generated proposal, decision authority, executed action, test result, escalation and outcome. Keep probabilistic interpretation distinct from deterministic validation. This is Ariftly's design interpretation, not a variable imputed from either government survey. Ariftly Forge's Record → Redesign → Operate direction is in development; the concrete local Oracle-to-Snowflake V1 has been tested only on synthetic inputs, not customer production workflows.

If the monitor is published as reusable data, include its source snapshot and transformation script or notebook. Make the script fail closed when crucial metric fields are absent or incompatible. Never auto-calculate EU rate - US rate without an explicitly documented, defensible harmonization method. Store row-level lineage and release a transparent revision log rather than silently replacing a statistic when an agency changes its dataset.

The goal is not to make adoption analysis impossible. It is to make a useful comparison reproducible and a misleading comparison difficult. Analysts can still show different official figures and learn from each, provided the visualization communicates what their denominators actually mean. The stronger enterprise takeaway is that adoption breadth and operational capability are different dimensions. One helps describe diffusion; the other requires measurable workflow evidence.

Primary source links and measurement details are in the canonical and evidence ledger; this adaptation is not a new statistical release.

Read the source-linked research

The full article includes the underlying official links, accessible comparison tables, methodology, limitations, and the downloadable Ariftly Enterprise AI Adoption Monitor v0.1:

The Enterprise AI Adoption Gap: What Official US and EU Data Actually Shows

Disclosure: This adaptation was prepared with AI assistance and reviewed against the cited U.S. Census and Eurostat sources. The statistics are agency-reported; Ariftly's contribution is the comparison method and derived arithmetic.

A 997 Passed. Did the Purchase Order?

Ariftly — Tue, 20 Oct 2026 16:11:50 +0000

Disclosure: This is Ariftly engineering research. The examples are synthetic, not customer evidence or production benchmarks. EDI onboarding remains a future workflow direction, not a shipped Ariftly product.

EDI partner onboarding is usually described as a mapping and transport exercise: convert the order, connect two endpoints, validate a sample and obtain certification. The harder operational issue is knowing what “finished” means. When the receiving organization uses several independent components, no single technical acknowledgment necessarily spans all of them.

Start by specifying the intended commercial outcome. In a typical X12 purchase-order exchange, an 850 requests goods or services. The sender may use AS2 for transport. The trading partner's systems may return an AS2 MDN, a 997 or 999, and, where agreed, an 855 from the seller's order process. The sequence sounds linear, but each response has its own scope and identifiers.

Transport boundary. RFC 4130 specifies AS2, including synchronous and asynchronous MDNs. A correctly verified signed receipt can offer important evidence about message delivery and integrity. The receiving party's signature, the original-message identifier, the returned MIC and the disposition provide pieces of that proof when required. A transport response or an MDN's arrival alone should not be interpreted as proof of the business outcome. The RFC's own example cautions that received EDI data may still be syntactically incorrect or not have reached the translator. Security configuration must follow current practice rather than copying 2005-era algorithms uncritically.

Syntax boundary. X12 defines the 997 to report syntactic-analysis results for encoded EDI transaction sets. Its definition expressly does not address the meaning of the business content. The 999 includes syntactic and relational analysis against the relevant implementation scope, yet it still does not establish the encoded business transaction's semantic meaning. A 997 with accepted counts is useful validation evidence. It is not the same thing as acceptance of quantities, dates, items or contractual terms.

Commercial response boundary. X12 describes the 855 as the response from a seller's purchase-order system after processing an 850. One official example depicts acceptance without line-item detail; another depicts a changed order involving date, quantity, price, substitution and a line rejection; another illustrates an order being rejected. The key observation is not that any one format is universal. It is that an engineer must interpret the received 855 under the exact agreed version and partner-specific guide. A bare message count or the presence of an 855 is not enough.

A disciplined implementation can represent these as a versioned evidence contract. The contract should state which responses are mandatory, which are optional, which identifier fields must match, what time window applies, how changes are handled, what counts as an authoritative order state and which person can approve an exception. The contract is not a standard transaction set. It is the organization's policy for determining whether the integration has met its promised outcome.

Correlation deserves explicit design. An AS2 MDN refers to an original AS2 message. A 997 or 999 may report on a group or multiple included transaction sets. An 855 refers to a purchase order and, in some cases, individual lines. These identifiers are not interchangeable. Build joins at the actual message, group, transaction and order levels and preserve sufficient context to explain why a particular business response belongs to a particular outbound PO. If the response omits optional line detail, do not fabricate a line-by-line proof from the header alone.

Negative tests should be engineered before go-live. Simulate a transport receipt arriving but syntax failing; an accepted 997 followed by a rejected 855; a response containing changed terms; a late or missing functional response; a wrong-PO reference; and a claimed acceptance that disagrees with the designated receiving system. Every scenario should specify the classification, escalation owner, safe retry behavior and record required for audit or handover.

The companion educational kit illustrates six such cases. Its CSV includes normalized fields for MDN state, 997 status, 855 status, PO-reference match and independent order-state check. The Python evaluator produces VERIFIED, WAIT_BUSINESS, BLOCK_SYNTAX, BLOCK_BUSINESS, REVIEW_DISCREPANCY or REVIEW_EVIDENCE_GAP under a deliberately strict fictional partner policy. The six outcomes are designed test coverage, not observed EDI traffic. The program is not a live AS2 validator or X12 parser and must not be installed as production acceptance logic.

This creates a useful division of responsibilities. Deterministic services validate protocol receipts, syntax and references. The seller's actual business response supplies commercially relevant evidence. A trusted system of record confirms whatever additional outcome the agreement requires. Operators resolve missing or contradictory evidence. AI may help analyze partner guides or summarize exception context, but its explanation is not the evidence. That is the difference between adding an AI feature and designing an operationally verifiable integration workflow.

A handover package should therefore contain the contract, test messages, negative-case results, correlated acknowledgment evidence, exception runbook, owner matrix and sign-off. The objective is not to make every status turn green. It is to be able to say exactly what was checked, what was not checked, and who accepted the remaining risk.

Sources: RFC 4130 (https://www.rfc-editor.org/rfc/rfc4130.html); X12 transaction sets (https://x12.org/products/transaction-sets); X12 supply chain flow (https://x12.org/flow/supply-chain); X12 855 public examples (linked in the canonical article).

From human-operated to bounded AI-operated workflows

The useful role for AI is not to invent an acceptance state. It is to compare partner documentation, prepare evidence, summarize exceptions and propose next actions. Deterministic software must still verify receipts, syntax, correlation, policy and business state. Consequential discrepancies stay with an accountable person.

A practical maturity path is: people perform and decide; AI prepares evidence; bounded steps execute under deterministic controls; autonomy expands only after representative work demonstrates reliability. That is the Ariftly View: redesign the workflow before adding a model, preserve an evidence chain and make every exception destination explicit.

Ariftly Forge is the in-development experience for recording, redesigning and operating workflows. Forward-Deployed AI Engineering is the implementation motion for one bounded workflow at a time. This article does not claim that Forge currently operates EDI onboarding in production.

Read, reproduce and cite

The full canonical guide includes the Ariftly Eight-Gate Acknowledgment Framework, methodology, limitations and downloadable six-case research pack:

Does an EDI 997 mean an order was accepted?

Documentation and data are available under CC BY 4.0; the evaluator code is MIT licensed.

Oracle to Snowflake migration validation: a six-gate acceptance framework

Ariftly — Tue, 20 Oct 2026 09:03:20 +0000

Originally published by Ariftly. This version focuses on the engineering method. The test scenarios below are synthetic specifications, not executed benchmark results.

A successful SQL conversion answers one question: can the target system accept the translated statement? A migration acceptance decision answers a harder one: does the translated workload preserve the behavior the team actually requires?

For Oracle-to-Snowflake work, those questions diverge around type semantics, input representation, session settings, DML behavior, transformed outputs, and approved business changes. A useful validation process therefore needs more than a compiler result or conversion score.

Define the behavior contract first

Before testing, separate three claims:

The SQL has been translated.
The translated SQL executes.
The translated SQL produces the accepted behavior for the intended workflow.

Each claim provides evidence. None proves the others.

A behavior contract records the source artifact, relevant DDL and dependencies, representative fixtures, session configuration, expected outputs, comparison rules, tolerances, and who can approve an intentional difference. Without it, reviewers cannot reliably distinguish a defect from an agreed modernization change.

Three risks that deserve explicit tests

1. NULL and zero-length strings

Oracle currently treats zero-length character values as NULL. Snowflake distinguishes empty strings from SQL NULL and exposes file-format controls that affect their treatment during loading and unloading.

A small fixture should include SQL NULL, zero-length text, one space, and ordinary text. Declare the expected representation before running the comparison. Do not silently normalize every blank-looking value to NULL.

Sources: Oracle Nulls and Snowflake data unloading considerations.

2. Oracle DATE contains time

Oracle DATE stores year, month, day, hour, minute, and second. Snowflake DATE stores a calendar date without time. A source value such as 2026-01-15 17:45:10 therefore needs an intentional target type and test.

Use two observations on the same day with different times. Record the selected Snowflake timestamp variant and relevant session settings instead of relying on an unspecified alias.

Sources: Oracle data types and Snowflake date/time types.

3. Duplicate MERGE matches

Snowflake documents cases in which multiple source rows match one target row. With ERROR_ON_NONDETERMINISTIC_MERGE=TRUE, the documented default, an ambiguous update or delete raises an error. When disabled, the selected source row is undefined.

Test whether the workflow should reject the ambiguity, deduplicate using a declared rule, or route it to an accountable owner. Source: Snowflake MERGE.

Use existing validation capabilities deliberately

Snowflake's documented migration validation has three levels:

L1 — schema: column structure, types, precision, scale, nullability, and row count.
L2 — metrics: counts, minimums, maximums, sums, averages, and other supported aggregates.
L3 — row: row signatures and cell-level mismatch investigation in supported configurations.

Row validation is optional and disabled by default in the documented configuration. Accepted transformations can reconcile explicitly declared source-to-target changes for applicable row-level validation.

These capabilities are valuable, but migrated-table validation is not automatically equivalent to validating every transformed query, procedure, scheduled job, or business calculation. The test boundary must match the workflow being accepted.

Source: Snowflake data validation.

A six-gate acceptance framework

We use a compact evidence chain:

Source contract: versioned SQL, DDL, dependencies, settings, fixtures, and expected outputs.
Translation: source and target SQL, diagnostics, and a record of material changes.
Execution: environment settings, run identifiers, output schemas, and captured results.
Comparison: typed assertions, predeclared tolerances, and mismatch reports.
Exceptions: classification, owner, and documented disposition for every consequential difference.
Approval: a versioned acceptance decision linked to the evidence above.

The framework is an engineering proposal, not a certification standard or performance claim.

Compare the right thing

Use stable business keys where they exist. When they do not, multiset comparison can preserve duplicate counts without treating arbitrary row position as identity. Enforce ordering only when the source contract requires it.

Compare typed values rather than display strings, especially for timestamps and decimals. Aggregate checks can pass even when keyed records differ, so validation depth should follow the required behavior and risk.

AI can help interpret legacy SQL, propose translations, and suggest tests. Exact value comparisons, schema assertions, duplicate detection, tolerances, and acceptance rules should remain deterministic wherever practical. A plausible explanation is not proof that a transformation is correct.

Start small, but preserve the evidence

A practical first suite can cover empty strings, temporal preservation, duplicate MERGE matches, decimal boundaries, aggregate masking, declared value mappings, session-sensitive time behavior, and missing or duplicate identities.

Ariftly has published the fixture specification and evidence-manifest template. They are test designs, not observed results.

The immediate next step is simple: define one source-to-target acceptance contract, build a representative fixture, and decide which differences are defects, which are intended transformations, and who has authority to accept the result.

A migration is ready for approval when its required evidence supports that decision—not merely when the SQL has been converted.