Stop Timing AI Tasks: Measure the Whole Work System Instead

An employee completes a memo in forty minutes instead of an hour. A developer produces a first draft of a function twice as fast. A support agent closes more conversations per shift. These observations look like productivity gains, and sometimes they are. But none of them, by itself, proves that the organization created more valuable output with the same or fewer resources.

The missing work may have moved elsewhere. A manager might spend longer checking the memo. A senior developer might repair subtle errors. A customer might reopen the conversation. A compliance reviewer might reconstruct sources the assistant omitted. The local task became faster while the system became slower.

The source benchmark on AI and professional productivity usefully emphasizes heterogeneity: effects differ by function, seniority, integration quality, training, and the type of work being measured. This field guide turns that caution into a practical measurement protocol.

The core idea is simple: measure an AI intervention at the boundary where the organization accepts useful work, not where the model finishes generating output. That requires a stable unit of work, a counterfactual, quality gates, downstream-cost accounting, segmented results, and a stop rule. It also requires the courage to report “no reliable gain” when the evidence does not support a stronger conclusion.

This is not an argument against AI adoption. It is an argument for measuring the thing leaders actually care about: more accepted outcomes, at known quality and risk, per constrained resource.

Productivity is a ratio, so define both sides

Productivity is usually expressed as output divided by input. The formula is easy; the definitions are not.

For knowledge work, output cannot safely mean words, tickets, slides, lines of code, or messages. Those are activity counts. A useful output is an artifact or decision that passes the receiving system's acceptance criteria and does not create disproportionate correction later.

Input cannot safely mean only the operator's visible task time. It may include:

• preparation and context assembly;
• prompt construction and retries;
• review, verification, and approval;
• integration into the system of record;
• rework after downstream discovery;
• incident response and customer recovery;
• training, licenses, governance, and administration;
• displaced work performed by another role.

A defensible operational ratio is:

accepted outcomes
────────────────────────────────────────────────────────────
operator time + review time + rework time + allocated overhead

You can express the denominator in hours, labor cost, or another constrained resource, but choose one before looking at results. If the intervention changes risk materially, report risk beside the ratio rather than pretending every consequence converts cleanly into money.

The OECD's analysis of AI, productivity, distribution and growth treats aggregate impact as conditional rather than automatic. Adoption, complementary investment, diffusion, competition, skills, and the distribution of gains all matter. That is a useful warning against extrapolating a ten-minute task experiment into an enterprise forecast.

Choose the decision before choosing the metric

An experiment should support a named decision. “Understand AI productivity” is too broad. Better decisions include:

• Should the finance team use an assistant for first-draft variance commentary?
• Should support agents receive retrieval-based answer suggestions for one queue?
• Should a legal team use summarization for document triage, with mandatory citation review?
• Should developers use an assistant for test scaffolding in a defined repository?
• Should a marketing team use a model to generate alternative briefs before human selection?

Each decision implies different failure costs and acceptance tests. A draft used only to provoke ideas can tolerate errors that would be unacceptable in a customer-facing answer. A code suggestion behind tests and review differs from an automated production change. A regulatory filing is not comparable with an internal brainstorming note.

Write a decision card before the pilot begins:

Field	Example
Decision	Adopt assisted variance-commentary drafting for monthly close
Population	Analysts producing standardized internal commentary
Exclusions	Novel transactions, restatements, regulated disclosures
Primary outcome	Accepted commentary packages per analyst-hour
Quality floor	No material factual error; every number traceable
Guardrails	Review time, correction rate, sensitive-data incidents
Evaluation window	Three close cycles
Stop rule	Stop if any material unsupported claim reaches approval
Owner	Finance transformation lead

The card prevents metric shopping. If speed improves but acceptance falls below the declared floor, the intervention fails its intended use even if another graph looks attractive.

Use a stable unit of work

Many productivity claims fail because the compared tasks are not equivalent. One worker receives a short, familiar case; another receives an ambiguous, high-risk case. One week contains ordinary volume; the next contains quarter-end exceptions. The AI condition looks faster because the work mix changed.

Define a unit with observable boundaries:

1. a start event that can be logged consistently;
2. the required input package;
3. an acceptance event performed by a receiving role or test;
4. a window for downstream rework;
5. complexity descriptors known before treatment;
6. exclusions that prevent unsafe experimentation.

For a support case, the unit might start when an eligible ticket is assigned and end when the answer is accepted with no reopen within seven days. For code, it might start from an approved issue and end when the change passes tests and review, with defects observed for a defined period. For a memo, it might end only after the accountable reviewer accepts every cited claim.

Do not redefine the unit after seeing results. If the initial unit proves unusable, close the experiment, document why, and begin a new version.

Build the counterfactual you can defend

A productivity gain is a comparison with what would have happened without the intervention. Before-and-after comparisons are convenient but fragile. Teams learn, workloads change, deadlines move, and people select different tasks once a new tool arrives.

Use the strongest feasible design:

Randomized assignment

Randomly assign eligible work units to assisted and control conditions. This is often possible for queues, document batches, or test cases. Stratify by complexity or team when simple randomization might create imbalance.

Crossover design

Participants alternate between conditions across comparable periods or task sets. This helps control stable differences between workers, though learning and carryover effects require attention.

Matched comparison

When randomization is impractical, match work by pre-treatment complexity, role, queue, and time period. State clearly that unobserved differences may remain.

Interrupted time series

For a broad rollout, collect enough pre-intervention and post-intervention observations to distinguish an existing trend from a level or slope change. Record other process changes that occur during the window.

Shadow evaluation

Let the AI produce output without exposing it to the operational decision. Review it against historical or live cases. Shadow mode is useful for quality and failure discovery, but it cannot fully measure behavior change, integration cost, or real user interaction.

Never compare volunteers enthusiastic about AI with nonusers and call the difference causal. That comparison mixes tool effect with motivation, skill, management support, and task selection.

Measure acceptance, not generation

The model's completion timestamp is rarely the business finish line. The measurement boundary should include the person or system that consumes the work.

For every unit, capture at least:

• total elapsed time;
• active operator time;
• model interaction time;
• review time by role;
• number of revision cycles;
• acceptance status and rejection reason;
• downstream rework within the observation window;
• severity of discovered errors;
• model, workflow, policy, and evaluator versions.

Then compute several complementary measures:

Measure	Question answered	Common trap
Accepted units per labor-hour	Did useful throughput improve?	Ignoring reviewer labor
Median time to acceptance	Did flow improve for a typical unit?	Hiding long-tail delays
First-pass acceptance	Did the draft reduce iteration?	Reviewers lowering standards
Rework minutes per unit	Was effort displaced downstream?	Too short an observation window
Material error rate	Did risk change?	Counting trivial and severe errors equally
Abandonment or escalation	Where did automation fail to complete?	Removing failures from denominator
Cost per accepted unit	Did total resource cost improve?	Excluding fixed and governance costs

Report distributions, not only averages. A tool may help routine cases while producing a small but consequential tail of expensive failures. Median, upper percentile, and worst credible outcome often matter more than a single mean.

A measurement pipeline you can execute

The following Node.js example compares accepted throughput and downstream cost between conditions. It deliberately rejects incomplete records rather than silently treating missing rework as zero. Save it as measure-productivity.mjs and run node measure-productivity.mjs.

const rows = [
  { condition: "control", accepted: true, operator: 52, review: 14, rework: 4, severe: 0 },
  { condition: "control", accepted: true, operator: 47, review: 18, rework: 9, severe: 0 },
  { condition: "control", accepted: false, operator: 41, review: 16, rework: 12, severe: 0 },
  { condition: "assisted", accepted: true, operator: 31, review: 24, rework: 8, severe: 0 },
  { condition: "assisted", accepted: true, operator: 29, review: 21, rework: 18, severe: 0 },
  { condition: "assisted", accepted: false, operator: 25, review: 37, rework: 30, severe: 1 }
];

function validate(row) {
  for (const key of ["operator", "review", "rework"]) {
    if (!Number.isFinite(row[key]) || row[key] < 0) {
      throw new Error(`Invalid ${key} in ${JSON.stringify(row)}`);
    }
  }
}

function summarize(items) {
  items.forEach(validate);
  const accepted = items.filter(x => x.accepted).length;
  const minutes = items.reduce(
    (sum, x) => sum + x.operator + x.review + x.rework,
    0
  );
  const severe = items.reduce((sum, x) => sum + x.severe, 0);
  return {
    units: items.length,
    accepted,
    acceptanceRate: accepted / items.length,
    totalLaborHours: minutes / 60,
    acceptedPerLaborHour: accepted / (minutes / 60),
    severeErrors: severe
  };
}

const byCondition = Object.groupBy(rows, row => row.condition);
const report = Object.fromEntries(
  Object.entries(byCondition).map(([name, items]) => [name, summarize(items)])
);

const relativeThroughput =
  report.assisted.acceptedPerLaborHour / report.control.acceptedPerLaborHour - 1;

console.log(JSON.stringify({ report, relativeThroughput }, null, 2));

This toy sample is far too small for an adoption decision. Its purpose is to expose the accounting. Generation looks faster in the assisted rows, but review and rework consume part of the apparent saving, and a severe error appears. A real analysis should add uncertainty intervals, predeclared exclusions, cluster effects when people contribute multiple observations, and sensitivity checks for missing data.

Quality gates must be independent of treatment

If reviewers know which outputs used AI, expectations may change. Some reviewers scrutinize assisted work more closely; others trust it or forgive it because leadership wants the pilot to succeed. Both behaviors bias results.

Where feasible, blind reviewers to condition. Use the same rubric, evidence access, and severity definitions for both groups. Randomize review order. Measure reviewer agreement on a sample, and resolve disagreements without changing the rubric midstream.

A useful quality rubric separates:

• factual correctness;
• completeness relative to the request;
• source traceability;
• policy or legal compliance;
• usability by the next role;
• style or presentation.

Do not let attractive presentation compensate for a factual or policy failure. Hard gates belong outside the weighted score.

The NIST AI Risk Management Framework organizes work around governing, mapping, measuring, and managing AI risk. It does not turn a productivity pilot into a compliance certificate. It does, however, reinforce a sound operational principle: measurement belongs inside a governed lifecycle, linked to context and subsequent management decisions.

Segment before you generalize

An overall average can conceal who benefits and where risk concentrates. Predefine segments that are plausibly related to the mechanism:

• task complexity;
• worker experience;
• input quality;
• language;
• function and workflow;
• degree of standardization;
• required factual precision;
• amount of available reference material.

Stanford HAI's 2025 AI Index reports rapid growth in organizational AI use and summarizes research finding productivity effects, often alongside narrowing skill gaps. That broad picture is informative, but an organization still has to test its own workflow. A result from a bounded writing or coding task does not establish performance in an ambiguous decision process, and effects on less experienced workers may include both immediate assistance and longer-term learning risks.

Segmentation should answer operational questions, not generate dozens of post hoc stories. Limit the number of primary segments, require a minimum amount of evidence, and label exploratory findings as exploratory.

Account for learning and skill retention

Initial pilot results mix tool performance with learning the tool. Productivity may rise as users develop better workflows, or fall as novelty wears off and hard cases dominate. Measure more than one time point.

Use a learning curve:

1. baseline without the intervention;
2. onboarding period reported separately;
3. early assisted work;
4. steady-state window after defined proficiency;
5. retention check after a gap;
6. periodic unassisted sample for critical skills where appropriate.

The last item matters when the organization needs humans to detect failures or operate during an outage. If assisted throughput rises while independent judgment deteriorates, the short-term productivity ratio omits a resilience cost.

Do not infer deskilling from discomfort or anecdote. Define a task that represents the retained capability, assess it consistently, and decide beforehand what decline would trigger training or workflow changes.

Convert potential into a business case carefully

Consulting research can identify where to investigate, but it is not a local forecast. A Deloitte survey on enterprise generative-AI value measurement found a revealing mismatch: 54 percent of surveyed organizations were seeking efficiency and productivity improvements, while 38 percent were tracking changes in employee productivity. The exact percentages belong to that survey population and period; they should not be generalized into a universal adoption rate. The durable lesson is that an investment objective and a measurement system are different things.

Build the local business case in layers:

Measured operational effect

Use the experiment's accepted throughput, quality, and total labor results with uncertainty.

Reachable volume

Count only eligible units that the workflow can realistically process. Exclude work outside the contract.

Adoption and persistence

Model observed use after onboarding, not licenses purchased. Include the possibility that use declines.

Full cost

Include licenses, integration, evaluation, training, support, governance, incident handling, and the opportunity cost of scarce reviewers.

Realization mechanism

Saved time creates value only if it changes capacity, service, cost, or quality. Ten minutes saved and immediately absorbed by unmeasured activity is not automatically cash.

Present at least three scenarios with explicit assumptions. Do not hide uncertainty behind a precise return number.

Treat uncertainty as part of the result

A pilot does not become reliable merely because it collected many rows. Repeated observations from the same person, team, customer, or document family are correlated. Ten thousand edits from five developers do not provide the same independent evidence as ten thousand randomly sampled edits from a broad population. The analysis must reflect how work was assigned and clustered.

Before the experiment, decide what difference would be large enough to change the operational decision. This is the minimum effect of interest. A two-percent throughput change may be irrelevant if switching workflows costs months of engineering effort. A two-percent reduction in a severe error may be decisive in a high-risk process. The threshold belongs to the decision context, not to a generic statistical convention.

Report an interval around the estimated effect. The interval should show which improvements, null effects, or harms remain compatible with the data and model. Do not translate “not statistically significant” into “the tools are equal.” The experiment may simply be too imprecise. Likewise, a narrow positive interval does not prove the effect will persist after the team, model, task mix, or incentive changes.

Use sensitivity analyses that correspond to plausible weaknesses:

• recalculate after assigning conservative rework to units with missing follow-up;
• separate first-time users from steady-state users;
• exclude the onboarding period without deleting it from the report;
• compare results with and without extreme cases, while still describing those cases;
• test alternative but predefensible complexity groupings;
• calculate outcomes at the worker or team level when assignment occurred there;
• show what happens when fixed governance cost is allocated across lower volume.

Avoid repeatedly checking results and stopping as soon as a favorable number appears. If safety monitoring requires continuous review, define safety thresholds separately from the success analysis. A data analyst or statistician should review designs where stakes, clustering, missingness, or sequential decisions materially affect the conclusion.

The most useful uncertainty sentence is operational: “Given this evidence, adopting for this bounded population is likely to improve accepted throughput enough to justify the reversible rollout, but the data do not support use for high-complexity cases.” It connects the estimate to a decision without pretending the estimate is permanent truth.

Preserve the evidence chain

Productivity dashboards often retain aggregates and discard the artifacts needed to audit them. That makes later questions impossible to answer. Preserve a privacy-appropriate chain from assignment through acceptance: work-unit identifier, eligibility decision, treatment assignment, versioned inputs, event timestamps, review result, revisions, terminal status, and downstream observations.

Do not store sensitive content merely because it could be useful later. Store references, hashes, structured reason codes, and access-controlled evidence appropriate to the risk. Define retention periods. Document transformations between raw events and reported measures. If a dashboard says review time fell, an auditor should be able to inspect the query and determine which roles, statuses, and time windows were counted.

Version the analysis itself. A corrected query should create a new result with an explanation, not silently rewrite the historical claim. This discipline prevents an encouraging pilot slide from becoming an untraceable organizational myth after the original team has moved on.

Contradictions a credible report must retain

Faster individuals may not produce a faster process

Local acceleration can create a review bottleneck. Measure queue time and the slowest constrained role.

Higher quality may reduce visible speed

An assistant can encourage more complete analysis, increasing task time while reducing errors or downstream cycles. A speed-only metric would call that a loss.

Novices may gain more and still need more oversight

The same worker can produce a better first draft yet remain unable to recognize a rare failure. Improvement and dependence can coexist.

Standardization improves measurement but narrows scope

Highly standardized tasks support clean comparison. They may represent only a small or less valuable portion of a role. State the boundary rather than generalizing to “the job.”

Adoption can correlate with performance without causing it

Strong teams may adopt earlier because they already possess better processes, data, and management. Observational differences require cautious language.

Quality scores can rise while trust falls

If users encounter occasional confident, high-severity failures, average quality may improve while willingness to rely on the system declines. Track severity and user behavior, not just mean ratings.

Failure modes and controls

Failure mode: stopwatch productivity

Only the operator's drafting minutes are counted.

Control: measure through acceptance and include review and rework by every role.

Failure mode: denominator deletion

Abandoned, escalated, or failed assisted cases disappear from the dataset.

Control: assign every eligible unit a durable identifier and terminal state. Report exclusions by condition.

Failure mode: task cherry-picking

Users choose the easiest work for the assisted condition.

Control: randomize eligible units or match on pre-treatment complexity. Log overrides.

Failure mode: moving quality standards

Reviewers accept lower quality to preserve speed gains.

Control: freeze the rubric and quality floor before treatment; audit a blinded sample.

Failure mode: contamination

Control participants use the tool informally, or assisted knowledge changes how control tasks are done.

Control: measure actual exposure, shorten crossover periods where appropriate, and interpret estimates accordingly.

Failure mode: version drift

The model, prompt, retrieval index, or policy changes during the evaluation.

Control: version every component and analyze material versions separately.

Failure mode: novelty theater

Extra attention, training, and executive support accompany the AI condition but not the control.

Control: document complementary interventions and provide comparable process support where feasible.

Failure mode: proxy victory

Activity rises but customer, risk, or financial outcomes do not improve.

Control: connect operational measures to at least one downstream outcome and state the attribution limit.

Failure mode: averaged harm

A large number of small gains hides a rare severe event.

Control: report severity counts and stop rules separately from the average productivity metric.

A six-week evaluation cadence

Week 1: map the workflow

Observe current work, handoffs, queues, rework, and acceptance. Define eligible units and sensitive exclusions.

Week 2: freeze the protocol

Write the decision card, assignment method, metrics, rubric, observation window, sample plan, and stop rules. Validate event capture.

Week 3: shadow and rehearse

Run the system without operational effect. Test whether reviewers can reproduce labels and whether logs capture all labor.

Weeks 4 and 5: controlled operation

Run the intervention within the approved boundary. Monitor safety stops but do not repeatedly inspect outcome differences and change the protocol.

Week 6: analyze and decide

Report the declared primary measure, guardrails, uncertainty, exclusions, segments, and downstream observations. Choose adopt, revise, extend, or stop. Record which evidence would change the decision.

If the work cycle is monthly or outcomes arrive slowly, extend the calendar. The point is not six weeks; it is separating preparation, operation, and decision.

Validation checklist

Before accepting a productivity claim, verify:

• [ ] The decision supported by the study is written clearly.
• [ ] The unit of work has stable start, acceptance, and rework boundaries.
• [ ] Eligibility and exclusions were defined before results.
• [ ] The comparison method supports the causal language used.
• [ ] Work complexity was randomized, matched, or explicitly adjusted.
• [ ] All eligible units have durable identifiers and terminal states.
• [ ] Operator, reviewer, and downstream rework time are included.
• [ ] The quality rubric and hard floor are identical across conditions.
• [ ] Reviewers are blinded where feasible.
• [ ] Severe errors are reported separately from average quality.
• [ ] Missing data and protocol deviations are disclosed by condition.
• [ ] Results are segmented only along predeclared operational dimensions.
• [ ] Model, workflow, evaluator, and policy versions are stored.
• [ ] Learning, persistence, and skill retention are considered.
• [ ] Full costs and the value-realization mechanism are explicit.
• [ ] Potential estimates are not presented as realized local returns.
• [ ] Stop rules and decision ownership are documented.

What a trustworthy conclusion sounds like

A trustworthy conclusion is bounded. It might say:

For eligible standardized internal memos during three monthly cycles, the assisted workflow increased accepted packages per total labor-hour, while first-pass acceptance remained above the predeclared floor. The estimate does not cover novel transactions or external disclosures. Review time rose, one high-severity unsupported claim triggered a workflow revision, and the next evaluation will test the revised evidence check.

That statement is less exciting than “AI made finance forty percent more productive.” It is also far more useful. It identifies the population, measure, duration, quality boundary, failure, and next decision.

The goal of productivity measurement is not to make an AI investment look successful. It is to discover where a controlled intervention creates more accepted value, where it merely moves work, where it changes risk, and where the organization should abstain.

Stop timing the model. Measure the whole work system.