DEV Community: yidao

We Split One Research Loop Across Three Agents. The Handoffs Became the Work.

yidao — Mon, 07 Sep 2026 07:26:32 +0000

I recently tried to turn a difficult research workflow into a small AI team.

The task involved improving an SVG reconstruction system. A researcher would inspect failures and propose a cause. An engineer would change the implementation. An evaluator would examine the result and decide whether another round was needed.

On paper, the roles looked sensible:

Researcher -> Engineer -> Evaluator

In practice, the team became very good at producing handoffs.

Six execution and handoff files grew to 12,757 lines. Over two days, five core workflow files received 125 commits; 116 of those commits repaired orchestration, bindings, evaluation state, or other workflow mechanics rather than advancing the reconstruction method itself. One version of the workflow called the full Researcher 47 times and spent 12,813 seconds doing so. We could identify only one call that produced a useful new algorithmic conclusion.

Those numbers come from one internal experiment, not a general benchmark. But they made the failure mode hard to ignore: the system was spending more effort preserving the appearance of a team than preserving the reasoning needed to solve the problem.

The task was one loop, not three jobs

The mistake was easiest to see after we stopped looking at the agent roles and looked at the information flow.

The real loop was:

observe an output
-> form a causal hypothesis
-> change the algorithm
-> inspect the new output
-> revise the hypothesis

Each step depends heavily on the previous one. A small visual detail can change the explanation. A code change can reveal that the original diagnosis was wrong. The next useful action often depends on something that is obvious while looking at the output but awkward to encode in a handoff document.

We had divided the loop according to familiar human job titles. We had not divided it according to independent information boundaries.

That distinction mattered. The Researcher compressed an observation into a report. The Engineer had to reconstruct the observation from that report before making a change. The Evaluator then reconstructed both the hypothesis and the intent of the change from another bundle of files. Every transition discarded context and introduced another opportunity for the agents to interpret the same state differently.

The reports kept getting longer because we tried to solve information loss by adding more information. That only made the contracts more expensive to produce and validate.

We used language models for compiler work

A second problem was more mundane.

Agents were repeatedly asked to maintain case identifiers, file paths, manifests, hashes, schema fields, evaluation bindings, and status records. These are exact transformations with exact failure conditions. They belong in ordinary code.

Instead, the system sometimes called an expensive reasoning agent to discover that a path was stale or a required field was missing. The agent would repair the record, explain the repair, and pass the corrected record to the next agent. The workflow looked active, but no research had happened.

Once we separated the calls by purpose, many of the so-called research iterations turned out to be handoff repair. A useful rule emerged:

If a machine can verify the answer exactly, do not make an agent negotiate it through prose.

We moved IDs, paths, hashes, schemas, manifests, and basic evaluation contracts into deterministic checks. Agent time was reserved for interpreting evidence and choosing what to try next.

More agents were maintaining different versions of reality

The third problem was state.

Research work changes its own premises. After each experiment, the best explanation of the defect may change. In our design, several agents were operating on compressed snapshots of that moving state. None of them had quite the same picture, and each handoff could lag behind the actual implementation.

This resembles what Cognition describes in its discussion of multi-agent coding: parallel writers make implicit choices that can conflict, while context does not naturally move between agents. Their useful pattern is closer to one writer receiving intelligence from other agents, followed by an independent review.

The comparison helped us see that our problem was not insufficient communication. It was shared decision authority over one changing implementation.

The replacement was deliberately less theatrical

We replaced the role-based team with one continuous owner of the research loop:

Research Lead
├─ inspect the real output
├─ form and revise the causal model
├─ implement one candidate
└─ optionally open two isolated hypothesis lanes
             |
             v
deterministic checks + independent evaluator
             |
             v
human perceptual review

The Research Lead is the only decision owner and the only writer for the active implementation. It keeps observation, hypothesis, implementation, and immediate inspection in the same working context.

This is not an argument for one enormous prompt or an agent that runs forever. The context has a lifecycle. It stays continuous while investigating one defect or one causal question. Once that question is resolved, rejected, or blocked, we archive the evidence and begin the next investigation with a cleaner context.

We still use additional agents, but their authority is narrower.

A read-only agent can search documentation or inspect a separate part of the repository. If two causal explanations can be tested independently, we can open two isolated lanes and compare their evidence. An evaluator receives a frozen target, the candidate output, the diff, and a rubric, but not the Lead's reasoning history. It can reject a candidate or report uncertainty; it cannot silently rewrite the implementation.

The final visual judgment remains human. A deterministic evaluator can detect empty output, structural damage, regressions, or missed constraints. It cannot reliably declare that a reconstruction simply looks right.

Multi-agent systems work when the work is actually parallel

This experiment did not convince me that multi-agent systems are useless. It made the boundary clearer.

Anthropic reports that its multi-agent Research system outperformed a single agent by 90.2% on an internal research evaluation. The example is breadth-first search: multiple subagents can investigate independent directions at the same time. Anthropic also notes that the architecture consumes substantially more tokens, so the extra coordination needs to buy something real. That is a good fit for parallel discovery, not necessarily for a tightly coupled implementation loop. The full account is worth reading in How we built our multi-agent research system.

A controlled study across 180 agent configurations found a similar task dependency. Centralized coordination improved performance on parallelizable tasks, while every tested multi-agent variant degraded performance on sequential reasoning tasks. The paper, Towards a Science of Scaling Agent Systems, is useful because it treats topology as a property to match to the task rather than a maturity ladder where more agents must be better.

OpenAI's practical guide makes a simpler recommendation: maximize a single agent's capabilities first, then split when complex logic or tool selection actually requires it.

That is now our default. Parallelize across independent repositories, pages, migrations, searches, audits, or falsifiable hypotheses. Keep writes single-threaded when several steps depend on the same evolving mental model.

Measure conclusions, not activity

The original workflow generated plenty of visible motion: calls, reports, commits, validations, and candidate outputs. None of those was a good measure of research progress.

For the next comparison, we care about a smaller set of outcomes:

time to the first human-validated improvement
cost per useful causal conclusion
percentage of agent calls spent repairing handoffs
workflow failure rate
quality improvement under an equal token and time budget

Throughput still matters when tasks are independent. In a tightly coupled research loop, it can be actively misleading. Ten candidates produced from the same broken causal model are not ten times the progress.

The main lesson was not that we needed a better manager agent, a stricter schema, or another reviewer. We had placed parallelism at the level of roles, where the information dependencies were strongest.

The better design was to keep one owner for the changing explanation and move parallelism outward, toward work that could genuinely be isolated.

*AI assistance disclosure: AI tools assisted with drafting and editing this article. The experiment data, architecture decisions, and final review are the author's.

Don't Send Every Audio File Straight to the AI Model

yidao — Tue, 01 Sep 2026 04:01:28 +0000

An audio file can pass every technical checkpoint and still produce a bad AI result.

The upload completes. The decoder opens it. The model returns a response. The encoder writes a playable file. From the system's point of view, the job succeeded.

Then the user presses play and hears a voice that is still too quiet, missing pieces, or buried under artifacts.

It is tempting to blame the model immediately. Sometimes that is fair. But many bad jobs begin earlier, when the product treats every decodable file as a valid model input.

A recording can be almost silent, heavily clipped, mostly empty, unexpectedly multichannel, or far outside the conditions a model was trained to handle. Sending all of those through the same pipeline with the same settings turns predictable input problems into mysterious model failures.

I prefer to put a small policy layer in front of inference:

upload -> probe -> inspect -> route -> process -> validate -> preview

The goal is not to diagnose audio perfectly. It is to catch obvious problems, choose a safer path when possible, and avoid claiming success when the result cannot be trusted.

A container is not the signal

Start with the inexpensive facts. Does the file contain an audio stream? What codec, duration, sample rate, channel count, and channel layout did the decoder find?

ffprobe can return that metadata as JSON without decoding the whole signal:

ffprobe -v error \
  -select_streams a:0 \
  -show_entries stream=codec_name,sample_rate,channels,channel_layout:format=duration \
  -of json \
  input.mp4

The ffprobe documentation covers -show_entries, stream selection, and its JSON writer in more detail.

These checks catch more than malformed uploads. They also stop quiet assumptions from spreading through the pipeline. A speech model expecting mono audio should not discover a six-channel layout halfway through inference. Blindly downmixing is not always harmless either: channels can contain different microphones, or partially cancel when combined.

Metadata still cannot tell us whether the file contains useful audio. For that, inspect at least a bounded portion of the decoded signal. Useful first-pass measurements include:

peak and RMS level
long silent regions
samples at or near full scale
active audio or speech ratio
channel imbalance

FFmpeg's astats and silencedetect filters are enough for a basic server-side pass:

ffmpeg -hide_banner -i input.mp4 \
  -af "astats=metadata=1:reset=0,silencedetect=n=-50dB:d=1" \
  -f null -

The silence level and duration above are examples, not standards. A whispered interview, a screen recording, and a field recording should not share thresholds just because they are all audio files. For speech products, a voice activity detector is also more useful than treating every non-silent sound as speech.

Low level is a routing decision

One especially awkward case is a recording whose digital level is extremely low.

The obvious fix is to normalize it before inference. That can help place the signal inside a model's expected operating range, but it does not create information that was never captured. Raising gain lifts the voice and the noise floor together. It does not improve signal-to-noise ratio, repair a poor microphone position, or restore detail lost during recording.

This is why I would not normalize every upload to the same target. First decide whether the input is unusually low for the pipeline. If it is, apply bounded gain with headroom, then run the model. If the recording is already clipped or contains almost no usable speech, gain is the wrong intervention.

While working on CleanAudio, this distinction became important: very low-level inputs may need a calibration step, but that is input conditioning, not a promise that every result should have the same loudness.

The thresholds belong to the model and the use case. They should come from tested failures, not from numbers copied out of a mastering guide.

Turn measurements into a plan

I find it more useful to produce a processing plan than a single valid boolean. A file can be valid enough to decode but still deserve conservative processing or manual review.

Here is a simplified policy in TypeScript:

interface AudioInspection {
  decodable: boolean;
  audioStreamCount: number;
  durationSeconds: number;
  peakDbfs: number;
  rmsDbfs: number;
  clippedSampleRatio: number;
  activeAudioRatio: number;
}

interface AudioPolicy {
  minDurationSeconds: number;
  maxDurationSeconds: number;
  minActiveAudioRatio: number;
  maxClippedSampleRatio: number;
  calibrateBelowDbfs: number;
  targetInputRmsDbfs: number;
  maxCalibrationGainDb: number;
  maxAllowedPeakDbfs: number;
}

interface ProcessingPlan {
  disposition: "process" | "review" | "reject";
  mode: "standard" | "conservative";
  preGainDb: number;
  reasons: string[];
}

function planAudio(
  input: AudioInspection,
  policy: AudioPolicy,
): ProcessingPlan {
  const reasons: string[] = [];

  if (!input.decodable || input.audioStreamCount === 0) {
    return {
      disposition: "reject",
      mode: "standard",
      preGainDb: 0,
      reasons: ["No decodable audio stream"],
    };
  }

  if (
    input.durationSeconds < policy.minDurationSeconds ||
    input.durationSeconds > policy.maxDurationSeconds
  ) {
    return {
      disposition: "reject",
      mode: "standard",
      preGainDb: 0,
      reasons: ["Duration is outside the supported range"],
    };
  }

  if (input.activeAudioRatio < policy.minActiveAudioRatio) {
    return {
      disposition: "review",
      mode: "conservative",
      preGainDb: 0,
      reasons: ["Too little active audio was detected"],
    };
  }

  let mode: ProcessingPlan["mode"] = "standard";

  if (input.clippedSampleRatio > policy.maxClippedSampleRatio) {
    mode = "conservative";
    reasons.push("Input contains substantial clipping");
  }

  let preGainDb = 0;

  if (input.rmsDbfs < policy.calibrateBelowDbfs) {
    const gainTowardTarget = policy.targetInputRmsDbfs - input.rmsDbfs;
    const availableHeadroom = policy.maxAllowedPeakDbfs - input.peakDbfs;

    preGainDb = Math.max(
      0,
      Math.min(
        gainTowardTarget,
        availableHeadroom,
        policy.maxCalibrationGainDb,
      ),
    );

    if (preGainDb > 0) reasons.push("Low-level input needs bounded gain");
  }

  return {
    disposition: "process",
    mode,
    preGainDb,
    reasons,
  };
}

There are deliberately no magic values in this example. A threshold is part of the product policy, not a universal property of audio. Version it alongside the model, record which route each job takes, and review the files clustered near a boundary.

This policy layer also makes failures easier to explain. "We could not detect enough usable audio" is more actionable than "processing failed." A conservative route can preserve more of the original signal instead of applying the strongest available effect to a risky input.

The output needs a postflight check

A successful inference request only proves that inference ran. Before offering the result, decode it again and check the boring invariants:

expected duration and timeline were preserved
output is not silent or unexpectedly clipped
sample rate and channel mapping are supported
the encoder produced a complete, playable file
any known model or codec delay was compensated

Compare those measurements with the input, not only with fixed limits. A large, unexplained duration change is suspicious even when both files are individually valid.

These checks catch catastrophic failures. They cannot tell whether consonants were softened, room tone started pumping, or a voice became less natural. That still requires listening. In a previous article, I described a synchronized A/B preview for making that comparison without restarting two separate players.

The distinction matters: automated validation protects the pipeline, while the preview protects the user's judgment.

Let the product admit uncertainty

AI interfaces often compress several states into one green message: Enhancement complete.

But there is a meaningful difference between a model completing, a file passing structural checks, and a person deciding that the result is better. Treating them as the same event makes the product sound more certain than the system actually is.

The original file should remain available. Risky inputs should get a clear explanation. A result that passes only basic checks should still be presented as something to review, not as an unquestionable improvement.

The model is only one stage in the feature. A small amount of inspection before and after it can prevent predictable failures, make routing decisions visible, and give the user a more honest result.

Sometimes the best AI processing decision is to do less. Sometimes it is not to run the model at all.

Disclosure: I am involved in building CleanAudio, an AI audio and video noise-removal tool. The implementation pattern and opinions in this article are presented as general product and engineering guidance.

AI assistance disclosure: AI tools assisted with drafting and editing this article. The technical direction, product observations, and final review are the author's.

How to Build a Fair A/B Audio Preview for AI Processing

yidao — Mon, 24 Aug 2026 09:32:34 +0000

Two audio players do not make a fair before-and-after test.

If the second player restarts from zero or takes half a second to load, the user is no longer comparing two versions of the same moment. They are comparing two memories.

That is a weak way to evaluate any audio effect. It is especially weak for AI processing.

A denoiser can remove a fan while softening consonants. A de-reverb model can reduce the room tail while making the voice sound less natural. The output may be cleaner without being better.

The preview therefore has one job: let the listener switch quickly enough to hear both the improvement and the damage.

The rule I use is deliberately boring. Both versions should contain the same edit and play from the same position. Switching should not restart playback or create a pause. The interface should not hint that one version is supposed to win.

Two independent <audio> elements fail surprisingly quickly. Each owns its playback state, buffering behavior, clock, and seek operation. The user ends up finding the same position twice and comparing one sound with a memory of another.

A better interface has one transport and one version control:

[ Play ]  [ Original | Processed ]  00:18 ━━━━━━━ 00:42

The transport decides where playback happens. The segmented control decides which signal is audible.

One transport, two signals

For a short preview, I decode both files into AudioBuffers, start them at the same AudioContext time and offset, and route each through its own GainNode. Both sources run; only one gain is open.

decodeAudioData() decodes complete file data and resamples it to the context's sample rate. The decoded buffers can then share the same audio clock. See the MDN documentation for format and loading details.

The core is small:

const context = new AudioContext();
const originalGain = context.createGain();
const processedGain = context.createGain();

originalGain.connect(context.destination);
processedGain.connect(context.destination);

async function loadBuffer(url) {
  const response = await fetch(url);
  return context.decodeAudioData(await response.arrayBuffer());
}

const buffers = {
  original: await loadBuffer("/audio/original.wav"),
  processed: await loadBuffer("/audio/processed.wav"),
};

let sources = [];

function startPair(fromSeconds = 0) {
  const when = context.currentTime + 0.03;
  const duration = Math.min(
    buffers.original.duration,
    buffers.processed.duration,
  );

  stopPair();
  sources = Object.entries(buffers).map(([name, buffer]) => {
    const source = context.createBufferSource();
    source.buffer = buffer;
    source.connect(name === "original" ? originalGain : processedGain);
    source.start(when, fromSeconds, duration - fromSeconds);
    return source;
  });

  setAudibleVersion(activeVersion, when, 0);
}

function stopPair() {
  for (const source of sources) source.stop();
  sources = [];
}

Call context.resume() and startPair() from a click or tap handler. Track the elapsed offset in your transport; pause and seek should stop the pair and create new source nodes at that offset. An AudioBufferSourceNode is intentionally single-use, while its decoded AudioBuffer can be reused. MDN explains the lifecycle here.

The switch itself can lie

An instant gain jump can click when it lands away from a zero crossing. A long crossfade hides the click but makes the comparison less precise because the listener hears a blend. For speech, I start with a ramp of a few milliseconds and tune it on real material.

const versionGains = {
  original: originalGain,
  processed: processedGain,
};

let activeVersion = "original";

function setAudibleVersion(version, now = context.currentTime, ramp = 0.012) {
  activeVersion = version;

  for (const [name, node] of Object.entries(versionGains)) {
    const target = name === version ? 1 : 0;

    node.gain.cancelScheduledValues(now);

    if (ramp === 0) {
      node.gain.setValueAtTime(target, now);
      continue;
    }

    node.gain.setValueAtTime(node.gain.value, now);
    node.gain.linearRampToValueAtTime(target, now + ramp);
  }
}

document.querySelector("#show-original").addEventListener("click", () => {
  setAudibleVersion("original");
});

document.querySelector("#show-processed").addEventListener("click", () => {
  setAudibleVersion("processed");
});

The 12 ms value is a starting point, not a standard. Listen for clicks, comb filtering during the overlap, and any sense that the button responds late. If the files are not sample-aligned, even a short overlap can sound strange.

The player cannot repair a misaligned file

Starting both buffers at the same Web Audio time only works if the files contain corresponding samples at corresponding positions.

An AI pipeline may introduce delay through frame padding, look-ahead context, silence trimming, resampling, or codec priming.

If the processed voice begins 80 ms later, the browser will faithfully play the wrong moments together.

Fix alignment before the files reach the comparison UI. Preserve the original timeline through processing, or estimate the delay against a reference and compensate by trimming or padding. Keep the preview lengths identical; do not quietly loop one version after the other ends.

Test this with a transient: switch repeatedly and confirm that its position does not move.

Most of the file does not matter

Most users do not need to compare an entire 40-minute recording. They need to inspect the four seconds where a keyboard overlaps a sentence or traffic passes behind a voice.

A shared timeline and a short loop region are more useful than two large players. Keep the A/B controls keyboard-accessible, show which version is audible, and do not stop the loop when the listener switches.

The waveform is useful for navigation, but it is not evidence of quality. A smoother waveform does not prove that speech sounds more natural. Avoid using visual cleanup as a substitute for listening.

This also keeps the buffer-based approach within its natural limits. Decoded PCM is much larger than MP3 or AAC, and the design holds two versions in memory. decodeAudioData() expects a complete file, too. For a long recording, generate a bounded preview, decode chunks, or move to a more deliberate streaming strategy.

On the browser side, expect autoplay blocking, mobile interruptions, codec failures, and aggressive toggling. Resume the AudioContext from a user gesture and make decode failure an ordinary UI state. MDN's autoplay guide is worth keeping nearby.

What changes when the source is video?

Do not switch between two video players. Keep one video element on screen and treat its timeline as the master clock. Mute its embedded audio, then audition aligned original and processed audio tracks through the A/B graph.

The video should drive playback, pause, seek, and rate-change events. After a seek, stop both audio sources and recreate them from video.currentTime. If playback speed changes, apply the same playbackRate to both new sources.

There is an important limit: HTMLMediaElement.currentTime and AudioContext.currentTime are different clocks. Listening for events does not guarantee sample-accurate sync across browsers. Check drift during playback and rebuild both audio sources at video.currentTime when it exceeds your tolerance. For short speech previews, this is often sufficient. Frame-accurate editing needs a shared media timeline built with a more controlled pipeline such as MSE or WebCodecs.

The processing backend must still preserve duration and compensate for model and codec delay. A perfect front-end clock cannot align files whose content starts at different offsets.

The preview is part of error handling

An AI tool naturally wants to say "enhancement complete" and move on to download. But the processed version may have less noise and worse speech. Both can be true.

I prefer plain labels such as Original and Processed, with neither option visually dominant. The preview is not decoration after a successful model run. It is how the user catches a bad run before committing to it.

While working on CleanAudio, this became a useful product principle for me: when AI changes a user's media, confidence should come from comparison, not from the success message.

Before shipping, I listen for the unglamorous failures: a transient that moves when I switch, a click at the boundary, drift after seeking a video, or a phone interruption that leaves the controls lying about playback.

The code is the smaller part. The harder part is resisting all the little ways a comparison can be made to favor the result we want users to choose.