DEV Community: KuoBa

I measured my agent's confidence. It was a constant.

KuoBa — Mon, 31 Aug 2026 15:13:51 +0000

Hackathon disclosure: I created this article for the purpose of entering
Fleet in the All Things Agentic Hackathon.

Fleet is a Slack-first dispatcher for a workforce made up of people and
specialized agents. It decides who should do each task, how much human review is
required, and when software should not attempt the work at all.

Try Fleet: https://fleet-jvmesmce2q-uc.a.run.app
Source code: https://github.com/KCL1104/fleet

The confidence problem

The design started with a nice idea: ask the model how sure it is and route the
work accordingly. Confident classifications go to an agent; uncertain ones go
to a person. Every agent framework makes this easy—ask for a confidence field
in the schema and a number comes back.

A number comes back. That is the problem.

Here is what four runs of the same customer email produced on Gemini 3.7 Flash,
at temperature 0, across two thinking levels:

LOW    run 1   feature flag 0.95   reconciliation 0.98   refund 0.95
LOW    run 2   feature flag 0.95   reconciliation 0.95   refund 0.95
MEDIUM run 1   feature flag 0.95   reconciliation 0.98   refund 0.95
MEDIUM run 2   feature flag 0.95   reconciliation 0.95   refund 0.95

Those requests have genuinely different consequences. One is a feature-flag
action whose arguments are present in the account snapshot. One is a document
that requires a billing period to be inferred. One is a refund—a decision about
money that software should never make here.

The result was almost always 0.95.

I do not think the model was lying. I think it was answering a question it had
no evidence to answer. It had never seen its own reconciliation drafts return
from a human reviewer, so it had no historical success rate to report. The
confidence value was still useful as one signal, but it was not a probability I
could safely use on its own.

So I stopped letting it have the last word.

What Fleet does

Fleet receives work where teams already communicate: Slack. A message shortcut,
/fleet, or an app mention can create a request. Fleet separates a multi-part
message into discrete tasks, classifies each task into a fixed taxonomy, assigns
a risk tier, and sends it down one of three paths:

Tier 0: an agent may complete low-risk work unattended.
Tier 1: an agent may prepare the work, but a person must review it.
Tier 2: software never attempts the task.

Interactive Slack cards and a persistent App Home let authorized teammates
review, send back, delegate, or complete work without adopting another inbox.
The Fleet web application is the control plane: it exposes the work queue,
routing evidence, workforce calibration, activity history, automations, and
connection health.

In the reproducible demo, one customer message asks Fleet to enable an export
feature, prepare a reconciliation document, and issue a refund. The first task
goes to an isolated Operator. The second is drafted by an Analyst and returned
for human review. The refund stays with a person and is never attempted by
software.

Routing to a person is not an agent failure. It is one of the system's intended
outputs.

The number that is not a constant

Fleet keeps a small table for each (task class, worker) pairing: how many times
a human approved that worker's output, and how many times they sent it back.

empirical_rate = (approvals + 2) / (approvals + send_backs + 3)
effective_confidence = model_confidence * empirical_rate

The model assessment is still present. It is multiplied by a rate grounded in
actual outcomes. An Analyst's record for reconciliation work does not affect a
Scribe's record for customer updates, and neither affects an Operator's record
for back-office actions.

The +2 / +3 Beta prior also creates a useful cold-start property. A new pairing
begins at 2/3 = 0.667. The Tier 1 automation threshold is 0.75, so clearing it
on the first attempt would require a model confidence greater than
0.75 / 0.667 = 1.12.

That number cannot exist. The first unfamiliar, higher-risk task therefore goes
to a person. The agent becomes eligible only after supervised attempts create a
real record.

The send-back is the signal

Human review is part of Fleet's routing loop, not a separate feedback form.

An approval records a successful outcome. A send-back lowers the empirical rate
for that worker-task pairing and stores the reviewer's correction for later
attempts. Slack then displays a Learning Receipt showing the before-and-after
track record, the calculation, the applicable threshold, and whether the agent
remains eligible next time.

before send-back: 0.98 x 0.83 = 0.82   threshold 0.75   -> Analyst eligible
after send-back:  0.98 x 0.71 = 0.70   threshold 0.75   -> route to a person

The request did not change. Fleet's evidence about its workforce did.

This also exposed a second problem: automatic demotion without a promotion path
would eventually make the product route everything to people. Fleet therefore
lets an authorized person delegate an eligible Tier 0 or Tier 1 task to an agent
under the same review boundary. Delegation alone does not improve the agent's
record. Only a later verdict on real agent output can do that. Tier 2 remains
human-only even if someone tries to override it.

A second reader, not a second vote

Fleet uses Gemini 3.7 Flash through Google's GenAI SDK on Vertex AI for
structured task separation, classification, and schema-constrained argument
extraction.

Gemma 4 26B independently reads each classified request against the same
catalogue and account snapshot. It is not a cheaper first pass, and agreement
does not inflate the score. When the models disagree on what a request is,
Fleet halves the first model's confidence before deterministic routing.

The lowest automation threshold is 0.55. Half of any model confidence is at most
0.50, and the empirical rate is always below 1. A contradicted classification
therefore cannot clear any automation threshold, regardless of prior success.

That matters for requests like:

"Any update on when the export feature will be working again?"

Gemini -> comms.status_update
Gemma  -> backoffice.feature_flag

Both readings are defensible. The customer may want information or an action.
Two competent readers disagreeing about the meaning is not a tie to break; it is
evidence that a person should decide.

What the models are not allowed to do

Three decisions stay outside the models:

Dispatch is arithmetic. Given the same task and track record, Fleet makes the same routing decision and can explain the exact calculation.
The task class selects the handler. A fixed taxonomy maps to typed code paths. Model-generated arguments must satisfy schemas built from the live system, so the model cannot invent a feature flag or action.
Business numbers are computed. Billing data is fetched and totaled in code. A model may write the covering note; it does not calculate the reconciliation.

The models contribute judgment where language is ambiguous. Deterministic code
owns policy, permissions, arithmetic, and the final routing decision.

How Fleet is built on Google Cloud

Fleet runs as four Cloud Run services with distinct responsibilities and trust
boundaries:

fleet-connectors verifies Slack signatures, acknowledges interactions quickly, and handles durable delivery. It has no model or business-system credentials.
fleet hosts the React/FastAPI control plane, intake pipeline, Gemini and Gemma calls, deterministic dispatcher, and general Analyst/Scribe workers.
fleet-operator is private and holds the only agent credential permitted to perform named write actions.
acme is a separate synthetic back-office service with its own identity, database, and per-action authorization.

Pub/Sub separates intake, human actions, general and isolated task lanes, Slack
notifications, retries, and dead-letter recovery. A filtered subscription makes
privileged Operator tasks reachable only by the isolated Operator service.

Firestore stores operational state, task decisions, calibration, corrections,
and audit evidence. A separate connector database holds replay claims and
delivery receipts, while the synthetic back office has its own database. Secret
Manager and service-specific IAM define capabilities. Model Armor screens
untrusted inbound text and approved outbound content. OpenTelemetry and Cloud
Trace join intake, routing, execution, and review under a shared trace ID.

The architectural goal was concrete: the component reading customer-authored
text should not also hold the credential that changes a business system.

What was harder than expected

Slack requires a fast acknowledgement, while two model calls and policy
evaluation can take much longer. Moving the work behind Pub/Sub solved the
latency problem, but introduced delivery concerns: replayed events, concurrent
button presses, stale task versions, retries, dead letters, and restarts. Fleet
uses idempotency claims, mutation leases, state versions, and durable receipts
so one human action produces one domain effect.

Calibration also had to follow the worker who actually produced the work. A
reassignment must not accidentally update the original assignee's record, and a
delegation must not count as a success before anyone judges the output.

Finally, timeouts and retries had to be designed as one budget. Stacked retries
can turn a graceful fallback into a ten-minute hang. Making those limits
explicit was less visible than another demo feature, but much more important for
a live system.

What I learned

The most useful memory in Fleet is not a transcript. It is operational evidence:
approvals, send-backs, concrete corrections, who actually performed the work,
and the context needed to apply that evidence to the next decision.

I also learned that an agent architecture should describe enforceable
capabilities, not character roles. Different names and prompts do not create
meaningful isolation. Different identities, credentials, message lanes,
databases, and failure boundaries do.

Fleet is a hackathon project, not a claim of production traction. But it
demonstrates the direction I believe agentic work systems should take: autonomy
that grows from evidence, remains inspectable, and can shrink again when the
evidence changes.

One prompt in, a finished film out: running an entire AI video pipeline on Qwen Cloud

KuoBa — Mon, 20 Jul 2026 13:07:40 +0000

I've been building extrovid — an AI-native director and editor. You give it one line of text, and it does the rest: it writes the brief and script, casts a consistent cast, develops a look, boards the shots, generates and reviews the video, adds voiceover, and hands you an edited rough cut. One prompt to a finished short, with a director you can talk to at every step.

The interesting part isn't any single model. It's that every creative decision in the pipeline is made by a model on Qwen Cloud (Alibaba DashScope) — the LLM that writes the script, the model that draws each frame, the voice that narrates, and the model that renders the video. This post is a tour of how those pieces fit together.

The whole crew is a Qwen model

extrovid is built like a film crew, and every role is played by a model on Qwen Cloud:

Job on set	Model on Qwen Cloud
Writing the script	`qwen3.7-max` — the flagship, for the one output that carries the whole film
Every other agent (brief, cast, look-dev, storyboard, director, review)	`qwen3.7-plus` — cheaper, huge context, used everywhere else
Concept frames, storyboard keyframes, cast portraits	`wan2.7-image-pro` (up to 4K)
Voiceover	`qwen3-tts`
Rendering the shots	HappyHorse or Wan 2.7 (text-, image-, and reference-to-video)

Six model families, four modalities — text, image, voice, and video — all reached through one API key and one endpoint. That single-vendor coherence turned out to be a real advantage: no juggling five providers, five billing accounts, and five sets of quirks. One key, one place to reason about cost, one place to reason about latency.

Talking to Qwen: one endpoint, typed answers

DashScope exposes an OpenAI-compatible endpoint, which means the planning agents can speak to Qwen through the same tooling the rest of the ecosystem already uses. On top of that, extrovid uses a typed agent framework, so the models don't just return prose — they return structured, validated data: a script with numbered scenes, a cast list with consistent character descriptions, a storyboard as machine-readable shots. The brief becomes the script, the script becomes the cast, the cast becomes portraits — each stage's structured output feeds the next.

That structure is what lets the pipeline be a real pipeline instead of a pile of chat prompts. And it's why the planning phase can stream back to the UI token by token, stage by stage, so you watch the film get planned in real time rather than staring at a spinner.

One integration lesson worth passing on: Qwen3 models run in a "thinking mode" by default that's excellent for reasoning, but it conflicts with the strict "you must return this exact schema" mode that structured-output frameworks rely on. Turning thinking mode off for the planning agents made their output deterministic and reliable. If you're getting Qwen to emit strict JSON or tool calls, that's the knob to reach for.

Images and voice

Once the plan exists, it needs to become something you can see and hear. wan2.7-image-pro draws the visual world — cast portraits so a character has a face, look-development frames so the film has a mood, and a keyframe for every shot in the storyboard. Because the Wan 2.7 image family handles both generation and editing, refining a frame later is the same kind of call, not a bolt-on service. Voiceover comes from qwen3-tts, one narration line per shot.

Video: the slow, interesting part

Video is where the integration gets genuinely interesting, because rendering a shot isn't instant — it takes minutes. Qwen Cloud handles this the right way: it's asynchronous. You submit a shot, get a ticket back immediately, and check on it until it's ready. That shape influences the whole backend, which has to track jobs in flight, notice when they finish, and stream live progress to the UI as each shot lands.

A few things I'm especially happy with here:

Two video models, one path. extrovid can render on HappyHorse (an Alibaba model that currently ranks #1 on the Artificial Analysis Video Arena, with native audio and multi-language lip-sync) or fall back to Wan 2.7 — a single config switch, because both live on the same Qwen Cloud transport. No second integration.
Best-of-N with an AI "dailies" review. Each shot is rendered several times, and a Qwen model reviews the takes and picks the winner automatically — the way a director watches dailies and chooses. You see it happen as a little status note: "picked best of 3." The machine owns the quality-control work, not just the generation.
Continuity that actually holds. Each shot is seeded with the previous shot's final frame and the cast portraits, so a character's face and the film's look carry across clips that were generated independently. Continuity turns out to be an architecture problem, not a prompting one.

The one hard-won operational lesson: asynchronous results don't wait for you forever. A finished video's download link expires, so the backend has to fetch it and re-host it in your own storage promptly — otherwise you've paid to generate something you can no longer retrieve. Planning for "fetch and keep" from the start saved a lot of pain.

The trick that made all of this pleasant to build

Every model in extrovid sits behind a provider seam — a thin boundary where a single setting decides whether a call hits real Qwen Cloud or a fast, deterministic offline stand-in. Flip one flag and the exact same pipeline runs with no key, no network, and no cost.

This one decision paid for itself over and over:

The whole thing is testable offline. The full idea-to-cut pipeline runs in tests with zero spend, because every model has an offline counterpart behind the same boundary.
Iterating is free. Image and video generation are billable; developing against the offline stand-ins (plus per-user daily caps in production) keeps costs bounded until you actually want pixels.
Going live is a config change, not a rewrite. The offline and real providers are interchangeable, so switching to production Qwen Cloud is flipping a flag and adding a key.

If you take one thing from this post, let it be that: when you build on a paid, multi-modal cloud, build the seam that lets you also run without it. It's the cheapest thing you'll build and the one that lets you move fastest.

Wrapping up

extrovid is an attempt to close the gap between "a model can make a shot" and "a tool can make a film" — the brief, the casting, the continuity, the take selection, the cut. Qwen Cloud made that feasible for a small project: one vendor covering text, image, voice, and video, reachable through one key, coherent enough that a single person could wire the whole crew together.

If you're building anything multi-modal for this hackathon, Qwen Cloud is a genuinely strong foundation to build the whole pipeline on — not just one piece of it.

Built with Qwen Cloud / Alibaba DashScope end to end.

Build with MeDo Hackathon Journey

KuoBa — Sat, 25 Apr 2026 18:25:49 +0000

I'm started to work on Build with MeDo Hackathon! I've tried many similar platform before and I'll share my experience in using MeDo compare with other platform during the Hackathon!