We let Qwen rewrite our scoring algorithm — but only through a clinical-style gate

State Of the Art TECHnology — Mon, 20 Jul 2026 12:49:09 +0000

TL;DR — We built a focus-measurement system whose scoring algorithm improves itself. Qwen3.7-Max reads the mistake log and proposes which parameters to change. Our server then refuses to believe a single number it says: it re-runs the simulation itself, and any claim off by more than 0.5 percentage points is thrown out. What survives goes through a train/holdout gate. What survives that waits for a human to click adopt. In our first Qwen-driven generation, holdout blank-stare sensitivity went from 0.273 to 1.000 with zero regressions — and Qwen deliberately avoided a value a previous generation had already failed with.

The problem nobody wants to solve with a camera

Every parent and teacher wants the same answer: is this student actually focusing, or just sitting in front of a book?
A camera can tell you. That's the easy part. The hard part is that nobody — correctly — wants a camera streaming their child's face to somebody's cloud.

So we drew a hard line in the architecture. Our device (a small NIR camera cube on the desk) processes video on-device and never stores or transmits it. What leaves the device is about 12,000–24,000 numeric records per session at 10 Hz: gaze coordinates, an on-page probability, pupil z-scores, blink events, PERCLOS. The cloud sees numbers. It never sees a face.

From those numbers we compute a Sustained Focus Index (SFI), 0–100. The scoring function is deliberately a pure function: score(records, param_set) → result. Same input, same parameters, same output. Always.

That purity is what makes the rest of this post possible.

Who's building this, and why

Concentration Cube is built by S-O-A TECH. Its founder chairs the Medical Committee of the Korean Olympic Committee and is an internationally recognized orthopedic surgeon — the physician who has treated and operated on most of Korea's elite athletes. Working at that level teaches you something quickly: at the top, every body is extraordinary. What separates a medal from fourth place isn't hours of training — it's the quality of focus. After watching that one scarce resource decide careers, year after year, the question becomes unavoidable: if focus is this decisive, why does nobody measure it properly?

A physician's reflex is to answer questions with measurements, not impressions. That reflex shaped everything downstream: measurement without surveillance (no video leaves the device), validation through an IRB process at a general hospital rather than around it, and a refusal to let any algorithm change go live without evidence. We didn't set out to build "an AI product". We set out to build a medical-grade answer to the question elite sport asks every day — and the only way to keep that answer honest at scale turned out to be a self-evolving engine behind clinical gates.

The actual hard problem: who tunes the parameters?

A scoring function like this lives or dies on its thresholds. When does low gaze dispersion mean deep focus versus a blank stare? How many saccades per second separates reading from zoning out?

We tuned these by hand. It was slow, and worse, it was unfalsifiable — we'd nudge a threshold, eyeball some sessions, and convince ourselves it was better.

The obvious 2026 answer is "let an LLM do it." We tried that, and immediately hit the thing everyone hits.

LLMs will happily lie to you about their own test results

Our first version ran a local coding-agent CLI in a sandboxed workspace. It could edit parameters, run our simulator, and report results.

It reported great results. Some of them were real. Some of them were the model telling us what we wanted to hear — numbers it had reasoned toward rather than measured. If you've given an agent a shell and then read the transcript carefully, you know exactly the feeling.

You cannot fix this with a better prompt. "Please don't hallucinate your test results" is not an engineering control.

So we stopped trying to make the model trustworthy, and made its trustworthiness irrelevant.

The hallucination firewall

Here is the whole idea, and it's almost embarrassingly simple:

The LLM is only allowed to decide which parameters to change. It is never allowed to tell us what happened as a result.

Concretely, in our Qwen adapter:

We hand Qwen the evidence — the mistake log (bins where the current parameters got the ground-truth label wrong), feature statistics, the current parameter set, and the history of previous generations.
Qwen returns a small JSON: a list of changes, each with a key, a target value, and its reasoning. That's it. No file access, no shell, no tools.
Our server applies those changes — filtered against an allowlist, clamped to valid bounds, with the key set preserved so nothing can be silently dropped.
Our server runs the simulation and computes the before/after numbers, using the same scoring core that production uses.
The validator compares what the agent claimed against what the server measured. Deviation over ±0.5 percentage points → the proposal is rejected outright.

Because the server computes the numbers with the same code and the same data, step 5 is not really "catching lies" anymore — it's a structural guarantee that the numbers in the record are the ones that actually happened.

Then: does it generalize, or did it just memorize?

Passing self-test isn't enough. A parameter change can fit the training sessions beautifully and fall apart on anything new.

So every candidate faces a train/holdout split. The mistake log the agent sees comes only from train sessions. The gate is measured on holdout sessions the proposal has never influenced.

The gate rule is deliberately conservative:

Every state's sensitivity and specificity on holdout must be within −2 percentage points of baseline, and at least one metric must actually improve.

No cherry-picking a win while quietly breaking something else.

And then a human clicks the button

Even after all of that, the system stops. A passing candidate sits in PASSED. Adoption happens when a person opens the report and clicks Adopt. The state machine has no path from PASSED to ADOPTED that doesn't go through a human.

This is not us being timid about AI. It's that this number gets shown to a parent about their child. Automation earns the right to propose. It doesn't get to decide.

Moving the brain to Qwen Cloud

Our CLI-based agent had a practical problem beyond hallucination: it authenticated through a browser OAuth session tied to the host machine. That meant the evolution engine could only ever run on a developer's laptop. It could not be deployed.

Switching to Qwen via Alibaba Cloud Model Studio's OpenAI-compatible endpoint removed that constraint entirely. An API key is an environment variable. An environment variable goes in a container. Suddenly the engine that had been stuck on one laptop could run on a server.

We ended up using Qwen in two distinct production roles:

Role	Model	Why
Proposer	`qwen3.7-max`	Parameter reasoning over statistical evidence — flagship reasoning matters here
Coach	`qwen3.7-plus`	Generates the student/parent report in Korean, English, and Chinese from a numeric summary containing no personal data

(We also keep qwen3.6-flash configured for smoke tests and debugging iterations.)

Splitting roles across models is a real design consideration, not an afterthought: Model Studio grants free quota per model, so separating roles across models separates their quota pools too. Role separation and cost control turned out to be the same decision.

What actually happened on the first Qwen generation

We seeded 18 synthetic LAB-5 sessions engineered so that blank staring would be misclassified as focus under the current parameters, then ran one generation.

Total elapsed: 2 minutes 10 seconds. (Our earlier CLI-agent rehearsal took 7.)

Qwen proposed two changes:

[{"key": "blank_stare.stare_dispersion_th", "from": 0.035, "to": 0.055},
 {"key": "blank_stare.max_saccade_count_1s", "from": 0.5,   "to": 0.6}]

Holdout results — five sessions the proposal never saw:

State	Sensitivity	Specificity
blank_stare	0.273 → 1.000	1.000 → 1.000
focus	1.000 → 1.000	0.619 → 1.000
off_task	1.000 → 1.000	1.000 → 1.000

Zero regressions. Gate passed.

But the part that made us stop and reread the log was the reasoning attached to the first change:

"…a value distinct from the previously rejected 0.06 / 0.062, to distribute holdout overfitting risk."

Qwen had read the generation history, found that an earlier agent had proposed 0.06 and had it rejected, and picked a different point in the separation gap specifically to avoid repeating that failure.

Nobody prompted it to do that. It was in the evidence, and it used it.

Why we call this a memory agent

There's a version of "agent memory" that means storing chat logs and retrieving them later. That's fine, but it isn't what's happening here.

Our agent's memory is a versioned lineage of scoring parameters, each attached to the evidence that justified it: the mistake log that motivated it, the server-computed before/after numbers, the gate verdict, whether a human adopted it, and whether it was later rolled back.

That's memory you can audit. When the SFI shown to a parent changes, you can walk backward through generations and see exactly which change caused it, what evidence motivated it, what it measured on data it hadn't seen, and who approved it.

Experience accumulating as evidence, not as transcript.

How this compares to everything else out there

The market is full of "focus apps" — webcam attention trackers, study timers with cameras, classroom monitoring tools. Almost all of them share two properties we consider disqualifying. First, they ship frames or video off the device — exactly what parents and regulators don't want pointed at children. Second, their scoring logic is frozen: fixed heuristics or a black-box model that interprets a 9-year-old and a 40-year-old with the same thresholds, forever.

Concentration Cube inverts both. Raw video never leaves the desk, and the interpretation itself evolves — against instructor-labeled clinical ground truth, through a holdout gate, with an auditable lineage and a human owning every adoption. We're not aware of another consumer focus product that combines on-device privacy, hospital-protocol ground truth, and a gated self-evolving interpreter. That combination is the moat — and it deepens daily, because every day of data makes the next generation better. As we like to put it: this device is at its least capable the day you buy it.

Where the daily evidence comes from

One thing that surprises people: the ground truth feeding this engine is not synthetic-only and not crowd-labeled. Our team runs an IRB-approved validation program at a general hospital — supervised, consented sessions, collected daily. Each session is an instructed experiment: an investigator tells the subject in real time — focus (actually read), blank-stare (keep looking at the book, stop reading), look away (leave the task entirely) — and logs each command at the moment it is spoken, so the 10 Hz eye/pupil stream is synchronized with ground truth to the second. The labeling method is literally called realtime_instructed in our API. What evolves against those labels is the interpretation method itself — how gaze dispersion, saccades, and pupil dynamics map onto mental states.

We chose to confront the ethics and safety questions of measuring children's attention through an institutional review process rather than around it. The architectural consequence is that the evolution server has a steady diet: every day of clinical ground truth is fresh evidence, and fresh evidence is a new generation candidate. The product consumers meet is a small cube on a desk. The thing that makes it smarter every day is this loop.

It also makes the two-server split make sense. The operations server is the product. The evolution server exists only to grow it — it owns the mistake logs, the lineage, the proposer, and the gates. Growth is a separate system from the thing that grows.

The whole stack is Alibaba-native

One thing worth stating plainly: this is an Alibaba-ecosystem project end to end, by choice. The cube itself is sourced for manufacturing through Accio, Alibaba's AI sourcing platform, with PCBA suppliers from the Alibaba network. The backend — both the operations server and the evolution server — runs on Alibaba Cloud ECS in Singapore. And the intelligence is Qwen on Model Studio, over the OpenAI-compatible endpoint: qwen3.7-max as the proposer that evolves the interpretation, qwen3.7-plus as the coach that writes trilingual reports, qwen3.6-flash for smoke tests. Splitting roles across models also splits Model Studio's per-model free quota — role separation and cost control turned out to be the same decision.

The roadmap only deepens this: age-stratified interpretation means one evolving lineage per 5-year age band — which means more generations, more evaluation, more Qwen. The heart of this product is the evolution server; the engine driving that heart is Qwen Cloud.

What we'd tell you to steal

If you're building anything where an LLM modifies a system that produces numbers people rely on:

Let the model choose, not report. Separate the decision (cheap to verify) from the measurement (expensive to trust). Never let the same entity do both.
Recompute, don't validate prose. Don't ask "does this claim look plausible?" Re-run it. A tolerance check against your own execution is worth more than any amount of prompt engineering.
Hold out data the proposal can't touch. Self-reported improvement on visible data is not improvement.
Put the human at the last step, not the first. Review every proposal and you've built a slow human. Review only what survived mechanical verification and a generalization gate, and you've built leverage.
Split roles across models. Better fit per task, and on Model Studio, separate free-quota pools.

Built for the Global AI Hackathon Series with Qwen Cloud. Everything runs on Alibaba Cloud — ECS for the stack, Model Studio (Singapore) for the Qwen models.

Code: github.com/S-O-A-TECH/concentration-cube-cloud · Demo video: youtu.be/S3M3wOPD8zE · Live demo: http://47.84.139.15:8200 · Track: MemoryAgent

DEV Community: State Of the Art TECHnology