DEV Community: Mohsen Seyedkazemi Ardebili

Whisper typed its own prompt into my document

Mohsen Seyedkazemi Ardebili — Tue, 03 Nov 2026 14:00:00 +0000

If you use faster-whisper or openai-whisper with an initial_prompt, try this sometime. Record two seconds of a fan, or a keyboard, or nothing in particular, loud enough to pass whatever loudness check you have. Transcribe it.

There is a decent chance you get your prompt back.

In YazSes, the offline dictation tool I work on, the built-in prompt is one short sentence:

BUILTIN_PROMPT = "The app is called YazSes."

It is there so Whisper spells the app's name right. And on a noisy burst with no speech in it, Whisper would sometimes transcribe it as... "The app is called YazSes." Typed straight into whatever window you had open. (#598 has the details.)

Why this happens

Whisper treats initial_prompt as if it were the text that came just before the audio. My working explanation (a hypothesis, not something I have proven inside the model) is that when the audio carries no words, the easiest thing for the decoder to produce is the text it was just shown. Loud noise is the worst case: a quiet clip gets dropped by the energy gate before Whisper sees it, but a loud one gets through.

Why the obvious fix doesn't work

Whisper reports a no_speech_prob for each segment. Surely you just drop segments where it is high?

I measured it, with the small, base.en and tiny.en models on synthetic noise bursts (white, brown and low-frequency noise and clicks, half a second to three seconds). On the echoes, no_speech_prob ranged from 0.43 to 0.97. On a public-domain speech sample it was 0.02 to 0.27.

Two problems with cutting on that:

faster-whisper's default no_speech_threshold is 0.6. Every echo between 0.43 and 0.6 sails straight through the default settings.
The gap between 0.27 and 0.43 comes from one speech sample and some synthetic noise. I would not bet a user's words on a cutoff placed inside a gap that narrow, measured that thinly.

And it wasn't only the built-in prompt. A personal vocabulary list came back too: "Ydotool, ydotool, Mohsen, Wayland." Whatever you prime the model with is what it reaches for when there is nothing to hear.

The fix: ask a different model

The fix is to stop asking Whisper whether there was speech, and ask an independent detector before decoding.

faster-whisper already bundles Silero VAD, so it costs no new dependency. Before a burst is decoded, YazSes asks Silero whether it holds any speech. If Silero positively says no, the burst is discarded and logged as discard_reason = "no_speech".

The most important design decision is the direction it fails in:

def holds_no_speech(
    audio: Any, sample_rate: int, *, detector: Detector | None = None
) -> bool:
    """True only when the detector *positively* found no speech. Never raises.

    ``None`` from the detector (unavailable, wrong sample rate) and any exception both
    read as "cannot tell", which is ``False``: the burst is decoded.
    """
    try:
        verdict = (detector or _default_detector)(audio, sample_rate)
    except Exception:  # noqa: BLE001 - a gate must never break dictation
        return False
    return verdict is True

It fails open. If Silero can't run, can't load, or gets audio at the wrong rate, the burst is decoded exactly as before. Think about the costs: a phantom sentence is annoying and you delete it. Silently swallowing words the user actually said is much worse, because they may not notice until later. A gate that guards against the small error must never cause the big one.

Choosing the threshold

Silero's default speech threshold is 0.5. I used 0.3, and here is the evidence, including its limits:

I cut 22 single-word bursts from the public-domain JFK speech sample and made 27 seeded noise bursts.
At 0.3: 0 words dropped, 0 noise bursts passed.
At 0.5: 2 quiet words were lost.
With a crude "whisper" proxy (random-phase audio), 3 of 22 words were lost at 0.3.

That is synthetic audio and a small set. I have not measured it on real whispered, accented or dysarthric speech, and those are exactly the users a dictation tool should care about most. So the check is skipped in the modes built for them: whispered/low-effort mode, the dysfluency-friendly mode and the quiet sotto-voce channel. It can also be turned off:

[accessibility]
speech_presence_check = true   # the default since v2.43.2

What I took away

Loudness can't tell a person from a fan. An energy gate is a first filter, not a speech detector.
Don't ask the model that is failing whether it is failing. Its own no_speech_prob let echoes through at default settings. A second, independent model that never saw the prompt answers the question more cleanly.
Decide which way a gate should fail before you write it. Then write the test for the failure, not just the happy path.

If you run Whisper with a prompt in any app (meeting notes, voice notes, subtitles), check what it returns on two seconds of loud noise. It takes a minute and you may be surprised.

The fix shipped in YazSes v2.43.2 (release notes). YazSes is offline, open source and Apache-2.0: github.com/MSKazemi/yazses.

I wrote this with help from an AI assistant for drafting and editing; the bug, the measurements and the design choices are mine, and I checked every fact against the code and release notes.

My dictation app typed words I never said. The silence gate was quieter than my room.

Mohsen Seyedkazemi Ardebili — Thu, 29 Oct 2026 14:00:00 +0000

One afternoon my own dictation tool started typing sentences I had not said.

Not garbage. Fluent, confident, plausible English. If you have used Whisper, you may already know where this is going.

Whisper does not do "nothing"

Speech models like Whisper are trained to produce text. Hand one a clip of an empty room, and it will very often give you words anyway, a phrase it has seen a thousand times, or something that sounds like a polite closing line. This is well known, and the usual defence is simple: don't send it silence.

In YazSes (an offline hold-to-talk dictation tool I maintain) the first defence is an energy gate. When you release the key, it computes the mean absolute level of the recording. If that is below a threshold, the clip is dropped and Whisper never sees it.

[accessibility]
vad_threshold = 0.01   # the default

My own config said 0.0005.

My room's noise floor, measured, was around 0.0039. The gate sat almost eight times below the room. Every time I pressed the key without speaking, or paused before talking, the fan and the street went straight through to Whisper, and Whisper wrote me some prose.

Three things that went wrong at once

It is tempting to blame "a bad config value". But three separate things had to fail for me to not notice for a while.

1. The value was below the noise. I don't actually know how 0.0005 got into my config. It does not matter much. Any threshold that is not measured against the room it runs in is a guess.

2. The warning that should have caught it was silent. yazses doctor --mic measured the room and compared it with the threshold. In exactly this case, it printed:

[OK] Mic level: ambient 0.0010 under vad_threshold 0.0005

Read that twice. 0.0010 is not under 0.0005.

The warning only fired when the room was "not silent", and "silent" was judged against a fixed floor of 0.002 that had nothing to do with my gate. So for any threshold below 0.002, the warning was switched off across exactly the band where it was needed. The one check designed for this situation reported success. (Fixed back in v2.25.1. It now says, plainly, that silence will pass the gate and be transcribed as invented words.)

3. The calibrator measured the wrong thing. There was a mic-level command to pick a threshold for you. It recorded one clip and assumed you were speaking in it. If you weren't, it calibrated to the room. In four empty-room recordings it measured 0.0036 to 0.0050 and recommended 0.002 to 0.0025: below the noise, every single time.

How to calibrate a gate properly

The fix is boring, and that is the point.

Measure the room and the voice separately. Since v2.36.0, yazses mic-level records a few seconds of silence first, then a few seconds of you talking. Now there are two numbers, and the gate must sit between them:

lo = max(_MIN_THRESHOLD, room * 1.5)   # comfortably above the room
hi = speech * 0.5                       # comfortably below your voice

Refuse when the numbers are too close. If hi < lo, the voice and the room are less than 3x apart. At that point any threshold you pick will either let noise through or eat quiet words. The honest output is not a number. It is "I can't calibrate this, move the mic or lower the gain." So it refuses.

That 3x is not tuned, by the way. It falls out of the two margins (1.5 / 0.5). For reference, across 1,646 real dictation events, speech had a median level of 0.0394 and clips that passed the gate but produced no text had a median of 0.0069, about 5.7x apart. Plenty of room, if you measure it.

The next release goes one step further (already on main): it aims the gate at twice the room level instead of halfway between room and voice, and it refuses recordings that are too short to say anything (under one second of room or two seconds of speech). That came from a bug report that mic-level --set picked a different threshold on every run.

What I took away

A threshold is meaningless without the noise it is meant to reject. Measure the noise, in place, on the real device.
One sample cannot tell you whether anyone spoke in it. If your calibration depends on the user doing something, record that something separately.
When two measurements are too close, refuse. A confident wrong number is worse than "I can't".
Test your warnings in the case they exist for. My warning was fine in every case except the one it was written for.

If you dictate with Whisper (any tool, not just mine) and you see phantom text, check your silence gate before you blame the model.

yazses mic-level          # measures room, then voice, and suggests a threshold
yazses mic-level --set    # writes it to your config

YazSes is offline, open source (Apache-2.0), and runs on Linux, macOS and Windows: github.com/MSKazemi/yazses. How the gate works in detail: silent audio discarding.

I wrote this with help from an AI assistant for drafting and editing; the incident, the numbers and the conclusions are mine, and I checked each fact against the code and changelog.

Can an AI agent's decision produce a verifiable receipt? An architecture proposal

Mohsen Seyedkazemi Ardebili — Wed, 28 Oct 2026 00:45:01 +0000

Explore how Jev and NovaFabric could support verifiable AI agent decision receipts, policy checks, and replayable execution evidence. This is a research proposal, not a shipped integration.

Imagine an AI agent working on a data-processing task.

At some point, it decides to upload a file to an external endpoint. The operation succeeds, and the workflow continues.

Six months later, someone asks:

Why did the agent decide to upload that file?
What information was available when the decision was made?
Which model evaluated the action?
Was the action authorized?
Can we verify the original record without trusting the running system?

A successful HTTP response cannot answer those questions.

What if every important AI-agent decision could produce a verifiable receipt?

This is a research direction I'm exploring around two technologies:

Jev, a System One model from TypeSafe AI designed for fast, structured decisions.
NovaFabric, an open-source system I've been developing to capture, replay, compare, and verify AI-agent execution evidence.

The idea is not to connect two APIs and call it an integration.

It's to investigate how we might preserve the evidence behind an AI decision so that engineers, researchers, and auditors can inspect it later.

Current status: This is an architectural proposal. The Jev integration described here has not been shipped or benchmarked, and there is no announced partnership with TypeSafe AI.

1. Why Jev is interesting

Many AI-agent decisions don't require an essay.

An agent might need to answer:

Is this tool call consistent with the assigned task?
Which risk category best describes an operation?
Does this answer satisfy a particular evaluation criterion?
Does a delegated action appear to exceed the original task?

Traditional LLMs can answer these questions, but generating explanatory text is not always the right interface.

TypeSafe AI's Jev exposes three useful primitives:

Primitive	Purpose
Choice	Select from predefined options
Score	Evaluate something against a defined rubric
Noul	Return a probability for a yes/no proposition

For example, we might ask Jev whether an agent's proposed file transfer appears unrelated to its assigned task.

But there is a critical distinction:

A model's probability is not proof that an action is safe, unsafe, or authorized.

It is a signal that must be evaluated against real outcomes and combined with independently enforced rules.

That's where the next part becomes interesting.

2. What NovaFabric adds

NovaFabric focuses on something different from model inference: preserving agent execution evidence.

Its central artifact is a Run Capsule, a portable directory containing recorded execution information from supported capture surfaces.

Depending on the workload, that includes model calls, tool calls, execution traces, environment information, and provenance.

NovaFabric supports structural comparison and scoped replay. Capsules can also be cryptographically sealed using an operator-configured key, making later modifications detectable.

If you want to try the verification side today, this tutorial walks through it end to end: Verify a sealed AI agent run for an auditor, offline.

This matters because an AI decision has two distinct questions:

Was the judgment useful or correct?

That requires evaluation against evidence and outcomes.

Is the archived decision record unchanged?

That can be checked using cryptographic integrity mechanisms, subject to their trust assumptions.

These are different problems. We should not confuse them.

For more background, see my arXiv preprint:

NovaFabric: Tamper-Evident, Replayable Evidence for Autonomous AI Agent Runs

3. The idea: a Decision Witness

I'm considering an optional component called a Decision Witness.

Think of it as a versioned record that connects an agent action with the information used to assess and authorize it.

A proposed record could contain:

An identifier for the tool call or agent action.
The model and version used for an assessment.
The question template and relevant input digest.
The model's structured output and probability distribution.
The applicable policy and authorization result.
Human approval information, when relevant.
A reference to the captured execution evidence.

Here's an illustrative schema:

{
  "schema_version": "decision-witness/0.1",
  "event_ref": "tool-call-42",
  "assessment": {
    "provider": "typesafe",
    "model_id": "<pinned-jev-version>",
    "question_id": "off_task_transfer",
    "type": "noul",
    "probability": 0.87
  },
  "authorization": {
    "policy_id": "outbound-policy-v3",
    "outcome": "review_required"
  },
  "human_approval": "pending",
  "origin": "observed",
  "evidence_ref": "sha256:<digest>"
}

This is a hypothetical design example, not an implemented NovaFabric schema or a real Jev response. The probability is illustrative.

The important feature isn't the number 0.87.

It's the ability to revisit the evidence, policy, and decision that accompanied it.

4. How could the architecture work?

A possible architecture:

Conceptual architecture: proposed, not implemented. Policy decides; Jev only advises; the witness records both.

The exact enforcement boundary matters.

Jev would not be allowed to override identity permissions, data policies, or hard authorization constraints.

A policy engine would remain responsible for those checks.

The Decision Witness would preserve the observed assessment and policy outcome. For post-run assessments, a separately linked attestation would avoid silently modifying an already sealed capsule.

5. Four experiments I'd like to explore

Experiment A: Detect unusual agent behavior

Take a collection of recorded agent tool calls, including benign and suspicious scenarios.

Run a Jev-based assessment in shadow mode.

Compare it against deterministic rules and NovaFabric's existing experimental security/provenance anomaly detector.

Research questions:

Does Jev identify useful semantic anomalies that structural detection misses?
How many false alarms does it produce?
How does its performance change across agent tasks and tool types?

Experiment B: A fast evaluation judge

Could Jev act as an additional judge for agent outputs?

For example, use rubric-based scores to evaluate whether an agent's output satisfies specific requirements.

Compare Jev against existing numerical, rule-based, and LLM-based judges.

The interesting question isn't just which judge is faster.

It's whether the scores remain useful and calibrated across different tasks.

Experiment C: Replay recorded decisions

Suppose a decision was captured today.

Three months later, the underlying model has changed.

Can we replay the recorded response without calling the model again, then separately compare it with a new model evaluation?

This would help distinguish the historical decision from a new assessment.

It would not prove that the model itself is deterministic.

Experiment D: Multi-agent delegation

Imagine several agents working across machines or organizations.

One agent delegates a task to another, which then invokes a tool.

Could we record the delegation chain, assess unusual handoffs, and determine whether authorization remained within the delegated scope?

I think this is particularly interesting for distributed and multi-agent AI systems.

6. How should we evaluate these ideas?

Before claiming that Jev improves agent safety or reliability, I'd want to run controlled experiments.

Useful measurements would include:

Dimension	Measurement
Detection quality	Precision, recall, PR-AUC
False alarms	False-positive rate
Probability quality	Brier score, calibration error
Performance	p50/p95/p99 latency
Cost	Cost per 1,000 evaluations
Robustness	Behavior under model/version changes
Privacy	Data exposed to the external assessment provider
Reproducibility	Recorded decision playback and verification

An independent Jev evaluation is also relevant:

Evaluating and Benchmarking the System One Model Jev

The authors benchmark Jev across 37 datasets and report results that depend on the task and, for some binary decisions, the chosen threshold.

That's a useful reminder that structured probabilities still require careful evaluation.

For a research prototype, I'd begin with offline shadow-mode assessment rather than live enforcement.

7. A question for the developer community

I think we're reaching a point where recording only an AI agent's final output will not be enough for many applications.

We need better ways to inspect not just what an agent did, but also the recorded assessments and authorization decisions surrounding its actions.

I'm particularly interested in feedback on three questions:

1. What should a verifiable AI decision receipt contain?

Would you preserve complete inputs, hashes, model versions, policy decisions, confidence distributions, or something else?

2. Where would this be most useful?

AI coding agents, autonomous infrastructure operations, financial workflows, multi-agent systems, scientific pipelines, or another area?

3. Would you start with historical decision analysis or real-time policy assistance?

My current preference is historical analysis and shadow-mode evaluation first, then carefully measured runtime integration.

I'd be interested to hear how other developers would approach the problem, including reasons this architecture might not work.

Projects and research references

NovaFabric

Website: https://novafabric.ai/
Open-source repository: https://github.com/MSKazemi/novafabric
Documentation: https://novafabric.ai/docs/
Archived software release: https://zenodo.org/records/23652607

Research papers

NovaFabric preprint: https://arxiv.org/abs/2609.12582
Independent Jev benchmarking paper: https://arxiv.org/abs/2609.37647

Jev

TypeSafe AI introduction: https://typesafe.ai/blog/introducing-system-one-models-and-jev
Documentation: https://docs.typesafe.ai/introduction

About the author

I'm Mohsen Seyedkazemi Ardebili, working on AI-agent systems, reproducibility, infrastructure, and verifiable execution evidence.

GitHub: https://github.com/MSKazemi

AI disclosure: This article was drafted with AI assistance. It presents a research proposal, not a completed integration or experimental result.

The tool that said yes and did nothing: ydotool 0.1.8 exits 0 when it refuses

Mohsen Seyedkazemi Ardebili — Tue, 27 Oct 2026 14:00:00 +0000

For about ten and a half hours on 24 September, my dictation app typed nothing at all on Ubuntu under Wayland.

You held the key, you spoke, the speech was transcribed correctly, and then... nothing landed in the window. No error in the log. The health check printed [OK] Injection: ydotool. The tests were green.

This is the story of that bug, because the lesson in it applies to almost every program that shells out to another program.

Some context

I work on YazSes, an open-source, offline hold-to-talk dictation tool. You hold a key, talk, release, and the text is typed into whatever app has focus. On Wayland you cannot just "send keystrokes" to another window the way you can on X11, so one common route is ydotool, which writes events to a virtual uinput keyboard.

The catch: there are two ydotools. Version 1.x and version 0.1.x share a name and a purpose, but they are different command-line tools. And distributions did not all move at the same pace. As of today (I checked Launchpad and sources.debian.org on 2026-10-22):

Distribution	ydotool version
Ubuntu 22.04, 24.04	0.1.8
Debian 11, 12	0.1.8
Ubuntu 26.04, Debian testing	1.0.4

YazSes only spoke 1.x.

How a correct fix broke things

The bug did not come from a careless change. It came from a correct one.

In v2.39.0 I taught the socket discovery to find /tmp/.ydotool_socket, which is where the 0.1.8 daemon listens. Before that, on Ubuntu, ydotool was simply never selected, and YazSes used another backend that worked fine.

After the fix, ydotool became reachable. It is first in the backend order. So on Ubuntu it pushed aside a path that worked and took over, speaking 1.x syntax to a 0.1.8 binary.

Three command lines were now wrong:

ydotool type -d 6 -H 6 was refused. 0.1.x uses --key-delay and has no hold time.
ydotool key 29:1 47:1 47:0 29:0 (Ctrl+V as raw keycodes): 0.1.x wants symbolic names.
ydotool key 97:0, a "release any stuck modifier" guard that ran after every dictation, typed the digit 9.

Why nothing noticed

Here is the part that matters. When 0.1.8 refuses an option, it does this:

$ ydotool type -d 6 -H 6 "hello"
error: unrecognised option '-d'
$ echo $?
0

It prints an error and exits 0.

My code ran it with subprocess.run(..., check=True). check=True only raises on a non-zero exit code. So no exception, so the clipboard fallback never ran, so nothing was logged, so the health check (which asked "is ydotool there and does it run?") said OK.

Every layer was telling the truth about what it was asked. Nobody was asking whether text reached the window.

It gets better. 0.1.8's key command has no error path for unknown names at all. It types the first letter of whatever you gave it. I measured this by grabbing the virtual keyboard device and counting events:

You send	0.1.8 types
`Return`	`r`
`Escape`	`e`
`space`	`s`
`KEY_BACKSPACE`	`k`
`29:1`	`2`

So "new line" would have typed r.

The fix

Two changes, released in v2.40.0 the same evening.

1. Stop trusting the exit code. The output is classified instead. 0.1.8 writes normal progress messages to stderr ("Key delay was set to 6 milliseconds"), so not every line on stderr is an error, but an explicit error is:

def _classify(stderr: str, returncode: int) -> type[YdotoolCliError] | None:
    """The exception for this result, or None when it succeeded.

    0.1.8 writes ordinary progress to stderr ("notice: Using ydotoold backend",
    "Key delay was set to 6 milliseconds."), so only an explicit error is an error.
    """
    low = stderr.lower()
    if any(marker in low for marker in _OPTION_ERROR_MARKERS):
        return YdotoolOptionError
    if ": error:" in low or returncode != 0:
        return YdotoolCliError
    return None

Now a refused command raises, and the fallback backend runs like it was always supposed to.

2. Detect the dialect, and only send what it understands. YazSes probes which ydotool it is talking to and keeps an allowlist of key names that were measured to work on 0.1.8. Anything else raises and goes to the fallback, instead of typing a random first letter. yazses doctor now says "ydotool 0.1.x" or "ydotool 1.x" instead of a bare OK.

And there is a regression test with an honest name: test_an_option_error_is_a_failure_even_though_ydotool_exits_zero.

What I took away

An exit code is a claim, not evidence. It tells you what the tool thinks happened. When a subprocess is the last step before something the user sees, read its stderr, and if you possibly can, measure the effect itself. For keyboard injection, that meant grabbing the uinput device and counting events. For your tool it might be "read the file back" or "query the API you just wrote to".

Making something reachable makes it load-bearing. My socket fix was right. But it promoted a backend from "never used on Ubuntu" to "first choice on Ubuntu", and I verified the probe, not the backend. If a change widens where code runs, test the code in its new home.

"Health check: OK" should name what it checked. "ydotool found" and "text arrives in the window" are very different sentences. Only one of them was being said.

If you maintain anything that wraps ydotool, xdotool, wtype or similar, I'd love to hear how you verify that input actually landed. That is still the hardest part of this whole project.

YazSes is an offline, open-source (Apache-2.0) dictation tool for Linux, macOS and Windows: github.com/MSKazemi/yazses. The full fix is in the v2.40.0 release notes.

I wrote this with help from an AI assistant for drafting and editing; the bug, the fix, the measurements and the opinions are mine, and I checked every fact against the code and release notes.

Anatomy of an offline dictation pipeline: where the time actually goes

Mohsen Seyedkazemi Ardebili — Thu, 22 Oct 2026 13:00:00 +0000

This is the first post in a series about the parts of offline voice dictation that turned out to be harder than they looked. Most of the series is about bugs: the subtle kind, where every check is green and the user still gets the wrong text. But first, a map.

The project is YazSes. You hold a key, speak, let go, and your words are typed into whatever app has focus: editor, terminal, browser, chat. Speech recognition runs on your own CPU. No account, no API key, and no audio leaves the machine.

The nine stages

When you press the key, audio starts going into a buffer in memory. Nothing is written to disk. When you release it, that burst goes through nine stages:

Hotkey: the press and release events (Right Alt on Linux, Right Option on macOS, Right Ctrl on Windows by default).
Capture: the audio buffer itself.
Silence gate: is this clip loud enough to contain speech at all? If not, drop it. (Post 3 of this series is about what happens when this is set wrong.)
Padding: a short silent lead-in is added before decoding, so Whisper doesn't clip your first word.
Speech-to-text: faster-whisper on the CPU, quantised to int8. Default model: base.en.
Clean-up: strips Whisper artefacts like [BLANK_AUDIO] and stray leading punctuation.
Disfluency filter: removes fillers, de-duplicates repeated words, and rolls back self-corrections like "no wait".
Command or dictation?: "undo that" should press Ctrl+Z, not type the words "undo that". A regex grammar decides, in well under a millisecond.
Inject: type the text into the focused window, by whatever means the OS allows.

Where the time goes

I assumed I'd have to optimise half of these. I measured instead.

On my laptop (an i7-1370P, int8 on CPU, 30 utterances with a median length of 7.3 seconds), speech-to-text with base.en took a median of 1,561 ms. Every other stage combined had a median of 0.29 ms. Speech-to-text is 99.98% of the processing time of a burst.

So the filters, the command grammar and the injector are not worth tuning for speed. The only latency setting that matters is which model you run.

That made the design question simple. Instead of tuning, offer choices:

Engine	Word error rate (LibriSpeech test-clean, 200 utterances)
Parakeet TDT 0.6B v2 (optional)	2.06%
Whisper `small.en`	2.66%
Moonshine base (optional)	3.17%
Whisper `large-v3`	3.23%
Whisper `base.en` (default)	4.01%

Two honest caveats on that table. The confidence intervals overlap, so "Parakeet beats large-v3" is not established by it. And large-v3 loses mostly on insertions (words that weren't spoken), not on recognising what was said. The full method is in docs/benchmarks.md.

The optional engines are not installed by default, because they are big. You turn one on when you want it:

yazses features enable stt-parakeet

If an engine you configured is missing, YazSes logs a warning and falls back to faster-whisper instead of refusing to start. (Post 12 is about the day that fallback hid a real bug.)

The hard part is the last stage

Speech recognition is the expensive stage. It is not the hard one. The hard one is stage 9: getting text into someone else's window.

macOS: CGEventKeyboardSetUnicodeString.
Windows: SendInput with KEYEVENTF_UNICODE.
Linux on X11: xdotool works.
Linux on Wayland: there is deliberately no general way for one app to type into another. YazSes probes, in order: ydotool (a virtual keyboard through uinput), wtype, the desktop portal's RemoteDesktop interface, and the clipboard as a last resort.

Every one of those has failure modes where the API reports success and the user gets nothing, or gets garbage. Posts 2 and 5 are two of those stories.

Why offline at all?

Partly privacy. What you dictate is often the most private text you write: messages, notes, half-formed ideas. But also reliability: it works on a train, it works when a service is down, and it has no per-minute cost. On a modern laptop CPU, small Whisper models are fast enough that you don't need the cloud for this.

"Nothing leaves the machine" is easy to say. Post 6 is about how the project turns that sentence into a test that fails the build.

The rest of the series

Anatomy of an offline dictation pipeline (this post)
The tool that said yes and did nothing
My silence gate was quieter than my room
Whisper typed its own prompt into my document
SendInput said success. The app got question marks.
Turning "nothing leaves the machine" into a test
Testing a speech app with no microphone, network or model
A guard is judged by how rarely it fires
"No wait, I meant...": when self-correction ate my sentences
Voice commands without an LLM
Offline meeting transcripts with speaker labels
The update button that uninstalled a feature

If you want to try it:

# Linux
bash <(curl -fsSL https://raw.githubusercontent.com/MSKazemi/yazses/main/install.sh)
# macOS (Apple Silicon)
brew tap MSKazemi/yazses && brew trust MSKazemi/yazses && brew install --cask yazses
# Windows
winget install MSKazemi.YazSes
# Any OS with Python 3.11+
pipx install yazses

Linux is the most tested platform. macOS and Windows are labelled preview, and reports from real machines help more than anything else.

YazSes is open source under Apache-2.0: github.com/MSKazemi/yazses. The architecture reference, with the measurements above: docs/architecture.md.

I wrote this with help from an AI assistant for drafting and editing; the project, the measurements and the opinions are mine, and I checked every number against the repository.

Find where two AI agent runs diverged

Mohsen Seyedkazemi Ardebili — Sat, 10 Oct 2026 09:41:50 +0000

Your agent gave one answer yesterday and a different one today. Same code, same input file, a different decision. Before you can fix anything you need to know where the two runs parted ways. Did the prompt change, did the model reply change, or did something after the model call change?

This tutorial is for engineers who run agents in Python and want to answer that from two recorded runs instead of from memory. You will capture two runs as Run Capsules, compare them with nova diff, read which model call changed, and turn the same comparison into a CI gate. Every command and output below was captured from the released novafabric 0.104.0. Long paths and skipped lines are marked with ….

What you need

Python 3.12 or newer (the package declares Requires-Python: >=3.12)
pip install novafabric openai
The examples/blackbox_demo/ directory from the public MSKazemi/novafabric repository, copied into an empty working directory. Run every command from that working directory.

No API key and no hosted model are involved. The example ships a small OpenAI-compatible mock server on 127.0.0.1:9099.

The scenario

agent.py reads a payment-service config, asks a model for one recommended change, and writes the answer to outputs/decision.json. The mock server answers in one of two ways, chosen by a request header the agent sends: a risky answer (disable rate limiting) or a safe one (reduce max_connections). That gives you two runs whose behaviour differs for a known reason, so you can check that the diff points at the right place.

Step 1: start the mock model

python blackbox_demo/mock_llm_server.py &
export OPENAI_API_KEY=sk-demo-no-key-needed
export OPENAI_BASE_URL=http://127.0.0.1:9099
export NOVAFABRIC_SUGGEST=0

Step 2: capture the first run

nova capture wraps the command with no code changes. Model calls from Python workloads are recorded automatically.

$ nova capture -- python blackbox_demo/agent.py --mode bad
✓ Capsule written: …/.novafabric/capsules/01M4FBTQ7FVQEZBWQGFD2RVTW2

Keep the capsule path in a variable (your run IDs will differ):

BAD=$HOME/.novafabric/capsules/01M4FBTQ7FVQEZBWQGFD2RVTW2

Step 3: capture the second run

$ nova capture -- python blackbox_demo/agent.py --mode fixed
✓ Capsule written: …/.novafabric/capsules/01M4FBVEZDG7XVBYJBB9WD6Q8D

Step 4: diff the two runs

$ nova diff "$BAD" "$FIXED"
  changed=3  added=0  removed=0

Model calls:
  ~ call at span 5fef2505ba0bd78e
Outputs:
  ~ outputs/decision.json
  ~ outputs/stdout.txt

Three items changed (~): one model call and two output files. Nothing was added or removed. For the machine-readable record, add --output-format json; the model-call pair reports request_changed: false and response_changed: true.

How to read it

aligned: 1: the diff paired the one model call in each run.
request_changed: false, response_changed: true: the agent sent the same request both times and got a different reply. The runs diverged at the model's answer, not in your prompt-building code.
environment.changes: []: no recorded environment difference to chase.
outputs: both files changed downstream of that reply.

Step 5: use it as a CI gate

--assert-no-regressions makes nova diff exit 1 when it finds any structural change between the two runs, including added or removed calls. Comparing a run with itself exits 0. On GitHub Actions, --output-format github-annotation prints one annotation per change.

Read the gate for what it is: it answers "did anything change?", not "did it get worse?". A safer answer fails it just as a riskier one does, as it did here.

What this does not do

It does not judge the change. The diff is structural: it tells you which call and which files differ, not whether the new behaviour is better.
Any difference counts. Against a live model, two runs of the same prompt can return different text, and the gate fails on that too.
Automatic model-call capture is for Python workloads. Other clients go through nova api-proxy.
Sealing is opt-in and was not used here. The capsule format is pre-1.0.

The full version, with every command output, the replay section and the CI examples, is on the original page: https://novafabric.ai/blog/find-where-two-agent-runs-diverged/

Originally published at novafabric.ai. NovaFabric is open source (Apache-2.0) and pre-1.0: github.com/MSKazemi/novafabric.

I got tired of typing, so I built a free, offline voice-typing tool (open source, contributors wanted)

Mohsen Seyedkazemi Ardebili — Thu, 08 Oct 2026 15:04:54 +0000

I built YazSes: hold a key, speak, release, and your words are typed into whatever app has focus. It runs speech-to-text locally (faster-whisper on CPU), so nothing leaves your machine: no cloud, no account, no API key.

Install

Linux

curl -fsSL https://raw.githubusercontent.com/MSKazemi/yazses/main/install.sh | bash

Snap

sudo snap install yazses
sudo snap connect yazses:audio-record   # microphone
sudo snap connect yazses:raw-input      # hold-to-talk key

macOS (Apple Silicon)

brew tap MSKazemi/yazses && brew trust MSKazemi/yazses && brew install --cask yazses

Windows

winget install MSKazemi.YazSes

Any OS with Python 3.11+

pipx install yazses

Then:

yazses quickstart   # 3 steps tailored to your machine, read-only
yazses doctor       # verify microphone, hotkey and text injection

Why offline?

Your voice never leaves the machine.
No subscription, no API key, works without internet.
It also transcribes recordings and captures meetings with speaker labels, locally.

Want to contribute? Here is where you can help in 15 minutes

You do not need to be a speech-recognition expert. Real, open tasks right now:

🌍 Native speaker of Persian, Urdu, Thai, Vietnamese, Telugu, Ukrainian, Traditional Chinese...? Several translations of the docs have never been read by a native speaker. Fixing a sentence is a valid first PR.
🧪 Pick a good first issue: https://github.com/MSKazemi/yazses/labels/good%20first%20issue
🐧 Test it on your distro / desktop (Wayland, X11, Windows, macOS) and open an issue with the output of yazses doctor. Bug reports from real machines are gold.
📖 Read the guide: https://github.com/MSKazemi/yazses/blob/main/.github/CONTRIBUTING.md

The test suite runs offline (no microphone, no model download). It is about 30,000 tests now and takes a few minutes on a laptop:

uv sync
uv run python -m pytest tests/ -v

Try it, then tell me

⭐ Star the repo so others can find it: https://github.com/MSKazemi/yazses
⬇️ Install it with the one-liner above and run yazses doctor.
💬 Comment below with your OS and what happened. I reply to every comment.

AI Agents Need More Than Logs: Why I Built NovaFabric

Mohsen Seyedkazemi Ardebili — Wed, 07 Oct 2026 10:10:58 +0000

AI agents are becoming more powerful.

They can:

modify code
call APIs
execute shell commands
change cloud infrastructure
read and write files
interact with other agents
make decisions

But there is a simple question that is still surprisingly hard to answer:

After an AI agent finishes its work, can we prove what it actually did?

Usually, the answer is: not completely.

That is the problem behind NovaFabric.

Logs are useful, but logs are not evidence

Today we have many good observability tools. They can show things like:

Agent started
↓
LLM called
↓
Tool called
↓
API called
↓
File modified
↓
Agent finished

This is useful for debugging. But imagine that six months later something goes wrong. An auditor, security engineer, researcher, or developer may ask:

Which model was used?
What prompt did it receive?
What did the model return?
Which tools did the agent call, and with what arguments?
Which files changed?
Which network services were contacted?
What software versions were installed?
Were secrets captured?
Was the record changed after the run?
Can we reproduce or inspect the execution again?

A traditional trace does not necessarily answer all of these questions. And more importantly, a normal database record can potentially be changed later.

That is where the idea of execution evidence becomes useful.

Think of an AI run as an artifact

NovaFabric takes a different approach. Instead of thinking:

AI Agent → Logs

it thinks:

AI Agent
   ↓
Execution
   ↓
Portable Evidence

NovaFabric captures a run into something called a Run Capsule: a structured folder describing one execution.

capsule/
├── capsule.yaml
├── trace.jsonl
├── model-calls.jsonl
├── tool-calls.jsonl
├── env.lock
├── redaction-proof.json
├── replay.yaml
└── ...

The idea is simple:

The execution should leave behind something you can keep, inspect, compare, replay, sign, and share.

The research paper models the capsule using 15 entity types, including model calls, tool calls, file events, network calls, human approvals, environment information, lineage, redaction records, replay information, and tamper-evidence.

Five verbs explain most of NovaFabric

Capture → Seal → Replay → Diff → Audit

1. Capture

First, capture what happened during the execution:

nova capture python my_agent.py

NovaFabric can wrap a command without requiring the application itself to be rewritten. It aims to capture model calls, tool calls, files, network activity, environment, inputs and outputs. The result is a portable Run Capsule.

2. Seal

Capturing information is not enough. If somebody modifies the record afterward, we want that modification to be detectable.

NovaFabric combines established technologies:

DSSE for signing
RFC 3161 trusted timestamps
Merkle logs for append-only history
cryptographic redaction records

The important point: NovaFabric is not inventing new cryptography. It combines existing standards into one evidence system for AI-agent executions. That integration, rather than a new signing algorithm, is the paper's main contribution.

Before sealing:  "This file says the agent did X."

After sealing:   "This file says the agent did X,
                  and we can detect if somebody changes it later."

There is an important limitation here. Tamper-evident does not mean tamper-proof. NovaFabric can detect later modification of properly sealed evidence. It cannot prove that a trusted signing party told the truth when the evidence was originally created. The paper explicitly defines this trust boundary.

3. Replay

AI systems have another problem: reproducibility. Run the same prompt tomorrow and you may not get the same answer. So NovaFabric does not pretend every AI execution can be reproduced bit-for-bit. Instead, it defines four replay modes:

Forensic replay - execute nothing; reconstruct and inspect what happened (incident investigation, security review, audit, debugging).
Mocked replay - use previously recorded model responses instead of calling the model again, to test the surrounding application logic.
Exact replay - try to recreate the same environment, model, data, and execution, when deterministic conditions exist.
Semantic replay - run against another model and compare the meaning of the new output with the original, e.g. when upgrading from Model v1 to v2.

The paper is deliberately careful here: only the mocked mode is deterministic by construction with respect to model inputs, while forensic mode executes nothing.

4. Diff

Run B produces a bad result. What changed? Maybe the model, the prompt, a dependency, a tool response, the environment, or the output. NovaFabric provides a structural diff between capsules, comparing the structure of runs instead of giant text logs.

In the paper's controlled experiment, structural diff correctly localized all 140 injected mutations across seven tested mutation classes. That could make questions like this easier to investigate:

"It worked yesterday. Why did the agent fail today?"

5. Audit

Finally, the evidence can be packaged into an Evidence Bundle. Another person should not have to trust your database; you can hand them a portable artifact containing the evidence and the cryptographic information needed to verify it.

Agent execution → Run Capsule → Cryptographic seal → Evidence Bundle → Auditor / Researcher / Security Team

The paper specifies an offline verification path using standard technologies rather than requiring the verifier to trust a NovaFabric server. However, it also states that this independent-verifier interoperability path had not yet been exercised by an independently implemented verifier at publication time. That distinction matters.

Why not just use OpenTelemetry?

I like OpenTelemetry, and NovaFabric uses OpenTelemetry concepts. But the problems are different:

Observability	Execution evidence
What is happening?	What happened?
Find errors	Reconstruct the execution
Metrics and traces	Portable run artifact
Operational debugging	Audit and forensics
Usually mutable storage	Tamper-evident evidence
Live system	Historical execution

NovaFabric is not intended to replace Prometheus, OpenTelemetry, Langfuse, LangSmith, or similar systems. It focuses on another question:

Can a past execution be replayed, compared, and proven?

What about secrets?

Capturing everything creates another problem: an agent may see API keys, tokens, passwords, or credentials, and you do not want them inside a bundle sent to another company or auditor. NovaFabric includes secret redaction and a redaction attestation.

But a redaction attestation means "the scanner says it detected and removed this secret." It does not prove that an unknown secret does not exist somewhere else in the capsule. The paper makes this explicit: completeness depends on the scanner, its rules, and the locations it checks. Being clear about these boundaries is especially important for security tools.

What did the research evaluation find?

Model calls captured:                 100% in tested scenarios
MCP tool calls captured:              100% in tested scenarios
Tested tampering classes detected:    3 / 3
Structural mutations localized:       140 / 140
Redaction (14 credential types):      14 / 14 removed
Provenance graph, 10M-edge blast-radius p99:   45.5 ms
Provenance graph, 100M-edge single-client p99: 167.9 ms

But some results exposed weaknesses too. Mocked replay avoided live model calls in 10/10 tested scenarios, but only 2/10 tool-using workloads completed, because tool responses were not yet being substituted in the evaluated version. That was one of the most useful outcomes of the research: the evaluation did not only produce good benchmark numbers, it found real defects.

An unexpected lesson

"The pipeline ran successfully" is not the same as "the evidence is correct."

Several problems were discovered because the tests checked what should have been recorded, rather than simply checking whether the program exited successfully. At one point some evidence streams were missing even though capture appeared to work. In another experiment, an ingest service returned successful HTTP responses even though its configured metadata database had no tables.

"Did it run?" is a weak test.
"Can we prove the expected things actually happened?" is a much stronger one.

NovaFabric today

NovaFabric is open source and self-hosted. The project has moved beyond the exact version evaluated in the paper, so the paper should be read as a research snapshot, not a complete description of today's repository.

pip install novafabric

nova capture python my_agent.py
nova validate <capsule>
nova replay <capsule> --mode forensic
nova diff <capsule-a> <capsule-b>

The project is still beta. Some local functionality is more mature, while server, cluster-scale, and some at-scale components remain experimental. I do not want to describe experimental infrastructure as production-proven.

The bigger idea

As AI agents move from chat interfaces into systems that actually do things - changing production infrastructure, approving financial operations, modifying software, moving important data, operating HPC systems, coordinating other agents - observability alone will not be enough. Eventually someone will ask: What exactly happened? Can you prove it? Can you reproduce it?

That is the space NovaFabric is exploring. Not another agent framework, not another tracing dashboard, but infrastructure for turning an execution into something we can capture → seal → replay → compare → audit.

Try it

NovaFabric is Apache-2.0 and open source:

GitHub: https://github.com/MSKazemi/novafabric
Website: https://novafabric.ai
Research paper: https://arxiv.org/abs/2609.12582

If you work on AI agents, MLOps, HPC, security, observability, provenance, or reproducibility, I would love your feedback. Especially this question:

When your AI agent makes an important decision today, what evidence will you still have six months from now?

YazSes

Mohsen Seyedkazemi Ardebili — Wed, 23 Sep 2026 11:32:26 +0000

Help Us Build an Open, Offline Hands-Free Desktop

Mohsen Seyedkazemi Ardebili — Wed, 23 Sep 2026 11:31:04 +0000

Update (2026-10-25): PR #413 was merged on 25 September 2026 as a planning and evaluation document, not a runtime release. Issue #524 (the ADR numbering collision) is now closed. PR #541 was closed as a duplicate, because the same status correction had already landed on main. Issue #102, the hands-free epic, is still open and is still the place to help.

YazSes started with a simple idea:

Hold a key, speak, and have your words typed into almost any desktop application — locally, without sending your audio to a cloud service by default.

GitHub:

👉 https://github.com/MSKazemi/yazses

But voice is only one part of hands-free computer interaction.

What if the same open-source project could eventually let you:

look toward a window and dictate into it,
look roughly at something and say “close this”,
move a pointer using your head,
use an intentional facial gesture as a switch,
use dwell when another physical switch is unavailable,
combine gaze + speech + head movement + switch input,
and safely fall back when the camera is uncertain?

That is the direction of the YazSes eye / camera control programme.

And this is an invitation to help us build it.

First: this is not “perfect eye tracking with a laptop webcam”

This distinction is important.

A normal laptop webcam is not the same thing as a dedicated infrared eye tracker.

We do not want to pretend otherwise.

Commodity-camera gaze may be useful for answering a coarse question such as:

Which window or region is the user looking toward?

It should not automatically be treated as a precision mouse pointer capable of selecting a tiny icon or placing a text caret.

Our current design gives different modalities different jobs.

Task	Input that may fit it best
Generate text	Speech
Choose a pane/window	Coarse gaze
Resolve “this” or “that window”	Gaze + speech
Move a pointer continuously	Head pose
Commit an action	Face switch, voice, dwell, keyboard, EMG, etc.
Precision gaze pointing	Future dedicated eye-tracker backend
Recover from uncertainty	Current focus, voice controls, keyboard/switch, or explicit abstention

One principle from the design summarizes this:

Perception proposes; an intent-bearing action commits.

Your eyes can help say where.

Your voice or another deliberate action can say what to do.

That matters because our eyes are always looking at something.

Looking at a button should not automatically mean clicking it.

Some gaze functionality already exists

This is not just a concept document.

YazSes already contains working gaze-related code.

Glance-Type gaze routing

The implemented gaze path can sample webcam gaze, use a calibration map, resolve a target window, and route the next dictation toward it.

When confidence is insufficient, it falls back rather than blindly choosing another window.

The relevant runtime code already exists in the repository:

src/yazses/gaze/mediapipe_backend.py
src/yazses/gaze/calibrate.py
src/yazses/gaze/targeter.py
src/yazses/gaze/route.py

Look-to-window routing is currently associated particularly with the X11 desktop path, and cross-platform semantics are one of the things we still need to improve.

Gaze + speech deixis

YazSes also has a particularly interesting interaction:

look + speak

For example:

“focus that window”

or:

“close this”

The speech provides the operation while the gaze snapshot provides context for what this or that refers to.

Destructive gaze-routed actions can go through confirmation instead of being executed blindly.

The important part is that gaze provides context rather than becoming an automatic click mechanism.

Other pieces already exist — but are not complete runtime features

This is where the project becomes interesting for contributors.

Several useful cores already exist.

Implicit gaze calibration

There is already a pure implicit-calibration implementation:

src/yazses/gaze/implicit.py

It can update a calibration model from (gaze, click) observations using recursive least squares.

It also contains important safeguards:

reject low-confidence samples,
reject extreme residuals,
use a forgetting factor,
compare a candidate calibration against a baseline,
keep the original calibration unless the candidate performs better on held-out samples.

But the pure estimator is not the same as a complete feature.

The runtime still needs proper opt-in click observation, sample association, persistence and rollback work.

Head-Pointer is another example

YazSes already contains the pure head-pose-to-pointer core:

src/yazses/headpointer/pointer.py

Including:

yaw/pitch → cursor delta mapping,
a configurable dead zone,
and a dwell-click state machine.

Conceptually:

head movement
      ↓
  yaw / pitch
      ↓
dead zone + mapping
      ↓
pointer movement
      ↓
optional dwell

But this is not currently a complete Head-Pointer desktop feature.

The missing engineering includes:

shared camera/head-pose input,
a platform-neutral pointer interface,
X11 output,
Wayland output,
macOS output,
Windows output,
pause/resume,
recentering,
signal-loss handling,
dwell feedback,
and reliable emergency stopping.

A continuous accessibility control without a dependable stop path is not something we want to ship.

Face gestures should become switches — not direct actions

Another planned capability is deliberate facial input.

Imagine intentionally:

opening your mouth,
raising your eyebrows,
making another calibrated expression,

and having that produce an abstract switch event.

Not:

eyebrow movement → directly click something

but:

camera
   ↓
face/blendshape signal
   ↓
gesture detector
   ↓
abstract switch event
   ↓
user-configured action

That difference matters.

The camera detector should recognize intent.

It should not contain desktop-action policy.

The eventual switch could be mapped to things such as:

start/stop dictation,
click,
confirm,
cancel,
pause the pointer,
participate in a gesture chord.

YazSes already has an input-agnostic Gesture Chord core:

src/yazses/gesture/chords.py

But the Face-Gesture Switch itself does not currently have a runtime detector/adapter implementation.

That distinction was important enough that we are currently correcting stale documentation which previously implied otherwise:

👉 https://github.com/MSKazemi/yazses/pull/541

One camera, not three competing camera loops

If gaze, head pose and facial switches all use the same webcam, each feature should not independently open the camera and run its own face model.

The architecture under review proposes a shared perception source:

                         camera
                            |
                            v
                  +-------------------+
                  | Shared Perception |
                  | one camera owner  |
                  | one face model    |
                  +---------+---------+
                            |
          +-----------------+-----------------+
          |                 |                 |
          v                 v                 v
      GazeSignal       HeadPoseSignal      FaceSignal
          |                 |                 |
          v                 v                 v
      calibration       Head-Pointer      Face Switch
          |                 |                 |
          +-----------------+-----------------+
                            |
                     safety / feedback
                            |
                         YazSes

Raw frames stay inside the perception layer.

Consumers should receive small derived signals.

That has several advantages:

one webcam owner,
less duplicated computation,
fewer camera-contention bugs,
easier testing,
clearer privacy boundaries,
easier failure handling.

It also means most of the system can be tested with deterministic numeric inputs rather than requiring a physical webcam in CI.

Privacy needs to be part of the architecture

Camera-based accessibility software deserves careful handling of data.

For this programme, the intended boundary is straightforward:

Camera processing stays local.

Raw webcam frames should not become part of normal logs, evaluation exports, issue reports or network requests.

For engineering evaluation, we care about derived measurements such as:

target successes,
wrong-target events,
fallback events,
calibration error,
confidence,
false activations,
accidental clicks,
pointer acquisition time,
recovery after signal loss,
display resolution and scaling,
OS/session,
broad camera category.

We do not need public contributors to upload:

face photos,
face videos,
raw webcam frames,
private screen contents,
private dictated text,
medical information,
passwords or tokens,
device serial numbers.

There is another important boundary.

Public GitHub QA is not automatically research participation

We eventually want good evidence about whether these interfaces actually work.

But a person who posts an engineering result on GitHub should not silently become a participant in a research paper.

The programme therefore separates:

CI / synthetic evidence
        ↓
community engineering QA
        ↓
separately consented research

A future human-participant study should have its own protocol and consent process.

That separation is already part of the planning work.

“It worked on my laptop” is not enough

Camera interaction can vary dramatically between environments.

We eventually need evidence across:

Windows,
macOS,
Linux X11,
GNOME Wayland,
KDE Wayland,
different webcams,
different laptops,
different screen resolutions,
display scaling,
multi-monitor configurations,
different people,
repeated sessions.

The programme therefore proposes an evidence ladder ranging from pure deterministic tests to real hardware and, separately, controlled human studies.

The basic idea is:

pure unit/state tests
        ↓
cross-platform contract tests
        ↓
synthetic trace replay
        ↓
one real computer
        ↓
another physical computer
        ↓
different people
        ↓
repeated sessions
        ↓
controlled research study

A green unit test does not prove webcam usability.

One working laptop does not prove Windows, macOS or Linux support.

And 1,000 trials from one person are still not 1,000 participants.

We also want to explore semantic grounding

One research/development direction is especially interesting.

Instead of asking the webcam to become dramatically more precise, perhaps precision can come from combining coarse spatial information with UI semantics.

Imagine looking approximately toward a group of controls and saying:

“click Save”

The gaze or head pointer gives an approximate region.

The operating system accessibility tree can provide structured nearby candidates.

Speech provides the requested operation.

Then the resolver can produce:

grounded target

or:

ambiguous — don't guess

or:

unresolved — don't act

Conceptually:

coarse gaze / pointer location
              +
structured UI candidates
              +
spoken intent
              ↓
       target resolver
              ↓
 grounded / ambiguous / unresolved
              ↓
         safety policy
              ↓
            action

The current proposal deliberately starts with structured accessibility information rather than making screenshots, OCR or vision-language models the default.

If structured UI data is insufficient, we should measure that gap first.

Where are we today?

This is important:

The complete hands-free system described in this post is not released.

As of September 23, 2026, the architecture, implementation sequence, evaluation model and contributor workflow are being reviewed in:

👉 https://github.com/MSKazemi/yazses/pull/413

The umbrella hands-free epic is:

👉 https://github.com/MSKazemi/yazses/issues/102

PR #413 is a planning / architecture / evaluation PR.

It does not itself ship:

Head-Pointer runtime,
Face-Gesture Switch runtime,
dedicated eye-tracker support,
a completed hands-free preset,
research results,
or new webcam data collection.

This separation is deliberate.

We want the architecture, privacy rules, safety boundaries, testing approach and dependencies to be understandable before dozens of implementation PRs start landing.

There is also active repository coordination work around the design records, including an ADR-numbering collision tracked in:

👉 https://github.com/MSKazemi/yazses/issues/524

So please treat the live issue status as authoritative rather than assuming every visible eye-control issue is ready for implementation.

We are looking for contributors

This is the main reason for this post.

If you have experience with any of these areas, we would really value your help:

Python
MediaPipe
computer vision
assistive technology
accessibility engineering
HCI
gaze interaction
head tracking
switch access
AAC
Windows UI Automation
macOS Accessibility APIs
Linux AT-SPI
X11
Wayland
XDG Desktop Portal / RemoteDesktop
pointer/input architecture
testing
privacy engineering
human-subject evaluation
technical documentation

You do not need expertise in everything above.

The roadmap is intentionally decomposed so contributors can work on small pieces.

Expected early implementation areas after the planning gate include things such as:

shared perception signal contracts,
a platform-neutral PointerSink,
deterministic evaluation schemas,
safe gaze-calibration plumbing,
accessibility-tree target contracts,
privacy-safe evaluation tooling.

But please check the issue before starting.

The rule is simple

For eye-control work:

Start an implementation issue only when its live issue says READY and has the help wanted label.

For coding-agent work, the repository uses an additional agent-ready gate.

A visible PLANNED issue exists so people can understand the roadmap. It does not mean the dependencies are ready.

You will also be able to help without writing code

Real hardware testing will eventually be extremely important.

The project has planned small validation slots for:

Windows,
macOS,
GNOME Wayland,
KDE Wayland,
X11,
HiDPI,
multi-monitor setups,
repeated-session testing.

Those tasks are intentionally being kept small.

The goal is eventually to let somebody run one defined test pack on one computer and report:

PASS / PARTIAL / FAIL / BLOCKED

A failure is useful evidence.

But those hardware slots are not ready yet.

They depend on the local evaluator and relevant runtime/safety features being completed first.

Please don't invent your own data-collection process for them.

Accessibility experience is as valuable as code

There is another kind of contribution that matters even more.

If you use or support someone who uses:

eye tracking,
switch access,
AAC,
head tracking,
dwell interaction,
alternative pointing devices,
hands-free computing,

we would like to hear from you.

Developers should not decide accessibility defaults simply by asking:

“What feels comfortable to me?”

Questions such as these need experience and evidence:

How long should dwell be?
Which facial gestures are practical?
Which gestures cause fatigue?
What should happen when tracking disappears?
How should pause/recovery work?
What causes accidental activation?
What feedback is necessary?
Which controls must always remain independently disableable?

If our assumptions are wrong, finding that out before implementation is much cheaper than finding it out after release.

How to get involved right now

Start here:

👉 YazSes repository
https://github.com/MSKazemi/yazses

Then look at:

👉 Hands-free / accessibility epic #102
https://github.com/MSKazemi/yazses/issues/102

👉 Eye/camera architecture and evaluation PR #413
https://github.com/MSKazemi/yazses/pull/413

👉 General contributor guide
https://mskazemi.com/yazses/contribute/start.html

Right now, reviewing PR #413 and challenging the design is genuinely useful.

Tell us:

what assumption is wrong,
which failure mode we missed,
which platform API we should use,
whether the proposed abstraction is unnecessarily complicated,
whether an accessibility interaction would be uncomfortable,
whether the test plan measures the wrong thing.

A thoughtful:

“This architecture will fail because…”

can be more valuable than a large pull request.

What I hope we can build

The goal is not an impressive webcam demo.

The goal is a practical, open-source multimodal hands-free desktop control layer.

Something where a person can:

Speak to create text.

Glance to provide context.

Move using head pose.

Commit using a deliberate switch, dwell, voice or another input.

Combine modalities instead of depending on one perfect sensor.

Stop the system immediately when needed.

Fall back safely when confidence disappears.

And do the processing locally on their own computer.

There is a lot left to build.

That's exactly why we are inviting people in now, while the architecture and implementation boundaries are still open to improvement.

If this is an area you care about, please look at the repository, read the plan, challenge our assumptions, and tell us where you would like to help.

⭐ GitHub: https://github.com/MSKazemi/yazses

👁️ Hands-free epic: https://github.com/MSKazemi/yazses/issues/102

🏗️ Current eye-control planning PR: https://github.com/MSKazemi/yazses/pull/413

Developers, accessibility users, researchers, testers and documentation contributors are all welcome.

IDKMesh: What if AI agents had to prove their work?

Mohsen Seyedkazemi Ardebili — Mon, 21 Sep 2026 19:15:11 +0000

A few days ago I wrote about one result from IDKMesh: a panel of 25 verifiers that, after measuring their shared errors, was effectively worth only one independent verifier.

That experiment is interesting, but it starts in the middle of the story.

So this post is the simpler version:

What is IDKMesh, why am I building it, and how is it supposed to work?

The problem

AI agents are getting very good at producing things.

They can write code, edit files, call tools, generate tests, open pull requests, and sometimes operate infrastructure.

But there is a basic problem:

An agent saying “I finished the task” is not the same as evidence that the task was done correctly.

If we scale from one agent to ten agents, or one hundred agents, generation becomes faster.

Verification can become the bottleneck.

And if all of those agents or reviewers make similar mistakes, adding more of them may only give us more confidence, not more truth.

That is the problem IDKMesh is exploring.

What is IDKMesh?

IDKMesh is an open-source research and engineering project about coordinating:

humans,
AI agents,
software tools,
verifiers,
and different kinds of compute

around uncertain work.

The name comes from a simple idea:

I don’t know. You don’t know. Together, the mesh can discover, build, verify, and learn.

For me, the most important word is not “mesh.”

It is verify.

IDKMesh tries to separate the thing that does the work from the thing that decides whether the work should be trusted.

A simple example

Imagine I give an AI agent this task:

Fix a bug in a Python project.

A normal agent workflow might look like this:

Task
  ↓
AI agent
  ↓
Code change
  ↓
“Done”

That is useful, but the same system that created the answer is also telling us the answer is good.

IDKMesh is aiming for something more like this:

Human goal
    ↓
Bounded Work Unit
    ↓
One or more replaceable workers
    ↓
Candidate results
    ↓
Independent verification
    ↓
Recorded evidence
    ↓
Human / governance decision

The worker can propose.

The verifier can check.

The evidence can be inspected later.

And the final authority stays separate.

Worker success != acceptance.

That small rule changes a lot.

How it works

At a high level, IDKMesh uses a few simple building blocks.

1. Give the worker a bounded job

Instead of giving an agent unlimited authority over a project, work is described as a Work Unit.

The task says what needs to be done, what the boundaries are, and what security constraints apply.

2. Let workers produce candidates

A worker is allowed to try.

There can be multiple attempts.

The first answer does not need to become the final answer.

Workers are replaceable. Their output is a candidate.

3. Verify separately

A verifier should not simply trust the worker’s own completion claim.

Verification has its own plan and produces its own result.

This is where tests, independent reviewers, policy checks, or other evidence can enter.

4. Keep the evidence

IDKMesh treats provenance and verification results as real artifacts, not temporary chat messages.

The goal is to be able to ask:

What was the task?
Who or what produced this result?
What checked it?
Which evidence supported the decision?
Can we replay or inspect it later?

5. Keep the final decision explicit

A worker does not get merge authority just because it says “success.”

A verifier recommendation is also not automatically the final decision.

The integration step stays explicit.

What can you actually use today?

This part is important because I do not want to describe research as a finished product.

The full Verified Swarm Runner is not yet a production-ready multi-agent platform.

The repository is still a research and engineering foundation.

But one useful tool is already available: idkmesh gate-audit.

It takes verdicts from a review panel you already have and asks a practical question:

How many independent votes is this panel actually worth?

The repository also contains experiments, schemas, validators, simulations, interoperability work, and pieces of the future runner.

Try it

The repository is private for now, so the clone and run commands that used to be here cannot be run from outside. The research code and its demo will be easier to try once the project is public.

What I want IDKMesh to become

The long-term idea is bigger than a review tool.

I want to explore whether large groups of humans and AI agents can work on real problems without making trust an afterthought.

That means learning how to:

divide work into bounded tasks,
run multiple independent attempts,
use different workers and tools,
measure whether verification is actually independent,
keep reproducible evidence,
learn from failures,
and scale without giving autonomous workers unlimited authority.

I do not know yet how far this model can scale.

That uncertainty is part of the project.

The repository is designed to test those questions rather than pretend they are already solved.

Why open source?

Because a system about verification should itself be inspectable.

If IDKMesh makes a claim, I want the experiment, code, result, or limitation to be visible.

And I especially want other people to find the assumptions that are wrong.

If you are interested in AI agents, verification, distributed systems, CI, developer tooling, or human-in-the-loop systems, take a look at the project on GitHub, which is private for now.

You do not need to understand the whole project to contribute.

Even showing where the newcomer path is confusing is useful evidence.

Our linter's "safe" autofix would have silently disabled RBAC

Mohsen Seyedkazemi Ardebili — Sun, 20 Sep 2026 21:54:58 +0000

KubeIntellect is an AI agent that runs kubectl against a live cluster, so its tools carry a role check and a human-approval gate. Both depend on one thing: the tool actually receiving the run config the graph injects. This is the parameter that receives it.

config: Annotated[RunnableConfig, InjectedToolArg] = None,  # type: ignore[assignment]

That line is wrong in every way a reviewer is trained to notice. The default is None, but the annotation does not say | None. mypy complains, which is why there is a type: ignore sitting on it. Ruff wants to rewrite the Optional[...] spelling of it. Every instinct says clean this up.

Cleaning it up disables role-based access control.

Why the annotation is load-bearing

LangChain finds the parameter to inject by walking the type hints and comparing with is:

for name, type_ in type_hints.items():
    if type_ is RunnableConfig:
        return name
return None

That is an identity comparison against one exact class object. RunnableConfig | None is a UnionType, not that object. It does not match, so no parameter is selected, so nothing is injected.

Three functions, run against langchain-core 1.6.2:

def bare(cmd: str, config: Annotated[RunnableConfig, InjectedToolArg] = None): ...
def widened(cmd: str, config: Annotated[RunnableConfig | None, InjectedToolArg] = None): ...
def optional_form(cmd: str, config: Annotated[Optional[RunnableConfig], InjectedToolArg] = None): ...

bare           -> injected param: 'config'
widened        -> injected param: None
optional_form  -> injected param: None

No error. No warning. The tool still runs. It just receives config=None forever.

What is actually lost

The caller's role comes off that config, and the fallback is the problem:

user_role = "admin"
if config:
    user_role = (config.get("configurable") or {}).get("user_role", "admin")

With the config gone, config is falsy and every call executes as admin. A readonly API key stops being read-only. The check that returns [Permission Denied] Your API key has read-only access is still there, still covered by its unit tests, still passing — and it never fires, because the role it compares against is no longer the caller's role.

To be precise about the other half, since overstating this would be easy: the human-approval gate does not break the same way. hitl_bypass also comes off the config and defaults to False, so losing the config makes the tool prompt for approval more often, not less. The RBAC default is the one that fails open.

The part that makes it a trap rather than a bug

Nobody has to be careless to introduce this. The tooling volunteers it.

$ ruff check --select UP045 probe.py
UP045 [*] Use `X | None` for type annotations
 --> probe.py:5:35
help: Convert to `X | None`
[*] 1 fixable with the `--fix` option.

The [*] means ruff classifies that rewrite as a safe fix. Not --unsafe-fixes. Plain ruff check --fix — the command people run without reading the diff — converts a working authorization boundary into a no-op.

A behavioral test will not save you either. The tools most likely to get "cleaned up" are the ones that never read config at all: they pass every test they have while silently receiving None. We found exactly that shape in four of our own read verbs. Harmless there, because those verbs make no authorization decision — but it is the same defect, one file away from a place where it matters.

What we did instead of a comment

There is a comment on the line. Comments do not fail CI.

The guard is a test that asserts the annotation itself. It scans every config: Annotated[..., InjectedToolArg] parameter in the source and fails if any is not bare RunnableConfig. Alongside it are two dynamic canaries that build a real tool with each spelling and assert, against whatever langchain version is actually installed, that the required form is injected and the forbidden form is not — so the day the library changes its matching rule, a test says so instead of production.

There is also a test asserting the scanner finds the known sites, because a regex that matched nothing would make every other assertion in the file vacuously true.

The general version

When a framework dispatches on type identity, your annotation is not documentation. It is runtime configuration written in the type language — and anything that "improves" your types can change behavior: a linter, a type checker, an IDE quick-fix, or an agent asked to clean up implicit Optional.

If you have a line like that, the fix is not a louder comment. It is a test that fails when someone improves it.

Code: https://github.com/MSKazemi/kubeintellect
The guard: v4/tests/test_injected_config_invariant.py

DEV Community: Mohsen Seyedkazemi Ardebili

Whisper typed its own prompt into my document

Why this happens

Why the obvious fix doesn't work

The fix: ask a different model

Choosing the threshold

What I took away

My dictation app typed words I never said. The silence gate was quieter than my room.

Whisper does not do "nothing"

Three things that went wrong at once

How to calibrate a gate properly

What I took away

Can an AI agent's decision produce a verifiable receipt? An architecture proposal

1. Why Jev is interesting

2. What NovaFabric adds

3. The idea: a Decision Witness

4. How could the architecture work?

5. Four experiments I'd like to explore

Experiment A: Detect unusual agent behavior

Experiment B: A fast evaluation judge

Experiment C: Replay recorded decisions

Experiment D: Multi-agent delegation

6. How should we evaluate these ideas?

7. A question for the developer community

Projects and research references

The tool that said yes and did nothing: ydotool 0.1.8 exits 0 when it refuses

Some context

How a correct fix broke things

Why nothing noticed

The fix

What I took away

Anatomy of an offline dictation pipeline: where the time actually goes

The nine stages

Where the time goes

The hard part is the last stage

Why offline at all?

The rest of the series

Find where two AI agent runs diverged

What you need

The scenario

Step 1: start the mock model

Step 2: capture the first run

Step 3: capture the second run

Step 4: diff the two runs

How to read it

Step 5: use it as a CI gate

What this does not do

I got tired of typing, so I built a free, offline voice-typing tool (open source, contributors wanted)

Install

Why offline?

Links

Want to contribute? Here is where you can help in 15 minutes

Try it, then tell me

AI Agents Need More Than Logs: Why I Built NovaFabric

Logs are useful, but logs are not evidence

Think of an AI run as an artifact

Five verbs explain most of NovaFabric

1. Capture

2. Seal

3. Replay

4. Diff

5. Audit

Why not just use OpenTelemetry?

What about secrets?

What did the research evaluation find?

An unexpected lesson

NovaFabric today

The bigger idea

Try it

YazSes

Help Us Build an Open, Offline Hands-Free Desktop

First: this is not “perfect eye tracking with a laptop webcam”

Some gaze functionality already exists

Glance-Type gaze routing

Gaze + speech deixis

Other pieces already exist — but are not complete runtime features

Implicit gaze calibration

Head-Pointer is another example

Face gestures should become switches — not direct actions

One camera, not three competing camera loops