YazSes started with a simple idea:
Hold a key, speak, and have your words typed into almost any desktop application — locally, without sending your audio to a cloud service by default.
GitHub:
👉 https://github.com/MSKazemi/yazses
But voice is only one part of hands-free computer interaction.
What if the same open-source project could eventually let you:
- look toward a window and dictate into it,
- look roughly at something and say “close this”,
- move a pointer using your head,
- use an intentional facial gesture as a switch,
- use dwell when another physical switch is unavailable,
- combine gaze + speech + head movement + switch input,
- and safely fall back when the camera is uncertain?
That is the direction of the YazSes eye / camera control programme.
And this is an invitation to help us build it.
First: this is not “perfect eye tracking with a laptop webcam”
This distinction is important.
A normal laptop webcam is not the same thing as a dedicated infrared eye tracker.
We do not want to pretend otherwise.
Commodity-camera gaze may be useful for answering a coarse question such as:
Which window or region is the user looking toward?
It should not automatically be treated as a precision mouse pointer capable of selecting a tiny icon or placing a text caret.
Our current design gives different modalities different jobs.
| Task | Input that may fit it best |
|---|---|
| Generate text | Speech |
| Choose a pane/window | Coarse gaze |
| Resolve “this” or “that window” | Gaze + speech |
| Move a pointer continuously | Head pose |
| Commit an action | Face switch, voice, dwell, keyboard, EMG, etc. |
| Precision gaze pointing | Future dedicated eye-tracker backend |
| Recover from uncertainty | Current focus, voice controls, keyboard/switch, or explicit abstention |
One principle from the design summarizes this:
Perception proposes; an intent-bearing action commits.
Your eyes can help say where.
Your voice or another deliberate action can say what to do.
That matters because our eyes are always looking at something.
Looking at a button should not automatically mean clicking it.
Some gaze functionality already exists
This is not just a concept document.
YazSes already contains working gaze-related code.
Glance-Type gaze routing
The implemented gaze path can sample webcam gaze, use a calibration map, resolve a target window, and route the next dictation toward it.
When confidence is insufficient, it falls back rather than blindly choosing another window.
The relevant runtime code already exists in the repository:
src/yazses/gaze/mediapipe_backend.py
src/yazses/gaze/calibrate.py
src/yazses/gaze/targeter.py
src/yazses/gaze/route.py
Look-to-window routing is currently associated particularly with the X11 desktop path, and cross-platform semantics are one of the things we still need to improve.
Gaze + speech deixis
YazSes also has a particularly interesting interaction:
look + speak
For example:
“focus that window”
or:
“close this”
The speech provides the operation while the gaze snapshot provides context for what this or that refers to.
Destructive gaze-routed actions can go through confirmation instead of being executed blindly.
The important part is that gaze provides context rather than becoming an automatic click mechanism.
Other pieces already exist — but are not complete runtime features
This is where the project becomes interesting for contributors.
Several useful cores already exist.
Implicit gaze calibration
There is already a pure implicit-calibration implementation:
src/yazses/gaze/implicit.py
It can update a calibration model from (gaze, click) observations using recursive least squares.
It also contains important safeguards:
- reject low-confidence samples,
- reject extreme residuals,
- use a forgetting factor,
- compare a candidate calibration against a baseline,
- keep the original calibration unless the candidate performs better on held-out samples.
But the pure estimator is not the same as a complete feature.
The runtime still needs proper opt-in click observation, sample association, persistence and rollback work.
Head-Pointer is another example
YazSes already contains the pure head-pose-to-pointer core:
src/yazses/headpointer/pointer.py
Including:
- yaw/pitch → cursor delta mapping,
- a configurable dead zone,
- and a dwell-click state machine.
Conceptually:
head movement
↓
yaw / pitch
↓
dead zone + mapping
↓
pointer movement
↓
optional dwell
But this is not currently a complete Head-Pointer desktop feature.
The missing engineering includes:
- shared camera/head-pose input,
- a platform-neutral pointer interface,
- X11 output,
- Wayland output,
- macOS output,
- Windows output,
- pause/resume,
- recentering,
- signal-loss handling,
- dwell feedback,
- and reliable emergency stopping.
A continuous accessibility control without a dependable stop path is not something we want to ship.
Face gestures should become switches — not direct actions
Another planned capability is deliberate facial input.
Imagine intentionally:
- opening your mouth,
- raising your eyebrows,
- making another calibrated expression,
and having that produce an abstract switch event.
Not:
eyebrow movement → directly click something
but:
camera
↓
face/blendshape signal
↓
gesture detector
↓
abstract switch event
↓
user-configured action
That difference matters.
The camera detector should recognize intent.
It should not contain desktop-action policy.
The eventual switch could be mapped to things such as:
- start/stop dictation,
- click,
- confirm,
- cancel,
- pause the pointer,
- participate in a gesture chord.
YazSes already has an input-agnostic Gesture Chord core:
src/yazses/gesture/chords.py
But the Face-Gesture Switch itself does not currently have a runtime detector/adapter implementation.
That distinction was important enough that we are currently correcting stale documentation which previously implied otherwise:
👉 https://github.com/MSKazemi/yazses/pull/541
One camera, not three competing camera loops
If gaze, head pose and facial switches all use the same webcam, each feature should not independently open the camera and run its own face model.
The architecture under review proposes a shared perception source:
camera
|
v
+-------------------+
| Shared Perception |
| one camera owner |
| one face model |
+---------+---------+
|
+-----------------+-----------------+
| | |
v v v
GazeSignal HeadPoseSignal FaceSignal
| | |
v v v
calibration Head-Pointer Face Switch
| | |
+-----------------+-----------------+
|
safety / feedback
|
YazSes
Raw frames stay inside the perception layer.
Consumers should receive small derived signals.
That has several advantages:
- one webcam owner,
- less duplicated computation,
- fewer camera-contention bugs,
- easier testing,
- clearer privacy boundaries,
- easier failure handling.
It also means most of the system can be tested with deterministic numeric inputs rather than requiring a physical webcam in CI.
Privacy needs to be part of the architecture
Camera-based accessibility software deserves careful handling of data.
For this programme, the intended boundary is straightforward:
Camera processing stays local.
Raw webcam frames should not become part of normal logs, evaluation exports, issue reports or network requests.
For engineering evaluation, we care about derived measurements such as:
- target successes,
- wrong-target events,
- fallback events,
- calibration error,
- confidence,
- false activations,
- accidental clicks,
- pointer acquisition time,
- recovery after signal loss,
- display resolution and scaling,
- OS/session,
- broad camera category.
We do not need public contributors to upload:
- face photos,
- face videos,
- raw webcam frames,
- private screen contents,
- private dictated text,
- medical information,
- passwords or tokens,
- device serial numbers.
There is another important boundary.
Public GitHub QA is not automatically research participation
We eventually want good evidence about whether these interfaces actually work.
But a person who posts an engineering result on GitHub should not silently become a participant in a research paper.
The programme therefore separates:
CI / synthetic evidence
↓
community engineering QA
↓
separately consented research
A future human-participant study should have its own protocol and consent process.
That separation is already part of the planning work.
“It worked on my laptop” is not enough
Camera interaction can vary dramatically between environments.
We eventually need evidence across:
- Windows,
- macOS,
- Linux X11,
- GNOME Wayland,
- KDE Wayland,
- different webcams,
- different laptops,
- different screen resolutions,
- display scaling,
- multi-monitor configurations,
- different people,
- repeated sessions.
The programme therefore proposes an evidence ladder ranging from pure deterministic tests to real hardware and, separately, controlled human studies.
The basic idea is:
pure unit/state tests
↓
cross-platform contract tests
↓
synthetic trace replay
↓
one real computer
↓
another physical computer
↓
different people
↓
repeated sessions
↓
controlled research study
A green unit test does not prove webcam usability.
One working laptop does not prove Windows, macOS or Linux support.
And 1,000 trials from one person are still not 1,000 participants.
We also want to explore semantic grounding
One research/development direction is especially interesting.
Instead of asking the webcam to become dramatically more precise, perhaps precision can come from combining coarse spatial information with UI semantics.
Imagine looking approximately toward a group of controls and saying:
“click Save”
The gaze or head pointer gives an approximate region.
The operating system accessibility tree can provide structured nearby candidates.
Speech provides the requested operation.
Then the resolver can produce:
grounded target
or:
ambiguous — don't guess
or:
unresolved — don't act
Conceptually:
coarse gaze / pointer location
+
structured UI candidates
+
spoken intent
↓
target resolver
↓
grounded / ambiguous / unresolved
↓
safety policy
↓
action
The current proposal deliberately starts with structured accessibility information rather than making screenshots, OCR or vision-language models the default.
If structured UI data is insufficient, we should measure that gap first.
Where are we today?
This is important:
The complete hands-free system described in this post is not released.
As of September 23, 2026, the architecture, implementation sequence, evaluation model and contributor workflow are being reviewed in:
👉 https://github.com/MSKazemi/yazses/pull/413
The umbrella hands-free epic is:
👉 https://github.com/MSKazemi/yazses/issues/102
PR #413 is a planning / architecture / evaluation PR.
It does not itself ship:
- Head-Pointer runtime,
- Face-Gesture Switch runtime,
- dedicated eye-tracker support,
- a completed hands-free preset,
- research results,
- or new webcam data collection.
This separation is deliberate.
We want the architecture, privacy rules, safety boundaries, testing approach and dependencies to be understandable before dozens of implementation PRs start landing.
There is also active repository coordination work around the design records, including an ADR-numbering collision tracked in:
👉 https://github.com/MSKazemi/yazses/issues/524
So please treat the live issue status as authoritative rather than assuming every visible eye-control issue is ready for implementation.
We are looking for contributors
This is the main reason for this post.
If you have experience with any of these areas, we would really value your help:
- Python
- MediaPipe
- computer vision
- assistive technology
- accessibility engineering
- HCI
- gaze interaction
- head tracking
- switch access
- AAC
- Windows UI Automation
- macOS Accessibility APIs
- Linux AT-SPI
- X11
- Wayland
- XDG Desktop Portal / RemoteDesktop
- pointer/input architecture
- testing
- privacy engineering
- human-subject evaluation
- technical documentation
You do not need expertise in everything above.
The roadmap is intentionally decomposed so contributors can work on small pieces.
Expected early implementation areas after the planning gate include things such as:
- shared perception signal contracts,
- a platform-neutral
PointerSink, - deterministic evaluation schemas,
- safe gaze-calibration plumbing,
- accessibility-tree target contracts,
- privacy-safe evaluation tooling.
But please check the issue before starting.
The rule is simple
For eye-control work:
Start an implementation issue only when its live issue says READY and has the help wanted label.
For coding-agent work, the repository uses an additional agent-ready gate.
A visible PLANNED issue exists so people can understand the roadmap. It does not mean the dependencies are ready.
You will also be able to help without writing code
Real hardware testing will eventually be extremely important.
The project has planned small validation slots for:
- Windows,
- macOS,
- GNOME Wayland,
- KDE Wayland,
- X11,
- HiDPI,
- multi-monitor setups,
- repeated-session testing.
Those tasks are intentionally being kept small.
The goal is eventually to let somebody run one defined test pack on one computer and report:
PASS / PARTIAL / FAIL / BLOCKED
A failure is useful evidence.
But those hardware slots are not ready yet.
They depend on the local evaluator and relevant runtime/safety features being completed first.
Please don't invent your own data-collection process for them.
Accessibility experience is as valuable as code
There is another kind of contribution that matters even more.
If you use or support someone who uses:
- eye tracking,
- switch access,
- AAC,
- head tracking,
- dwell interaction,
- alternative pointing devices,
- hands-free computing,
we would like to hear from you.
Developers should not decide accessibility defaults simply by asking:
“What feels comfortable to me?”
Questions such as these need experience and evidence:
- How long should dwell be?
- Which facial gestures are practical?
- Which gestures cause fatigue?
- What should happen when tracking disappears?
- How should pause/recovery work?
- What causes accidental activation?
- What feedback is necessary?
- Which controls must always remain independently disableable?
If our assumptions are wrong, finding that out before implementation is much cheaper than finding it out after release.
How to get involved right now
Start here:
👉 YazSes repository
https://github.com/MSKazemi/yazses
Then look at:
👉 Hands-free / accessibility epic #102
https://github.com/MSKazemi/yazses/issues/102
👉 Eye/camera architecture and evaluation PR #413
https://github.com/MSKazemi/yazses/pull/413
👉 General contributor guide
https://mskazemi.com/yazses/contribute/start.html
Right now, reviewing PR #413 and challenging the design is genuinely useful.
Tell us:
- what assumption is wrong,
- which failure mode we missed,
- which platform API we should use,
- whether the proposed abstraction is unnecessarily complicated,
- whether an accessibility interaction would be uncomfortable,
- whether the test plan measures the wrong thing.
A thoughtful:
“This architecture will fail because…”
can be more valuable than a large pull request.
What I hope we can build
The goal is not an impressive webcam demo.
The goal is a practical, open-source multimodal hands-free desktop control layer.
Something where a person can:
Speak to create text.
Glance to provide context.
Move using head pose.
Commit using a deliberate switch, dwell, voice or another input.
Combine modalities instead of depending on one perfect sensor.
Stop the system immediately when needed.
Fall back safely when confidence disappears.
And do the processing locally on their own computer.
There is a lot left to build.
That's exactly why we are inviting people in now, while the architecture and implementation boundaries are still open to improvement.
If this is an area you care about, please look at the repository, read the plan, challenge our assumptions, and tell us where you would like to help.
⭐ GitHub: https://github.com/MSKazemi/yazses
👁️ Hands-free epic: https://github.com/MSKazemi/yazses/issues/102
🏗️ Current eye-control planning PR: https://github.com/MSKazemi/yazses/pull/413
Developers, accessibility users, researchers, testers and documentation contributors are all welcome.
Top comments (0)