A safety gate with no AI in it: building JAMES, a touch-free maintenance copilot

Built: first version over 25 to 26 Sep 2026 for AgentX-2026, public on GitHub since 26 Sep.
Status: still being built. ATLAS, CANTUS, SIGHT and the hash-chained ledger landed through October; next is a tuning session on real hands.

A technician with gloves on and both hands busy cannot scroll a PDF manual.

That one sentence is the whole reason JAMES exists. The fix for a vibrating pump lives in four places at once: the manual, the machine's sensor log, the notes from the last time someone repaired it, and the head of whoever did that repair. The technician is standing in front of the pump, holding a spanner, and every one of those sources assumes a free hand and a screen.

So I built a computer you operate by pointing at the air.

JAMES (Joint Autonomous Multimodal Execution System) watches a webcam, listens to a microphone, finds the right procedure, checks the sensor log and past repairs, and walks the technician through the fix one step at a time. Point at the machine, ask out loud, pinch to confirm each step. It runs entirely on a MacBook Air: local model, local search, local memory, cloud off.

It took first prize at AgentX-2026, a national-level hackathon at TKREC Hyderabad. But the part I want to write about is not the demo. It is the three ideas that made it safe to put in front of a jury, and the ugly bugs that taught me each one.

Idea 1: the model can suggest a step. Only plain code can allow it.

JAMES is a small team of agents, each with one job:

Agent	Job	Built with
LENS	Sees hands and the machine's asset tag	MediaPipe Hand Landmarker, OpenCV ArUco
ECHO	Hears the request	faster-whisper, Qwen3-VL 4B via Ollama
PAGE	Finds the procedure and cites the page	PyMuPDF, BM25, nomic-embed-text in Qdrant
PULSE	Reads the sensor log	pandas, and it never invents a reading
LEDGER	Remembers approved jobs	append-only SQLite
FOREMAN	Plans and decides the likely cause	LangGraph fan-out to PAGE, PULSE, LEDGER
WARDEN	Blocks unsafe steps	plain Python and a rule file. No AI.

WARDEN is the important one. The model is allowed to be creative about what to check next. It is never allowed to decide whether a step is safe. That decision lives in a rule engine anyone can read, with rules that ship in an "industry pack" next to the manual and the asset tags.

In the demo, step 4 of the pump repair is "Remove the coupling guard". If the technician skips lockout, WARDEN refuses:

BLOCKED BY SAFETY POLICY   LOTO-01, ZERO-01
Remove the coupling guard
Lockout and zero energy are not confirmed.

Two pinches later (lockout confirmed, zero energy confirmed), WARDEN re-checks and the step opens. The seeded safety suite runs 8 known violations and 4 legitimate sequences: 8 of 8 violations blocked, 0 of 4 false blocks.

The pack is hash-checked too. If anyone edits a rule file after its manifest was written, the app refuses to start. A safety rule you can quietly change is not a safety rule.

Idea 2: the pinch has to be a pinch

The first live run on the Mac taught me that hand tracking fails quietly, and that the failures look exactly like intent.

• A fist reads as a pinch. In a fist the thumb lies on the index finger, so the thumb-to-index distance drops under the pinch threshold. Closing your hand to rest it was confirming steps.
• Reaching for a tool reads as a pinch. Hold a pinch, move your hand toward the camera to grab a spanner, and you have "confirmed" whatever was on screen.
• Losing the hand was silent. In low light or with dark gloves the tracker just stopped, and the app kept showing the last known state as if nothing happened.

None of these were tuning problems. They were design problems, so each got a design fix:

• Fist latch. A pinch that begins as a fist is ignored until the fingers open again. The test for "this is a pinch, not a fist" is that the index finger is at least half extended.
• Reach guard. While a pinch is held, JAMES tracks hand size (the geometric mean of palm length and knuckle width, so tilting moves it about half as much as reaching). If it changes more than 30%, the hand is moving to or from the camera. The pinch is dropped and must be released before it counts again. A reach never confirms a step.
• SIGHT. A small module that watches frame brightness and how often the hand appears and vanishes. When tracking can't be trusted it says so on screen, and a lost hand is marked with a still amber X where it was last seen, the way cockpit displays mark invalid data. SIGHT can warn. It can never press or unlock anything.

The research backed this up. One study of 52 person-hours of everyday activity found a naive gesture spotter fired a false positive about every 5.1 seconds. That number changed how I thought about every gesture in the app: the default assumption is "this was an accident", and the gesture has to prove otherwise.

Idea 3: every answer has a source, and every record can be checked later

A maintenance copilot that makes things up is worse than no copilot. So:

• Steps come from real numbered lines in the manual, word for word, each with its page.
• Readings come only from the supplied sensor log. PULSE never estimates a value.
• Past repairs are recalled only if a senior approved them, they were not revoked, and they are for the same class of equipment. Memory can suggest. It never decides a cause on its own.
• The status bar always says which backend is actually live (local model or keyword rules, BM25 or BM25 plus Qdrant), so the demo can't overstate itself.

And because buyers in regulated plants ask whether a record has been altered, every logged event is hash-chained: each event stores SHA-256 of the previous hash plus the event itself. A read-only checker walks the chain and names the first edited, reordered or removed event. It is tamper-evident, not tamper-proof, and the README says exactly that.

How it was built

In short iterations with the same loop every time:

1. Review the build and list findings as P0 (must fix before a live demo) or P1 (should fix).
2. Fix each one and add a regression test that fails without the fix.
3. Run it live on the Mac. Every sentence that misbehaved in the app log became a new test, and the whole log was replayed through old and new code.

Some of my favourite bugs from that log:

• Playing the neural voice through PortAudio in the same process as the microphone crashed on macOS. Each sentence now plays through afplay in its own process.
• A model plan searched a website the user never named. Search and collect steps must now name a site the user actually said.
• Gesture timers reset when the hand moved at all, so "explode the model" felt dead, then fired all at once.

The repo has 607 tests passing, and the build log marks every claim as LIVE (seen on the Mac), TESTED (automated) or UNVERIFIED. That labelling is the most useful habit I picked up on this project. It stops me quoting a number I never actually saw.

Then I made it fun

Two side apps grew out of the hand tracking:

• ATLAS draws the targeted machine as an exploded line drawing and lights the part the current step touches: amber when WARDEN allows it, red when WARDEN is blocking it. It also runs as its own app with a library of openly licensed models (NASA's Perseverance rover, Ingenuity, the James Webb Space Telescope) that you take apart by opening your hand.
• CANTUS is a rhythm game in the launcher. Blocks fly towards you on the beat; swing through them, left hand red, right hand blue.

A pinch inside ATLAS can never confirm a maintenance step. ATLAS owns the hand while it is open. Fun is allowed. Ambiguity about what a gesture means is not.

What is not done

• It ships with sample data and demo safety rules. Nothing here is certified for real safety use.
• Industry packs are hash-checked but not signed yet.
• It runs on macOS only today.
• SIGHT thresholds, the reach tolerance and the gesture timings still need a proper tuning session on the Mac before I quote them as final.

Try it

The code is public under MIT: github.com/victorisaacchinta-eng/james. The product site is jamesagent.vercel.app. python app.py --sim runs a full rehearsal with no camera or microphone.

If you have built gesture interfaces and have opinions about false activations, I would genuinely like to hear them in the comments.

---

I wrote this with AI assistance, working from the repo and its build log. Every number above comes from the repo.