When a designer says “remove these yellow icons,” the intent is clear to the person sitting in front of the screen. It is much less clear when a coding agent receives only a screenshot, a window title, and a sentence.
I built UI Screen Context to close that gap.
The handoff
Start a recording, point at the element, and speak. The app follows the active window as you work, records the pointer target, captures screenshots and timing, and transcribes speech locally on the Mac. When you stop, it creates a structured context folder that you can paste into a coding agent.
The agent gets:
- the recorded windows and their timing;
- pointer events and the text target when one can be confirmed;
- recommended evidence images;
- the original transcript and the speech language;
- clear limits when the capture cannot prove something.
That last part matters. A brief window switch can be missed. A frame held during a transition is not proof that the UI changed. The export says this plainly so the agent has less room to guess.
Why local?
The capture and speech models run locally. The app can use macOS speech or a local Whisper model, including an existing model folder. The generated context keeps the language the user actually spoke instead of silently translating it to English.
Who it is for
It is useful when a product designer, developer, or QA person needs to show a change across a real interface and wants the agent to start with evidence rather than a long explanation. It is also useful for debugging the handoff itself: you can inspect exactly what the agent will see before asking it to edit code.
The app is free and Apache-2.0 licensed.
Source: https://github.com/behavio1/behavio-context
Demo video:
Website: https://behavio.one
I’m sharing it because the format is still evolving. If you use coding agents for UI work, what is the one piece of context you wish they received every time?
Top comments (1)
The detail about stating clear limits in the export — "a frame held during a transition is not proof that the UI changed" — is the right design choice. A context package that overstates certainty forces the agent to resolve ambiguity on its own, usually by guessing. Explicit "I couldn't confirm this" is more useful than confident-sounding noise.
The question you close with is interesting. The piece of context I most consistently find missing is the reason the change was requested, not just what changed. A screenshot + pointer target tells an agent what element to touch, but "remove these yellow icons" without knowing they were added as an accessibility workaround three months ago leads to changes that technically satisfy the request but break something upstream. Intent plus evidence together are the real context package — a short "why" alongside the recorded pointer events would make this significantly more useful.
Local transcription with language preservation is also the right call. Silent English normalization creates subtle failures in multilingual teams where the original phrasing carries meaning the translation drops.