What Breaks When You Let an LLM Edit Screen Recordings (and How We Built a Deterministic CLI Harness for macOS)

Zhihuang Lin — Fri, 13 Nov 2026 14:23:34 +0000

Coding agents such as Claude Code and Codex work well on source code for three reasons: source files are plain text, git diff makes every proposed change readable, and compilers or test suites reject broken edits deterministically.

Screen recordings have none of those properties. A flattened MP4 bakes cursor movement, zoom state, and audio into a single video stream. Dumping raw frames into a model context window is slow and expensive. Worst of all, if an LLM edits timeline JSON directly, a single timestamp miscalculation shifts downstream cuts, clips spoken consonants, or deletes the exact two seconds where a user clicked a button on screen.

While building Oil Studio—a native macOS screen recorder and editor written in Swift and Metal for Apple silicon (macOS 15+)—we wanted terminal agents to handle two tedious parts of shipping software demos: cutting retakes out of rough voice-over recordings, and recording clean UI walkthroughs without manual timeline scrubbing.

To make that reliable in practice, we had to move all timeline math, media validation, and safety gates out of the prompt and into a deterministic local CLI harness (oilstudio). Here are the five concrete failure modes we ran into when connecting LLMs to video editing, and how the architecture handles each one.

Architecture overview: separating semantic judgment from media execution

Instead of embedding a chat sidebar inside the video editor or asking users to install external scripts manually, Oil Studio bundles both the oilstudio command-line binary and its Agent Skill directly inside the macOS app (Settings › AI Agent → Install).

Installing from the app sets up the CLI and Skill in one click and keeps them version-locked across app updates. Before running any task, the agent performs a read-only handshake:

OILSTUDIO_SKILL_VERSION=0.9.15 oilstudio status

If skill.compatible is false, the CLI stops immediately and returns an explicit upgrade hint rather than letting an older prompt schema guess at newer project fields.

From there, responsibilities are strictly divided:

+-------------------------------------------------------------------+
| External Agent (Claude Code / Codex)                              |
| - Reads word-timed transcripts, audio metrics, and OCR timelines  |
| - Identifies semantic retakes, false starts, and terminology bugs |
| - Inspects ambiguous spans via on-demand storyboards or WAV clips |
+---------------------------------+---------------------------------+
                                  | Structured JSON (CLI only)
                                  v
+-------------------------------------------------------------------+
| oilstudio CLI & Validator                                         |
| - Enforces projectFingerprint & contextFingerprint consistency    |
| - Validates word boundaries, keptRange, and UI activity overlaps  |
| - Projects post-cut caption timing (--plan) before edits apply    |
+---------------------------------+---------------------------------+
                                  | Staged proposal (proposalId)
                                  v
+-------------------------------------------------------------------+
| Oil Studio macOS App (.oilstudio bundle)                          |
| - Displays cuts, zooms, overlays, and redactions in review panel  |
| - Applies approved changes atomically (single-step Cmd+Z undo)    |
+-------------------------------------------------------------------+

The agent never reads or writes files inside a .oilstudio package directly. Every edit goes through plan validate and plan submit, producing an inspectable proposal inside the native editor.

Failure mode 1: Output-time drift when combining multiple cuts

When a human watches a 10-minute video and writes notes like "cut 01:10 to 01:22, then cut 03:05 to 03:18", they are referring to current playback time. If an LLM proposes 15 cuts in playback/output coordinates, two things go wrong:

Cumulative offset errors. Models frequently miscalculate cumulative time shifts across a dozen edits, causing later cuts to land half a second into the next sentence.
Partial rejection breaks downstream edits. If the user unchecks cut #3 in the review UI because they want to keep that remark, cuts #4 through #15 would shift if they were anchored to output timestamps.

Solution: immutable source-time addressing + pre-cut caption projection

In oilstudio, cuts (decisions), zoom blocks (zooms), and privacy masks (redactions) never use output time. Every edit references either utterance IDs (U0012–U0015) or integer source milliseconds (sourceMs), pinned against two hashes returned by oilstudio context: projectFingerprint and contextFingerprint.

If the user makes a manual trim in the app while the agent is thinking, the fingerprint changes and plan validate rejects the stale plan instead of applying edits to shifted state.

What if the user asks the agent to "cut out my retakes AND fix the misspelled CLI flags in the captions" in a single prompt? Waiting for the user to approve the video cuts before proofreading captions would force a slow two-turn workflow. Instead, subtitles get and subtitles apply accept --plan:

# 1. Export editing context and generate a word-protected pause draft
oilstudio context "Demo.oilstudio" --out "$WORK/01-context.json"
oilstudio pauses "Demo.oilstudio" --out-plan "$WORK/02-plan.json"

# 2. After appending semantic retake cuts to 02-plan.json, validate and stage
oilstudio plan validate "$WORK/02-plan.json"
oilstudio plan submit "$WORK/02-plan.json"

# 3. Project visible post-cut captions and diagnostics without waiting for approval
oilstudio subtitles get "Demo.oilstudio" --plan "$WORK/02-plan.json" --out "$WORK/04-subtitles.json"
oilstudio subtitles apply "Demo.oilstudio" "$WORK/05-subtitle-edits.json" --plan "$WORK/02-plan.json"

Passing --plan projects which captions remain visible after the pending cuts, computes diagnostics (inconsistent casing, residual spoken fillers, line-wrap warnings), and queues the caption fixes right behind the cut proposal (pendingReason: "review_in_progress"). When the user clicks Apply once in the app, timeline cuts, captions, and chapter markers all land in sync.

Failure mode 2: Waveform cutters clip consonants, while text-only LLMs miss vocal hesitations

Automated speech editing fails in opposite ways depending on whether you use audio thresholds or transcripts alone:

Audio-only silence removal clips unvoiced consonants. Sounds like s, t, k, and p carry low RMS energy. Slicing strictly on volume thresholds trims word edges and makes speech sound abrupt. Worse, if an LLM tries to "help" by adding its own ±50 ms padding on top of an existing silence detector, cuts either leave awkward dead air or swallow adjacent syllables.
Transcript-only editing misses untranscribed hesitations and misreads discourse words. When a speaker mumbles an aborted half-word or hesitates ("uh"), local Whisper may emit no text token at all—leaving a visible lump on the audio waveform with an empty transcript span. Conversely, words like "so", "actually", or "like" can be either removable verbal clutter or load-bearing logical transitions depending on the sentence.

Solution: deterministic breath margins + multi-modal evidence on demand

First, oilstudio pauses --out-plan generates a deterministic baseline pause-cleanup draft (defaulting to pauses >= 300 ms, a 50 ms word-boundary guard around verified speech tokens, and optional --keep breath retention—while re-checking Whisper tokens longer than 800 ms against the session noise floor so DTW-stretched tokens do not accidentally protect long silences). The agent appends semantic cuts to that same plan file rather than inventing its own silence offsets.

Second, oilstudio context enriches each utterance with acoustic and visual metadata so the model knows where text alone is insufficient:

utterances[].audio: recognition confidence, speaking pace, loudness, internal pauses, and unrecognized spans (timestamps where voice activity exists on the waveform without recognized words).
screen: an OCR text timeline of what was visible in the recorded window.
fillerCandidates: detected hesitation sounds and candidate filler words that require contextual judgment.

Rather than extracting frames for an entire 15-minute recording, the agent reads the text transcript end-to-end first, then inspects media only around ambiguous candidates:

# Render one composite sheet (frames + waveform + word timing) for utterances 12..18
oilstudio storyboard "Demo.oilstudio" --utterances U0012-U0018 --out "$WORK/check-u12.png"

# Or render a short WAV preview across a proposed splice to verify the audio join
oilstudio plan preview "$WORK/02-plan.json" --index 0 --out "$WORK/join-0.wav"

In a storyboard sheet, two consecutive utterances with similar waveform contours where the first stops mid-phrase alongside a repeated UI state confirm a retake. A read-only coverage script distributed with the Skill (check_filler_coverage.py) then verifies that every fillerCandidates item is explicitly resolved (cut, keep, review, or covered by a larger passage cut) and calculates time savings from the mathematical union of source intervals rather than double-counting overlapping ranges.

Failure mode 3: Deleting a "verbal retake" that contained a non-repeated UI click

In a talking-head podcast, if you say a sentence twice and the second take is cleaner, you can always cut the first take.

In a software tutorial, the audio might be a retake while the screen is not. Suppose the presenter clicks "Create API Key" during take #1, stumbles over their explanation, and repeats the explanation smoothly in take #2 without clicking the button again because the modal is already open. Deleting take #1 removes the actual UI action from the video, leaving viewers staring at a modal that appears out of nowhere.

Solution: activity-overlap validation and `uncheckedByDefault`

We tried warning against this in the system prompt first. Under long-context pressure, models still occasionally proposed cutting across active UI steps. So we moved the rule into plan validate.

Here is what an actual semantic cut entry looks like in 02-plan.json:

{
  "schemaVersion": 1,
  "projectFingerprint": "sha256:9c4e...",
  "contextFingerprint": "sha256:31a8...",
  "summary": "Remove 3 retakes and 2 verbal slips, shortening recording by 48s",
  "decisions": [
    {
      "decision": "cut",
      "confidence": "high",
      "range": { "from": "U0012", "to": "U0015" },
      "category": "abandoned_take",
      "removedText": "Now let's open the export menu and set the frame rate to—sorry, let me redo that.",
      "keptText": "Next, open the export menu and choose 60 fps.",
      "keptRange": { "from": "U0016", "to": "U0017" },
      "reason": "Later take repeats the full instruction cleanly.",
      "screenAction": "redundant",
      "visualAssessment": "The same export popover is opened again from scratch at U0016."
    },
    {
      "decision": "review",
      "confidence": "low",
      "sourceMs": [61200, 64850],
      "category": "filler",
      "removedText": "I usually leave hardware encoding on default here.",
      "keptText": "",
      "reason": "Sounds conversational, but mentions a setting not repeated later."
    }
  ]
}

plan validate enforces three structural rules on these entries:

Mandatory replacement proof (keptRange). Every abandoned_take cut must point to the exact retained utterances (keptRange) that cover the same information.
UI activity protection. The validator cross-checks every proposed cut against recorded pointer clicks, keystrokes, and screen changes in activity. If a cut overlaps an interaction and does not include "screenAction": "redundant" paired with a concrete "visualAssessment", the validator automatically flags the entry as uncheckedByDefault in Oil Studio's review panel so it will not apply unless the human explicitly ticks it.
Duration guardrails. Standard non-silence cuts longer than 30 seconds are also marked uncheckedByDefault unless they are high-confidence abandoned_take spans backed by a valid keptRange (up to 3 minutes). Low-confidence review items stay out of the executable cut list entirely and are reported separately for human judgment.

Failure mode 4: UI automation recordings have teleporting cursors and dead loading waits

The other workflow we wanted was letting an agent record a product demo of a native macOS window or web app after a UI change.

If you record the screen while an automation tool clicks DOM selectors or accessibility elements, the raw video looks unnatural: the cursor jumps instantaneously from coordinate to coordinate in a single frame, followed by 2–4 seconds of motionless waiting between steps while pages load or the agent plans its next action.

Solution: unbaked vector cursor synthesis + post-capture idle stripping

Because a .oilstudio bundle stores video at native window resolution and keeps pointer trajectories as separate vector metadata rather than baking the OS cursor into the video frames, oilstudio record decouples the rendered cursor from the physical mouse:

# 1. Inspect available windows and center/resize the target window to 1280x800
oilstudio record sources
oilstudio record prepare --source window-123 --size 1280x800 --center --activate
oilstudio record start --source window-123

# 2. Declare each target coordinate before clicking (or pass --perform)
oilstudio record click --at-fraction 0.74,0.18 --perform

# 3. Stop capture and strip silent waiting gaps while rebuilding click zooms
oilstudio record stop
oilstudio idle "/path/to/New Recording.oilstudio" --submit

Two details make the resulting video look clean:

Declared intent trajectories. During agent recording, the physical macOS pointer is not recorded at all. Each record click or record move report supplies target coordinates (either screen points via --at or normalized 0–1 window fractions via --at-fraction), from which Oil Studio synthesizes a continuous spring-smoothed cursor path, click animation, and motion blur at 60 fps.
Idle compression with zoom realignment. Running oilstudio idle <project> --submit afterward detects silent waiting spans (default threshold 1500 ms, keeping 400 ms around actions), aligns any slightly delayed click reports with actual visual frame changes, and recalculates automatic zoom blocks across the tightened timeline. (Permission prompts remain strictly human-controlled: only the user can grant macOS Screen Recording access or enable Oil Studio's agent-recording toggle, and a visible recording pill stays on screen throughout capture.)

Failure mode 5: Privacy redactions drift under auto-zoom or leak into agent vision

Screen recordings of developer tools regularly contain API keys, customer emails, or internal URLs in corners of the window. Combining auto-zoom composition with external AI agents introduces two easy ways to leak those secrets:

Canvas-space coordinate drift. If you attach a blur or pixelation box to the output video canvas, the moment an auto-zoom block scales the window to 1.8× and pans toward a button, the underlying window moves while the mask stays put—uncovering the secret.
Unmasked frame extraction. If a user masks an API key in the editor and then asks an external cloud LLM agent to adjust zooms or proofread captions, sending raw source frames to frame or storyboard would expose the unredacted pixels to the model provider.

Solution: pre-compositor source-frame coordinates + pre-redacted agent frames

In Oil Studio's Metal rendering pipeline, redactions (pixelate, blur, or solid fill) are defined in normalized 0–1 source-frame coordinates (region: {x, y, width, height}) over [sourceStartMs, sourceEndMs] and rendered before camera zoom, cursor overlays, or canvas framing run. When the virtual camera zooms in or pans across the window, the pixelation stays locked to the underlying UI element.

Just as importantly, that redaction pass runs before oilstudio frame or oilstudio storyboard exports any PNG or runs OCR for an external agent, and any plan proposal that removes or shrinks an existing redaction requires manual confirmation in the app.

One deliberate limitation we document clearly in the Skill: a redaction is a fixed rectangle in source-frame coordinates, not an optical-flow object tracker. If a sensitive string scrolls up a terminal or web page, the agent (or user) must specify separate time ranges and rectangles and verify coverage on the output timeline (oilstudio frame <project> --output-ms ... --text --out ...) rather than assuming a static box will chase scrolling text.

Trade-offs and when another tool is a better fit

Designing around native macOS frameworks, local structured project bundles, and a local CLI meant making deliberate scope trade-offs. Depending on your workflow, those trade-offs may be dealbreakers:

macOS 15+ on Apple silicon only. Keeping the capture pipeline, Metal compositor, and on-device Whisper engine small and fast meant skipping Electron and legacy cross-platform layers. If your team needs Windows, Linux, or Intel Mac support, OBS Studio, Descript, or Loom are better choices.
Structured .oilstudio projects, not arbitrary MP4 editing. Both the editor and oilstudio CLI rely on separate video, audio, cursor, and click metadata tracks (though oilstudio import can convert existing third-party auto-zoom recording projects read-only without touching the originals). If you need a general-purpose NLE for camera footage or flat MP4s, DaVinci Resolve or Final Cut Pro is the right tool.
Local file export rather than hosted share links. Oil Studio exports standard 60 fps MP4, GIF, and SRT files locally with no cloud account. If your primary use case is recording a quick bug report and pasting an instant hosted URL with timestamped team comments into Slack or Linear, Loom or CleanShot Cloud fits that workflow much better. (We maintain a detailed 7-tool Mac screen recorder technical comparison covering exact pricing models, audio tracks, redaction behavior, and platform limits across these tools.)

Wrapping up

The biggest lesson from building oilstudio was that making non-text software "agent-friendly" has very little to do with clever prompt writing. Prompts cannot reliably add millisecond offsets, protect unvoiced consonants, or notice that a verbal stumble happened at the exact moment a user clicked a critical UI button.

Putting immutable source-time addressing, acoustic/activity metadata, and strict structural validation inside a local CLI lets the LLM focus on what it is actually good at—reading a messy 10-minute rough take and understanding human intent—while keeping every edit inspectable and reversible with a single ⌘Z.

If you want to inspect the full CLI command reference and workflow rules, see the AI Agent CLI technical documentation on studio.oiloil.org. Recording, auto-zoom editing, local Whisper transcription, and the built-in Agent CLI are free to try locally on Apple silicon Macs; if you experiment with driving native desktop apps from Claude Code or Codex, I would love to hear how you structure your own validation harnesses.

DEV Community: Zhihuang Lin