From Screen Recording to AI Agent: How I Built a Skill Extraction Pipeline

The Problem

I've been building browser automation for years. The workflow was always the same:

1. Record a video of the task
2. Write a Playwright script based on memory
3. Debug for 3 hours because I missed a step
4. Repeat

Writing automation scripts from memory is error-prone. You always miss edge cases, timing issues, or subtle UI changes.

The Insight

What if the AI could watch the recording and extract the exact steps? Not just generate code, but understand the intent behind each action?

That's what led me to build SkillForge.

How It Works

SkillForge (currently on Product Hunt) takes a different approach:

1. Record your screen doing any browser workflow
2. AI analyzes frame-by-frame — clicks, typing, navigation, waits
3. Exports structured SKILL.md — a standard format any AI agent can execute
4. Test and refine — run the skill, see where it breaks, re-record

The key insight: screen recordings capture ground truth. Unlike written documentation, they don't lie about what actually happened.

The Pipeline

Here's what happens under the hood:

• Vision AI processes the recording, identifying UI elements and actions
• Semantic extraction understands what each action means (not just "click at x,y" but "click the submit button")
• SKILL.md generation outputs a structured file with steps, selectors, and context
• Agent execution — Claude Code, Codex, or any agent framework can run the skill

Real Example

I recorded myself setting up a new AWS EC2 instance. The old way: 200 lines of bash script that broke on the third run. The SkillForge way: 5-minute recording → SKILL.md file that works every time.

The skill file includes:

• Exact selectors for each form field
• Timing information (wait for page load, wait for API response)
• Error handling (what to do if the instance type is unavailable)
• Context (why each step matters)

Why This Matters

We're moving from writing automation to demonstrating automation.

Instead of learning Playwright syntax, you just do the task once. The AI extracts the knowledge and makes it reusable.

Try It

SkillForge is free to try — record your first workflow at skillforge.expert. 20 free credits on signup.

We're also on Product Hunt this week: https://www.producthunt.com/products/skillforge-2

Would love feedback from the Dev.to community. What workflows would you want to automate?

---

Built with vision AI, browser automation, and a lot of frustration with brittle scripts.