Quinn

Posted on Mar 17

I Built an AI Crochet Pattern Engine Using Browser Agents (Debugg.ai + Claude Code + OpenAI) for my Sister

#ai #browser #agenticbrowser

Most people think browser agents are for QA, scraping, or growth automation.

I wanted to test something weirder:

Can you fully automate a creative, messy, human workflow like crocheting?

Turns out — yes. And it’s a perfect stress test for agent reliability.

🧠 The Idea

Crochet patterns are essentially programs written for humans:

Step-by-step instructions
Stateful execution (you can’t skip rows)
Error-prone (one missed stitch breaks everything)
Inconsistent formatting across the internet

This makes them ideal for testing agentic workflows over real UIs.

To ground this, I used real-world pattern flows from sites like the one I built for my sister:
👉 Crochets.site Crocheting Page

There are many types of general resources sites that provide structured + semi-structured patterns, materials, and UI interactions — perfect for agent testing.

🏗️ System Architecture

                ┌──────────────────────┐
                │   Surfs.dev Agents   │
                │ (browser execution)  │
                └─────────┬────────────┘
                          │
        ┌─────────────────┼─────────────────┐
        │                 │                 │
        ▼                 ▼                 ▼
 Pattern Sites     Pattern Editor UI   Test Runner (Debugg.ai)
 (scraping)         (Next.js app)        (validation layer)

        │
        ▼
   OpenAI / Claude
 (parsing + generation)

        │
        ▼
 Structured Pattern JSON

🔍 Step 1: Scraping Real Patterns

Using Surfs.dev agents (real browser sessions), I:

Navigate crochet sites
Extract rows, stitch counts, materials
Handle messy UI (expandable notes, dynamic sections)

Why not traditional scraping?

Because most pattern sites:

Aren’t clean HTML
Require interaction
Break easily

Browser agents solve this.

🧩 Step 2: Structuring with LLMs

Raw pattern example:

Row 3: Ch 1, *sc in next st, inc; rep from * around. (18 sts)

Converted into structured JSON:

{
  "row": 3,
  "stitches": 18,
  "instructions": [
    { "type": "sc", "count": 1 },
    { "type": "inc", "repeat": true }
  ]
}

Model Roles

OpenAI → deterministic parsing + schema output
Claude Code → reasoning, cleanup, transformations

🪄 Step 3: Generating Variations

Once structured, patterns become programmable.

Examples:

“Make it longer” → extend BODY rows
“Add ribbing” → inject BLO stitches
“Convert to child size” → adjust stitch counts

Crochet becomes:

A DSL for physical objects

🧪 Step 4: Testing with Debugg.ai

This is the key unlock.

We treat patterns like end-to-end tests.

What gets validated:

Stitch counts per row
Logical progression
UI rendering
Instruction clarity

Because:

One bad row = broken output

🤖 Step 5: Execution via Browser Agents

Agents:

Load pattern UI
Click through steps
Expand notes
Simulate a human following instructions

This creates:

Replayable sessions
Failure logs
Deterministic debugging

⚠️ Challenges

1. Data is messy

Different sites = different formats
→ solved with LLM normalization

2. Fully stateful system

Each row depends on the last
→ solved with full agent simulation

3. Tiny errors compound

→ solved with Debugg.ai validation loops

🚀 Why This Matters

Crochet is just a proxy.

The real insight:

Most real workflows are not APIs — they’re UI-driven systems.

Examples:

SaaS dashboards
Internal tools
Marketplaces
Creative workflows

If agents can handle crochet patterns reliably:
they can handle almost anything.

🧠 Key Takeaway

Crochet patterns are:

Programs for humans

And with Debugg.ai + Surfs.dev:

They become programs that can be tested, executed, and improved by agents.

If you’re building agent workflows (growth, ops, QA, etc.), I’d love to see what you’re working on.

DEV Community