DEV Community

Cover image for Your LLM can scrape the web — locally, without writing throwaway code
Iurii Panarin
Iurii Panarin

Posted on • Originally published at github.com

Your LLM can scrape the web — locally, without writing throwaway code

#mcp #ai #webscraping #go

Every time you ask an AI assistant to "get data from this website", one of two things happens: it hallucinates the data, or it writes a one-off Python script you'll never run again. Both are wasteful. The model already knows where the data lives and how to select it — what's missing is a safe, reusable way to execute that knowledge.

That's what fitter does. It's a small Go engine with one idea at its core: web extraction should be a config, not code. A fitter config declares where the data lives (HTTP request, headless browser, static value, file) and what to extract (gjson paths, CSS selectors, XPath). The engine does the rest.

And because configs are plain JSON/YAML, an LLM can author them. Fitter ships an MCP server, so Claude Code, Claude Desktop, or any MCP client can write a config, validate it, and run it on your machine — in one conversation turn.

60-second setup

Grab a binary from the releases page (or go build -o fitter_mcp ./cmd/mcp), then:

claude mcp add fitter -s user -- /path/to/fitter_mcp
Enter fullscreen mode Exit fullscreen mode

Claude Desktop users: download the .mcpb bundle for your platform from the same release page and open it — that's the whole install.

The server exposes five tools: fitter_run, fitter_run_file, fitter_run_url, fitter_validate_config, and fitter_config_reference — that last one returns a condensed format reference, so the model can teach itself the config format without leaving the conversation.

"Get me the latest Hacker News stories"

Ask exactly that. Behind the scenes, the model authors something like this and calls fitter_run:

{
  "item": {
    "connector_config": {
      "response_type": "json",
      "url": "https://hacker-news.firebaseio.com/v0/newstories.json",
      "server_config": { "method": "GET" }
    },
    "model": {
      "array_config": {
        "root_path": "@this",
        "length_limit": 10,
        "item_config": {
          "field": {
            "type": "object",
            "generated": { "model": {
              "type": "object",
              "connector_config": {
                "response_type": "json",
                "url": "https://hacker-news.firebaseio.com/v0/item/{PL}.json",
                "server_config": { "method": "GET" }
              },
              "model": { "object_config": { "fields": {
                "title": { "base_field": { "type": "string", "path": "title" } },
                "by":    { "base_field": { "type": "string", "path": "by" } },
                "score": { "base_field": { "type": "int",    "path": "score" } },
                "url":   { "base_field": { "type": "string", "path": "url" } }
              } } }
            } }
          }
        }
      }
    }
  },
  "limits": { "host_request_limiter": { "hacker-news.firebaseio.com": 5 } }
}
Enter fullscreen mode Exit fullscreen mode

Read it top to bottom: fetch the ID list, take the first 10, and for each element fire a nested request ({PL} is the current value) to fetch the story object — rate-limited to 5 parallel requests against the HN API. The result is clean JSON:

[
  {"title": "Fields Medals 2026", "by": "nill0", "score": 2, "url": "https://www.mathunion.org/..."},
  {"title": "Writing by Hand is Good for your Brain", "by": "dwwoelfel", "score": 2, "url": "..."}
]
Enter fullscreen mode Exit fullscreen mode

Here's the part that matters: that config is an artifact, not an afterthought. Save it to a file and it's a CLI command, a cron job, or a service — the model's one-off answer became infrastructure.

One config, four data sources

My actual morning briefing — weather, HN, world headlines, and the Bitcoin price — is one config with four independent branches. It also shows off placeholders:

"connector_config": {
  "response_type": "json",
  "url": "https://wttr.in/{{{FromInput=.}}}?format=j1"
}
Enter fullscreen mode Exit fullscreen mode

{{{FromInput=.}}} is filled from the tool's input argument at run time, so "run my briefing for Berlin" and "for Paris" reuse the same config. There are placeholders for env vars, cached references (think: fetch a JWT once, use it in every request's headers), array indices, and more.

Join sources: scrape what has no API, enrich from the one that does

GitHub's trending page famously has no API. But the repos on it do. So: scrape the HTML for the repo slugs, then fan each one out into the real GitHub REST API:

{
  "item": {
    "connector_config": {
      "response_type": "HTML",
      "url": "https://github.com/trending",
      "server_config": {
        "method": "GET",
        "headers": { "User-Agent": "Mozilla/5.0 (fitter demo)" }
      }
    },
    "model": {
      "array_config": {
        "root_path": "article.Box-row h2 a",
        "length_limit": 5,
        "item_config": {
          "field": {
            "type": "string",
            "html_attribute": "href",
            "generated": { "model": {
              "type": "object",
              "connector_config": {
                "response_type": "json",
                "url": "https://api.github.com/repos{PL}",
                "server_config": { "method": "GET", "headers": { "User-Agent": "fitter-demo" } },
                "null_on_error": true
              },
              "model": { "object_config": { "fields": {
                "repo":        { "base_field": { "type": "string", "path": "full_name" } },
                "stars":       { "base_field": { "type": "int", "path": "stargazers_count" } },
                "language":    { "base_field": { "type": "string", "path": "language" } },
                "open_issues": { "base_field": { "type": "int", "path": "open_issues_count" } },
                "description": { "base_field": { "type": "string", "path": "description" } }
              } } }
            } }
          }
        }
      }
    }
  },
  "limits": { "host_request_limiter": { "api.github.com": 2 } }
}
Enter fullscreen mode Exit fullscreen mode

Today's actual output:

[
  {"repo": "block/buzz", "stars": 6214, "language": "Rust", "open_issues": 460,
   "description": "A hive mind communication platform"},
  {"repo": "koala73/worldmonitor", "stars": 71179, "language": "TypeScript", "open_issues": 272,
   "description": "Real-time global intelligence dashboard. AI-powered news aggregation..."},
  {"repo": "shiyu-coder/Kronos", "stars": 32946, "language": "Python", "open_issues": 241,
   "description": "Kronos: A Foundation Model for the Language of Financial Markets"},
  {"repo": "Pumpkin-MC/Pumpkin", "stars": 8808, "language": "Rust", "open_issues": 207,
   "description": "Empowering everyone to host fast and efficient Minecraft servers."},
  {"repo": "citrolabs/ego-lite", "stars": 1485, "language": "JavaScript", "open_issues": 18,
   "description": "The best browser for both you and your AI agents work in parallel."}
]
Enter fullscreen mode Exit fullscreen mode

The details worth stealing:

  • HTML and JSON mix freely. The outer connector parses HTML with a CSS selector (article.Box-row h2 a); each matched element fans out into a JSON API request. One config, two worlds.
  • html_attribute reads attributes, not text. The anchor's href is already /owner/repo — which is exactly what https://api.github.com/repos{PL} needs. The join key comes from the page itself, so it can't drift out of sync.
  • length_limit + host_request_limiter keep you polite. Top 5 repos, max 2 concurrent API calls. Unauthenticated GitHub API allows 60 requests/hour — a config that respects that is a config you can run on a schedule.

This is the general pattern for the modern web: the interesting ranking lives in some HTML page, the facts live in an API, and the join between them is exactly one {PL} away.

When the join key is a field, not the whole value

{PL} injects the entire current value — perfect when your array items are scalars (story IDs, repo slugs). But often the items are objects and the join key lives inside them. That's what expression placeholders are for. Book search → author enrichment on OpenLibrary:

{
  "item": {
    "connector_config": {
      "response_type": "json",
      "url": "https://openlibrary.org/search.json?q={{{FromInput=.}}}&limit=4&fields=title,author_key,first_publish_year",
      "server_config": { "method": "GET" }
    },
    "model": {
      "array_config": {
        "root_path": "docs",
        "item_config": {
          "fields": {
            "title": { "base_field": { "type": "string", "path": "title" } },
            "year":  { "base_field": { "type": "int", "path": "first_publish_year" } },
            "author": {
              "base_field": {
                "type": "object",
                "generated": { "model": {
                  "type": "object",
                  "connector_config": {
                    "response_type": "json",
                    "url": "https://openlibrary.org/authors/{{{FromExp=fromJSON(fRes).author_key[0]}}}.json",
                    "server_config": { "method": "GET" },
                    "null_on_error": true
                  },
                  "model": { "object_config": { "fields": {
                    "name": { "base_field": { "type": "string", "path": "name" } },
                    "born": { "base_field": { "type": "string", "path": "birth_date" } },
                    "died": { "base_field": { "type": "string", "path": "death_date" } }
                  } } }
                } }
              }
            }
          }
        }
      }
    }
  },
  "limits": { "host_request_limiter": { "openlibrary.org": 2 } }
}
Enter fullscreen mode Exit fullscreen mode

Run with input = dune:

[
  {"title": "Dune",             "year": 1965,
   "author": {"name": "Frank Herbert", "born": "8 October 1920", "died": "11 February 1986"}},
  {"title": "Dune Messiah",     "year": 1969,
   "author": {"name": "Frank Herbert", "born": "8 October 1920", "died": "11 February 1986"}},
  {"title": "Children of Dune", "year": 1976,
   "author": {"name": "Frank Herbert", "born": "8 October 1920", "died": "11 February 1986"}}
]
Enter fullscreen mode Exit fullscreen mode

The key line is the nested URL: {{{FromExp=fromJSON(fRes).author_key[0]}}} is an expr-lang expression evaluated against the current item (fRes) — parse it, take the first author key, splice it into the URL. Anything expr-lang can compute can become part of a request: pick a field, concatenate, add an offset, branch on a condition. And notice the output shape: each book keeps its own scalar fields while the enrichment lands as a nested author object — the config's structure is the response's structure.

Don't just return data — write the report

Sometimes the deliverable isn't JSON in a chat window, it's a file on disk. The file_storage generated field turns any field into a write: the top-5 crypto coins, straight into a CSV:

{
  "item": {
    "connector_config": {
      "response_type": "json",
      "url": "https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=5&page=1",
      "server_config": { "method": "GET" }
    },
    "model": {
      "array_config": {
        "root_path": "@this",
        "item_config": {
          "field": {
            "type": "object",
            "generated": {
              "file_storage": {
                "content": "{HUMAN_INDEX},{{{name}}},{{{current_price}}},{{{price_change_percentage_24h}}}\n",
                "file_name": "coins.csv",
                "path": "/tmp/fitter-report",
                "append": true
              }
            }
          }
        }
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The tool call returns the written paths, and on disk:

$ sort -n /tmp/fitter-report/coins.csv
1,Bitcoin,64778,-2.3
2,Ethereum,1881.01,-3.4
3,Tether,0.999265,0
4,BNB,566.81,-1.6
5,USDC,0.999672,0
Enter fullscreen mode Exit fullscreen mode

What's going on in that content template:

  • Bare {{{json.path}}} placeholders pull fields straight out of the current item — {{{name}}}, {{{current_price}}} — no expressions needed for simple lookups. (Type the field as object so the template sees the item's raw JSON.)
  • {HUMAN_INDEX} stamps the item's position (1-based; {INDEX} is the 0-based sibling). Fitter processes array items concurrently, so appends land in completion order — the rank column is what makes sort -n restore truth. A nice reminder that this thing is genuinely parallel.
  • There's a sibling file field type that downloads — point it at an image/PDF URL and the field's value becomes the local file path. Scrape a gallery, keep the pictures.

Combined with schedule-based service mode, this is a tiny ETL pipeline: fetch → shape → append to a growing CSV, every hour, defined entirely in one JSON document.

When the site has no API

The examples above hit JSON APIs, but the response_type can just as well be HTML (CSS selectors), xpath, or XML — and the connector can be a real browser when the page needs JavaScript:

"connector_config": {
  "response_type": "HTML",
  "url": "https://example.com/spa-page",
  "browser_config": { "playwright": { "browser": "Chromium", "install": true, "stealth": true } }
}
Enter fullscreen mode Exit fullscreen mode

Chromium, Playwright (with stealth), and dockerized browsers are all built in — one static binary, no npm install.

Why local-first matters

The hosted scraping APIs are good products, but for agent workflows the local model has real advantages:

  • Auditable — you can read exactly what the agent fetched and how. A config diff is reviewable; generated code in a sandbox somewhere is not.
  • No keys, no metering — your machine makes the requests. Nothing to sign up for, no per-page billing, no data passing through a third party.
  • Reusable by construction — the same config runs via MCP, CLI (fitter_cli --path config.json), a Go library call, or fitter's service mode with schedulers and notifiers (Telegram, webhook, Redis, file). Authored once by the model, promoted to a cron job when it proves useful.

Hosting it for your team

New in the latest release: the MCP server also speaks streamable HTTP, so you can run one fitter for the whole team. A minimal docker-compose.yml:

services:
  fitter-mcp:
    image: ghcr.io/pxyup/fitter-mcp:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      FITTER_MCP_HTTP_ADDR: ":8080"
      FITTER_MCP_AUTH_TOKEN: "${FITTER_MCP_AUTH_TOKEN:?set a token}"
    volumes:
      # optional: mount reusable configs for fitter_run_file
      - ./configs:/configs:ro
Enter fullscreen mode Exit fullscreen mode 
export FITTER_MCP_AUTH_TOKEN=$(openssl rand -hex 16)
docker compose up -d
curl http://localhost:8080/healthz   # -> ok
Enter fullscreen mode Exit fullscreen mode

Every /mcp request must then carry the bearer token, and the image's built-in healthcheck keeps your orchestrator honest. Register the endpoint in Claude Code:

claude mcp add --transport http fitter http://your-host:8080/mcp \
  --header "Authorization: Bearer $FITTER_MCP_AUTH_TOKEN"
Enter fullscreen mode Exit fullscreen mode

The image is slim by design (binary + CA certs, linux/amd64 + arm64): API/HTML scraping works out of the box, while browser-rendered pages need a release binary on a host with Chromium or Playwright.

Client-side cancellation propagates all the way down to in-flight HTTP requests and browser sessions, so an abandoned agent run doesn't keep scraping in the background.

Try it

Open Claude Code, add the server, and ask for some data you actually want every morning. The config it writes is yours to keep — that's the point.

Questions and config contributions welcome — the examples folder takes PRs.

Top comments (0)