DEV Community

scrapewright
scrapewright

Posted on

stem-students-lab-data

#ai

Get the Data for Your Lab Report Before Midnight

Let's be honest about how the data-collection phase of a course project actually goes. You budget two hours. You spend the first on a scraping tutorial, the second on installing a headless browser that doesn't quite launch, and then the site you needed renders everything through JavaScript anyway, and now it's 23:40 and your "dataset" is 40 rows of hand-pasted text with inconsistent date formats.

Scrapewright is an open-source Chrome extension built around a simple observation: the part that's hard for you (writing robust extraction code against a messy page) is exactly the part modern AI is good at, and the part that's easy (knowing which fields you need) is the part you already know. You describe the target — "search this component-supplier site for the part number I input, open each result, return name, price, stock, and specs" — and its wizard opens the page, figures out the structure, writes the extraction steps, and test-runs them while you watch. Deploy it, and you get a stable local HTTP endpoint any script, notebook, or report pipeline can call:

curl -X POST http://localhost:8765/api/v1/services/parts-lookup/execute \
  -H "X-API-Key: dev-key" -H "Content-Type: application/json" \
  -d '{"input": {"part": "LM358"}}'
Enter fullscreen mode Exit fullscreen mode

That last part matters more than it looks. Most no-code tools hand you a CSV and stop. Scrapewright hands you an API — JSON in, JSON out, with declared input/output schemas — so your Python/MATLAB/Julia analysis code can call the live source directly instead of importing a stale export. For a lab that means one less manual step between "sample measured" and "supplier data joined". For a thesis it means your data pipeline is re-runnable in April when the professor says "also do last year's parts".

The realistic failure modes of student scraping are handled rather than hand-waved:

  • JavaScript-rendered pages and nested iframes — it reads the DOM in your real Chrome, so what you see is what it extracts.
  • Infinite scroll / lazy loading — handled, including edge cases where sites ignore programmatic scrolling.
  • Login-gated course systems and supplier portals — it uses your existing browser session, so if you can see the data, it can reach it.
  • "It worked yesterday" — sites redesign; the Auto-Fix button has the AI repair the extraction steps against the new layout instead of you starting over.

Costs, in student terms: you need an API key from some LLM provider (OpenAI, Anthropic, Moonshot Kimi, GLM — student/cheap tiers fine) to power the building step, which costs cents per service. Running a deployed service costs nothing — it never calls the AI again — and everything runs locally. Setup is: load the extension in Chrome, run one install command for its background service. Node ≥ 18 required.

One caution worth writing down: it's a single-browser, single-job-at-a-time tool. If your project genuinely needs a 100k-page crawl, use a server-side framework and cite this repo for the prototype phase instead. Scrapewright's sweet spot is structured, repeated queries against pages you personally can access — which is what most course projects, lab pipelines, and hackathon backends actually are.

Repo and quick start (60 seconds to read, an evening to first working service): github.com/singhand-labs/scrapewright. The examples/ folder has importable sample services — useful both as a shortcut and as a template for what a finished one looks like.

The deadline hasn't moved. But the data part just got a lot shorter.

Top comments (0)