DEV Community: Anakin

When a scraping platform is too much for an LLM workflow

Anakin — Fri, 29 May 2026 09:30:02 +0000

You start with a simple requirement: give the model fresh data from a web page. Then the implementation grows into browser automation, job queues, dataset exports, retry handling, selector maintenance, and a parser that exists only to turn someone else's output into the JSON your LLM actually needs.

That mismatch is common. Scraping platforms solve a broad problem. LLM workflows usually need a narrower thing: take a URL or search query, extract a few fields, and pass structured data into the next model call.

The shape of the problem

A lot of scraping tooling is built around orchestration:

pick or write a scraper
configure inputs
run it as a job
wait for completion
fetch results from storage
normalize whatever format the scraper returned
pass that into your app

That model makes sense for scheduled crawls, large data pipelines, and teams maintaining many scrapers. If you run nightly competitor monitoring across thousands of pages, orchestration is not overhead. It is the product.

For an LLM feature, it can be too much.

Most agent or RAG flows want something closer to this:

user asks question
→ fetch relevant web data
→ extract structured fields
→ validate shape
→ call the model with clean context

The painful part is usually not the HTTP request. It is everything around it: client-side rendered pages returning empty HTML, CSS selectors drifting, different scrapers returning different schemas, and async jobs failing somewhere three layers away from the code that needs the result.

A failure often looks boring but expensive:

{
  "title": "",
  "price": null,
  "description": "Sign in to continue"
}

Your pipeline does not crash. It just gives the model bad context. Then the model confidently answers from garbage.

Prefer a narrow extraction contract

For LLM workflows, the useful abstraction is not "run a scraper". It is "return this kind of structured object or fail clearly".

That means your integration should have a contract like this:

type ExtractArticleResult = {
  url: string;
  title: string;
  author?: string;
  published_at?: string;
  markdown: string;
};

Once you define the shape, the rest of the pipeline becomes easier to reason about. You can validate it, cache it, embed it, summarize it, or pass it as tool output.

Wire by Anakin exposes this kind of submit-and-poll extraction flow over REST, returning structured JSON for LLM and agent workflows without requiring an SDK or actor lifecycle management: https://anakin.io/wire.

The specific provider matters less than the pattern: keep extraction behind a small interface and make the rest of your app depend on typed data, not scraper internals.

A concrete submit-and-poll wrapper

Many extraction APIs are asynchronous because pages may require rendering, retries, or remote execution. Do not let that async lifecycle leak throughout your app. Wrap it once.

Here is a minimal TypeScript example:

type JobResponse = {
  status: "processing";
  job_id: string;
  poll_url: string;
};

type CompletedJob<T> = {
  status: "completed";
  data: T;
  execution_ms?: number;
};

type FailedJob = {
  status: "failed";
  error_code: string;
  message: string;
};

async function submitExtraction(params: {
  apiKey: string;
  actionId: string;
  payload: Record<string, unknown>;
}): Promise<JobResponse> {
  const res = await fetch("https://api.anakin.io/v1/holocron/task", {
    method: "POST",
    headers: {
      "X-API-Key": params.apiKey,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      action_id: params.actionId,
      params: params.payload
    })
  });

  if (res.status !== 202) {
    throw new Error(`Extraction submit failed: ${res.status} ${await res.text()}`);
  }

  return res.json();
}

async function pollJob<T>(apiKey: string, pollUrl: string): Promise<T> {
  const deadline = Date.now() + 30_000;

  while (Date.now() < deadline) {
    const res = await fetch(`https://api.anakin.io${pollUrl}`, {
      headers: { "X-API-Key": apiKey }
    });

    if (!res.ok) {
      throw new Error(`Polling failed: ${res.status} ${await res.text()}`);
    }

    const body = (await res.json()) as CompletedJob<T> | FailedJob | { status: "processing" };

    if (body.status === "completed") return body.data;

    if (body.status === "failed") {
      throw new Error(`${body.error_code}: ${body.message}`);
    }

    await new Promise(resolve => setTimeout(resolve, 1000));
  }

  throw new Error("Extraction timed out after 30s");
}

The important part is not the URL. It is the boundary.

Your agent code should call something like extractArticle(url) and receive an ExtractArticleResult. It should not know about job IDs, polling URLs, retries, browser sessions, or dataset storage.

Validate before the model sees it

Do not pass extracted data straight into a prompt. Validate it first.

import { z } from "zod";

const ArticleSchema = z.object({
  url: z.string().url(),
  title: z.string().min(1),
  author: z.string().optional(),
  published_at: z.string().optional(),
  markdown: z.string().min(200)
});

const raw = await pollJob<unknown>(apiKey, job.poll_url);
const article = ArticleSchema.parse(raw);

This catches the common silent failures: login walls, empty rendered output, cookie banners, blocked requests, or pages where the extractor found navigation text instead of article content.

A schema error is much cheaper than a bad model answer.

When a bigger platform is still the right choice

A direct extraction API is not always better. It is better for a specific class of problem.

Use a scraping orchestration platform when you need:

scheduled batch jobs
shared scraper reuse across teams
dataset storage and historical exports
proxy management at large scale
multi-step scraping pipelines
webhooks and job chaining

Tools like Apify fit this world well. Actor marketplaces, queues, storage, and proxy controls are useful when scraping is a platform concern inside your company.

Use the narrower API pattern when web data is just one tool inside an LLM system. In that case, every extra concept becomes something your agent code has to handle or hide.

Keep the scraper out of your agent logic

The practical rule I use is simple: the model should never depend on a scraping provider's native output format.

Put a small adapter in front of it:

provider response
→ adapter
→ validated domain object
→ LLM prompt or tool result

That lets you swap providers, add fallbacks, cache successful extractions, and test your agent without hitting the network.

A good next step is to pick one existing LLM workflow that fetches web data and write down the exact JSON shape it needs. Then add validation at that boundary. You will usually find that the scraping code can get smaller once the contract is explicit.

Reliable Web-Connected AI Agents Start at the Fetch Layer

Anakin — Thu, 28 May 2026 09:30:03 +0000

A web-connected agent usually fails before the model gets a chance to reason. The request hits a 403. The page hydrates after your scraper reads it. A selector returns null because the site shipped an A/B test. Then the agent tries to continue with partial data and you get a confident answer built on garbage.

The fix is not a better prompt. Treat web access as an unreliable infrastructure dependency and put a hard boundary between fetching data and reasoning over it.

Classify failures before retrying

A lot of agent loops retry too blindly. They send the same request again after every failure, which wastes time and makes rate limits worse.

Classify the failure first:

502, 503, network reset: probably transient, retry with backoff
429: rate limited, respect Retry-After if present
401, 403: credentials, permissions, or bot detection; do not blindly retry
malformed extraction: page shape changed or JavaScript did not finish rendering
repeated timeout: escalate configuration, not prompt

Here is a small TypeScript example. It is intentionally boring because the boring version is easier to debug at 3 a.m.

type FetchMode = "http" | "browser";

type FetchResult = {
  ok: boolean;
  status?: number;
  html?: string;
  error?: string;
  mode: FetchMode;
};

const sleep = (ms: number) => new Promise(r => setTimeout(r, ms));

function retryableStatus(status?: number) {
  return status === 502 || status === 503 || status === 504;
}

function shouldEscalateToBrowser(result: FetchResult) {
  return (
    result.status === 403 ||
    result.error?.includes("selector returned null") ||
    result.error?.includes("missing required field")
  );
}

async function fetchHttp(url: string): Promise<FetchResult> {
  try {
    const res = await fetch(url, { headers: { "user-agent": "Mozilla/5.0" } });
    const html = await res.text();
    return { ok: res.ok, status: res.status, html, mode: "http" };
  } catch (err) {
    return { ok: false, error: String(err), mode: "http" };
  }
}

async function fetchWithPolicy(url: string): Promise<FetchResult> {
  for (let attempt = 0; attempt < 3; attempt++) {
    const result = await fetchHttp(url);

    if (result.ok) return result;

    if (result.status === 429) {
      await sleep(2_000 * (attempt + 1));
      continue;
    }

    if (retryableStatus(result.status)) {
      await sleep(1_000 * 2 ** attempt);
      continue;
    }

    if (shouldEscalateToBrowser(result)) {
      return runBrowserFetch(url); // Playwright, Browserbase, etc.
    }

    return result;
  }

  return { ok: false, error: "retry budget exhausted", mode: "http" };
}

The important part is not the exact code. It is the policy: retries are bounded, errors have categories, and browser automation is an escalation path rather than the default.

Wire is Anakin's web data layer for agents, and in this context it is useful because it keeps API-style extraction, browser fallback, and structured output outside the model loop.

Use HTTP first, browser automation only when needed

Browser automation solves real problems. It executes JavaScript, preserves cookies, handles client-side routing, and looks more like an actual user session than a raw HTTP client.

It also costs more and runs slower. You pay for browser startup, rendering, hydration waits, proxy routing, and heavier infrastructure. If you run browser-first across a large crawl, the cost difference compounds quickly.

Start with plain HTTP when the page is server-rendered or exposes structured data:

curl -s https://example.com/products/123 \
  -H 'user-agent: Mozilla/5.0' \
  | pup 'h1 text{}'

Move to browser mode when you see symptoms like these:

the HTML contains only <div id="root"></div> and script tags
required data appears in DevTools after hydration, but not in curl
HTTP requests return 403 Forbidden while a browser session succeeds
content depends on cookies, localStorage, or a multi-step login
extraction fails intermittently because modals or region prompts appear

A good production pipeline can use both. HTTP handles the cheap path. Browser sessions handle the pages that actually need a browser. Wire routes between these modes per action, so the agent does not need to embed that decision logic itself.

Log data-layer failures separately from model failures

If an agent returns a bad answer, you need to know whether the model reasoned badly or the fetch layer handed it bad input.

Log at least these fields per job:

{
  "job_id": "price-check-8271",
  "url": "https://example.com/item/123",
  "fetch_mode": "http",
  "status": 403,
  "attempts": 2,
  "latency_ms": 4810,
  "extraction_valid": false,
  "error": "missing required field: price"
}

Track success rate, p95 latency, block rate, and cost per completed job. Average latency hides retry cascades. Total spend hides waste. A rising p95 with stable success rate often means your fallback path is working but getting more expensive. A rising block rate means you should inspect fingerprints, request pacing, or proxy routing before touching prompts.

Also store enough response data to replay failures. If a selector starts returning null, you want the actual HTML that caused it. Otherwise you are debugging a memory of a page that no longer exists.

Keep credentials out of the agent’s reach

Authenticated browsing creates a different class of risk. The page can contain untrusted instructions, hidden text, or injected markup. If the agent has direct access to long-lived credentials, prompt injection can become credential theft.

Use scoped credentials and short session lifetimes:

never put API keys in prompts or tool descriptions
keep cookies in an isolated browser context per job or tenant
redact secrets from logs before storing request traces
rotate credentials used by automation separately from human accounts
destroy sessions after sensitive workflows unless reuse is required

For Playwright-style automation, that usually means creating a fresh context and closing it explicitly:

const context = await browser.newContext({ storageState: "auth-state.json" });
const page = await context.newPage();
await page.goto("https://example.com/account/orders");
// extract only the fields the agent needs
await context.close();

Persistent sessions are faster, but they expand the blast radius when something goes wrong. Use them where the performance gain is worth the security tradeoff.

A practical next step: pick one existing web-connected workflow, add explicit failure classification and fetch-mode logging, then review the next 100 failures before changing the prompt.

Giving n8n AI Workflows Fresh Web Data Without Babysitting Scrapers

Anakin — Wed, 27 May 2026 10:02:16 +0000

Most AI workflows in n8n look great right up until they need facts from the internet.

The demo works. The Slack bot summarizes a static document. The CRM enrichment flow pulls from a clean test payload. Then someone asks for competitor pricing, lead research, or a daily market brief, and suddenly your “AI automation” is mostly a scraper held together with retries and regret.

That is the part tutorials usually skip. Getting GPT or Claude into n8n is easy. Getting fresh web data into them without waking up to broken selectors is the actual job.

The boring problem: AI nodes are only as current as their input

A model can write a lovely summary of stale data. It will sound confident. That is not the same as being useful.

If your workflow needs live company info, pricing pages, job listings, directories, documentation, or news, you have a few options:

Use an official API, if one exists and the pricing does not make finance appear in your doorway
Build scrapers and maintain them forever
Pay someone else to handle the scraping layer

The second option is where teams lose time.

Not because scraping one page is hard. Scraping one page is usually fine. The pain starts when the site uses client-side rendering, changes its layout, blocks your requests, or returns different markup depending on geography, cookies, or mood. Surprising to no one who has actually tried it.

For n8n workflows, I usually want something simpler: call an HTTP endpoint, get clean markdown or JSON back, and move on with the workflow.

That is where Anakin.io fits. It gives you API endpoints for URL scraping, web search, and deeper research jobs. n8n calls those endpoints with the HTTP Request node. Nothing mystical. Just HTTP, credentials, payloads, and a bit of polling.

Start by treating the API key like a credential, not a sticky note

Before building nodes, put the API key into n8n credentials.

In n8n:

Go to Settings > Credentials
Create a Header Auth credential
Set the header name to X-API-Key
Set the value to your Anakin key, like ak-your-key-here

Now every HTTP Request node can reuse it. This matters more than it sounds. Hardcoded API keys have a way of ending up in copied workflows, screenshots, and repos. Ask me how I know.

Scraping a page into markdown

The basic workflow shape is:

A trigger fires in n8n
An HTTP Request node submits a URL to Anakin
Anakin returns a jobId
n8n waits or loops
Another HTTP Request node fetches the result
Your AI node summarizes, extracts, classifies, or routes the content

The first request looks like this:

{
  "url": "https://example.com/product-page",
  "useBrowser": false,
  "generateJson": false
}

Send it as a POST request to:

https://api.anakin.io/v1/url-scraper

The response gives you a jobId. Then poll:

GET https://api.anakin.io/v1/url-scraper/{{ $json.jobId }}

When the job status is completed, you get the page content back as clean markdown.

That markdown is usually what you want to feed into an AI node. It strips away a lot of the page furniture: nav, scripts, styling, and other junk that makes models waste tokens on nonsense.

When the page is rendered by JavaScript

Some pages return almost nothing unless a browser runs the JavaScript first. React apps, pricing pages, dashboards, fancy marketing sites. The usual suspects.

For those, set:

{
  "useBrowser": true
}

That tells Anakin to render the page with a browser before extracting content.

Do not turn this on for everything by default. Browser rendering is the slow lane compared with plain HTML fetching. Use it when you need it, not because it feels safer.

When you want fields instead of prose

Markdown is good for summaries. JSON is better when the next node needs structured fields.

For example, a lead enrichment workflow probably does not need a beautiful essay about a company. It needs fields like:

company description
target customer
product category
pricing hints
team size signals

Set:

{
  "generateJson": true
}

Then pass the structured result into your CRM update node, Airtable row, database insert, or whatever system owns the record.

Still validate the output. AI extraction is useful, not holy scripture. If a field controls money, routing, compliance, or customer messaging, add checks before writing it downstream.

Search is different from scraping

Sometimes you do not have a URL. You have a question.

For that, Anakin’s search endpoint is synchronous, which means n8n gets the answer back immediately instead of polling a job.

Example request:

{
  "prompt": "latest funding rounds in enterprise AI 2025",
  "limit": 5
}

Send it to:

POST https://api.anakin.io/v1/search

The response includes a summary, ranked results, relevance scores, and citations.

That fits nicely into things like daily Slack briefings. Schedule a workflow for 8am, run a few searches, combine the summaries, send the digest. Not glamorous. Very useful.

Use deeper research when a search result is not enough

There is also an async research-style endpoint:

POST https://api.anakin.io/v1/agentic-search

Payload:

{
  "prompt": "competitive analysis of no-code automation tools 2025"
}

This returns a jobId, so you use the same submit-and-poll pattern:

GET https://api.anakin.io/v1/agentic-search/{{ $json.jobId }}

The output is a longer report built from search, scraping, and synthesis.

This is not what I would use inside a latency-sensitive webhook. If a user is waiting on the other end, polling a research job is a bad time. But for scheduled briefs, Notion reports, internal analysis, and agent context refreshes, it makes sense.

A few workflows that are actually worth building

A competitor pricing monitor is the obvious one. Schedule a daily run, scrape pricing pages with generateJson: true, store the results, diff against yesterday, and send Slack alerts only when something changed. Do not send “no change” alerts unless you enjoy being muted.

Lead enrichment is another practical case. When a new company enters your CRM, scrape its website, extract useful fields, and write them back. Keep a confidence field or raw source link so a human can inspect weird cases.

For RAG pipelines, scrape documentation pages into markdown, chunk by headings, and push the chunks into your vector database. This is less exciting than an agent demo and far more likely to help users.

What I would do next

Start with one boring workflow. One URL. One HTTP Request node. One poll. One AI summary.

Then add structure, retries, and validation.

If your team does not want to own scraper infrastructure, Anakin.io is a reasonable layer to put between n8n and the messy web. It will not remove the need to design good workflows, handle async jobs, or sanity-check AI output. Nothing does.

But it does mean your 2am problem is less likely to be “the pricing page moved a div.”

And honestly, that is enough of a win.