Type-Safe Web Scraping for AI Agents: Integrating Firecrawl into a TypeScript Monorepo

When building AI agents that need fresh web data, you hit two friction points fast: runtime API shape drift and monorepo cache invalidation. Firecrawl solves the scraping problem elegantly, but without strict TypeScript guards and proper Turborepo pipeline configuration, you're one upstream schema change away from silent data corruption in production.

Here's how I hardened our integration.

The Problem: Loose API Contracts Kill Reliability

Firecrawl's REST API returns rich document metadata—markdown content, structured data, HTTP status codes. But raw fetch() calls return any soup. When Firecrawl added a new actions field in their latest release, our agent pipeline silently ignored it for three days until a customer complained about missing form interactions.

Root cause: no compile-time contract enforcement between our agent logic (packages/agents/) and the Firecrawl client wrapper (packages/web-data/).

Deliverable 1: Strict Firecrawl Response Types with Template Literal Validation

First, model Firecrawl's actual API surface with precise branded types:

// packages/web-data/src/firecrawl-types.ts

export type FirecrawlFormat = 'markdown' | 'html' | 'rawHtml' | 'links' | 'screenshot';

export type ScrapeResponse<F extends FirecrawlFormat = 'markdown'> = {
  success: true;
  data: {
    markdown: F extends 'markdown' ? string : never;
    html: F extends 'html' | 'rawHtml' ? string : never;
    links: F extends 'links' ? string[] : never;
    screenshot: F extends 'screenshot' ? string : never;
    metadata: {
      title: string;
      description?: string;
      language?: string;
      statusCode: number;
    };
  };
} | {
  success: false;
  error: string;
};

export type CrawlStatusResponse = 
  | { status: 'scraping'; current: number; total: number }
  | { status: 'completed'; data: ScrapeResponse['data'][] }
  | { status: 'failed'; error: string };

// Enforce valid URL patterns at the type level
export type ValidatedUrl<T extends string> = 
  T extends `http://${string}` | `https://${string}` 
    ? T 
    : never;

Now the client wrapper enforces compile-time correctness:

// packages/web-data/src/firecrawl-client.ts

import type { ScrapeResponse, FirecrawlFormat, ValidatedUrl } from './firecrawl-types';

export class FirecrawlClient {
  constructor(private apiKey: string) {}

  async scrape<F extends FirecrawlFormat>(
    url: ValidatedUrl<string>,
    format: F
  ): Promise<ScrapeResponse<F>> {
    const response = await fetch('https://api.firecrawl.dev/v0/scrape', {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${this.apiKey}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ url, pageOptions: { onlyMainContent: true }, formats: [format] }),
    });

    if (!response.ok) {
      return { success: false, error: `HTTP ${response.status}` };
    }

    return response.json();
  }
}

Attempting scrape('not-a-url', 'markdown') now fails at compile time. Accessing data.html when you requested 'markdown' format? Type error.

Deliverable 2: Turborepo Pipeline with Strict Cache Invalidation

The second trap: Firecrawl client changes don't auto-invalidate dependent agent packages because Turborepo doesn't see the runtime API as an input.

Here's the lockdown turbo.json:

{
  "$schema": "https://turbo.build/schema.json",
  "pipeline": {
    "build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**"],
      "inputs": ["src/**/*.ts", "package.json", "tsconfig.json"]
    },
    "web-data#build": {
      "dependsOn": ["^build"],
      "outputs": ["dist/**"],
      "inputs": [
        "src/**/*.ts",
        "package.json",
        "../../.env.firecrawl"
      ],
      "env": ["FIRECRAWL_API_VERSION"]
    },
    "agents#build": {
      "dependsOn": ["^build", "web-data#build"],
      "outputs": ["dist/**"],
      "inputs": ["src/**/*.ts", "package.json"]
    },
    "test": {
      "dependsOn": ["build"],
      "inputs": ["src/**/*.ts", "tests/**/*.ts"],
      "outputs": []
    }
  },
  "remoteCache": {
    "signature": true
  }
}

Key moves:

1. Explicit inputs for cache hashing: Adding .env.firecrawl ensures API key rotation invalidates the build.
2. Environment variable tracking: FIRECRAWL_API_VERSION forces rebuild when we bump client compatibility.
3. Hard dependency chain: agents#build explicitly depends on web-data#build, preventing stale agent builds from using an outdated client.

The War Story: Recursive Type Instantiation Depth

First iteration of ScrapeResponse<F> used recursive conditional types to model paginated crawl results. TypeScript compiler hit max instantiation depth and crashed with a 2GB memory spike during turbo build.

Fix: flattened the discriminated union and moved pagination metadata out of the generic. Lesson: conditional type recursion and large monorepo incremental builds don't mix—keep generics shallow.

Verification

turbo build --filter=agents --dry=json | jq '.tasks[] | select(.command == "build") | .inputs'

Confirm .env.firecrawl appears in the hash inputs. Change the API version, verify cache miss:

echo "FIRECRAWL_API_VERSION=v1" >> .env.firecrawl
turbo build --filter=agents  # Should show cache miss for web-data + agents

Next Steps

Wrap Firecrawl's batch crawl API with a similar branded type pattern, and add Zod runtime validation at the package boundary to catch upstream breaking changes before they poison your agent's knowledge base.