DEV Community

Deduped
Deduped

Posted on

Turn any web page into clean markdown for ChatGPT or Claude

#ai #webscraping #python #llm

Paste a web page into ChatGPT or Claude and most of what you send is navigation, cookie banners and script tags. Fetch it with plain requests and a JavaScript-heavy page often comes back nearly empty. Firecrawl solves both: give it a URL and it returns clean markdown of the page's main content, ready for a prompt or a vector store. Here is how to set it up from Python, Node, curl or the CLI, with the commands from the README. The short version lives on the Deduped use-case page.

What Firecrawl does

Firecrawl is an API to search, scrape and interact with the web, built for AI agents. Its scrape endpoint converts any URL to markdown, HTML, screenshots or structured JSON. It handles rotating proxies, rate limits and JavaScript-blocked content, and it can parse PDFs and DOCX files into markdown too.

It is open source under AGPL-3.0 (the SDKs are MIT) and also runs as a hosted service. The examples below use the hosted API, which needs a key from firecrawl.dev. The README links a self-hosting guide if you would rather run it yourself.

Step 1: get a key and keep it out of your code

Sign up at firecrawl.dev to get your API key. It starts with fc-. Put it in an environment variable rather than in a file you commit:

export FIRECRAWL_API_KEY="fc-YOUR_API_KEY"
Enter fullscreen mode Exit fullscreen mode

Step 2: scrape a page to markdown

Python

pip install firecrawl
Enter fullscreen mode Exit fullscreen mode 
import os
from firecrawl import Firecrawl

app = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])

doc = app.scrape("https://firecrawl.dev", formats=["markdown"])
print(doc.markdown)
Enter fullscreen mode Exit fullscreen mode

Node.js

npm install firecrawl
Enter fullscreen mode Exit fullscreen mode 
import { Firecrawl } from 'firecrawl';

const app = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const doc = await app.scrape('https://firecrawl.dev', { formats: ['markdown'] });
Enter fullscreen mode Exit fullscreen mode

curl

curl -X POST 'https://api.firecrawl.dev/v2/scrape' \
-H "Authorization: Bearer $FIRECRAWL_API_KEY" \
-H 'Content-Type: application/json' \
-d '{
  "url": "firecrawl.dev"
}'
Enter fullscreen mode Exit fullscreen mode

CLI, with only the main content of the page:

firecrawl https://firecrawl.dev --only-main-content
Enter fullscreen mode Exit fullscreen mode

The output is plain markdown: headings, lists and text, without the page chrome. That is what you paste into a prompt or split into chunks for retrieval.

Step 3: go wider when one page is not enough

The same client covers the jobs that usually come next:

# Find URLs on a site, ordered by relevance to a word
result = app.map("https://firecrawl.dev", search="pricing")

# Scrape several URLs at once
job = app.batch_scrape([
    "https://firecrawl.dev",
    "https://docs.firecrawl.dev",
], formats=["markdown"])

# Crawl a docs site (the SDK waits for the job to finish)
docs = app.crawl("https://docs.firecrawl.dev", limit=50)
for d in docs.data:
    print(d.metadata.source_url, d.markdown[:100])
Enter fullscreen mode Exit fullscreen mode

Crawling a whole docs site this way is the fastest route to a knowledge base for a docs chatbot.

Step 4: give Claude Code or Cursor the same power

If you want your coding agent to read current docs instead of guessing from training data, install Firecrawl as a skill and restart the agent:

npx -y firecrawl-cli@latest init --all --browser
Enter fullscreen mode Exit fullscreen mode

Or add it as an MCP server in any MCP client:

{
  "mcpServers": {
    "firecrawl-mcp": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "fc-YOUR_API_KEY"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

That config holds your key, so keep it out of shared or committed config files.

Caveats

  • Respect each site's terms and robots rules when you scrape.
  • Crawls cost credits per page on the hosted service, so set a limit while testing.
  • The README examples hardcode fc-YOUR_API_KEY; reading it from the environment, as above, is the safer habit.

More like this

Compare Firecrawl with other options on best AI repos for web scraping for LLMs, or see every Firecrawl use case, including giving your coding agent live web access.

Want the setup commands for 50 AI repos people actually ship with? Get the free AI Repos That Actually Ship pack: https://dedupedai.com/ship-pack/?utm_source=devto&utm_medium=article&utm_campaign=shippack

Top comments (0)