DEV Community

Preecha
Preecha

Posted on

How Can You Install Firecrawl CLI and Use Firecrawl CLI

Firecrawl CLI: A Practical Guide to Scraping, Searching, Crawling, and Browser Automation

Firecrawl CLI is a terminal-native tool for AI agents and developers. It can scrape, search, map, crawl, and automate browser sessions while saving clean Markdown, JSON, screenshots, and HTML directly to your filesystem. Run it with npx firecrawl-cli without a permanent installation, or install it globally and connect it to Claude Code, Cursor, or OpenCode with a single initialization command.

Try Apidog today

AI agents need reliable, current web data, but custom scrapers often break on JavaScript-rendered pages, dynamic sites, and protected flows. Firecrawl CLI combines web scraping, search, site mapping, recursive crawling, and cloud browser sessions in one tool.

The CLI writes results to local files instead of sending large payloads directly to your terminal. That makes it easy to search output with standard shell tools, keep LLM context focused, and build repeatable research pipelines.

This guide covers:

  1. Preparing your environment
  2. Installing and authenticating Firecrawl CLI
  3. Scraping pages and selecting output formats
  4. Searching and mapping websites
  5. Crawling sites recursively
  6. Automating browser sessions
  7. Integrating with AI coding agents
  8. Troubleshooting and operational best practices

Why Use Firecrawl CLI?

Firecrawl CLI renders JavaScript through cloud browsers and is designed for sites that basic HTTP clients, Cheerio-based scrapers, or simple Puppeteer scripts may not handle reliably. It returns LLM-friendly Markdown by default, removing much of the navigation, advertising, and boilerplate content that consumes context-window space.

The CLI supports:

  • Web scraping
  • Web search with optional result scraping
  • URL discovery through site maps
  • Recursive site crawling
  • Screenshots and full-page screenshots
  • Cloud browser sessions
  • Markdown, JSON, HTML, links, and image outputs
  • Local file output for shell scripts and agent workflows

For example, adding --only-main-content to a scrape can significantly reduce irrelevant output:

firecrawl scrape https://example.com --only-main-content
Enter fullscreen mode Exit fullscreen mode

Prepare Your Environment

Firecrawl CLI depends on modern Node.js and npm features. Verify that Node.js 18 or later is installed:

node --version
Enter fullscreen mode Exit fullscreen mode

If the version is too old, update Node.js with your package manager or with nvm.

Create a dedicated workspace for output files:

mkdir firecrawl-cli-projects
cd firecrawl-cli-projects
Enter fullscreen mode Exit fullscreen mode

A separate directory keeps generated datasets organized and makes them easy to track with Git.

If you do not want telemetry enabled, set the environment variable before running the CLI:

export FIRECRAWL_NO_TELEMETRY=1
Enter fullscreen mode Exit fullscreen mode

Install Firecrawl CLI

Initialize Firecrawl CLI for AI Agents

The initialization command is the quickest way to install Firecrawl CLI, authenticate, and configure agent skills:

npx -y firecrawl-cli@latest init --all --browser
Enter fullscreen mode Exit fullscreen mode

The command opens a browser for Firecrawl account login or signup, stores your API key, and configures compatible agent integrations. Restart your coding agent after initialization so it can detect the new capabilities.

Install Globally with npm

For frequent use across multiple projects, install the CLI globally:

npm install -g firecrawl-cli
Enter fullscreen mode Exit fullscreen mode

Verify the installation:

firecrawl --version
Enter fullscreen mode Exit fullscreen mode

A global installation lets you run firecrawl from any directory without the npx startup overhead.

Authenticate and Inspect Configuration

Authenticate with the browser-based login flow:

firecrawl login
Enter fullscreen mode Exit fullscreen mode

You can also provide an API key directly through an environment variable:

export FIRECRAWL_API_KEY=fc-your-key-here
Enter fullscreen mode Exit fullscreen mode

Check authentication and account status:

firecrawl --status
Enter fullscreen mode Exit fullscreen mode

This displays the current authentication state and available account information, such as credits and concurrency limits.

To inspect the complete local configuration, run:

firecrawl view-config
Enter fullscreen mode Exit fullscreen mode

Switch accounts by logging out and signing in again:

firecrawl logout
firecrawl login
Enter fullscreen mode Exit fullscreen mode

For a local or self-hosted Firecrawl instance, specify a custom API URL:

firecrawl --api-url http://localhost:3002
Enter fullscreen mode Exit fullscreen mode

Scrape a Web Page

Extract the primary content from a page with:

firecrawl scrape https://example.com --only-main-content
Enter fullscreen mode Exit fullscreen mode

By default, Firecrawl CLI returns clean Markdown. Save the result to a specific file with -o:

firecrawl scrape https://example.com \
  --only-main-content \
  -o output.md
Enter fullscreen mode Exit fullscreen mode

Using a named output file is useful when you want to process the result in a later shell step or commit it to a dataset repository.

Request Multiple Formats

You can request several representations in one operation:

firecrawl scrape https://example.com \
  --format markdown,json,html,links,images \
  --pretty
Enter fullscreen mode Exit fullscreen mode

This is useful when an agent needs readable content alongside structured metadata or when you want to inspect the page's links and images separately.

Capture Screenshots

Capture a viewport screenshot:

firecrawl scrape https://example.com --screenshot
Enter fullscreen mode Exit fullscreen mode

For a full-page image, use:

firecrawl scrape https://example.com --full-page-screenshot
Enter fullscreen mode Exit fullscreen mode

Wait for Dynamic Content

If a page loads content asynchronously, add a wait timer:

firecrawl scrape https://example.com --wait-for 5000
Enter fullscreen mode Exit fullscreen mode

The value is specified in milliseconds. Increase it when important content appears only after client-side rendering completes.

Filter Page Content

Limit extraction to relevant HTML elements:

firecrawl scrape https://docs.example.com \
  --include-tags main,article \
  --exclude-tags nav,footer,script
Enter fullscreen mode Exit fullscreen mode

Use --timing when diagnosing slow requests or comparing different extraction settings:

firecrawl scrape https://example.com --only-main-content --timing
Enter fullscreen mode Exit fullscreen mode

Search the Web

Search for a topic and scrape the top results in one command:

firecrawl search "latest AI agent benchmarks" \
  --scrape \
  --limit 8 \
  --scrape-formats markdown
Enter fullscreen mode Exit fullscreen mode

Firecrawl CLI saves the search and scraped results locally. You can then inspect them with standard command-line tools:

find . -type f -maxdepth 2
grep -R "benchmark" .
Enter fullscreen mode Exit fullscreen mode

Depending on the research task, you can also filter by recency, location, or source type. Combine search with mapping and browser sessions when the initial results require deeper verification.

Map a Website

Use map to discover URLs before extracting content from a site:

firecrawl map https://example.com -o sitemap.json
Enter fullscreen mode Exit fullscreen mode

The command returns a structured list of URLs and associated metadata. Filter the resulting list before sending selected URLs to scrape or crawl.

A map-first workflow helps you understand a site's structure and keeps large extraction jobs focused on the pages you actually need.

Crawl a Site Recursively

Start a recursive crawl with progress reporting:

firecrawl crawl https://example.com \
  --wait \
  --progress \
  -o crawl-output.json
Enter fullscreen mode Exit fullscreen mode

Firecrawl CLI follows internal links, extracts page content, and stores the results locally. For larger sites, configure limits such as crawl depth, maximum pages, and concurrency to control runtime and usage.

The --progress option provides visibility into a running job, while --wait keeps the command active until the crawl completes.

A practical crawl workflow is:

firecrawl map https://example.com -o sitemap.json
# Filter sitemap.json for the sections you need.
firecrawl crawl https://example.com --wait --progress -o crawl-output.json
Enter fullscreen mode Exit fullscreen mode

Start with a small scope, validate the output, and then expand the crawl.

Automate Browser Sessions

Use cloud browser sessions for interactive pages and multi-step flows.

Launch a session:

firecrawl browser launch-session
Enter fullscreen mode Exit fullscreen mode

The command returns a session ID. Use that ID for browser actions:

firecrawl browser execute \
  "open https://news.ycombinator.com" \
  --session <id>
Enter fullscreen mode Exit fullscreen mode

Click an element:

firecrawl browser execute \
  "click .titleline > a" \
  --session <id>
Enter fullscreen mode Exit fullscreen mode

Extract the current page:

firecrawl browser execute \
  "scrape" \
  --session <id>
Enter fullscreen mode Exit fullscreen mode

Browser sessions support navigation, clicking, typing, and extraction after dynamic interactions. Close sessions when they are no longer needed so resources are released.

This approach replaces a large amount of custom Puppeteer orchestration with simple, agent-readable commands.

Configure Global Options

Persist common settings with the config command:

firecrawl config \
  --api-url https://your-custom-endpoint \
  --concurrency 5
Enter fullscreen mode Exit fullscreen mode

You can also configure output preferences and headers where supported by your installation. Keep the API key in an environment variable or shell profile rather than hard-coding it in scripts:

export FIRECRAWL_API_KEY=fc-your-key-here
Enter fullscreen mode Exit fullscreen mode

Before starting a large search or crawl, check account status:

firecrawl --status
Enter fullscreen mode Exit fullscreen mode

This helps you review available credits and concurrency limits before launching a long-running job.

Integrate Firecrawl CLI with AI Coding Agents

Initialize the Firecrawl CLI skill for compatible agents:

npx -y firecrawl-cli@latest init --all
Enter fullscreen mode Exit fullscreen mode

In CLI-and-skills mode, the agent runs Firecrawl commands explicitly when web data is needed. In MCP mode, the agent can call the configured tools directly.

Because Firecrawl CLI stores results as local files, an agent can receive a file path and inspect only the relevant sections instead of loading an entire response into context. This keeps research workflows more predictable and reduces unnecessary prompt data.

A simple agent-oriented workflow looks like this:

firecrawl search "topic to research" \
  --scrape \
  --limit 5 \
  --scrape-formats markdown \
  -o research.json

grep -R "key phrase" .
Enter fullscreen mode Exit fullscreen mode

The exact agent integration depends on the coding tool and its configuration, but the underlying workflow remains the same: fetch current data, save it locally, and pass focused files or excerpts to the agent.

Troubleshoot Common Problems

Authentication Errors

Run the login flow again:

firecrawl login
Enter fullscreen mode Exit fullscreen mode

If you are switching accounts or keys, clear the existing credentials first:

firecrawl logout
firecrawl login
Enter fullscreen mode Exit fullscreen mode

Rate Limits or Concurrency Errors

Reduce concurrency in your configuration:

firecrawl config --concurrency 2
Enter fullscreen mode Exit fullscreen mode

Check account status and available limits before retrying:

firecrawl --status
Enter fullscreen mode Exit fullscreen mode

Missing Content on JavaScript-Heavy Pages

Increase the wait time:

firecrawl scrape https://example.com --wait-for 10000
Enter fullscreen mode Exit fullscreen mode

Also try extracting only the primary content:

firecrawl scrape https://example.com --only-main-content
Enter fullscreen mode Exit fullscreen mode

Use timing information to compare different settings:

firecrawl scrape https://example.com --wait-for 10000 --timing
Enter fullscreen mode Exit fullscreen mode

Unexpected or Empty Output

Test the page with a broader extraction first:

firecrawl scrape https://example.com --format markdown,json,html
Enter fullscreen mode Exit fullscreen mode

Then add content filters after you understand the page structure:

firecrawl scrape https://example.com \
  --include-tags main,article \
  --exclude-tags nav,footer,script
Enter fullscreen mode Exit fullscreen mode

Best Practices

Follow these practices to keep Firecrawl CLI workflows efficient and reproducible:

  • Use --only-main-content when you need concise Markdown.
  • Give output files descriptive names.
  • Store results in a dedicated directory.
  • Test a small set of pages before starting a full crawl.
  • Use a search → map → scrape workflow for focused research.
  • Use crawl when you need broad internal-link coverage.
  • Set concurrency deliberately for large jobs.
  • Use --timing when diagnosing performance issues.
  • Version-control output directories when reproducibility matters.
  • Review credit usage regularly.
  • Keep API keys in environment variables instead of source files.
  • Close browser sessions after interactive work is complete.

Test Firecrawl API Requests with Apidog

Apidog can complement Firecrawl CLI when you need to inspect or validate API requests before automating them in a terminal workflow. Import Firecrawl endpoints such as scrape, search, and crawl into an Apidog collection, then use variables for the API key and other request parameters.

This lets you inspect:

  • Request headers
  • API key handling
  • Custom parameters
  • Response shapes
  • Different extraction options

You can also mock responses and run automated tests before moving a request into a shell script or agent workflow. This is useful when debugging complex options or validating an integration independently from the CLI.

Learn more at Apidog.

Conclusion

Firecrawl CLI provides a terminal-first workflow for scraping, searching, mapping, crawling, and browser automation. Its local file output makes results easy to inspect with shell tools, pass to AI coding agents, and preserve as reproducible datasets.

Start with the initialization command:

npx -y firecrawl-cli@latest init --all --browser
Enter fullscreen mode Exit fullscreen mode

Then authenticate, run a small scrape, and expand into search, mapping, crawling, or browser automation as your workflow requires. Careful use of format selectors, content filters, wait timers, and concurrency settings can improve both output quality and efficiency.

Additional Resources

Top comments (0)