DEV Community

Charles
Charles

Posted on

Introducing xcrawl-cli: A Command-Line Web Scraper in Your Terminal

#webscraping #cli #javascript #node

Introducing xcrawl-cli: A Command-Line Web Scraper in Your Terminal

In my previous posts, I covered the xcrawl-scraper npm package and the XCrawl API. Today, I want to show you xcrawl-cli — the command-line interface that puts web scraping power directly in your terminal.

What is xcrawl-cli?

xcrawl-cli is a Node.js CLI tool that wraps the XCrawl API into simple terminal commands. No code required — just pipe URLs and get structured data.

Installation 

npm install -g xcrawl-cli
Enter fullscreen mode Exit fullscreen mode

Quick Start

Scrape a single page 

xcrawl scrape https://example.com --format markdown
Enter fullscreen mode Exit fullscreen mode

Search the web 

xcrawl search "latest AI news" --count 10
Enter fullscreen mode Exit fullscreen mode

Output to file 

xcrawl search "Python tutorials" --output results.json
Enter fullscreen mode Exit fullscreen mode

Features

  • Zero config — Just install and run
  • Multiple output formats — JSON, CSV, Markdown
  • Smart retry — Automatic retry with JS rendering when pages block you
  • Concurrent scraping — Up to 5 parallel requests
  • Proxy rotation — Residential proxies included

Real-World Example: Monitor HN Front Page 

xcrawl search "site:news.ycombinator.com" --count 20 --output hn.json
Enter fullscreen mode Exit fullscreen mode

Why Terminal?

Not every scraping task needs a full script. Sometimes you just want to:

  • Quickly grab page content for debugging
  • Test a search query before writing code
  • Schedule a crawl via cron

xcrawl-cli is for those moments.

Built on XCrawl API — handle JS rendering, CAPTCHAs, and IP blocks automatically.

Top comments (0)