DEV Community

Cover image for Launching Crawlee, the web scraping and browser automation library for Node.js
Ondra Urban
Ondra Urban

Posted on

7 1

Launching Crawlee, the web scraping and browser automation library for Node.js

Hello world,

Today, drawing on our team's years of experience, we're launching Crawlee, the the web scraping and browser automation library for Node.js that's built for fastest development and maximum reliability in production.

Main features

๐Ÿ–ผ Supports headless browsers with Playwright or Puppeteer

โšก๏ธ Supports raw HTTP crawling with Cheerio or JSDOM

๐ŸŽ› Automated parallelization and scaling of crawlers for top performance

๐Ÿพ Avoids blocking using smart sessions, proxies, and browser fingerprints

๐ŸšŽ Simple management and persistence of queues of URLs to crawl

๐Ÿ—œ Written completely in TypeScript for type safety and code autocompletion

๐Ÿ“š Comprehensive documentation, code examples, and tutorials

๐Ÿ’ช๐Ÿผ Actively maintained and developed by Apifyโ€”we use it ourselves!

Getting started

Visit crawlee.dev or run the following command:

npx crawlee create my-crawler
Enter fullscreen mode Exit fullscreen mode

Liked Crawlee?

๐Ÿ’› You can support the project on GitHub, Product Hunt, or Hacker News

Top comments (3)

Collapse
 
joelbonetr profile image
JoelBonetR ๐Ÿฅ‡ โ€ข

Amazing! ๐Ÿ˜

Do you plan on writing more posts using it to showcase?

I would love to see the typical stand-alone example of a google maps crawler using Playwright and Crawlee! ๐Ÿ˜

Collapse
 
mnmkng profile image
Ondra Urban โ€ข

If you don't mind an actual production grade example then this Google Maps scraper uses Apify SDK v2, from which we spun Crawlee off. Apify SDK and Crawlee interface is almost the same.

Crawlee is essentially Apify SDK v3, without the things that could only be used on the Apify platform. If you wish, you can read here, why we did the spin off.

We will migrate the Google Maps scraper to Crawlee soon, to leverage some of the new features.

Collapse
 
joelbonetr profile image
JoelBonetR ๐Ÿฅ‡ โ€ข

Than you for all this information, much appreciated ๐Ÿ˜

SurveyJS custom survey software

JavaScript Form Builder UI Component

Generate dynamic JSON-driven forms directly in your JavaScript app (Angular, React, Vue.js, jQuery) with a fully customizable drag-and-drop form builder. Easily integrate with any backend system and retain full ownership over your data, with no user or form submission limits.

Learn more

๐Ÿ‘‹ Kindness is contagious

Please leave a โค๏ธ or a friendly comment on this post if you found it helpful!

Okay