DEV Community

Cover image for Apify, step by step: how to automate web data collection (no code needed)
0xGollum
0xGollum

Posted on

Apify, step by step: how to automate web data collection (no code needed)

It's 10 p.m. An empty spreadsheet, thirty browser tabs open, one job: for every listing, copy the title, the company, the city and the link. Listing #1. Listing #2. Listing #3. 197 to go.

I build data actors on Apify and I share them here, and I keep noticing the same thing: a lot of people see them without really knowing what Apify is. So this is the walkthrough I wish I'd had on day one, with an example you can redo yourself in a few minutes. (Full disclosure: the actor in the example is one of mine, and my other actors live here.)

What is Apify?

A platform that hosts tens of thousands of ready-to-use programs you can run without installing anything. Each one does one specific job: collect a venue's reviews, track a product's price, extract job listings. Apify calls them actors. An actor is just a script hosted and executed on Apify's servers: you give it parameters, it hands you back a data file. The Apify Store is the catalog, like an app store.

Web scraping in brief

What these programs do is called web scraping: automatically collecting information displayed on web pages and storing it in a structured form (a table, a JSON file). It's copy-paste, automated. It only applies to public data, and while respecting the terms of the websites you visit: it is neither hacking nor a way around a login.

A complete example: job listings, in three steps

Step 1 — Choose the actor. In the Apify Store, search for "LinkedIn Jobs" and open its page (here is the one I use for this example). It shows what the actor returns, how it's priced, and the button to try it.

The actor page in the Apify Store

Step 2 — Fill in the input. The input is the parameters of the search. Here: a keyword ("data analyst"), a location (Paris, France) and a time window (posted this week). It works like a form, no code at all.

The actor input form

Step 3 — Run it and get the result. Click Start. A minute or two later the output (the dataset) is ready: job title, company, location, contract type, seniority level, number of applicants, direct link. Export it to Excel, CSV or JSON, or query it through the API.

The results table
Result columns: 1. job title · 2. company · 3. city · 4. number of applicants

What it changes in practice

  • Job seekers: compare hundreds of listings in a spreadsheet instead of reading them one by one.
  • Recruiters and sales: see which companies are hiring, for which profiles, at what level: a good indicator of their plans.
  • Analysts: follow a market (number of listings, cities, seniority) by rerunning the same search every week.

And it goes well beyond jobs: prices, classified ads, customer reviews, sports results. Any public data on a page can end up in a table.

The cost

Apify gives you a free monthly credit to try things out. Beyond that, most actors are paid per result rather than by subscription: for this one, starting at $1 per 1,000 listings. The example run (15 listings) comes to about 1.5 cents.

For developers

Nothing forces you through the UI: every actor can be called through the API or an SDK (Python, JavaScript), scheduled (every Monday, say) and plugged into other tools (Make, n8n, Zapier, Google Sheets). And you can write and publish your own actors in the Store.

Limits you should know about

An actor isn't magic:

  • Some websites block bots (anti-bot protections, CAPTCHAs, IP blocking). The actor may return fewer results than expected, or fail. Good actors handle this (switching addresses, retrying) and don't bill for blocked pages, but none can promise 100%.
  • Websites change their pages without warning. What worked yesterday can break tomorrow until its author adapts it, so pick a maintained actor (last update date, reviews, number of users).
  • The data must be public, and its use must comply with the websites' terms and with regulations, including GDPR whenever personal data is involved. Some websites also forbid automated collection in their terms of use: check what is allowed for your use case.
  • Results vary day to day: listings change, and so does their order. A result is a snapshot.

Glossary

  • Actor: a program hosted by Apify and run on demand.
  • Web scraping: automated collection of public data displayed on web pages.
  • Input: the parameters you give the actor.
  • Output / dataset: the data it returns, exportable as CSV, Excel or JSON.

Going further

To try it: create a free account on apify.com, search for "LinkedIn Jobs" and run it with your own search. Questions or a data need you can't find an actor for? Drop it in the comments.

Top comments (0)