DEV Community

Cover image for google hid the links in its search results. how i still pull linkedin employee lists
George Kioko
George Kioko

Posted on

google hid the links in its search results. how i still pull linkedin employee lists

#webscraping #seo #javascript #webdev

aug 30 my linkedin scraper failed 452 runs in one day. 251 more the next day.

google had started swapping every search result link for a google.com/goto redirect with a token inside. google confirmed the rollout to search engine land and said its a measure to protect against abuse.

searchers dont notice. you click, google redirects you, you land on the page. scrapers notice right away.

what the pages look like now

on sep 19, 25 of the 28 result pages i pulled came back with hidden links. three still had normal ones.

the token is not something you can decode, and asking google for it through my proxy returned an error.

the strange part is everything else is still there. the name, the job title, the company, even the city. only the link is gone. so it doesnt stop scraping. it makes it slower and more expensive, and that hurts small tools the most.

what i changed

three things, in order of how much they helped

  1. read the names and titles straight off the hidden pages and ship those rows without a link, free
  2. retry a page when it comes back hidden, because the layout google serves changes from one request to the next
  3. ask for googles basic html page, which still had real links about 1 time in 4 in my tests

same test input, three companies, 25 people each. before the change 16 rows came back. after it, 75.

the tool, if you need who works where

its an apify actor. you give it linkedin company urls or names. it finds public profiles through google, keeps the people whose own headline names the company, and tags each one as a decision maker or not, with a seniority band and a department.

it never logs in to linkedin and never touches an account.

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('george.the.developer/linkedin-company-employees-scraper').call({
    companies: ['https://www.linkedin.com/company/stripe/'],
    maxEmployees: 25,
    decisionMakersOnly: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((p) => `${p.fullName} | ${p.headline} | ${p.seniorityBand}`));
Enter fullscreen mode Exit fullscreen mode

set decisionMakersOnly: true and everyone below director gets skipped before any charge.

price is $25 per 1,000 profiles tied to the company plus $0.05 a run. rows it cant tie to the company come back free, and so do the rows with hidden links.

a test today on stripe with 10 people finished in about two minutes. 9 rows tied to stripe got billed, so that run cost $0.275.

turn on monitoring and run it again next week. it tells you who joined, who left and who changed title. that part is free.

what it wont do. no emails, no phone numbers. and it only sees what google has indexed, so small companies come back thin.

https://apify.com/george.the.developer/linkedin-company-employees-scraper

Top comments (0)