DEV Community

Crawler Bros
Crawler Bros

Posted on

How to Scrape LinkedIn Profile Posts for Competitor Analysis

#linkedin #scraping #automation #socialmedia

Tracking competitor content and monitoring industry thought leaders on LinkedIn is one of the most effective ways to refine your B2B marketing strategy. However, manually scrolling through individual profile activity feeds to copy post copy, engagement counts, and media formats is slow, tedious, and impossible to scale across dozens of key accounts.

Whether you are performing content research, measuring influencer engagement rates, or analyzing personal brand performance, automated data extraction provides the structured insights you need.

The linkedin-profile-posts-scraper actor on Apify solves this by scraping all posts, articles, and reposts directly from any LinkedIn profile's activity feed at scale.

Why Scrape LinkedIn Profile Activity Feeds?

Personal profiles often generate significantly more organic engagement on LinkedIn than company pages. By extracting post data from founders, executives, and creator accounts, you can unlock valuable market intelligence:

  • Analyze Winning Content Formats: Discover whether text-only posts, carousels, documents, polls, or videos generate the highest interaction in your industry.
  • Track Competitor Messaging: Monitor product updates, positioning shifts, and key narratives shared by competing leaders.
  • Evaluate Influencer Reach: Review verified engagement metrics—including reaction counts, comments, and reposts—before launching collaboration campaigns.
  • Identify Active Prospects for Lead Generation: Identify key decision-makers who actively post and engage on the platform.

Key Features of LinkedIn Profile Posts Scraper

The scraper interacts with LinkedIn's Voyager API to retrieve comprehensive data from individual activity feeds. Key capabilities include:

  • Comprehensive Feed Extraction: Captures text posts, images, videos, articles, documents, carousels, polls, events, and reposts.
  • Granular Engagement Metrics: Extracts total reactionsCount, commentsCount, and repostsCount, along with a complete reactionBreakdown object covering specific reaction types (like, celebrate, love, insightful).
  • Batch Processing: Accepts multiple profile URLs or bare usernames in a single run.
  • Deep Pagination: Retrieves up to 500 posts per profile.
  • Targeted Filtering: Filter data before exporting using built-in parameters such as minimum reaction counts or media types.

Supported Inputs and Configuration

Setting up a run requires only basic configuration:

  • profileUrls (Required): An array of target profiles. It accepts multiple formats, including full URLs like https://www.linkedin.com/in/williamhgates/ or https://uk.linkedin.com/in/williamhgates, bare usernames like williamhgates, or paths like /in/williamhgates.
  • cookie (Required): A valid LinkedIn session cookie. You can provide either the raw li_at string from your browser's DevTools or a full cookie JSON array.
  • maxPostsPerProfile: Sets the maximum number of posts to scrape per profile (default is 50, maximum is 500).
  • minReactions: Filters output to only include posts meeting a specific reaction threshold.
  • mediaType: Restricts results to specific formats, such as text, image, video, article, document, carousel, repost, poll, or event.
  • includeReposts: A boolean parameter (default: true) to include or exclude reshared content.
  • proxyConfiguration: Proxy settings to route requests; residential proxies are recommended.

Step-by-Step: How to Scrape Posts from LinkedIn Profiles

Step 1: Retrieve Your LinkedIn Session Cookie

Because LinkedIn requires authentication to access feed data via the Voyager API, you need your session cookie:

  1. Open LinkedIn in your browser and log in.
  2. Open Chrome DevTools (F12 or right-click and select Inspect).
  3. Navigate to Application > Cookies > https://www.linkedin.com.
  4. Find the cookie named li_at and copy its value.

Step 2: Configure the Actor Input

Navigate to the linkedin-profile-posts-scraper on Apify and set your parameters:

  1. Paste the target URLs or usernames into the profileUrls array.
  2. Paste your li_at token into the cookie field.
  3. Optional: Set maxPostsPerProfile (e.g., 100) and set a minReactions filter if you only want top-performing posts.

Step 3: Run the Scraper and Export Data

Click Start. The scraper processes profiles sequentially while using realistic delays to follow platform API patterns. Once complete, you can view or download the structured dataset in JSON, CSV, or Excel formats.

Understanding the Scraped Data Output

Each post extracted from an activity feed is saved as a structured record with detailed metadata:

  • postId: The unique LinkedIn activity ID.
  • postUrl: Direct link to the post (https://www.linkedin.com/feed/update/urn:li:activity:.../).
  • authorName, authorProfileUrl, and authorHeadline: Complete author profile information.
  • postedAt: ISO 8601 UTC timestamp of publication.
  • content: The full text body of the post.
  • mediaType and mediaUrls: The classified format and associated media CDN links.
  • articleTitle and articleUrl: Metadata captured when mediaType is article.
  • reactionsCount, commentsCount, and repostsCount: Numerical engagement totals.
  • reactionBreakdown: Breakdown object detailing specific reaction counts (like, celebrate, love, insightful).
  • inputProfileUrl and scrapedAt: Source tracking and scrape execution timestamps.

Frequently Asked Questions

Do I need a LinkedIn account to use this scraper?

Yes. A valid li_at session cookie from an active LinkedIn account is required to authenticate requests to the Voyager API.

How many posts can be collected per profile?

The actor can retrieve up to 500 posts per profile. Note that LinkedIn's feed API may return fewer posts for older accounts with limited historical activity.

Can I run multiple profiles in one execution?

Yes. You can provide an array of multiple profile URLs or usernames in profileUrls, and the actor will process each profile sequentially.

How do I connect this with other LinkedIn scraping workflows?

This actor belongs to a broader suite of LinkedIn automation tools on Apify. Because all actors in the suite share the same cookie input conventions and omit-empty data structures, you can easily combine profile post scraping with tools like the linkedin-comments-scraper, linkedin-post-reactions-scraper, or linkedin-profile-scraper for end-to-end social media intelligence.

Ready to try it yourself? Run *LinkedIn Profile Posts Scraper** on the Apify Store -- no setup required.*

Top comments (0)