DEV Community

DIvy97
DIvy97

Posted on

Help: LinkedIn Profile Analytics Automation Challenges

#webdev #automation #browser #javascript

TL;DR: Building a LinkedIn analytics scraper using Playwright/Puppeteer but hitting multiple roadblocks. Need advice from developers who've tackled similar challenges.

What I'm Building
A service that automatically fetches LinkedIn profile analytics (post impressions, followers, profile views, etc.) for users. Think of it as a dashboard that pulls your LinkedIn metrics without manual logging in each time.

Technical Stack

  • Browser Automation: Playwright with Chromium
  • Deployment: AWS Lambda
  • Authentication: Stored cookies + user agents
  • Session Management: Persistent browser profiles

Problems I've Hit (and semi-solved)

  1. Cookie Authentication Issues
    Stored LinkedIn cookies but still getting login screens
    Tried: Session cookies, proper domain/path settings, complete storage state
    Partial Fix: Detecting and clicking saved login profiles

  2. LinkedIn's Bot Detection
    Getting "Try again later" pages even with valid credentials
    Tried: Stealth mode, realistic user agents, proper headers, timing delays
    Partial Fix: Anti-detection measures + automatic "try again" button clicking

  3. Page Loading Issues
    Profile pages showing skeleton/placeholder content instead of actual data
    Analytics buttons not appearing or clickable
    Tried: Various wait strategies, network idle detection, content-specific selectors
    Current: 45+ second waits + content validation

  4. Navigation Reliability
    Me menu → View Profile → Analytics flow was unreliable
    Solution: Direct URL navigation to specific profile

  5. Rate Limiting & Session Persistence
    Frequent blocks when running multiple attempts
    Tried: Persistent browser directories, proper session state saving
    Current Status

Current Status

✅ Login automation working (saved profiles)
✅ Profile navigation working (direct URLs)
❌ Still struggling: Page content loading + Analytics button detection
❌ Main blocker: LinkedIn's aggressive bot detection

Questions for the Community

  1. Has anyone successfully automated LinkedIn analytics collection? What approach worked?
  2. Better alternatives to cookie-based auth? OAuth feels overkill for personal analytics.
  3. LinkedIn-specific anti-detection techniques? Beyond standard stealth measures.
  4. Page loading detection strategies? How do you know when LinkedIn's dynamic content is actually ready?

Top comments (0)