DEV Community: Anurag Panda

ig-harvester: an open-source Instagram OSINT toolkit built on Playwright + CDP

Anurag Panda — Fri, 09 Oct 2026 06:05:07 +0000

Instagram doesn't have a public API for the kind of data OSINT work actually needs —
comment threads with reply hierarchies, full follower lists, carousel slides at original
resolution. So most tools in this space either reverse-engineer the private API (fragile)
or drive a browser (slow, easy to get wrong).

ig-harvester takes the second route,
but with a twist that solves the hardest part of browser-based collection: it attaches to a
Chrome window you're already logged into via the Chrome DevTools Protocol (CDP).

Disclosure: I built ig-harvester. It's MIT-licensed and free. This post explains how it
works, not just what it does.

The core idea: bring your own session

Most scrapers try to become logged in — storing cookies, solving challenges, rotating
accounts. That's a losing arms race, and it puts the tool in a bad spot legally and
operationally.

ig-harvester instead launches Chrome with remote debugging (one PowerShell script), you log
in with a burner account in that window, and the tool attaches to it:

.\dependencies\launch-chrome-debug.ps1    # Chrome with CDP on :9222
# log into your burner account, keep the window open
.\ig-harvester.ps1 scrape someuser --posts 50 --comments --followers --sqlite

Everything the harvest does, your session could do by hand. No API keys, no token storage,
nothing leaves your machine.

Triple-source extraction

The single biggest failure mode of Instagram scrapers is relying on one signal.
Instagram serves content three different ways, and each one breaks at different times:

Embedded Relay JSON — the __ NEXT_DATA__-style blob in the page source. Rich (carousels, comment threads, exact timestamps) but occasionally absent.
Semantic DOM — walking the accessibility-oriented markup. Always present, but shallower.
Screenshots — the ground truth for anything the first two disagree on.

ig-harvester mines all three and cross-checks. When live Instagram stopped embedding sidecar
JSON for carousels (it did, recently), the tool didn't break — it fell back to walking the
post with the Next button and reading each slide's <img>/srcset at full resolution.

Resumability is not optional

A 500-post harvest takes a while, and browsers crash. Every run writes a SQLite cache keyed
by shortcode:

out/someuser/
  someuser.json            # payload + analytics
  someuser-posts.csv       # one row per post
  someuser-comments.csv    # one row per comment
  someuser-followers.csv
  someuser-following.csv
  someuser.db              # --sqlite
  .cache.db                # resume cache

Kill the process halfway, run it again, and it continues where it stopped instead of
re-collecting the first 250 posts. --no-resume disables it when you want a clean sweep.

Pacing like a human, not a loop

Rate limiting is where "it works on my machine" dies in production. The tool uses:

jittered delays between actions (--delay-min 900 --delay-max 2600, in ms)
a token bucket for burst control
exponential backoff on transient failures
optional HTTP/SOCKS5 proxy support for operational security

node dependencies/scrape-ig.mjs --profile someuser --proxy http://127.0.0.1:8080 --delay-min 1200

Data out, insights out

Raw data is the start, not the end. Every run computes engagement rate, posting cadence,
best posting hour/day, top hashtags and mentions, follower/following ratio, and bio signals
(email/phone/URL detection) — straight into the JSON payload.

Then there's ig-analyzer, a live dashboard on localhost:8080 that charts likes per post,
posts by hour/day, and content mix for any scraped or archived profile.

For the archive-minded: import-data.ps1 copies each run into a dated snapshot (re-scrapes
never overwrite history), so follower counts stay comparable over time.

One entry point, no CLI archaeology

.\ig-harvester.ps1                                    # interactive menu
.\ig-harvester.ps1 scrape someuser --posts 50 --comments
.\ig-harvester.ps1 -Action Full -User someuser -Yes    # scrape -> images -> archive
.\ig-harvester.ps1 -Action Doctor                      # environment check

-Yes makes it non-interactive for CI, -DryRun prints the exact commands, and exit codes
are deterministic (0 ok, 1 step failed, 2 usage error). On macOS/Linux, run the Node
CLIs directly — same flags, same behavior.

Get it

Repo: github.com/anurag-panda-dev/ig-harvester
Site & full CLI reference: anurag-panda-dev.github.io/ig-harvester
Requires Node ≥ 20 and Chrome. npm install && npm test to verify your setup.

Ethical use: ig-harvester is intended for legitimate OSINT research, journalism,
security audits and education. You are responsible for complying with applicable laws,
Instagram's Terms of Service and privacy regulations (GDPR, CCPA) — and for confirming
consent before collecting data about someone.

Have you built browser-based collectors for sites with no API? I'd love to hear how you
handled extraction fallbacks — the carousel problem was the interesting one for me.

🧑‍💼 Face Attendance Maker using Streamlit and OpenCV 😎

Anurag Panda — Sun, 06 Jul 2025 03:17:04 +0000

📌 Published by @anurag_panda

🚀 GitHub Repo: Face Attendance Maker

👋 Introduction Attendance tracking is a crucial part of any educational institution or organization, but the traditional method of calling names or marking attendance manually is outdated and error-prone.

So I built Face Attendance Maker — a modern solution that uses face recognition to automate attendance using Streamlit for the UI and OpenCV for the camera and image processing.

🧠 Features
✅ Real-time face detection using OpenCV
✅ Attendance tracking with time stamps
✅ Streamlit-based UI for ease of use
✅ CSV export of attendance records
✅ Simple, clean, and fast
🧰 Tech Stack
Python

Streamlit – for the front-end UI

OpenCV – for face detection and camera access

face_recognition – for comparing and identifying faces

NumPy & Pandas – for data handling

CSV – for storing attendance records

🚦 How It Works Upload or register known faces in a folder (ImagesAttendance/)

Launch the Streamlit app

Camera opens and starts detecting faces

If a known face is found:

Name and timestamp are recorded

Data is saved into Attendance.csv

🖥️ Getting Started Clone the repo:

git clone https://github.com/anurag-panda-nshm/face-attendance-maker
cd face-attendance-maker

Install dependencies:

pip install -r requirements.txt

Run the app:

streamlit run app.py

📸 Make sure your webcam is enabled.

📂 Folder Structure

📁 face-attendance-maker/
├── app.py
├── Attendance.csv
├── ImagesAttendance/
│   └── anurag.jpg
├── requirements.txt
└── ...

Add more images to ImagesAttendance/ with filenames as the person’s name.

🔍 Use Cases Classroom attendance

Workplace check-ins

Event registrations

Lab/Library access tracking

⚙️ Future Improvements Face registration through the UI

Multi-user support and login system

Dashboard with analytics

Integration with cloud databases like Supabase or Firebase

🙌 Let's Collaborate!
Feel free to fork, contribute, or open issues!
⭐ Star the repo if you like the project!

GitHub: https://github.com/anurag-panda-nshm/face-attendance-maker

💬 Final Thoughts Building this project taught me how powerful and fun real-time computer vision can be. With Streamlit, it's easier than ever to turn Python scripts into beautiful web apps.

Let me know your thoughts or improvements!

📧 Reach out on LinkedIn or drop a comment below 👇