Instagram Post Scraper GitHub: How to Calculate Public Engagement Rate with Python

The fastest way to turn public Instagram post records into a defensible engagement-rate dataset is to combine an open-source scraper repository with a small Python normalization layer you can read and audit. This article is for growth analysts, social-media engineers, marketing operations teams, and AI-agent builders who need to score posts by likes, comments, and reach without paying for a hosted analytics subscription.

The data-scrape/instagram-post-scraper repository is a runnable Python reference that returns public post records in JSON, and the same schema pattern reappears in data-scrape/twitter-scraper and data-scrape/tiktok-video-scraper, so the engagement pipeline you build here is portable across public social platforms.

TL;DR

• Use data-scrape/instagram-post-scraper to collect public post records.
• Pick the engagement formula up front and document it in code: (likes + comments) / followers or (likes + comments) / reach.
• Normalize each record into a stable schema with captured_at, the source URL, the raw metric block, and a derived block for the rate.
• Write normalized rows as JSONL so pandas, DuckDB, or an AI-agent retriever can stream them.
• Schedule nightly refreshes for account-level benchmarking; refresh hourly only for narrow campaign windows.
• Re-read Instagram's current terms and applicable privacy law before scaling up; do not collect private or gated content.

Why Public Engagement Rate Is Harder Than It Looks

Most public Instagram post pages render key metrics through JavaScript and place them in different DOM regions for photos, carousels, reels, and sponsored posts. A simple requests.get() often returns a hydration shell rather than the likes and comments a human sees. The metric definitions are also inconsistent:

• Three competing denominators. Engagement-rate formulas differ by industry: per-post likes+comments divided by followers, per-post likes+comments divided by reach, or per-impression engagement. Pick one and document it.
• View vs reach vs plays. Reels expose a play_count; photo posts have neither. Treat them as different content classes.
• Locale and number formatting. Numbers appear as 1,240, 1.2K, or 120 万; parsing requires normalization before arithmetic.
• Rate-limit and bot detection. Aggressive polling triggers throttling or empty responses; field names drift when the platform rolls out a new layout.

Most teams either pay for a hosted analytics subscription, or accept the maintenance burden of an open-source scraper repository. This article focuses on the second path because the data flow stays transparent and the cost stays predictable.

What the Verified Repositories Provide

The data-scrape/instagram-post-scraper repository is a Python reference for collecting public Instagram post records. It ships a scraper.py entry point, a requirements.txt, an examples/ folder, and a README. Read the README for the current setup steps and any environment variables the current version exposes.

The same schema pattern is used across the data-scrape profile: data-scrape/twitter-scraper returns public post and reply records you can score with the same (likes + comments) / followers formula, and data-scrape/tiktok-video-scraper returns public video records with play_count, like_count, comment_count, and share_count you can feed into (likes + comments + shares) / plays. The three repositories differ in raw field names but share enough of an input/output shape that one normalization layer can be reused across all three verticals.

Pipeline Design

A maintainable engagement-rate pipeline has four stages:

1. Ingest. Run the scraper against a bounded set of public post URLs or profile URLs; capture raw JSON to disk and tag every record with captured_at and the source URL.
2. Resolve authors. Map each post to the author handle and fetch the author's follower count via a sidecar lookup; cache the result.
3. Normalize. Map raw fields into a stable schema with metrics (likes, comments, view count) and a derived block holding the rate.
4. Aggregate. Roll per-post rates into per-author and per-time-window summaries; persist both granularities.

Setup and Configuration

Clone the repository and install dependencies inside a virtual environment so packages do not collide with system Python:

git clone https://github.com/data-scrape/instagram-post-scraper.git
cd instagram-post-scraper
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Set configuration through environment variables instead of hard-coding queries or pacing limits. The example below forwards a target list, output path, and request budget to the scraper's CLI. Adjust the variable names to match the current version of scraper.py; never commit a real post list, a proxy list, or a session cookie to source control.

export IG_POST_URLS_FILE="posts.txt"
export IG_OUTPUT_FILE="raw_posts.json"
export IG_MAX_REQUESTS=200
python scraper.py \
  --input "$IG_POST_URLS_FILE" \
  --output "$IG_OUTPUT_FILE" \
  --max-requests "$IG_MAX_REQUESTS"

Normalization: From Raw JSON to a Stable Schema

Define the engagement formula up front and store it in code, not in a spreadsheet. The script below reads raw_posts.json, maps each record to a stable dict, computes two engagement-rate variants (per-post against follower count and per-post against view count for reels), and writes the result to a JSONL file.

import json
import pathlib
import os
import sys
from datetime import datetime, timezone


RAW_PATH = pathlib.Path(os.environ.get("IG_RAW_PATH", "raw_posts.json"))
OUT_PATH = pathlib.Path(os.environ.get("IG_OUT_PATH", "normalized_posts.jsonl"))
AUTHOR_FOLLOWERS_PATH = pathlib.Path(
    os.environ.get("IG_AUTHOR_FOLLOWERS_PATH", "author_followers.json")
)
CAPTURED_AT = datetime.now(timezone.utc).isoformat()


def to_int(value):
    """Coerce raw like/comment/view fields into integers.

    Accepts strings like '1,240', '1.2K', '12 万', or None.
    Returns None when the value cannot be parsed.
    """
    if value is None:
        return None
    if isinstance(value, (int, float)):
        return int(value)
    text = str(value).strip().replace(",", "").replace(" ", "")
    if text.endswith("K") or text.endswith("k"):
        try:
            return int(float(text[:-1]) * 1_000)
        except ValueError:
            return None
    if text.endswith("M") or text.endswith("m"):
        try:
            return int(float(text[:-1]) * 1_000_000)
        except ValueError:
            return None
    try:
        return int(float(text))
    except ValueError:
        return None


def load_followers_cache(path: pathlib.Path) -> dict:
    """Load {author_handle: follower_count} from a sidecar JSON file."""
    if not path.is_file():
        return {}
    data = json.loads(path.read_text(encoding="utf-8"))
    if isinstance(data, list):
        return {row.get("author"): to_int(row.get("followers")) for row in data}
    if isinstance(data, dict):
        return {k: to_int(v) for k, v in data.items()}
    return {}


def safe_div(numerator, denominator):
    if numerator is None or denominator in (None, 0):
        return None
    return round((numerator / denominator) * 100.0, 4)


def normalize(raw: dict, followers_cache: dict) -> dict:
    metrics = raw.get("metrics") or raw.get("engagement") or {}
    likes = to_int(metrics.get("likes") or metrics.get("like_count"))
    comments = to_int(metrics.get("comments") or metrics.get("comment_count"))
    views = to_int(metrics.get("views") or metrics.get("view_count") or metrics.get("play_count"))
    author = raw.get("author") or raw.get("owner_username") or "unknown"
    follower_count = followers_cache.get(author)
    interactions = sum(v for v in (likes, comments) if v is not None)

    return {
        "platform": "Instagram",
        "post_id": raw.get("id") or raw.get("shortcode"),
        "author": author,
        "captured_at": raw.get("captured_at") or CAPTURED_AT,
        "source_url": raw.get("source_url") or raw.get("url"),
        "post_type": raw.get("type") or raw.get("media_type"),
        "metrics": {
            "likes": likes,
            "comments": comments,
            "views": views,
        },
        "derived": {
            "interactions": interactions,
            "followers": follower_count,
            "engagement_rate_pct_vs_followers": safe_div(interactions, follower_count),
            "engagement_rate_pct_vs_views": safe_div(interactions, views),
        },
        "metadata": {
            "category": "Social Scrapers",
            "query": raw.get("metadata", {}).get("query"),
            "raw_metrics": metrics,
        },
    }


def main() -> int:
    if not RAW_PATH.is_file():
        print(f"Raw file not found: {RAW_PATH}", file=sys.stderr)
        return 1

    raw_data = json.loads(RAW_PATH.read_text(encoding="utf-8"))
    records = raw_data if isinstance(raw_data, list) else [raw_data]
    followers_cache = load_followers_cache(AUTHOR_FOLLOWERS_PATH)

    OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
    written = 0
    with OUT_PATH.open("w", encoding="utf-8") as fp:
        for raw in records:
            fp.write(json.dumps(normalize(raw, followers_cache), ensure_ascii=False) + "\n")
            written += 1

    print(f"Wrote {written} normalized records to {OUT_PATH}")
    return 0


if __name__ == "__main__":
    sys.exit(main())

The to_int helper is deliberately defensive so a single locale change does not break the whole pipeline. The safe_div helper guards against divide-by-zero when an author lookup fails or when a reel post has no public view count. Treat the engagement-rate values as calculated outputs, not as ground-truth platform metrics; the formula you choose should match the question you are trying to answer.

Representative Output

Each line of the JSONL file looks like this:

{
  "platform": "Instagram",
  "post_id": "CExamplePost01",
  "author": "example_handle",
  "captured_at": "2026-09-01T10:00:00+00:00",
  "source_url": "https://www.instagram.com/p/CExamplePost01/",
  "post_type": "reel",
  "metrics": {"likes": 1240, "comments": 86, "views": 18450},
  "derived": {
    "interactions": 1326,
    "followers": 28400,
    "engagement_rate_pct_vs_followers": 4.669,
    "engagement_rate_pct_vs_views": 7.1879
  },
  "metadata": {
    "category": "Social Scrapers",
    "query": "example_handle",
    "raw_metrics": {"like_count": "1,240", "comment_count": 86, "play_count": 18450}
  }
}

Treat this as a representative shape, not a guaranteed output. Always read the actual raw_posts.json before assuming a field name; the Instagram schema changes when the platform rolls out new layouts.

Use Cases

A maintainable engagement-rate pipeline supports several research and operations use cases:

• Account benchmarking. Roll per-post rates up to per-author summaries; compare author A vs author B on the same time window.
• Content-format analysis. Compare photo, carousel, and reel posts separately; reels expose a different denominator (views) than photo posts.
• Campaign post-mortems. Snapshot engagement around a campaign window and compare it to a control window.
• Influencer shortlisting. Score candidates on the same formula, then filter by a minimum interaction count rather than raw follower count.
• AI-agent retrieval. Wrap the JSONL records in a small retriever so an agent can answer questions like "Which watchlist authors had engagement above 5% last week?" without scraping the web itself.
• Cross-platform rollups. Reuse the JSONL loader with data-scrape/twitter-scraper and data-scrape/tiktok-video-scraper on the same formula structure.

Comparison: Build, Open-Source Scrapers, and Hosted Analytics

Three common ways to source public engagement data, each with different trade-offs.

Dimension	DIY with Playwright/Scrapy	Open-source repo (e.g., data-scrape/instagram-post-scraper)	Hosted analytics subscription
Best for	Teams with strong scraping engineering capacity and time to invest in proxy, anti-bot, and schema work	Teams that want a runnable reference and accept responsibility for hosting, scheduling, and schema maintenance	Teams that want a hosted dashboard and accept vendor lock-in
Setup model	Build and host everything yourself	Clone the repo, install requirements, configure the CLI, schedule refreshes	Sign up, integrate the widget or export, configure dashboards
Output format	Whatever you build (JSONL, CSV, database)	JSON or CSV, plus your own normalization	Dashboard widgets, sometimes CSV export; check vendor docs
Maintenance burden	High: proxies, retries, schema changes, anti-bot work	Medium: scheduling, retries, schema drift, proxy policy	Low: vendor maintains the integration
Engagement formula	Yours to choose and document	Yours to choose and document	Vendor's formula; usually fixed, sometimes configurable
Pricing verification	N/A	Free (open source)	The vendor's current pricing page; do not trust third-party summaries

Operational Checklist

Before you run an engagement pipeline in any non-trivial environment, walk through this checklist:

• [ ] Read Instagram's current terms of service and any jurisdiction-specific platform guidelines.
• [ ] Your post list is narrow enough to return useful results without hammering the platform.
• [ ] You set request pacing (delays, jitter) to avoid rate limits or CAPTCHA walls.
• [ ] You capture captured_at, the source URL, and the raw metric block on every record.
• [ ] Your engagement formula is documented in code with the denominator spelled out.
• [ ] You have a dedupe strategy based on post_id or source_url, not caption text.
• [ ] You have a backoff and retry policy and you log the failures.
• [ ] You have a refresh cadence for the author follower cache so per-author rates stay accurate.
• [ ] You have reviewed applicable privacy and data-protection laws for the records you collect.

Limits, Maintenance, and Compliance

Instagram's HTML and JavaScript evolve, and the repository's README reflects a snapshot of that evolution. The repositories do not ship a hosted proxy, a CAPTCHA solver, or an unlimited request budget; you provide those, or you accept a smaller, slower dataset.

Public Instagram data still touches trademark, database, and consumer-protection law in many jurisdictions, and Instagram's terms restrict certain automated access. Restrict your collection to public, non-authenticated posts, respect the platform's terms, do not attempt to bypass access controls, and do not republish records in misleading ways. Engagement-rate formulas are an analyst convention, not a platform metric, so always label the formula you used when you share a chart or a number.

FAQ

Is there an official Instagram API I can use instead?
The Instagram Graph API requires a Business or Creator account and an app review for most use cases. The path in this article is the open-source, self-managed path for teams without API access. Verify current Instagram Graph API availability on the official Meta developer documentation page.

What data fields does the repository return?
Field names depend on the current version of scraper.py and the query. Treat the example record as a representative shape, not a guarantee. Read raw_posts.json and the README before integrating.

How often should the workflow run?
Daily is enough for most account-level benchmarking. Hourly refreshes during a narrow campaign window are reasonable; avoid sub-hourly refreshes without a documented reason and a sustainable request budget.

What happens when Instagram changes the page layout?
Layout changes break CSS selectors and JSON keys. Keep a small set of "smoke test" posts that fail loudly if a core field disappears; treat a missing metric as a hard error.

How do I extend this to Twitter and TikTok?
Clone data-scrape/twitter-scraper and data-scrape/tiktok-video-scraper next to the Instagram project. Map each platform's metric names into the same metrics block and reuse derived to score them on the same formula.

Next Steps

Clone the data-scrape/instagram-post-scraper repository, follow the README, and start with the normalization script. Layer data-scrape/twitter-scraper and data-scrape/tiktok-video-scraper into the same JSONL loader when you want Instagram, Twitter, and TikTok scored on one formula. The rest of the data-scrape profile holds adjacent open-source repositories you can adopt as your pipeline grows.

Related Articles

• YouTube Channel Scraper: How to Build Creator Analytics with Python
• X Scraper GitHub: How to Extract Public Posts and Profiles with Python
• TikTok Scraper GitHub: How to Build a Public Video and Creator Pipeline with Python