Instagram Post Scraper GitHub: How to Extract Public Content Data with Python

The fastest way to turn public Instagram post pages into a structured research dataset is to combine an open-source scraper repository with a Python normalization pipeline that you control. This article is for social media analysts, content researchers, growth marketers, and data engineers who want to collect publicly visible post metadata — captions, timestamps, like and comment counts, media URLs — normalize it into a stable schema, and store the results for downstream analysis without relying on a paid third-party API.

You do not need a commercial scraping subscription to prototype this workflow. The data-scrape/instagram-post-scraper repository provides a runnable reference implementation, and the broader data-scrape profile hosts complementary social media scraper repositories. Your job is to wrap the scraper in a pipeline that deduplicates records, enforces a consistent schema, and respects Instagram's platform terms.

TL;DR

• Use the open-source data-scrape/instagram-post-scraper repository to collect public Instagram post metadata as JSON.
• Normalize raw fields into a consistent schema: post URL, post ID, username, caption, timestamp, like count, comment count, media type, and media URL.
• Store normalized records as JSONL so downstream tools can stream them without parsing a large array.
• Schedule the workflow with cron, Airflow, or a simple queue; refresh cadence depends on how frequently your target accounts publish.
• Always verify Instagram's terms of service, rate limits, and robots directives before running at scale.
• For cross-platform social media scraping, the data-scrape/tiktok-video-scraper and data-scrape/twitter-scraper repositories follow a similar pattern.
• For a Chinese-language reference on lawful public web data collection, see this public-web-data compliance guide.

Why Instagram Public Post Data Is Difficult to Collect

Instagram serves most content through a single-page application that loads data via internal GraphQL endpoints. A plain requests.get() against a post URL typically returns a skeleton HTML page with minimal metadata embedded in <meta> tags or JSON-LD blocks, not the full structured payload that a logged-in browser sees. The common pitfalls are:

• JavaScript-rendered content: Captions, like counts, and comment counts load dynamically after the initial page request.
• Rate limiting and IP blocking: Instagram aggressively throttles repeated requests from the same IP without authenticated sessions.
• Layout variability: Post pages differ between Reels, carousels, and single-image posts, requiring flexible parsing.
• Authentication walls: Some data is only visible to logged-in users; only genuinely public content should be collected.
• Carousel and Reel complexity: Multi-item posts and short-form video formats require different parsing paths than single-image posts, and the metadata structure can vary across formats.
• Hashtag and location data: Discovery by hashtag or location tag may return a mix of top posts and most recent posts, with different pagination behavior for each.

A repository-led workflow helps because it already handles these edge cases with tested parsing logic, retry strategies, and structured output formatting. Instead of building a parser from scratch and discovering each failure mode yourself, you start with a reference implementation that has already been tested against real post pages and can be adapted as the platform evolves.

What the Verified Repository Provides

The data-scrape/instagram-post-scraper repository is an open-source Python project that demonstrates how to collect publicly visible Instagram post data. It is not a managed service and does not promise unlimited coverage, but it gives you a working foundation for fetching and structuring public post pages.

Verified capabilities from the repository:

• Post metadata extraction: Captures post URL, post ID, username, caption, timestamp, like count, comment count, and media URLs from public post pages.
• Media type detection: Distinguishes between single-image, carousel, and video posts.
• Structured JSON output: Returns consistent field names so downstream normalization is straightforward.
• Retry and timeout handling: Configurable retry attempts and request timeouts to manage transient failures.
• Python classes and CLI: Supports both scripted and batch usage patterns.

The extracted fields typically include:

post_url | post_id | username | caption | timestamp | like_count | comment_count | media_type | media_urls | scrape_time

Step-by-Step Setup

1. Clone the repository

git clone https://github.com/data-scrape/instagram-post-scraper.git
cd instagram-post-scraper
pip install -r requirements.txt

2. Configure environment variables

Create a .env file or export variables in your shell:

# Optional: proxy URL for rotating requests (do not hardcode credentials)
export INSTAGRAM_PROXY_URL="http://your-proxy:8080"

# Retry configuration
export SCRAPE_MAX_RETRIES=3
export SCRAPE_TIMEOUT=30

# Output directory
export OUTPUT_DIR="./data"

3. Run the scraper against a list of public post URLs

python -m instagram_post_scraper --input post_urls.txt --output ./data/results.json

Runnable Python Normalization Pipeline

Once you have raw JSON from the scraper, the next step is normalization. The following script reads raw results, deduplicates by post ID, enforces a consistent schema, and writes JSONL output:

import json
import os
import pathlib
import sys
from datetime import datetime, timezone

# Configuration via environment variables
RAW_INPUT = pathlib.Path(os.environ.get("RAW_INPUT", "./data/results.json"))
OUTPUT_DIR = pathlib.Path(os.environ.get("OUTPUT_DIR", "./data"))
OUTPUT_FILE = OUTPUT_DIR / "normalized_posts.jsonl"

# Target schema fields
SCHEMA_FIELDS = [
    "post_url", "post_id", "username", "caption",
    "timestamp", "like_count", "comment_count",
    "media_type", "media_urls", "scrape_time",
]

def normalize_record(raw: dict) -> dict:
    """Normalize a raw scraper record into the target schema."""
    return {
        "post_url": raw.get("post_url", ""),
        "post_id": str(raw.get("post_id", "")),
        "username": raw.get("username", ""),
        "caption": raw.get("caption", ""),
        "timestamp": raw.get("timestamp", ""),
        "like_count": int(raw.get("like_count", 0) or 0),
        "comment_count": int(raw.get("comment_count", 0) or 0),
        "media_type": raw.get("media_type", "unknown"),
        "media_urls": raw.get("media_urls", []),
        "scrape_time": raw.get("scrape_time") or datetime.now(timezone.utc).isoformat(),
    }

def main() -> None:
    if not RAW_INPUT.is_file():
        print(f"Input file not found: {RAW_INPUT}")
        sys.exit(1)

    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
    seen_ids = set()
    written = 0

    with RAW_INPUT.open("r", encoding="utf-8") as f:
        records = json.load(f)

    with OUTPUT_FILE.open("w", encoding="utf-8") as out:
        for raw in records:
            record = normalize_record(raw)
            post_id = record["post_id"]
            if not post_id or post_id in seen_ids:
                continue
            seen_ids.add(post_id)
            out.write(json.dumps(record, ensure_ascii=False) + "\n")
            written += 1

    print(f"Normalized {written} unique posts → {OUTPUT_FILE}")

if __name__ == "__main__":
    main()

Representative Output

After running the normalization pipeline, your JSONL file will contain one record per line:

{
  "post_url": "https://www.instagram.com/p/CxABCD1234/",
  "post_id": "CxABCD1234",
  "username": "public_account_example",
  "caption": "Example caption text from a public post",
  "timestamp": "2026-08-20T14:30:00+00:00",
  "like_count": 1842,
  "comment_count": 73,
  "media_type": "carousel",
  "media_urls": ["https://scontent.example.com/image1.jpg", "https://scontent.example.com/image2.jpg"],
  "scrape_time": "2026-08-25T02:10:00+00:00"
}

This format is directly consumable by pandas, BigQuery loaders, or any JSONL-aware pipeline.

Comparison: Repository Workflow vs Managed API vs Manual Export

Dimension	Open-source repository	Managed scraping API	Manual browser export
Best for	Developers who want full control and flat costs	Teams that need scale without infrastructure	One-off lookups, quick checks
Setup model	Clone, install, configure, run	Sign up, get API key, call endpoint	Open browser, navigate, copy
Data coverage	Public posts only; depends on parsing logic	Varies by provider; may include more fields	Only what is visible on screen
Output format	JSON / JSONL, fully customizable	Provider-defined JSON schema	Manual, unstructured
Maintenance burden	You fix parsers when Instagram changes markup	Provider handles maintenance	No maintenance, no automation
Cost model	Free + your infrastructure	Per-request or subscription pricing	Free but labor-intensive

Use Cases

• Creator research: Track public post frequency, engagement trends, and content format distribution across accounts in a niche.
• Brand monitoring: Detect when a brand's public posts mention specific keywords or hashtags, then normalize the data for a dashboard.
• Content audit: Archive your own public posts with engagement metrics for quarterly reporting.
• Cross-platform benchmarking: Compare Instagram engagement metrics with TikTok data from the data-scrape/tiktok-video-scraper repository or public post data from the data-scrape/twitter-scraper repository.

Limitations and Maintenance

• Markup changes break parsers. Instagram periodically updates its page structure. When a scraper stops returning expected fields, inspect the raw HTML and update the parser selectors.
• Rate limits apply. Even with proxies, aggressive request volumes can trigger temporary blocks. Use delays, retries, and rotating user agents.
• Public data only. Private posts, stories, and direct messages are not accessible. Never attempt to bypass login walls or access controls.
• Freshness decay. Like and comment counts change over time. A single scrape is a snapshot; for trend analysis, schedule recurring runs.
• No guaranteed completeness. An open-source scraper may not capture every field available in the browser. Verify coverage against your requirements before production use.

Compliance Notes

Collecting public web data comes with responsibilities:

• Respect Instagram's Terms of Service and robots directives. Do not attempt to access private content, bypass authentication, or evade CAPTCHA challenges.
• Do not collect personal data that is not publicly visible without consent.
• Be transparent about your data practices if you publish or share derived datasets.
• Consult legal counsel for your jurisdiction before commercial use of scraped data.
• For a broader reference on lawful public web data collection practices, see this public-web-data compliance guide.

FAQ

Is there an official Instagram API?

Yes. The Instagram Graph API provides access to business and creator account data. It requires a linked Facebook Page and an approved app. For public post metadata at scale without authentication constraints, an open-source repository workflow may be more flexible for research purposes.

What data fields does the repository return?

The core fields include post URL, post ID, username, caption, timestamp, like count, comment count, media type, and media URLs. Verify the current field list in the repository README before building downstream logic.

How often should I run the scraper?

That depends on your use case. For creator research, a daily or weekly run may be sufficient. For brand monitoring during a campaign, you might run it more frequently. Always respect rate limits and add delays between requests.

What happens when Instagram changes its page structure?

The scraper's parsing logic will need to be updated. This is the primary maintenance burden of any open-source scraper. Monitor the repository for updates and test your pipeline against known URLs after changes.

Can this connect to Python, a queue, or a CRM?

Yes. The JSONL output is directly consumable by Python scripts, pandas, BigQuery, or any downstream system. You can pipe it into a queue worker, write it to a database, or transform it for CRM ingestion.

What should I verify before production use?

Verify the current field list, test against known post URLs, confirm rate limits and proxy requirements, check that your normalization schema matches downstream needs, and review Instagram's current terms of service.

Can I scrape private profiles or direct messages?

No. This workflow is for public web data only. Attempting to access private content, bypass authentication, or intercept direct messages violates platform terms and potentially applicable law.

Next Steps

Start by cloning the data-scrape/instagram-post-scraper repository and running it against a small set of public post URLs. Review the raw JSON output, adapt the normalization script to your schema, and schedule a recurring run. For cross-platform social media research, explore the full data-scrape profile, which includes repositories for TikTok video scraping, Twitter/X public posts, and other social platforms.