A client came to me in June with a request that sounded like a Tuesday afternoon and turned into a small research project. They'd been posting to Instagram for three years, roughly 380 posts, and nobody could say which of them earned their keep. "Which posts actually worked, and is there a pattern?" That's it. That's the whole brief.
The trouble is that the honest answer lives in data the platform doesn't hand you nicely. Instagram's own export covers a rolling window that stops well short of three years. To compare a post from last spring against one from two summers ago, I needed captions, hashtags, timestamps, and engagement counts sitting together in one table. So, like every developer who has ever been handed this problem, I opened a terminal and started to write a scraper. Then I remembered the last three times I did that.
Let me save you the evening I already spent.
The dead path: static requests
The first thing everyone tries is the cheapest thing, a plain HTTP GET and a parser:
import requests
from bs4 import BeautifulSoup
resp = requests.get(
"https://www.instagram.com/some_brand/",
headers={"User-Agent": "Mozilla/5.0"},
)
soup = BeautifulSoup(resp.text, "html.parser")
posts = soup.select("article a") # optimism
print(len(posts)) # prints 0
This returns an empty shell. The page you get back is a scaffold that JavaScript fills in later, and the tidy JSON blob that older tutorials tell you to fish out of a <script> tag has been moved, renamed, or gated behind a request signature that changes. Static scraping of Instagram has been effectively dead for years. If a Stack Overflow answer suggests ?__a=1, check the date on it, then close the tab.
The fragile path: headless browser
Next rung up is driving a real browser. Playwright is genuinely good at this, and for a lot of sites I'd stop here:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://www.instagram.com/some_brand/")
# Now you inherit a queue of problems:
# - the login/consent wall that appears for logged-out sessions
# - the cookie banner that steals your first click
# - infinite scroll, so you loop scroll + wait + collect
# - selectors like div._aagv that mean nothing and change monthly
for _ in range(30):
page.mouse.wheel(0, 4000)
page.wait_for_timeout(1500)
cards = page.query_selector_all("article img")
# ...and half of these are avatars, not posts
Every one of those comments is a real bug I have chased at an hour I'd rather not admit. The scroll-and-collect loop is fiddly but fine. The login wall is where it gets grim: to see a full profile reliably you end up feeding it a session, which means storing credentials, rotating them when challenges fire, and quietly hoping the account doesn't get flagged for behaving like a bot. Add a datacenter IP to that mix and you've built a machine whose main job is looking suspicious.
And the selectors. div._aagv is not an API. It's an obfuscated class name that Instagram's build tooling regenerates, and when it flips, your scraper doesn't error loudly — it collects zeros and keeps a straight face. You find out days later when the numbers look wrong.
Here's the thing that finally landed for me: you don't write an Instagram scraper. You adopt one, and then you feed it forever. For a product where scraping is the product, fine, that's the job. For a one-off content audit billed as a fixed fee, maintaining bespoke browser automation is just lighting money on fire.
The economics, stated plainly
The client needed this data once. Maybe again next quarter if the audit proved useful. Not hourly, not daily. So the calculus wasn't "which scraper is most powerful," it was "how do I get a clean CSV without signing up for a maintenance contract with myself."
I evaluated a handful of hosted options against boring criteria: does it export the fields I need, does it handle public profile data without me managing proxies, and does it treat platform terms as real. The one I shipped with was the AllyHub Instagram Scraper, and the deciding factor was architectural rather than featural. It runs as a browser extension inside my own logged-in session. That's a meaningfully different design from the datacenter-IP tools: it reads the same pages I can already see, behaves like a person actually browsing, and there are no proxies or session tokens for me to babysit.
There was a second thing I only appreciated on the third run. The initial collection is slower because it's learning the page structure — walking the grid, figuring out where the counts live. After that, the setup is saved as a reusable one-click job, so re-running it skips the exploration and just goes. When a profile layout shifted mid-project (Instagram redesigns something roughly whenever I get comfortable), the re-run recovered on its own, because it reads the live page rather than trusting a selector I wrote down last month. My workflow didn't start from scratch again, which is exactly the part I was dreading.
I pointed it at the account, let it collect, and exported captions, hashtags, post dates, and engagement counts to CSV. Total setup time was a coffee. Then I got to do the work I was actually hired for.
The part I wanted to spend time on
Everything below is pandas, which is where a data question deserves to live:
import pandas as pd
df = pd.read_csv("brand_posts.csv", parse_dates=["posted_at"])
# normalize engagement so likes and comments are comparable-ish
df["eng"] = df["likes"] + df["comments"] * 3
# format = carousel / image / video, inferred at export
by_format = (
df.groupby("format")["eng"]
.agg(["median", "count"])
.sort_values("median", ascending=False)
)
print(by_format)
For this account, carousels ran away with it — the median carousel outperformed the median single image by a wide margin, and it wasn't close. Video sat in the middle. That alone reframed their whole 2025 plan.
Then the timing question, the one the client was sure about:
df["dow"] = df["posted_at"].dt.day_name()
df["hour"] = df["posted_at"].dt.hour
print(df.groupby("hour")["eng"].median().round(1))
They believed posting time was everything. The medians by hour were nearly flat. Whatever advantage a "best time to post" gave them was drowned out by format and topic. Uncomfortable, useful finding.
The hashtags were the fun part. I exploded the tag column and joined it back to engagement:
tags = df.assign(tag=df["hashtags"].str.split()).explode("tag")
tag_perf = (
tags.groupby("tag")["eng"]
.agg(["median", "count"])
.query("count >= 10")
.sort_values("median")
)
print(tag_perf.head(8)) # their worst-associated tags
Two hashtags the client used on nearly every post correlated with their lowest performers. Not causation — probably those tags rode along on low-effort posts — but it was enough to start a genuinely interesting conversation about what they were signaling to the algorithm versus to humans.
The unglamorous but non-optional part
Some ground rules, because a technical post that skips them is doing you a disservice.
Collect public data only. This was a business account's own public posts, aggregated. Respect rate limits even when a tool would happily let you steamroll them — polite volume keeps you out of trouble and keeps the platform usable for everyone. Read the platform terms and your local regulations before you scrape anything, because "it was technically reachable" is not a legal theory. And if your dataset touches individual people, ask whether you'd be comfortable explaining it to their faces. Aggregate analysis of a brand's public content is a normal thing to do. Assembling profiles of private individuals is not, and nothing here is a recipe for that.
Start-to-delivery on the whole engagement was about half a day, and nearly all of it was spent in the notebook arguing with the data — which is the only part worth billing for. The scraper stayed exactly where I wanted it: not my problem, not my codebase, not my 11pm.
Top comments (0)