DEV Community: Automate Better

Three Lines to Draw Before You Scrape Instagram

Automate Better — Wed, 19 Aug 2026 03:55:28 +0000

Most write-ups on this subject are about technique. This one is about the three decisions you should make before you write any code, because in my experience every project that went badly went badly for a reason that was decided on day one and not noticed until much later.

I have built this kind of collection twice, for competitive analysis and for a partner-vetting workflow. Neither of them needed to touch anything behind a login, and I want to explain why that turned out to be the useful constraint rather than the limiting one.

Line one: the login wall is a boundary
A login wall is a statement about who the content is for. Treating it as an engineering obstacle to be routed around is the decision that puts a project on the wrong side of everything: terms of service, the platform's own detection, and in several jurisdictions the law.

So the first line is simply: if it requires an account to see, it is out of scope. Not "hard," not "for later." Out of scope.

I am not going to discuss techniques for getting past one, and I would be sceptical of any article that does. The interesting engineering question here is not how to see more. It is how much you can actually do with what is openly published, and the honest answer is: considerably more than people assume before they check.

This constraint also has a practical benefit that is easy to miss. A pipeline built only on openly available data does not break when authentication changes, does not require credential management, and does not put an account at risk. Mine has survived two platform changes that took down colleagues' authenticated collectors.

Line two: public does not mean unrestricted
The second line is the one developers get wrong most often, and it has nothing to do with access.

Data being publicly visible says nothing about whether you may store it, for how long, or what you may do with it. In the EU and UK, information about an identifiable person is personal data whether or not they published it themselves, and a public profile is very obviously about an identifiable person.

That does not make collection impossible. It makes it something you have to have a reason for, and the reason has to be written down before rather than reconstructed afterwards.

Three practical consequences I build in from the start.

Have a stated purpose, in a document, before collecting. "Competitive benchmarking of brand accounts in our category" is a purpose. "We might want it later" is not, and it is the one that will not survive any subsequent conversation with a lawyer.

Set a retention period and enforce it in code. Mine is 90 days for post-level records and indefinite for aggregate statistics, because aggregates about a category stop being about a person once they are aggregated. Enforce it with a scheduled deletion job, not a calendar reminder.

Distinguish business accounts from personal ones. An account operating openly as a business, with a category and a contact method it has chosen to publish, is a different proposition from a private individual's account that happens not to be locked. I collect the first and exclude the second, and that single filter removes most of the genuinely uncomfortable cases.

Line three: rate limits are a contract
The third line is about behaviour rather than scope.

A rate limit is the platform telling you what load it is prepared to serve. Backing off when you hit one is the correct response. Engineering around one is a decision to take more than you were offered, and it is also, practically, how a collection project ends.

My rules are unglamorous. Sequential rather than parallel. Delays that are generous rather than minimal. A hard daily cap on total requests set well below anything I have ever seen enforced. Respect for anything the site's own robots directives express.

Slow collection is fine. Almost nobody actually needs this data in real time. The competitive analysis I run is a weekly job, and the difference between finishing in ten minutes and finishing in four hours is invisible to everyone including me.

The one thing I would push back on is the idea that this is only about avoiding a block. Load you generate is load someone else pays for. That framing has been more useful to me for making good decisions than any risk calculation.

What is actually available inside those lines
Data Openly available How I treat it
Handle, display name, bio Yes, on public accounts Collect; the bio is free-text and may contain personal detail, so it is not exempt from line two
Follower, following, post counts Yes Collect; pure numbers, lowest sensitivity
Category and published contact on a business profile Yes, when the account chooses to show it Collect for business accounts only; this is the clearest legitimate-interest case
Post caption, timestamp, media type Yes, on public accounts Collect; captions are free text and get the same treatment as bios
Public like and comment counts Yes Collect; this is what engagement rate is computed from
Hashtags, mentions, tagged accounts in a post Yes Collect the tags; mentions identify third parties who did not post, so I aggregate rather than store
Hashtag feeds and post counts Yes Collect; the least personal data on this list and often the most useful
Individual commenter identities Sometimes visible Do not store. High sensitivity, low analytical value, easy to skip
Stories, DMs, follower lists, private accounts No Out of scope, permanently
Impressions, reach, saves, audience demographics No Not observable from outside. Anyone offering these for accounts they do not own is describing data they cannot have
The bottom row deserves emphasis because it comes up in every requirements conversation I have had. Someone will ask for a competitor's reach or their audience breakdown. Those numbers exist only in that account's own analytics, and no external collection produces them. The correct answer is that the question cannot be answered rather than that it is difficult.

What the open data is actually good for
The obvious use is engagement rate, computed from public likes and comments against follower count on recent posts. It is a crude measure and it is still the single most useful thing available from outside, because a follower count on its own tells you nothing about whether anyone is there.

Anyone can compute it by hand for one account. The reason to build or buy anything is the set: a category is a benchmark and one account is a data point. Tools built for this operate the same way, and the ones worth using are explicit that they scrape Instagram posts without login, covering public profiles, posts and hashtag feeds and nothing beyond them. AllyHub computes the engagement figure from the public likes and comments rather than implying access to anything private, which is the disclosure I would look for in any option here.

The less obvious use, and the one I have got more from, is hashtag ecosystem mapping. Which tags co-occur, which accounts sit at the intersections, how a topic's vocabulary shifts across quarters. It involves almost no personal data, it is genuinely hard to get any other way, and it answers strategy questions rather than vanity ones.

The test I apply to every field
Before a field goes into the schema, three questions.

Can I see it without an account? Would I be comfortable if the account owner read my purpose statement? Is there a use for it already written in that statement?

Two yeses and a no is a no. The third question kills the most fields and kills them for the right reason. Anything collected because it happened to be there is precisely the data that makes a dataset harder to defend and that nothing downstream ever reads.

I dropped four fields from my first schema on this test and have not once wanted them back. The instinct to collect broadly and decide later is the same instinct that produces the retention problem in line two, arriving a step earlier.

Things I have declined to build
A request to identify individual commenters and enrich them against an email list. Technically feasible from visible data, and I turned it down. Building an identity graph of individuals from social activity is a different category of thing from benchmarking brand accounts, regardless of every item in it being visible.

A request to monitor a specific person's account activity over time. Same data types as a competitive brand monitor, entirely different in what it is. If replacing the account name with a person's name in the project description makes it sound like surveillance, it is surveillance.

A request to reconstruct historical data from before collection started by inferring it. The honest answer is that you cannot, and an inferred number in a dataset of measured ones is worse than a gap because nothing downstream will remember it was invented.

Practical setup
Write the purpose statement first, in the repository, next to the code. Two paragraphs. It will feel like ceremony and it is the artifact that makes every later decision easy.

Log what you collected and when, at the field level. When someone asks whether you hold data on X, you want that to be a query and not an investigation.

Separate collection from analysis with a storage boundary. Collection writes raw records with a timestamp and a retention flag. Analysis reads aggregates. That separation is what makes a 90-day deletion policy implementable rather than aspirational.

And build the exclusion list before the include list. Mine names account types I do not collect, and it has been more useful for keeping the project on the right side of things than any positive specification.

Put a calendar entry on re-reading the platform's terms twice a year. They change, the changes are not announced to you, and "it was allowed when we built it" is not a position anyone wants to be arguing from. Twice in three years a re-read has caused me to narrow something, both times cheaply because the collection was narrow to begin with.

Where this leaves you
The three lines rule out a lot: anything authenticated, anything about private individuals, and anything that treats a rate limit as a challenge.

What is left is still a substantial and genuinely useful dataset, and it has the property of being defensible, which the alternative does not. I have never wanted the excluded data badly enough to reconsider, and the two projects that survived contact with a compliance review survived because these decisions were made on day one rather than argued about in month four.

I Treat My Video Descriptions Like a Build Artifact Now. Here's the Pipeline.

Automate Better — Mon, 10 Aug 2026 17:00:03 +0000

I have a CI job that lints my commit messages. I have a pre-commit hook that yells at me about trailing whitespace. And until about four months ago, I wrote the description for every conference talk recording and tutorial video by opening an empty text box at 11pm and typing until it looked long enough.

That asymmetry started to bother me. Not because descriptions are precious writing. Because they're the opposite: a structured artifact with a fixed schema, assembled from inputs I already have, that I was producing by hand with no validation step and no way to reproduce.

So I did the obvious thing and treated it like one. Input contract, generation step, validation step, reuse. This post is what that pipeline looks like after four months and about thirty videos, including the parts that don't work.

Why a description is a build artifact and a blog post isn't

The distinction matters, because I wasted a month trying to systematise the wrong things.

A blog post is not a build artifact. The argument is the value, the structure follows the argument, and no two posts have the same shape. Automating that produces slop, and I stopped trying.

A video description is different. Mine have had exactly the same schema for two years: a first line or two carrying the thing people would actually search for, three or four lines of summary, chapter timestamps, links to the repo and the slides, a call to action, hashtags. The words change every time. The schema never changes at all. That is a template with variable interpolation, and I was doing the interpolation in my head, at midnight, badly.

Once I framed it that way, the design followed.

Step one: the input contract

The single biggest improvement had nothing to do with generation. It was writing down what the inputs are, so I stop trying to remember them at the end of an edit session when I'm tired.

I keep a small file next to the project for each video:

# video-meta.yml
topic: >
  Walking through how we cut cold-start time on a Lambda-backed
  GraphQL API from 2.4s to under 400ms, including the two things
  that didn't work.
target_keyword: lambda cold start optimization
audience: backend devs who already ship serverless, not beginners
model_channels:
  - the three channels currently ranking for this keyword
links:
  repo: https://github.com/...
  slides: https://...
chapters:
  - "00:00 The problem"
  - "02:15 Provisioned concurrency, and why we dropped it"
  - "07:40 Bundle size, the boring win"
  - "14:05 Results and what we'd do next"

Nothing clever here. But writing the contract down did two things. It made the generation step deterministic enough to be worth automating, and it moved the thinking to Tuesday afternoon when I'm planning the video, instead of Thursday midnight when I'm exhausted and will accept anything.

The target_keyword field is the one people skip. Skip it and you get a description that summarises your video accurately and matches nothing anyone types into a search box.

Step two: generation

For the generation step itself I use the AllyHub YouTube Description Generator, and the reason I use that one rather than a general chat model comes down to what it does before it writes.

A general model writes from priors. Ask it for a YouTube description about Lambda cold starts and you get the average of every description in its training data, which is fine and completely interchangeable. This one reads the descriptions of the videos already ranking for the keyword you gave it, then mirrors how those are structured and where they place terms. You're starting from what the niche actually looks like rather than a blank box.

I want to be precise about what that does and doesn't tell you, because I've seen this oversold. It can see what those descriptions say, because they're public text on a public page. It cannot see how those videos perform beyond what's on screen, and nothing here is a claim about why they rank. It's structural evidence, not a causal story. Treat it as "this is the shape of the neighbourhood" and you'll calibrate correctly.

The output comes back as one formatted block: keyword up front, body, chapter markers, links, CTA, hashtags. On a good run I change one or two lines because they don't sound like me. On a bad run I change four.

Step three: validation, which is the part everyone skips

This is where a pipeline earns its keep, and it's the part I see missing from every "just use AI for your descriptions" post.

Generation is cheap and mostly fine. The failure modes are boring and specific, and they're catchable. Here's what I check, in order, before anything gets pasted:

Check	What it catches	Automatable?
Keyword appears in the first 150 characters	The most common failure. The model writes a lovely opening sentence and buries the search term in paragraph three	Yes, string match
Every chapter timestamp exists in the actual video	Hallucinated or shifted timestamps after a re-edit. This one has bitten me twice	Yes, diff against the edit list
All links resolve and point where you meant	Repo link to the wrong branch, slides link still pointing at a draft	Yes, HEAD request
No claim in the description that the video doesn't make	The single most damaging failure, and the only one that hurts your credibility rather than your reach	No. Human read, every time
Hashtags are relevant and under the platform cap	Tag stuffing, or tags copied from a different video	Partly
It sounds like a person, specifically you	Generic phrasing that reads like every other description in the niche	No

The two rows marked "No" are why I don't call this automated. I call it assisted, and the distinction is load-bearing. The pipeline gets me from an empty box to a draft that's structurally complete. I still read it, and I'm still the one who hits publish, because the failure that actually costs something is a description promising a benchmark the video never shows.

Rows one through three I've since scripted, because they're pure string and network checks and I was doing them by eye at midnight, which is exactly the wrong time to do anything by eye.

The timestamp one deserves a story, because it's the failure I'd least expected and the one I now care most about. I re-cut a talk after publishing, trimming about forty seconds of a false start near the beginning. The video updated. The description didn't. So every chapter marker after the first was off by roughly forty seconds for two weeks, which meant every person who clicked "Bundle size, the boring win" landed mid-sentence in the previous section. Nobody told me. I found it myself, by accident, while linking someone to a specific chapter.

That is not a generation failure. The generator produced correct timestamps for the cut it was given. It's a pipeline failure, in exactly the way a stale build artifact is a pipeline failure, and the fix is the same: make the check cheap enough that it runs every time rather than when you remember. Mine now diffs the chapter list against the final edit's marker export and refuses to pass if the counts differ.

Step four: reuse, which is where the time actually comes back

The first video through this pipeline was slower than doing it by hand. That's normal for any pipeline and I want to say it clearly, because posts like this tend to skip the part where the setup costs you an evening.

The gain shows up on repetition. The channel's voice, my link block, my chapter formatting, the fact that I want the keyword front-loaded and the CTA short, all of that gets saved as a reusable setup rather than re-explained per video. AllyHub calls the saved version a Playbook; the name doesn't matter, the property does. Run two starts from run one's configuration instead of from zero, and the corrections I made on run one are part of that configuration rather than sitting in a chat log I'd have to go find.

This is the bit I'd push back on if someone described the whole category as "AI writes your descriptions." Plenty of things will write you a description. The question I'd ask of any of them is what survives the run: if next week's video starts from the same empty box as this week's, you've bought an output, not a pipeline, and the thirty minutes comes back exactly once.

The second effect I didn't expect: batching stopped being painful. I record talks in clusters, so I'll have four videos edited and none published because writing four descriptions in a row is miserable. Handing the whole queue over in one pass, each one drafted against its own topic and keyword rather than one template smeared across all four, took the backlog from a Thursday evening to about twenty minutes of reading and editing.

That's the actual win, and it's not a dramatic one. Roughly thirty minutes per video down to under ten, on something I do weekly.

What I got wrong

I tried to validate quality. I spent a while building checks for whether the description was "good," which is not a thing a string match can tell you, and I eventually deleted all of it. The pipeline checks structure and facts. Quality is a human read, and it's a fast one once structure is guaranteed.

I also over-fit to my first keyword. The structural evidence for lambda cold start optimization is not the structural evidence for graphql schema design, and for a while I was carrying over conventions from one niche into another because they'd worked once. Regenerate per keyword. It costs nothing.

And I under-invested in the input contract for about six videos, filling it in hastily and then being annoyed at the output. Garbage in, structurally-valid garbage out. The contract is the leverage point, not the generator.

If you want to try this

Take your last five video descriptions and diff them. If they share a schema, you have a pipeline problem, not a writing problem, and everything above applies. If they don't share a schema, you probably don't need any of this.

Then write the input contract before you write anything else. Topic, keyword, audience, links, chapters. Five fields, filled in while you still remember what the video is about.

Generate, validate, read it yourself, publish.

One thing worth saying plainly at the end: none of this is a ranking strategy, and I'd be lying if I presented it as one. Whether a video gets found comes down to the video and who else made one about the same thing. What a pipeline fixes is the part where a finished video sits unpublished for three days because writing the box underneath it felt like more work than editing the thing did.

For me that was the real cost. It turned out to be a build problem all along.