DEV Community: Angel Junior

The Real Cost of Building Your Own Scraper Isn't the Scraper

Angel Junior — Thu, 06 Aug 2026 15:14:11 +0000

There's a specific kind of engineering optimism I recognize instantly now, because I've had it myself: "It's just an HTTP request."

You need follower counts for a client dashboard. Or post metrics for a creator analytics tool. Or video data to train something. The platform serves that data to a browser, publicly, to anyone who visits. So it's just an HTTP request, and you'll have it working before lunch.

And you will. That's the trap. The first version genuinely does work before lunch. What nobody prices in is the second year of that decision.

The four costs that don't appear in the estimate

When engineers estimate "build a scraper," they estimate the extraction logic. That's the one part that's actually easy. Here's what sits underneath it.

The maintenance tax is permanent and unpredictable. Scraping infrastructure isn't a project that ships — it's a subscription you pay in engineering hours, and the bill arrives on the platform's schedule, not yours. A markup change on a Saturday is your Saturday. There's no version pinning, no deprecation notice, no changelog. Worse, the maintenance is bursty: months of quiet, then a week where nothing works and everything is on fire, always coinciding with something else you'd committed to.

The infrastructure isn't the fun kind. Proxy pools, session management, request pacing, retry orchestration, geographic distribution. None of this is your product. None of it differentiates you. All of it needs monitoring, and all of it degrades silently — which is the expensive kind of degradation, because you find out from a customer rather than from an alert.

Nobody owns it. This is the one that actually kills teams. Scraping infrastructure has no natural owner. It's not the backend team's roadmap, it's not the data team's roadmap, it's a thing someone built once. So it rots between sprints until it breaks loudly, then gets a panicked patch from whoever is free, and the patch adds to the rot. Two years in, you have a critical dependency that three people are afraid of.

The opportunity cost is invisible. Every hour on proxy rotation is an hour not spent on the thing your users actually pay for. That trade never shows up in a ticket, so it never gets debated — it just quietly happens for a couple of years.

The build-versus-buy question isn't "can we build this." You can. It's "do we want to own this forever, and is owning it worth anything to our customers." For most teams the answer to the second half is no. Nobody has ever chosen a product because its vendor maintains their own proxy pool.

What "developer-first" means when it isn't a slogan

Every API in this category says "built for developers." Here's the concrete version — the specific decisions, and what each one is a reaction to.

You get a key in under a minute, and nobody has to approve you. No sales call, no demo request, no sandbox tier that behaves differently from production, no waiting for a human to unlock your account. Sign up, key appears, you're live. This exists because the standard experience in enterprise data is a two-week procurement dance before you can determine whether the thing even returns the field you need. You should be able to answer that in five minutes, for free, before anyone talks about money.

Authentication is one header. Pass a key, get JSON. No OAuth handshake, no SDK to install, no client library that lags three versions behind the API. If your language can make an HTTP request — curl, Python, Node, Go, PHP, a Bash script, a spreadsheet macro — you're already compatible. SDKs are a convenience that becomes a liability the moment they're under-maintained; a plain HTTP contract never is.

Pricing is one request, one credit. Flat. No field-level multipliers, no tiered math where a "heavy" endpoint silently costs six times more, no bill that requires a spreadsheet to predict. Credits don't expire and there's no subscription, so a month where you don't use it costs nothing. This is a direct reaction to the standard model in this space, where the pricing page is optimized to make cost comparison hard.

There are no artificial concurrency caps. Run one request or a thousand in parallel. Your credit balance is the only limit — no per-minute throttle bolted on to push you toward a higher tier. Throttles in this category are usually a business decision dressed as a technical constraint, and you can tell because the limit is always exactly one tier below where you are.

Responses are flat, typed, and boringly consistent. Predictable schemas across every endpoint, an OpenAPI schema you can generate types from, and no undocumented fields quietly changing shape in production. Boring is the highest compliment you can pay a response body. And nothing is cached — every call is a live fetch, so engagement numbers reflect the moment you asked, not the moment someone else asked.

Support, honestly described

I'm skeptical of support claims in general, so let me describe the actual shape of it rather than promising a number I'd have to defend.

Support is handled by the people who wrote the API. Not a tier-one script, not a chatbot that suggests clearing your cache. When you report that an endpoint returns something unexpected for a particular account, that goes to someone who can read the parser and fix it — which is the difference between "we've escalated your ticket" and a fix.

Support scales with the plan: email support on Starter, priority on Growth and Pro, dedicated on Scale. But the free tier isn't a support desert either — if something is genuinely broken, it's broken for everyone, and hearing about it early is worth more to me than the tier you're on.

The most useful thing I can say about support is that the docs are meant to make it unnecessary. Full endpoint reference, real request and response examples, OpenAPI schema. If you have to open a ticket to figure out what a field means, that's a documentation bug and I'd rather fix the docs than answer the ticket twice.

Where this argument breaks down

Since I'm the one selling, here's the honest boundary.

Build it yourself if scraping is your product. If your differentiation is data acquisition — if that's the thing customers are buying — then owning the pipeline is owning your moat. Don't outsource your moat.

Build it yourself if your volume is enormous and steady. Per-request pricing is excellent at low and medium volume and stops being obviously right at very high sustained volume. Do the arithmetic. If dedicated infrastructure comes out cheaper at your scale, that's the correct answer and I'd rather you run the numbers than take my word for it.

Build it yourself if your requirements are unusual. We cover Instagram, TikTok, and YouTube, and go deep on each rather than shipping a shallow wrapper over twenty platforms. If you need a platform we don't support, or a field we don't return, a managed API is the wrong shape for you today. X, LinkedIn, Reddit, and Pinterest are in progress, but "in progress" isn't something you can build a roadmap on.

And one boundary that isn't about build-versus-buy at all: this is public data only. No login bypass, no private accounts, no data behind an authentication wall. If your use case needs any of that, we're not the vendor, and I'd think carefully about who is.

For everyone else — the majority, in my experience — the calculation is simpler than it looks. The question was never whether you're capable of building it. It's whether the fifth time you rebuild it will feel like a better use of the week than shipping the feature your users have been asking for.

There are 100 free credits and no card required, which is enough to answer that for yourself without talking to me: scrapingisnotacrime.com.

I'm curious where other people landed on this. If you built your own and don't regret it, I'd like to hear what made it worth it — that's genuinely more useful to me than agreement.

Your Instagram Scraper Breaks Every 48 Hours. The Problem Isn't the Scraper.

Angel Junior — Thu, 06 Aug 2026 15:10:20 +0000

Every engineer who has needed social media data has written the same script. curl the profile, regex the JSON blob out of the HTML, json_decode, done. It works beautifully on a Tuesday afternoon.

By Thursday it returns an empty array. You add a User-Agent header. It works again for a week. Then it returns a login wall. You add a proxy. Then the JSON key renames itself from edge_owner_to_timeline_media to something else entirely and your foreach throws on null.

I did this three times before admitting the real problem: I was treating an unreliable, mutating, rate-limited remote source as if it were a local database call. Every architectural decision downstream of that assumption was wrong.

Here's what the ingestion layer actually needs to look like.

The four failure modes you're designing for

Before any code, name the enemy. Social data ingestion fails in four distinct ways, and they need four different responses:

Transient failure — timeout, 502, connection reset. Retry with backoff. Costs you nothing but time.
Rate/block failure — 429, or a 200 with an empty payload that means "blocked." Back off much harder, or route elsewhere.
Schema drift — the response is valid JSON, status 200, but the field you need moved or vanished. Retrying does nothing. This one is the silent killer because it doesn't throw — it writes null into your database for three weeks before anyone notices.
Semantic absence — the account is private, deleted, or has zero posts. A correct, permanent, empty answer. Retrying is pure waste. Most homemade pipelines only handle #1. Collapsing #2, #3 and #4 into "an error occurred" is why they degrade so quietly.

Architecture: the pipeline in three stages

[ scheduler ]  ->  [ fetch workers ]  ->  [ normalize ]  ->  [ store ]
      |                    |                    |
   dedupe            budget guard          schema contract

Three rules that matter more than the diagram:

The fetch stage never writes to your domain tables. It writes raw payloads. Normalization is a separate, replayable step.
The scheduler owns the budget. Whether your budget is API credits, proxy bandwidth, or your own tolerance for getting blocked, one component decides how much of it gets spent — never the workers.
The normalizer owns a contract. It validates against an expected shape and fails loudly. This is your schema-drift alarm. That first rule is the one people skip. Storing the raw response costs you a few KB and buys you the ability to re-parse six months of history when you discover your engagement calculation was wrong — without re-fetching (and re-paying for) a single record.

Stage 1: The fetch worker, with a budget guard

Hyperf's coroutine model makes concurrent HTTP fetching cheap, but "cheap" is exactly how you burn a credit balance in ninety seconds. The guard is not optional.

<?php

declare(strict_types=1);

namespace App\Ingestion;

use GuzzleHttp\Client;
use GuzzleHttp\Exception\ConnectException;
use Hyperf\Coroutine\Parallel;
use Psr\Log\LoggerInterface;

final class ProfileFetcher
{
    private const MAX_ATTEMPTS = 3;

    public function __construct(
        private readonly Client $http,
        private readonly BudgetGuard $budget,
        private readonly LoggerInterface $log,
    ) {}

    /**
     * @param list<string> $usernames
     * @return array<string, FetchResult>
     */
    public function fetchMany(array $usernames, int $concurrency = 20): array
    {
        $parallel = new Parallel($concurrency);

        foreach ($usernames as $username) {
            $parallel->add(
                fn (): FetchResult => $this->fetchOne($username),
                $username
            );
        }

        return $parallel->wait();
    }

    private function fetchOne(string $username): FetchResult
    {
        if (! $this->budget->tryConsume(1)) {
            return FetchResult::deferred($username, 'budget exhausted');
        }

        $attempt = 0;

        while (++$attempt <= self::MAX_ATTEMPTS) {
            try {
                $response = $this->http->get('/v1/instagram/profile', [
                    'query'       => ['username' => $username],
                    'headers'     => ['X-API-Key' => $this->budget->key()],
                    'timeout'     => 30,
                    'http_errors' => false,
                ]);

                $status = $response->getStatusCode();
                $body   = (string) $response->getBody();

                // Semantic absence — permanent, do not retry.
                if ($status === 404) {
                    return FetchResult::absent($username);
                }

                // Rate/block — back off hard, return budget to the pool.
                if ($status === 429 || $status === 503) {
                    $this->budget->refund(1);
                    $this->sleepBackoff($attempt, aggressive: true);
                    continue;
                }

                if ($status >= 500) {
                    $this->sleepBackoff($attempt);
                    continue;
                }

                if ($status === 200) {
                    // Raw payload only. No parsing here — that is stage 2.
                    return FetchResult::ok($username, $body);
                }

                $this->log->warning('unexpected status', [
                    'username' => $username,
                    'status'   => $status,
                ]);

                return FetchResult::failed($username, "http {$status}");
            } catch (ConnectException $e) {
                $this->sleepBackoff($attempt);
            }
        }

        return FetchResult::failed($username, 'attempts exhausted');
    }

    private function sleepBackoff(int $attempt, bool $aggressive = false): void
    {
        $base   = $aggressive ? 5.0 : 1.0;
        $jitter = random_int(0, 1000) / 1000;

        // Jitter matters: without it, N coroutines that failed together
        // will retry together, and fail together again.
        \Hyperf\Coroutine\Coroutine::sleep($base * (2 ** ($attempt - 1)) + $jitter);
    }
}

Two details worth stealing regardless of your stack:

http_errors => false. Let the status code be data you branch on, not an exception you catch. A 429 and a 500 deserve different treatment, and exception-based control flow flattens that distinction.

Refunding budget on a block. If the request never returned data, it shouldn't count against your quota accounting. Sounds obvious. Almost nobody does it, and then the "why did we spend 4,000 credits yesterday" investigation takes an afternoon.

Stage 2: The normalizer as a contract

This is the stage that catches schema drift, and the whole point is that it's strict.

<?php

declare(strict_types=1);

namespace App\Ingestion;

final class ProfileNormalizer
{
    /**
     * @throws SchemaDriftException
     */
    public function normalize(string $rawJson, string $username): ProfileSnapshot
    {
        $decoded = json_decode($rawJson, true, 512, JSON_THROW_ON_ERROR);

        $data = $decoded['data'] ?? throw new SchemaDriftException(
            "missing 'data' envelope for {$username}"
        );

        return new ProfileSnapshot(
            username:   $this->requireString($data, 'username', $username),
            followers:  $this->requireInt($data, 'followers', $username),
            verified:   (bool) ($data['verified'] ?? false),
            capturedAt: new \DateTimeImmutable(),
            rawHash:    hash('xxh128', $rawJson),
        );
    }

    private function requireInt(array $data, string $key, string $ctx): int
    {
        $value = $data[$key] ?? null;

        if (! is_int($value)) {
            throw new SchemaDriftException(
                sprintf('expected int at "%s" for %s, got %s', $key, $ctx, get_debug_type($value))
            );
        }

        return $value;
    }

    private function requireString(array $data, string $key, string $ctx): string
    {
        $value = $data[$key] ?? null;

        if (! is_string($value) || $value === '') {
            throw new SchemaDriftException(
                sprintf('expected non-empty string at "%s" for %s', $key, $ctx)
            );
        }

        return $value;
    }
}

The instinct is to write $data['followers'] ?? 0. Resist it. A null-coalesce here converts a structural break into a plausible-looking zero, and plausible-looking zeros are how you end up presenting a dashboard where an account "lost" 600 million followers overnight.

Throw. Alert on the exception. Keep the raw payload so you can replay once you've fixed the mapping.

The rawHash field earns its keep too: it lets you skip re-normalizing identical payloads, and it gives you a cheap change-detection signal for polling.

Stage 3: Idempotent storage

The one thing that will happen, guaranteed, is that some job runs twice. A worker dies mid-batch, a deploy restarts the queue, someone re-runs a backfill. Design for it now rather than deduplicating rows later.

// Migration: the uniqueness constraint is the real defense.
Schema::create('profile_snapshots', function (Blueprint $table) {
    $table->id();
    $table->string('platform', 32);
    $table->string('username');
    $table->unsignedBigInteger('followers');
    $table->boolean('verified')->default(false);
    $table->string('raw_hash', 32);
    $table->timestamp('captured_at');
    $table->timestamps();

    // One snapshot per account per payload-state.
    $table->unique(['platform', 'username', 'raw_hash']);
    $table->index(['platform', 'username', 'captured_at']);
});

ProfileSnapshot::upsert(
    $rows,
    uniqueBy: ['platform', 'username', 'raw_hash'],
    update:   ['captured_at'],
);

Now a duplicate run touches captured_at and moves on. No duplicate rows, no application-level "does this exist yet" query per record.

What I'd tell my past self

The scraper was never the hard part. Getting HTML and pulling fields out of it is an afternoon. The hard part is everything that assumes the afternoon's work will keep being true — and the fix is to stop pretending a remote social platform is a stable dependency and start treating it like the flaky third party it is.

Concretely, in priority order:

Store raw payloads. Cheapest insurance in the whole pipeline.
Separate fetch from parse. Lets you replay history without re-fetching.
Make normalization strict and loud. Silent nulls are worse than crashes.
Put budget control in exactly one place. Concurrency without a budget guard is a bill.
Make storage idempotent from day one. Retrofitting this is miserable. Whether you run your own infrastructure or use a managed API for the fetch stage is genuinely a build-vs-buy call — it depends on how much of your week you want to spend on the parts that aren't your product. But stages 2 and 3 are yours either way, and they're the ones that decide whether the data you're sitting on is trustworthy.

If you want to skip stage 1, that's the problem ScrapingIsNotACrime solves — public-data endpoints for Instagram, TikTok and YouTube, flat per-request pricing, 100 free credits to try. Public data only.

What does your ingestion layer look like? I'm especially curious how people handle schema drift alerting without drowning in false positives — drop it in the comments.