Every engineer who has needed social media data has written the same script. curl the profile, regex the JSON blob out of the HTML, json_decode, done. It works beautifully on a Tuesday afternoon.
By Thursday it returns an empty array. You add a User-Agent header. It works again for a week. Then it returns a login wall. You add a proxy. Then the JSON key renames itself from edge_owner_to_timeline_media to something else entirely and your foreach throws on null.
I did this three times before admitting the real problem: I was treating an unreliable, mutating, rate-limited remote source as if it were a local database call. Every architectural decision downstream of that assumption was wrong.
Here's what the ingestion layer actually needs to look like.
The four failure modes you're designing for
Before any code, name the enemy. Social data ingestion fails in four distinct ways, and they need four different responses:
- Transient failure — timeout, 502, connection reset. Retry with backoff. Costs you nothing but time.
- Rate/block failure — 429, or a 200 with an empty payload that means "blocked." Back off much harder, or route elsewhere.
-
Schema drift — the response is valid JSON, status 200, but the field you need moved or vanished. Retrying does nothing. This one is the silent killer because it doesn't throw — it writes
nullinto your database for three weeks before anyone notices. - Semantic absence — the account is private, deleted, or has zero posts. A correct, permanent, empty answer. Retrying is pure waste. Most homemade pipelines only handle #1. Collapsing #2, #3 and #4 into "an error occurred" is why they degrade so quietly.
Architecture: the pipeline in three stages
[ scheduler ] -> [ fetch workers ] -> [ normalize ] -> [ store ]
| | |
dedupe budget guard schema contract
Three rules that matter more than the diagram:
- The fetch stage never writes to your domain tables. It writes raw payloads. Normalization is a separate, replayable step.
- The scheduler owns the budget. Whether your budget is API credits, proxy bandwidth, or your own tolerance for getting blocked, one component decides how much of it gets spent — never the workers.
- The normalizer owns a contract. It validates against an expected shape and fails loudly. This is your schema-drift alarm. That first rule is the one people skip. Storing the raw response costs you a few KB and buys you the ability to re-parse six months of history when you discover your engagement calculation was wrong — without re-fetching (and re-paying for) a single record.
Stage 1: The fetch worker, with a budget guard
Hyperf's coroutine model makes concurrent HTTP fetching cheap, but "cheap" is exactly how you burn a credit balance in ninety seconds. The guard is not optional.
<?php
declare(strict_types=1);
namespace App\Ingestion;
use GuzzleHttp\Client;
use GuzzleHttp\Exception\ConnectException;
use Hyperf\Coroutine\Parallel;
use Psr\Log\LoggerInterface;
final class ProfileFetcher
{
private const MAX_ATTEMPTS = 3;
public function __construct(
private readonly Client $http,
private readonly BudgetGuard $budget,
private readonly LoggerInterface $log,
) {}
/**
* @param list<string> $usernames
* @return array<string, FetchResult>
*/
public function fetchMany(array $usernames, int $concurrency = 20): array
{
$parallel = new Parallel($concurrency);
foreach ($usernames as $username) {
$parallel->add(
fn (): FetchResult => $this->fetchOne($username),
$username
);
}
return $parallel->wait();
}
private function fetchOne(string $username): FetchResult
{
if (! $this->budget->tryConsume(1)) {
return FetchResult::deferred($username, 'budget exhausted');
}
$attempt = 0;
while (++$attempt <= self::MAX_ATTEMPTS) {
try {
$response = $this->http->get('/v1/instagram/profile', [
'query' => ['username' => $username],
'headers' => ['X-API-Key' => $this->budget->key()],
'timeout' => 30,
'http_errors' => false,
]);
$status = $response->getStatusCode();
$body = (string) $response->getBody();
// Semantic absence — permanent, do not retry.
if ($status === 404) {
return FetchResult::absent($username);
}
// Rate/block — back off hard, return budget to the pool.
if ($status === 429 || $status === 503) {
$this->budget->refund(1);
$this->sleepBackoff($attempt, aggressive: true);
continue;
}
if ($status >= 500) {
$this->sleepBackoff($attempt);
continue;
}
if ($status === 200) {
// Raw payload only. No parsing here — that is stage 2.
return FetchResult::ok($username, $body);
}
$this->log->warning('unexpected status', [
'username' => $username,
'status' => $status,
]);
return FetchResult::failed($username, "http {$status}");
} catch (ConnectException $e) {
$this->sleepBackoff($attempt);
}
}
return FetchResult::failed($username, 'attempts exhausted');
}
private function sleepBackoff(int $attempt, bool $aggressive = false): void
{
$base = $aggressive ? 5.0 : 1.0;
$jitter = random_int(0, 1000) / 1000;
// Jitter matters: without it, N coroutines that failed together
// will retry together, and fail together again.
\Hyperf\Coroutine\Coroutine::sleep($base * (2 ** ($attempt - 1)) + $jitter);
}
}
Two details worth stealing regardless of your stack:
http_errors => false. Let the status code be data you branch on, not an exception you catch. A 429 and a 500 deserve different treatment, and exception-based control flow flattens that distinction.
Refunding budget on a block. If the request never returned data, it shouldn't count against your quota accounting. Sounds obvious. Almost nobody does it, and then the "why did we spend 4,000 credits yesterday" investigation takes an afternoon.
Stage 2: The normalizer as a contract
This is the stage that catches schema drift, and the whole point is that it's strict.
<?php
declare(strict_types=1);
namespace App\Ingestion;
final class ProfileNormalizer
{
/**
* @throws SchemaDriftException
*/
public function normalize(string $rawJson, string $username): ProfileSnapshot
{
$decoded = json_decode($rawJson, true, 512, JSON_THROW_ON_ERROR);
$data = $decoded['data'] ?? throw new SchemaDriftException(
"missing 'data' envelope for {$username}"
);
return new ProfileSnapshot(
username: $this->requireString($data, 'username', $username),
followers: $this->requireInt($data, 'followers', $username),
verified: (bool) ($data['verified'] ?? false),
capturedAt: new \DateTimeImmutable(),
rawHash: hash('xxh128', $rawJson),
);
}
private function requireInt(array $data, string $key, string $ctx): int
{
$value = $data[$key] ?? null;
if (! is_int($value)) {
throw new SchemaDriftException(
sprintf('expected int at "%s" for %s, got %s', $key, $ctx, get_debug_type($value))
);
}
return $value;
}
private function requireString(array $data, string $key, string $ctx): string
{
$value = $data[$key] ?? null;
if (! is_string($value) || $value === '') {
throw new SchemaDriftException(
sprintf('expected non-empty string at "%s" for %s', $key, $ctx)
);
}
return $value;
}
}
The instinct is to write $data['followers'] ?? 0. Resist it. A null-coalesce here converts a structural break into a plausible-looking zero, and plausible-looking zeros are how you end up presenting a dashboard where an account "lost" 600 million followers overnight.
Throw. Alert on the exception. Keep the raw payload so you can replay once you've fixed the mapping.
The rawHash field earns its keep too: it lets you skip re-normalizing identical payloads, and it gives you a cheap change-detection signal for polling.
Stage 3: Idempotent storage
The one thing that will happen, guaranteed, is that some job runs twice. A worker dies mid-batch, a deploy restarts the queue, someone re-runs a backfill. Design for it now rather than deduplicating rows later.
// Migration: the uniqueness constraint is the real defense.
Schema::create('profile_snapshots', function (Blueprint $table) {
$table->id();
$table->string('platform', 32);
$table->string('username');
$table->unsignedBigInteger('followers');
$table->boolean('verified')->default(false);
$table->string('raw_hash', 32);
$table->timestamp('captured_at');
$table->timestamps();
// One snapshot per account per payload-state.
$table->unique(['platform', 'username', 'raw_hash']);
$table->index(['platform', 'username', 'captured_at']);
});
ProfileSnapshot::upsert(
$rows,
uniqueBy: ['platform', 'username', 'raw_hash'],
update: ['captured_at'],
);
Now a duplicate run touches captured_at and moves on. No duplicate rows, no application-level "does this exist yet" query per record.
What I'd tell my past self
The scraper was never the hard part. Getting HTML and pulling fields out of it is an afternoon. The hard part is everything that assumes the afternoon's work will keep being true — and the fix is to stop pretending a remote social platform is a stable dependency and start treating it like the flaky third party it is.
Concretely, in priority order:
- Store raw payloads. Cheapest insurance in the whole pipeline.
- Separate fetch from parse. Lets you replay history without re-fetching.
- Make normalization strict and loud. Silent nulls are worse than crashes.
- Put budget control in exactly one place. Concurrency without a budget guard is a bill.
- Make storage idempotent from day one. Retrofitting this is miserable. Whether you run your own infrastructure or use a managed API for the fetch stage is genuinely a build-vs-buy call — it depends on how much of your week you want to spend on the parts that aren't your product. But stages 2 and 3 are yours either way, and they're the ones that decide whether the data you're sitting on is trustworthy.
If you want to skip stage 1, that's the problem ScrapingIsNotACrime solves — public-data endpoints for Instagram, TikTok and YouTube, flat per-request pricing, 100 free credits to try. Public data only.
What does your ingestion layer look like? I'm especially curious how people handle schema drift alerting without drowning in false positives — drop it in the comments.
Top comments (0)