Our trend-tracking pipeline at ViralVidVault pulls from a dozen upstream video APIs — YouTube Data, Vimeo, a couple of regional European providers, and our own ingestion service. For two years the client that talked to all of them was a pile of PHP that did json_decode($body, true) and then reached blindly into associative arrays. It worked until an upstream provider renamed view_count to viewCount on one endpoint and left it untouched on another. Nothing threw. We silently recorded zero views for 40,000 videos over a weekend before a dashboard looked wrong enough for someone to notice.
That outage is why we moved the hot path of our fetcher into a small Rust SDK built on reqwest and serde. The goal was not raw speed — PHP 8.4 with an SQLite WAL backend is plenty fast for what we do. The goal was that a shape change upstream should be a compile error or a loud, typed runtime error, never a silent zero. This article walks through how we built that SDK: the type model, error handling, pagination, rate limiting, and how we still call it from our PHP stack over a thin FFI-free boundary.
Modeling the API response, not the transport
The first mistake people make with serde is deriving Deserialize on a struct that mirrors the raw JSON one-to-one, including the provider's naming quirks and nullable-everything fields. Then those quirks leak into your whole codebase. We separate two layers: a wire type that matches the JSON exactly, and a domain type that the rest of the SDK actually uses.
Here is the wire layer for a video resource. Notice that every optional field upstream is Option<T>, and we use serde's attributes to absorb the provider's naming instead of fighting it.
use serde::Deserialize;
#[derive(Debug, Deserialize)]
pub struct WireVideo {
pub id: String,
pub title: String,
// Provider sends camelCase; we normalize at the boundary.
#[serde(rename = "viewCount", default)]
pub view_count: Option<u64>,
#[serde(rename = "publishedAt")]
pub published_at: Option<String>,
#[serde(default)]
pub region: Option<String>,
// Unknown/extra fields are ignored by default, which is what we want
// for forward compatibility with new provider fields.
}
#[derive(Debug, Deserialize)]
pub struct WirePage {
pub items: Vec<WireVideo>,
#[serde(rename = "nextPageToken", default)]
pub next_page_token: Option<String>,
}
The key attributes doing work here:
-
#[serde(rename = "...")]maps the ugly wire name to a clean Rust field. When the provider renamedview_counttoviewCount, we would have changed exactly one line here — and if they had removed it, theOptionplusdefaultmeans we get an explicitNone, not a crash and not a silent zero. -
#[serde(default)]means a missing field deserializes toOption::Nonerather than failing the whole parse. This is the single most important defensive habit: a missing optional field should never poison an otherwise-valid page of 50 videos. - Unknown fields are dropped silently by default. If you want the opposite — to be told when a provider adds fields — you can add
#[serde(deny_unknown_fields)], but for a consumer that is usually too brittle.
The domain type is where we make decisions. A video with no view_count is a real thing that happened; we represent it honestly rather than defaulting to zero.
use time::OffsetDateTime;
#[derive(Debug, Clone)]
pub struct Video {
pub id: String,
pub title: String,
pub views: Option<u64>,
pub published_at: Option<OffsetDateTime>,
pub region: Option<String>,
}
impl TryFrom<WireVideo> for Video {
type Error = SdkError;
fn try_from(w: WireVideo) -> Result<Self, Self::Error> {
let published_at = match w.published_at {
Some(s) => Some(
OffsetDateTime::parse(&s, &time::format_description::well_known::Rfc3339)
.map_err(|e| SdkError::BadField {
field: "publishedAt",
detail: e.to_string(),
})?,
),
None => None,
};
Ok(Video {
id: w.id,
title: w.title,
views: w.view_count,
published_at,
region: w.region,
})
}
}
This TryFrom is the seam. Parsing JSON is infallible-ish (serde handles it), but interpreting a field — turning an RFC 3339 string into an actual timestamp — can fail, and when it does we get a SdkError::BadField that names the exact field. That error message is what turns a 2 a.m. debugging session into a 2-minute one.
Errors that tell you where it broke
A video API SDK fails in categories, and callers need to react to them differently. A 429 means back off. A 401 means fix your credentials and stop retrying. A deserialization failure means the provider changed something. Collapsing all of these into one Box<dyn Error> throws away exactly the information the caller needs.
We use thiserror to build a flat, matchable error enum:
use thiserror::Error;
#[derive(Debug, Error)]
pub enum SdkError {
#[error("transport error: {0}")]
Transport(#[from] reqwest::Error),
#[error("rate limited, retry after {retry_after_secs}s")]
RateLimited { retry_after_secs: u64 },
#[error("authentication failed (status {status})")]
Auth { status: u16 },
#[error("provider returned {status}: {body}")]
Http { status: u16, body: String },
#[error("could not parse response: {0}")]
Decode(String),
#[error("invalid field `{field}`: {detail}")]
BadField { field: &'static str, detail: String },
}
The #[from] reqwest::Error lets us use ? on transport calls and get automatic conversion. Everything else we construct explicitly so the variant carries structured data — RateLimited holds the retry delay, Http holds the status and body so a human can read what the provider actually complained about. When our PHP layer receives one of these as JSON, it can branch on the variant name without string-matching an error message.
The client: one place for auth, timeouts, and base URL
The reqwest::Client is designed to be created once and shared — it holds a connection pool. Creating a new client per request is the classic performance footgun that shows up as mysterious latency and exhausted file descriptors. We wrap a single client in our SDK type and hang configuration off it.
use reqwest::{Client, StatusCode};
use std::time::Duration;
pub struct VideoApi {
http: Client,
base_url: String,
api_key: String,
}
impl VideoApi {
pub fn new(base_url: impl Into<String>, api_key: impl Into<String>) -> Result<Self, SdkError> {
let http = Client::builder()
.timeout(Duration::from_secs(15))
.connect_timeout(Duration::from_secs(5))
.user_agent("viralvidvault-sdk/1.0")
.build()?;
Ok(Self {
http,
base_url: base_url.into(),
api_key: api_key.into(),
})
}
async fn get_page(&self, region: &str, page_token: Option<&str>) -> Result<WirePage, SdkError> {
let mut req = self
.http
.get(format!("{}/videos", self.base_url))
.bearer_auth(&self.api_key)
.query(&[("region", region)]);
if let Some(token) = page_token {
req = req.query(&[("pageToken", token)]);
}
let resp = req.send().await?;
let status = resp.status();
match status {
StatusCode::OK => {
let text = resp.text().await?;
serde_json::from_str::<WirePage>(&text)
.map_err(|e| SdkError::Decode(format!("{e} — body was: {}", truncate(&text, 300))))
}
StatusCode::TOO_MANY_REQUESTS => {
let retry_after_secs = resp
.headers()
.get("retry-after")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse().ok())
.unwrap_or(30);
Err(SdkError::RateLimited { retry_after_secs })
}
StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN => {
Err(SdkError::Auth { status: status.as_u16() })
}
other => {
let body = resp.text().await.unwrap_or_default();
Err(SdkError::Http { status: other.as_u16(), body: truncate(&body, 500) })
}
}
}
}
fn truncate(s: &str, max: usize) -> String {
if s.len() <= max { s.to_string() } else { format!("{}…", &s[..max]) }
}
Two details worth calling out. First, on a 200 we read the body as text before handing it to serde_json::from_str instead of using resp.json(). This costs one extra allocation but buys us the raw body in the error message when parsing fails — you cannot debug a shape change from "expected struct WirePage" alone; you need to see what actually arrived. Second, we parse Retry-After from the header and fall back to a sane default. Providers are wildly inconsistent about that header, so defensive parsing with unwrap_or(30) keeps us from panicking on a missing or malformed value.
Pagination as an async stream
Most video APIs paginate with an opaque nextPageToken. The naive approach is a loop that accumulates a giant Vec, but that means holding every video from every region in memory before you write a single one to disk. For our nightly full-sync across European regions that is hundreds of thousands of rows. We expose pagination as an async iterator instead so the caller processes each page as it arrives and memory stays flat.
impl VideoApi {
/// Fetch every video for a region, invoking `sink` once per page.
/// Stops on the first error and returns it.
pub async fn for_each_video<F>(&self, region: &str, mut sink: F) -> Result<u64, SdkError>
where
F: FnMut(Vec<Video>) -> Result<(), SdkError>,
{
let mut token: Option<String> = None;
let mut total: u64 = 0;
loop {
let page = self.get_page(region, token.as_deref()).await?;
let videos: Vec<Video> = page
.items
.into_iter()
.map(Video::try_from)
.collect::<Result<_, _>>()?;
total += videos.len() as u64;
sink(videos)?;
match page.next_page_token {
Some(t) if !t.is_empty() => token = Some(t),
_ => break,
}
}
Ok(total)
}
}
The sink closure is where the caller writes to SQLite, pushes to a queue, or in our case streams straight into a batch insert. The collect::<Result<_, _>>()? line is a small piece of Rust elegance: converting a Vec of Results into a Result of Vec, short-circuiting on the first bad field. One malformed timestamp in a page fails that page loudly instead of silently dropping the row — which is exactly the behavior our old PHP client lacked.
A few production notes on this loop:
-
Bound your retries per page. The version above stops on the first
RateLimited. In our real build the caller wrapsget_pagein a retry-with-jitter that respectsretry_after_secs, capped at 3 attempts, so a single 429 does not abort an eight-hour sync. -
Deduplicate at the sink, not in memory. Regional feeds overlap heavily — a video trending in Germany also trends in Austria. We let SQLite's
INSERT ... ON CONFLICT DO UPDATEhandle dedup on the primary key rather than holding aHashSetof seen IDs in the SDK. -
Cap total pages defensively. A misbehaving provider that always returns a non-empty
nextPageTokenwill loop forever. We add amax_pagesguard in production; it has fired exactly once, and that once was a provider bug.
Calling Rust from a PHP and Cloudflare Workers stack
We did not rewrite the site in Rust, and you probably should not either. The SDK is a single command-line binary that the PHP fetcher shells out to, handing it a region and reading newline-delimited JSON back on stdout. No FFI, no shared library ABI to keep in sync, no segfaults taking down the web server. The boundary is a process boundary, which is the most debuggable boundary there is.
On the PHP 8.4 side the consumer looks like this — and because the Rust layer already validated every field, the PHP code can trust the shapes it receives:
<?php
declare(strict_types=1);
function sync_region(string $region, PDO $db): int
{
$cmd = sprintf(
'/usr/local/bin/vvv-sdk fetch --region %s',
escapeshellarg($region)
);
$proc = popen($cmd, 'r');
if ($proc === false) {
throw new RuntimeException("could not start sdk for {$region}");
}
$stmt = $db->prepare(
'INSERT INTO videos (id, title, views, region, published_at)
VALUES (:id, :title, :views, :region, :published_at)
ON CONFLICT(id) DO UPDATE SET
views = excluded.views,
title = excluded.title'
);
$count = 0;
while (($line = fgets($proc)) !== false) {
$v = json_decode($line, true, flags: JSON_THROW_ON_ERROR);
$stmt->execute([
':id' => $v['id'],
':title' => $v['title'],
':views' => $v['views'], // may be null — column is nullable
':region' => $v['region'],
':published_at' => $v['published_at'],
]);
$count++;
}
pclose($proc);
return $count;
}
The SQLite database runs in WAL mode, so this writer and the LiteSpeed-served read traffic do not block each other — the sync can run mid-day without the site stalling. The whole videos table is served to the public through a Cloudflare Worker that caches read responses at the edge, so the Rust binary only ever touches the origin. That division of labor is the point: Rust owns the untrusted, shape-shifting boundary with upstream providers; PHP and SQLite own the boring, well-understood storage and rendering; Cloudflare owns distribution. Each layer does the thing it is good at.
Why null for views instead of 0 matters here, one more time: our trend-scoring query ranks videos by view velocity. A NULL view count is correctly excluded from the ranking as unknown data. A 0 would have ranked those 40,000 videos as the least popular content on the platform and buried them. The type system carried that distinction all the way from the JSON boundary to the SQL WHERE views IS NOT NULL clause.
What the SDK bought us
Six months in, the concrete wins:
-
Zero silent zeros. Every field-level parse failure now produces a
BadFielderror with the field name, logged and alertable. When a provider changed a date format in April we knew within one sync cycle, not one weekend. - One place to absorb provider quirks. Every rename, every camelCase-vs-snake_case fight, lives in the wire structs. The rest of the SDK and all of the PHP never see it.
- Flat memory during full syncs. The streaming pagination means a region with 300,000 videos uses the same memory as one with 300.
- Matchable errors. Rate limits back off, auth failures page a human, decode failures alert the data team. The caller never has to grep an error string.
If you are building a client for any JSON API that you do not control — and a video API is the canonical example, because upstream providers change fields on their schedule, not yours — the pattern generalizes. Keep a wire type that matches the JSON exactly and absorbs its quirks with serde attributes. Keep a domain type that models what your application actually believes. Put a fallible TryFrom between them so interpretation failures are named and loud. Use one shared reqwest::Client, a thiserror enum with one variant per failure category, and streaming pagination so memory never surprises you. The compiler will not catch a provider renaming a field on their end — but it will make absolutely sure you handle the Option when they do.
Top comments (0)