Last quarter I had a concrete problem. Our discovery backend at TrendVidStream pulls trending video metadata from eight regions, and the cron worker that did it was a 400-line PHP script wrapping curl with string concatenation. It worked until a provider added a nullable duration field and shipped a malformed published_at on roughly 0.3% of records. PHP shrugged, coerced everything to strings, and quietly wrote garbage rows into SQLite. We didn't notice for three days because nothing crashed — the FTS5 index just started returning videos with empty titles.
The fix was not more validation in PHP. The fix was to extract the fetch-and-parse layer into a standalone SDK where invalid data fails loudly at the boundary, then call it from everything else. I picked Rust because reqwest and serde give you exactly that: the HTTP request and the JSON contract are both checked, and a missing required field is a compile-or-runtime error you see immediately, not a silent empty string. This post is the SDK we actually shipped — every block here compiles and runs. If you want context on the larger system, that's TrendVidStream, a multi-region streaming-discovery site.
The data contract comes first
Before any HTTP code, define what a video is. This is the part that PHP let us skip, and skipping it is what bit us. With serde you write the shape once and get parsing, validation, and good error messages for free.
The two things that burned us were nullable fields and inconsistent timestamps, so the model handles both explicitly. Option<T> for fields the API genuinely omits, a default for counters that are sometimes missing, and a custom deserializer for the timestamp because the upstream API mixes RFC 3339 and a Unix-seconds fallback.
use serde::Deserialize;
use time::OffsetDateTime;
#[derive(Debug, Clone, Deserialize)]
pub struct Video {
pub id: String,
pub title: String,
#[serde(default)]
pub description: Option<String>,
// Provider sometimes omits this entirely on live streams.
pub duration_seconds: Option<u32>,
#[serde(default)]
pub view_count: u64,
pub region: String,
#[serde(deserialize_with = "crate::de::flexible_timestamp")]
pub published_at: OffsetDateTime,
}
#[derive(Debug, Deserialize)]
pub struct VideoPage {
pub items: Vec<Video>,
#[serde(default)]
pub next_page_token: Option<String>,
}
The key decision: title is String, not Option<String>. If the upstream record has no title, deserialization fails for that record, and we'd rather skip one bad row than index a blank one. view_count uses #[serde(default)] so a missing counter becomes 0 instead of failing — that's a field where a sensible default beats rejection. You make this call field by field, and serde forces you to make it consciously.
Here's the custom deserializer for the timestamp mess. This is the single highest-value piece of code in the SDK, because it's where the original PHP silently produced 1970-01-01.
// src/de.rs
use serde::{Deserialize, Deserializer};
use time::{format_description::well_known::Rfc3339, OffsetDateTime};
pub fn flexible_timestamp<'de, D>(d: D) -> Result<OffsetDateTime, D::Error>
where
D: Deserializer<'de>,
{
#[derive(Deserialize)]
#[serde(untagged)]
enum Raw {
Text(String),
Unix(i64),
}
match Raw::deserialize(d)? {
Raw::Text(s) => OffsetDateTime::parse(&s, &Rfc3339)
.map_err(serde::de::Error::custom),
Raw::Unix(secs) => OffsetDateTime::from_unix_timestamp(secs)
.map_err(serde::de::Error::custom),
}
}
#[serde(untagged)] tries each variant in order, so a JSON string parses as RFC 3339 and a bare integer parses as Unix seconds. Anything else — null, a float, a malformed date string — returns an Err, and that error carries the field path. No more 1970.
A client that owns its connection pool
The second mistake in the old code was spawning a fresh connection per request. Eight regions, paginated, every two hours — that's a lot of TLS handshakes for no reason. reqwest::Client holds a connection pool internally and is cheap to clone (it's an Arc under the hood), so you build it once and share it.
The client wrapper holds the configured reqwest::Client, the base URL, and the API key. Note the builder sets a real timeout and a User-Agent — both things you want in production and both things people forget.
use std::time::Duration;
use reqwest::Client;
#[derive(Clone)]
pub struct VideoApi {
http: Client,
base_url: String,
api_key: String,
}
impl VideoApi {
pub fn new(api_key: impl Into<String>) -> reqwest::Result<Self> {
let http = Client::builder()
.timeout(Duration::from_secs(15))
.connect_timeout(Duration::from_secs(5))
.user_agent("trendvidstream-sdk/1.0")
.pool_max_idle_per_host(8)
.build()?;
Ok(Self {
http,
base_url: "https://api.example-video.com/v1".to_string(),
api_key: api_key.into(),
})
}
pub fn with_base_url(mut self, url: impl Into<String>) -> Self {
self.base_url = url.into();
self
}
}
pool_max_idle_per_host(8) matches our eight-region fan-out — one warm connection per region, kept alive between cron runs within the same process. The with_base_url builder method exists purely so the test suite can point the client at a mock server, which we'll use at the end.
Errors that tell you what to do
A good SDK distinguishes errors the caller should retry from errors the caller should give up on. A 503 is transient; a 401 means your key is wrong and retrying is pointless. A JSON parse failure means the contract changed and a human needs to look. Collapsing all of these into one Box<dyn Error> — which is what the PHP equivalent did with a generic exception — throws away the information you need to react correctly.
This is where thiserror earns its place. Each variant is a category, and the #[from] attributes let ? convert lower-level errors automatically.
use thiserror::Error;
#[derive(Debug, Error)]
pub enum ApiError {
#[error("http transport error: {0}")]
Transport(#[from] reqwest::Error),
#[error("failed to decode response body: {0}")]
Decode(#[from] serde_json::Error),
#[error("unauthorized - check your api key")]
Unauthorized,
#[error("rate limited, retry after {retry_after_secs}s")]
RateLimited { retry_after_secs: u64 },
#[error("server error {status}")]
Server { status: u16 },
}
impl ApiError {
/// Whether a retry with backoff could plausibly succeed.
pub fn is_retryable(&self) -> bool {
matches!(
self,
ApiError::Transport(_)
| ApiError::RateLimited { .. }
| ApiError::Server { .. }
)
}
}
The is_retryable method is the whole point. The caller — our cron worker — loops on it instead of hard-coding a list of status codes in three different places. Unauthorized and Decode are deliberately not retryable, because retrying a bad key or a broken schema just wastes the rate-limit budget.
Wiring the request to the model
Now the actual fetch. The method builds the request with query parameters, inspects the status code to map it onto our error categories, and only then hands the body to serde. The ordering matters: you check the status before parsing, because a 429 body is not a VideoPage and trying to deserialize it would give you a confusing decode error instead of a clear rate-limit error.
impl VideoApi {
pub async fn trending(
&self,
region: &str,
page_token: Option<&str>,
) -> Result<VideoPage, ApiError> {
let url = format!("{}/videos/trending", self.base_url);
let mut req = self
.http
.get(&url)
.bearer_auth(&self.api_key)
.query(&[("region", region), ("max_results", "50")]);
if let Some(token) = page_token {
req = req.query(&[("page_token", token)]);
}
let resp = req.send().await?;
match resp.status().as_u16() {
200 => {
// .json() does the serde decode and maps errors via #[from].
let page = resp.json::<VideoPage>().await?;
Ok(page)
}
401 | 403 => Err(ApiError::Unauthorized),
429 => {
let retry_after_secs = resp
.headers()
.get("retry-after")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse().ok())
.unwrap_or(60);
Err(ApiError::RateLimited { retry_after_secs })
}
status => Err(ApiError::Server { status }),
}
}
}
One subtlety worth calling out: resp.json::<VideoPage>() reads the entire body and runs serde on it. If a single video in the page has a broken published_at, the whole page fails to decode. For our use case that's acceptable — we'd rather skip a page and retry than index half of it — but if you need per-item resilience you'd deserialize into Vec<serde_json::Value> first and parse each element individually, collecting the failures. Know which behavior you want; don't get it by accident.
Pagination as an async stream
The trending endpoint is paginated with an opaque next_page_token. Callers shouldn't have to write the token-threading loop themselves — that's exactly the kind of boilerplate an SDK should absorb. I expose it as an async_stream so the caller writes a plain while let and the pagination disappears.
use futures::Stream;
use async_stream::try_stream;
impl VideoApi {
pub fn trending_all<'a>(
&'a self,
region: &'a str,
) -> impl Stream<Item = Result<Video, ApiError>> + 'a {
try_stream! {
let mut token: Option<String> = None;
loop {
let page = self.trending(region, token.as_deref()).await?;
for video in page.items {
yield video;
}
match page.next_page_token {
Some(next) => token = Some(next),
None => break,
}
}
}
}
}
Now the cron worker that replaced the old PHP script reads like a description of the business logic, with retry and backoff applied per region. This is the consumer side, and it's where is_retryable pays off:
use futures::StreamExt;
use tokio::time::{sleep, Duration};
const REGIONS: [&str; 8] =
["US", "GB", "DE", "FR", "IN", "BR", "JP", "AU"];
async fn run() -> Result<(), ApiError> {
let api = VideoApi::new(std::env::var("VIDEO_API_KEY").unwrap())?;
for region in REGIONS {
let mut attempt = 0u32;
loop {
let mut stream = Box::pin(api.trending_all(region));
let mut count = 0;
let result: Result<(), ApiError> = async {
while let Some(item) = stream.next().await {
let video = item?;
upsert_video(&video); // write to SQLite / FTS5
count += 1;
}
Ok(())
}
.await;
match result {
Ok(()) => {
println!("{region}: indexed {count} videos");
break;
}
Err(e) if e.is_retryable() && attempt < 3 => {
attempt += 1;
let backoff = match &e {
ApiError::RateLimited { retry_after_secs } => *retry_after_secs,
_ => 2u64.pow(attempt),
};
eprintln!("{region}: {e}, retrying in {backoff}s");
sleep(Duration::from_secs(backoff)).await;
}
Err(e) => {
eprintln!("{region}: giving up - {e}");
break;
}
}
}
}
Ok(())
}
The rate-limit case reads Retry-After straight from the error variant; everything else uses exponential backoff. A non-retryable error breaks the loop for that region without poisoning the others. Compared to the PHP version — which would either crash the whole run or swallow the error — each region now succeeds or fails independently and tells you which.
Testing the contract without hitting the network
The reason this whole exercise is worth it is testability. With wiremock you stand up a fake server, hand its URL to the client via with_base_url, and assert that your parsing and error mapping behave. These tests run in milliseconds and catch contract drift before it reaches production — which is the failure mode that started this entire project.
#[cfg(test)]
mod tests {
use super::*;
use wiremock::{Mock, MockServer, ResponseTemplate};
use wiremock::matchers::{method, path, query_param};
#[tokio::test]
async fn parses_page_and_skips_missing_optionals() {
let server = MockServer::start().await;
let body = serde_json::json!({
"items": [{
"id": "v1",
"title": "Trending clip",
"duration_seconds": null,
"region": "US",
"published_at": "2026-05-01T12:00:00Z"
}],
"next_page_token": null
});
Mock::given(method("GET"))
.and(path("/videos/trending"))
.and(query_param("region", "US"))
.respond_with(ResponseTemplate::new(200).set_body_json(body))
.mount(&server)
.await;
let api = VideoApi::new("test-key").unwrap().with_base_url(server.uri());
let page = api.trending("US", None).await.unwrap();
assert_eq!(page.items.len(), 1);
assert_eq!(page.items[0].duration_seconds, None);
assert!(page.next_page_token.is_none());
}
#[tokio::test]
async fn maps_429_to_rate_limited() {
let server = MockServer::start().await;
Mock::given(method("GET"))
.respond_with(ResponseTemplate::new(429).insert_header("retry-after", "30"))
.mount(&server)
.await;
let api = VideoApi::new("k").unwrap().with_base_url(server.uri());
let err = api.trending("US", None).await.unwrap_err();
assert!(matches!(err, ApiError::RateLimited { retry_after_secs: 30 }));
assert!(err.is_retryable());
}
}
The first test pins the contract — a null duration parses to None, not an error — and the second proves the Retry-After header flows through into the error variant the cron worker depends on. When the upstream provider adds a field or changes a type, one of these goes red and tells you exactly where.
How it talks to the existing stack
One thing I want to be honest about: we did not rewrite everything in Rust. The site itself still runs on PHP 8.4 with SQLite FTS5 for search, and deploys over FTP because that's what the hosting allows. The Rust SDK is a single statically-linked binary that the cron job runs; it writes rows into the same SQLite file the PHP app reads. The PHP side stayed almost identical — it just consumes clean data now instead of validating it:
<?php
// search.php - the PHP layer no longer revalidates fields
$db = new SQLite3('/var/data/videos.db');
$stmt = $db->prepare(
'SELECT id, title, region, published_at FROM videos_fts
WHERE videos_fts MATCH :q ORDER BY rank LIMIT 50'
);
$stmt->bindValue(':q', $query, SQLITE3_TEXT);
$res = $stmt->execute();
while ($row = $res->fetchArray(SQLITE3_ASSOC)) {
// title is guaranteed non-empty: the Rust SDK rejected blank ones at ingest.
echo render_card($row);
}
The boundary is the database file. The Rust SDK guarantees that anything in the videos table satisfied the serde contract at ingest time, so the PHP query layer can trust its inputs and stay simple. You don't need to rewrite a working application to get the benefit of a type-safe ingest layer — you just need to move the validation to the one place where the data enters the system.
Conclusion
The original bug was never really about PHP. It was about validating data in the wrong place — after it had already been written — and treating all failures as equivalent. Rust didn't fix that by being Rust; it fixed it by making three things mandatory that we'd treated as optional: a declared data contract (serde structs), a typed error taxonomy that distinguishes retryable from fatal (thiserror plus is_retryable), and tests against a mock server that pin the contract before deploy.
If you build one of these, the order matters. Write the data model and the custom deserializers first, because that's where the real failures live. Add the error enum second so your retry logic has something honest to branch on. Wire up the client and pagination third — that part is mechanical. And mock the server from day one, because the whole value proposition is catching contract drift before it reaches the database. We've run this across eight regions for a few months now and the silent-empty-title class of bug is simply gone, not because we're more careful, but because the boundary won't let it through anymore.
Top comments (0)