Compact Video Metadata Serialization With Protobuf Across Services

The 40KB Problem Nobody Noticed Until Cloudflare Billed Us

Every viral video we ingest at ViralVidVault crosses three service boundaries before it ever reaches a user. A PHP 8.4 ingestion worker pulls the raw metadata, hands it to a Go trend-scoring service, which in turn feeds a Python analytics pipeline that computes velocity and acceleration curves for the European feeds. For a long time these three services spoke JSON to each other, because JSON is what everyone reaches for first. It worked, it was debuggable, and nobody questioned it.

Then we looked at the numbers. A single enriched VideoMetadata record — title, tags, region, published timestamp, view counts, and a nested block of trend signals — averaged just under 40KB as pretty-printed JSON, and around 28KB minified. Multiply that by the ~2.1 million records that move between services every day during a trend spike, and the internal egress alone was measurably showing up on our Cloudflare bill. Worse, the Go service was burning real CPU on encoding/json reflection, and the PHP worker spent more time in json_encode than it did doing the actual HTTP fetch.

We migrated the inter-service contract to Protocol Buffers. Payloads dropped by roughly 68%, parse CPU on the Go side fell by more than half, and — the part I care about most as someone shipping under GDPR — the schema became a single enforced contract instead of a loose bag of keys. This is the write-up I wish I'd had before starting. If you want to see the end result in production, it's the discovery engine behind ViralVidVault, our GDPR-compliant European viral video tracker.

Why JSON Was Costing Us More Than Bytes

The byte count is the obvious problem, but it wasn't the expensive one. Three things hurt more than payload size:

• No schema enforcement. A field renamed in the PHP worker would silently become null in the Go consumer. We caught these in production, not in review.
• Type ambiguity. JSON numbers are all doubles. A view_count of 4,300,000,000 quietly lost precision once it crossed 2^53, which for a genuinely viral clip is not hypothetical.
• Reflection cost. Both encoding/json in Go and json_encode in PHP walk the structure at runtime. At our volumes that reflection was a real slice of CPU, and CPU on our LiteSpeed origin is finite.

Protobuf addresses all three at once: a compiled schema, explicit integer widths, and generated marshalling code that doesn't reflect at runtime. The tradeoff is that the wire format is no longer human-readable, which matters less than you'd think once you have decent tooling.

Defining the Schema

Everything starts with the .proto file. This is the single source of truth that all three languages compile against. I keep it in a dedicated contracts/ repository that the PHP, Go, and Python services each vendor in, so nobody can drift.

syntax = "proto3";
package viralvidvault.metadata.v1;

message VideoMetadata {
  string video_id            = 1;
  string title               = 2;
  uint32 duration_seconds    = 3;
  uint64 view_count          = 4;
  Region region              = 5;
  repeated string tags       = 6;
  int64  published_at_unix   = 7;
  TrendSignals signals       = 8;
  bool   gdpr_pii_stripped   = 9;
}

enum Region {
  REGION_UNSPECIFIED = 0;
  REGION_DE = 1;
  REGION_FR = 2;
  REGION_ES = 3;
  REGION_IT = 4;
  REGION_NL = 5;
  REGION_PL = 6;
}

message TrendSignals {
  float  velocity        = 1;  // views/hour, normalised
  float  acceleration    = 2;  // d(velocity)/dt
  uint32 shares_per_hour = 3;
}

A few deliberate choices worth calling out, because they are the ones that bite people later:

• Field numbers are permanent. The number 4 is what goes on the wire, not the name view_count. You can rename the field freely, but never reuse or renumber a tag. If you retire a field, mark it reserved.
• uint64 for view counts. This is the precision fix. Protobuf varints encode small numbers in one byte and only grow as the value grows, so you pay nothing for the wide type until you actually need it.
• The region enum is versioned by the v1 package. European regions rarely change, but the enum's zero value being REGION_UNSPECIFIED means an unset region is unambiguous rather than defaulting to a real country.
• gdpr_pii_stripped is a boolean gate. Our contract says no record leaves the ingestion worker for analytics unless PII has been removed, and this flag is asserted downstream. More on that below.

Enum choice matters for size too: an enum is a varint on the wire, so REGION_DE costs one byte, versus the string "DE" costing three plus framing in JSON.

Encoding in PHP 8.4

Our ingestion worker is PHP. The official google/protobuf package ships a pure-PHP runtime, but for our throughput I strongly recommend installing the protobuf C extension as well — the pure-PHP path is correct but noticeably slower on hot loops. With protoc and the PHP plugin you generate classes under a namespace that mirrors the package.

<?php
declare(strict_types=1);

use ViralVidVault\Metadata\V1\VideoMetadata;
use ViralVidVault\Metadata\V1\TrendSignals;
use ViralVidVault\Metadata\V1\Region;

function buildMetadata(array $row): string
{
    $signals = (new TrendSignals())
        ->setVelocity((float) $row['velocity'])
        ->setAcceleration((float) $row['acceleration'])
        ->setSharesPerHour((int) $row['shares_per_hour']);

    $meta = (new VideoMetadata())
        ->setVideoId($row['video_id'])
        ->setTitle($row['title'])
        ->setDurationSeconds((int) $row['duration_seconds'])
        ->setViewCount((int) $row['view_count'])
        ->setRegion(Region::REGION_DE)
        ->setPublishedAtUnix((int) $row['published_at_unix'])
        ->setGdprPiiStripped(true)
        ->setSignals($signals);

    // repeated fields take an array; the runtime handles the packing
    $meta->setTags($row['tags'] ?? []);

    // returns the compact binary wire format, ready for the next hop
    return $meta->serializeToString();
}

// Decoding a payload that came back from another service:
function parseMetadata(string $binary): VideoMetadata
{
    $meta = new VideoMetadata();
    $meta->mergeFromString($binary); // throws on malformed input
    return $meta;
}

Two practical notes. First, serializeToString() gives you the raw binary — you send it as the request body with Content-Type: application/x-protobuf, not as a JSON string. Second, mergeFromString() throws on genuinely malformed bytes but happily ignores unknown fields, which is exactly the forward-compatibility behaviour you want during a rolling deploy where the encoder is a version ahead of the decoder.

One PHP-8.4-specific gotcha: int in PHP is a 64-bit signed integer on any 64-bit build, so a uint64 view count near the top of the range round-trips fine as a native int. If you ever run on a 32-bit SAPI, the runtime falls back to string representation for large integers — worth an assertion in your tests if you can't guarantee the build.

Consuming in Go

The trend-scoring service is Go, and this is where the CPU win showed up most clearly. Generated code plus google.golang.org/protobuf/proto gives you zero-reflection unmarshalling.

package scoring

import (
    "fmt"

    "google.golang.org/protobuf/proto"
    pb "github.com/viralvidvault/contracts/gen/go/metadata/v1"
)

// DecodeAndScore unmarshals a wire payload and rejects anything that
// has not been through the GDPR PII strip in the ingestion worker.
func DecodeAndScore(payload []byte) (float64, error) {
    var meta pb.VideoMetadata
    if err := proto.Unmarshal(payload, &meta); err != nil {
        return 0, fmt.Errorf("decode metadata: %w", err)
    }

    if !meta.GetGdprPiiStripped() {
        return 0, fmt.Errorf("refusing record %s: pii not stripped", meta.GetVideoId())
    }

    sig := meta.GetSignals()
    if sig == nil {
        return 0, nil // no signals yet, score is zero
    }

    // A cheap composite score; the real one is a weighted model.
    score := float64(sig.GetVelocity())*0.6 +
        float64(sig.GetAcceleration())*0.3 +
        float64(sig.GetSharesPerHour())*0.1

    return score, nil
}

The Get* accessors are the important habit here: they are nil-safe. meta.GetSignals() on a message where signals was never set returns a typed nil pointer, and sig.GetVelocity() on that nil returns the zero value rather than panicking. This is proto3's answer to the JSON null problem — there's a defined default for every scalar, so downstream code doesn't need defensive existence checks scattered everywhere.

Benchmarked against our old json.Unmarshal path over a representative sample, the protobuf decode was consistently 2–3x faster and allocated far less, because there's no map construction and no reflection walk.

Storing the Binary in SQLite WAL

Our origin uses SQLite in WAL mode as the local cache on each LiteSpeed node. Here's a pattern that surprised people on my team: you don't have to unpack protobuf to store it. The compact binary is a perfectly good BLOB, and SQLite treats it as an opaque byte string.

import sqlite3
from viralvidvault.metadata.v1 import video_metadata_pb2

con = sqlite3.connect("cache.db")
con.execute("PRAGMA journal_mode=WAL")
con.execute(
    "CREATE TABLE IF NOT EXISTS video_meta ("
    "  video_id TEXT PRIMARY KEY,"
    "  region   INTEGER,"        # denormalised for WHERE filters
    "  payload  BLOB NOT NULL"   # the raw protobuf bytes
    ")"
)

def store(payload: bytes) -> None:
    meta = video_metadata_pb2.VideoMetadata()
    meta.ParseFromString(payload)  # validate before we trust it
    con.execute(
        "INSERT OR REPLACE INTO video_meta (video_id, region, payload) VALUES (?, ?, ?)",
        (meta.video_id, meta.region, payload),
    )
    con.commit()

def load(video_id: str) -> video_metadata_pb2.VideoMetadata | None:
    row = con.execute(
        "SELECT payload FROM video_meta WHERE video_id = ?", (video_id,)
    ).fetchone()
    if row is None:
        return None
    meta = video_metadata_pb2.VideoMetadata()
    meta.ParseFromString(row[0])
    return meta

The trick is denormalising the couple of fields you actually filter on — here region — into real columns while keeping the full record as an opaque blob. You get indexable queries on the hot dimensions and a compact single-blob store for everything else, and the blob is already in the exact format you'll ship to the next service. No re-encoding on read. In WAL mode these blob writes don't block concurrent readers, which keeps the analytics pipeline from stalling the ingestion path.

Decoding at the Edge in a Cloudflare Worker

Because our audience is European and latency-sensitive, some responses are assembled at the Cloudflare edge. Protobuf travels well here too — you decode it in the Worker using protobufjs and emit whatever shape the browser needs, so the compact format lives all the way to the edge and only becomes JSON at the last possible moment.

import protobuf from "protobufjs";
import schema from "./metadata.v1.json"; // protobufjs JSON descriptor

const root = protobuf.Root.fromJSON(schema);
const VideoMetadata = root.lookupType("viralvidvault.metadata.v1.VideoMetadata");

export default {
  async fetch(request, env) {
    const origin = await env.META.fetch(request); // bytes from origin
    const buf = new Uint8Array(await origin.arrayBuffer());

    const meta = VideoMetadata.decode(buf);

    // Strip anything the browser has no business seeing, then hand back JSON.
    const body = JSON.stringify({
      id: meta.videoId,
      title: meta.title,
      region: meta.region,
      velocity: meta.signals?.velocity ?? 0,
    });

    return new Response(body, {
      headers: { "content-type": "application/json; charset=utf-8" },
    });
  },
};

The origin-to-edge hop stays compact, and the JSON expansion happens once, at the edge, right before it hits the browser. That is the smallest possible amount of JSON in the whole path.

Schema Evolution Without Breaking Deploys

The question everyone asks: how do you change the schema without a big-bang deploy across three languages? Proto3 makes this genuinely safe if you follow a small set of rules.

• Only add fields with new numbers. Old decoders ignore unknown fields; new decoders see the defaults for fields that old encoders didn't set.
• Never change a field's number or wire type. Renaming is fine, renumbering is not.
• Reserve retired fields. Add reserved 6; and reserved "tags"; so nobody accidentally reuses the slot.
• Deploy decoders before encoders. Roll out the readers that understand the new field first, then the writers that emit it. This ordering means there's never a moment where a new field arrives at a service that can't parse it.

Because unknown fields are preserved rather than dropped by the binary runtimes, a record can pass through a service that's a version behind and come out the other side intact — the intermediate service round-trips fields it doesn't even know about. That property alone eliminated a class of data-loss bugs we used to hit whenever the JSON contract drifted mid-deploy.

GDPR Notes Specific to This Setup

Because we operate under GDPR, the schema itself is part of our compliance story, not an afterthought:

• The gdpr_pii_stripped flag is enforced downstream, not just set upstream. The Go consumer above rejects any record where it's false. A single boolean in the contract is worth more than a policy document nobody reads.
• No raw personal identifiers ever enter VideoMetadata. We deliberately kept uploader identity out of the message. If it isn't in the schema, it can't leak through the schema.
• The binary format is not obscurity-as-security — anyone with the .proto can decode it — but keeping the compact payload internal and only expanding a whitelisted subset at the edge means the browser sees strictly less than what moves between services.

What I'd Tell Someone Starting Today

Protobuf is not free complexity. You take on a build step (protoc), a generated-code artifact per language, and payloads you can't curl | jq without a decoder. For a small app talking to itself, JSON is still the right call.

But the moment you have multiple services in multiple languages exchanging the same records at volume, the math flips hard. For us the wins were concrete: ~68% smaller payloads, 2–3x faster decode in Go, precise 64-bit counts, and — the quiet one — a schema that turns contract drift into a compile error instead of a 3am incident. If your metadata crosses a language boundary more than once, it's worth the build step.

Start with one message, one boundary, and the four evolution rules above pinned somewhere your whole team can see them. Everything else is incremental.