How Protobuf Cut Our Video Metadata Payloads by 71% Across Services

Our trend-detection pipeline was choking on JSON. Every two hours a Cloudflare Worker pulls a batch of trending videos, hands them to a Python enrichment job, which writes to a PHP 8.4 service backed by SQLite in WAL mode, which then fans the same records out to a ranking service. The video metadata record itself is small — an ID, a few timestamps, view counts, region codes, a language tag, some engagement ratios. But we were moving millions of them per day, and the JSON representation of that tiny record was costing us more than the data was worth.

The specific number that made me stop and rewrite the serialization layer: our average metadata record was 412 bytes as JSON and 119 bytes as Protobuf. That is a 71% reduction on the wire, before any gzip. At ViralVidVault we track viral video trends across the European market, and cross-region fan-out means the same record hops between three or four services before it lands. Multiply 293 saved bytes by tens of millions of records and the egress bill, the Worker CPU time, and the SQLite write amplification all move in the same direction.

This is a walkthrough of how we did it — the schema design, the PHP encoding path, the Python and Go interop, and the specific traps around evolving a schema without breaking a running fleet.

Why JSON Was the Wrong Default

JSON is the right default for a lot of things. Human-readable, self-describing, debuggable with curl. But for a fixed-shape record that machines produce and consume at high volume, every one of those virtues becomes a cost:

• Field names ship on every record. "view_count": is 13 bytes repeated on every single message. Protobuf sends a field number — one byte for the common cases.
• Numbers are text. 1483920 is 7 bytes as a JSON string of digits. As a protobuf varint it is 3 bytes, and a count of 200 is a single byte.
• No native integer/enum types. Region codes like "DE", "FR", "GB" are strings in JSON. As a protobuf enum they are one byte each.
• Parsing is expensive. JSON parsing means scanning for delimiters and allocating strings. Protobuf parsing is length-prefixed and reads fields by number.

The self-describing property is genuinely useful during development, so we kept a JSON debug endpoint. But the transport between services had no business paying that tax.

Designing the Schema

Here is the actual .proto definition we run, trimmed to the fields that matter for this article. The design choices are the interesting part, not the syntax.

syntax = "proto3";

package viralvidvault.metadata.v1;

enum Region {
  REGION_UNSPECIFIED = 0;
  REGION_DE = 1;
  REGION_FR = 2;
  REGION_GB = 3;
  REGION_ES = 4;
  REGION_IT = 5;
  REGION_NL = 6;
  REGION_PL = 7;
}

enum Platform {
  PLATFORM_UNSPECIFIED = 0;
  PLATFORM_YOUTUBE = 1;
  PLATFORM_TIKTOK = 2;
  PLATFORM_INSTAGRAM = 3;
}

message VideoMetadata {
  string video_id = 1;
  Platform platform = 2;
  Region region = 3;

  uint64 view_count = 4;
  uint32 like_count = 5;
  uint32 comment_count = 6;

  // Unix seconds. int64, not a timestamp string.
  int64 published_at = 7;
  int64 first_seen_at = 8;

  string language = 9;          // BCP-47, e.g. "de-DE"
  float velocity_score = 10;    // views/hour, our trend signal

  repeated string tag = 11;

  // Reserved for GDPR: never reuse these numbers.
  reserved 12, 13;
  reserved "uploader_email", "uploader_ip";
}

A few decisions worth calling out:

• Field numbers 1–4 are the hot path. Protobuf encodes the field number and wire type into a single "tag" byte for field numbers 1–15. Numbers 16 and up need two bytes for the tag. So the fields present on every record — video_id, platform, region, view_count — got the low numbers. Rarely-populated fields can live above 15.
• uint64 for view_count, uint32 for likes. A viral video can cross a billion views, so view_count needs 64 bits. Likes and comments realistically fit in 32. Right-sizing the integer type keeps the varint short for the common small values.
• Enums for region and platform. This is where a big chunk of the savings lives. "de-DE" region strings became one-byte enums.
• reserved for the fields we deliberately do not carry. We had two personally-identifying fields early in the design — uploader email and IP. GDPR data-minimization says we do not move those between services, so we removed them and reserved the numbers. Reserving means no future engineer can accidentally reintroduce a PII field on the same wire number that an old client might misread. That is a compliance guarantee enforced by the compiler.

Encoding in PHP 8.4

Our write service is PHP. The Google protobuf runtime for PHP is available as a C extension (fast) or a pure-PHP package (portable). On LiteSpeed with the extension installed, encoding is cheap. Here is the encode/decode path, using classes generated by protoc --php_out.

<?php
declare(strict_types=1);

use Viralvidvault\Metadata\V1\VideoMetadata;
use Viralvidvault\Metadata\V1\Region;
use Viralvidvault\Metadata\V1\Platform;

final class MetadataCodec
{
    /**
     * Build a protobuf message from an ingest row and return the
     * binary wire bytes ready to store or forward.
     */
    public function encode(array $row): string
    {
        $msg = new VideoMetadata();
        $msg->setVideoId($row['video_id']);
        $msg->setPlatform($this->platformEnum($row['platform']));
        $msg->setRegion($this->regionEnum($row['region']));
        $msg->setViewCount((int) $row['view_count']);
        $msg->setLikeCount((int) $row['like_count']);
        $msg->setCommentCount((int) $row['comment_count']);
        $msg->setPublishedAt((int) $row['published_at']);
        $msg->setFirstSeenAt($row['first_seen_at'] ?? time());
        $msg->setLanguage($row['language'] ?? '');
        $msg->setVelocityScore((float) ($row['velocity_score'] ?? 0.0));

        if (!empty($row['tags'])) {
            $msg->setTag($row['tags']); // repeated string
        }

        return $msg->serializeToString();
    }

    public function decode(string $bytes): VideoMetadata
    {
        $msg = new VideoMetadata();
        $msg->mergeFromString($bytes); // throws on malformed input
        return $msg;
    }

    private function regionEnum(string $code): int
    {
        return match (strtoupper($code)) {
            'DE' => Region::REGION_DE,
            'FR' => Region::REGION_FR,
            'GB' => Region::REGION_GB,
            'ES' => Region::REGION_ES,
            'IT' => Region::REGION_IT,
            'NL' => Region::REGION_NL,
            'PL' => Region::REGION_PL,
            default => Region::REGION_UNSPECIFIED,
        };
    }

    private function platformEnum(string $name): int
    {
        return match (strtolower($name)) {
            'youtube'   => Platform::PLATFORM_YOUTUBE,
            'tiktok'    => Platform::PLATFORM_TIKTOK,
            'instagram' => Platform::PLATFORM_INSTAGRAM,
            default     => Platform::PLATFORM_UNSPECIFIED,
        };
    }
}

The match expressions bridging our string ingest data to enums are the boring but important glue. Do this mapping in exactly one place. If you scatter string-to-enum conversion across the codebase you will eventually store REGION_UNSPECIFIED for a typo'd "de" and spend an afternoon finding out why a whole region dropped off your trend charts.

Storing Blobs in SQLite Without Losing Queryability

We run SQLite in WAL mode. The naive move is to store the whole protobuf blob in a BLOB column and call it done. That works for the transport payload, but it destroys your ability to run WHERE region = 'DE' ORDER BY velocity_score. Protobuf is opaque to SQL.

Our compromise: store the blob for fast fan-out, and denormalize the handful of columns you actually query alongside it. The blob is the source of truth for the full record; the columns are a query index.

<?php
declare(strict_types=1);

final class MetadataStore
{
    public function __construct(private \PDO $db) {}

    public function upsert(VideoMetadata $msg): void
    {
        // Denormalize only the query dimensions. The blob holds
        // the complete record for downstream services.
        $sql = <<<SQL
            INSERT INTO video_metadata
                (video_id, region, platform, velocity_score, published_at, payload)
            VALUES
                (:vid, :region, :platform, :velocity, :published, :payload)
            ON CONFLICT(video_id) DO UPDATE SET
                velocity_score = excluded.velocity_score,
                payload        = excluded.payload
            SQL;

        $stmt = $this->db->prepare($sql);
        $stmt->execute([
            ':vid'       => $msg->getVideoId(),
            ':region'    => $msg->getRegion(),      // stored as int enum
            ':platform'  => $msg->getPlatform(),
            ':velocity'  => $msg->getVelocityScore(),
            ':published' => $msg->getPublishedAt(),
            ':payload'   => $msg->serializeToString(), // BLOB column
        ]);
    }
}

The payoff of WAL mode here is that the enrichment writers and the read-heavy ranking queries do not block each other. The protobuf blob keeps the row narrow — storing a 119-byte blob instead of a 412-byte JSON text column measurably reduced page churn in our write-heavy tables, which matters when your database is a single file and every writer contends on it.

One caution: SQLite BLOBs are binary-safe, but if you ever pipe a row through something that assumes UTF-8 text (a careless CSV export, some ORMs), it will mangle the bytes. Keep the payload column strictly typed as BLOB and never SELECT payload into a text-handling code path.

Cross-Service Interop: Python Enrichment, Go Ranking

The whole point of protobuf is that the same schema generates types in every language, and the wire bytes are identical. Our enrichment step is Python. It reads the raw ingest, computes the velocity score, and emits protobuf bytes that the PHP service stores verbatim.

import time
from metadata.v1 import metadata_pb2 as pb


def enrich(raw: dict) -> bytes:
    msg = pb.VideoMetadata()
    msg.video_id = raw["video_id"]
    msg.platform = pb.PLATFORM_YOUTUBE
    msg.region = pb.Region.Value(f"REGION_{raw['region'].upper()}")
    msg.view_count = int(raw["view_count"])
    msg.like_count = int(raw.get("like_count", 0))
    msg.published_at = int(raw["published_at"])
    msg.first_seen_at = int(time.time())
    msg.language = raw.get("language", "")

    # Trend signal: views accrued per hour since we first saw it.
    age_hours = max((time.time() - msg.first_seen_at) / 3600.0, 1.0)
    msg.velocity_score = msg.view_count / age_hours

    msg.tag.extend(raw.get("tags", []))
    return msg.SerializeToString()


def parse(blob: bytes) -> pb.VideoMetadata:
    msg = pb.VideoMetadata()
    msg.ParseFromString(blob)  # raises DecodeError on garbage
    return msg

The ranking service is Go, chosen for its concurrency when scoring large batches. It reads the same blobs and never once has to agree with PHP or Python on a JSON field name — the schema is the contract.

package ranking

import (
    "sort"

    "google.golang.org/protobuf/proto"
    pb "viralvidvault/gen/metadata/v1"
)

// DecodeBatch turns stored blobs into typed messages, skipping
// any record that fails to parse rather than aborting the batch.
func DecodeBatch(blobs [][]byte) []*pb.VideoMetadata {
    out := make([]*pb.VideoMetadata, 0, len(blobs))
    for _, b := range blobs {
        var m pb.VideoMetadata
        if err := proto.Unmarshal(b, &m); err != nil {
            continue // one bad record must not sink the batch
        }
        out = append(out, &m)
    }
    return out
}

// TopByVelocity returns the n hottest videos for a region.
func TopByVelocity(msgs []*pb.VideoMetadata, region pb.Region, n int) []*pb.VideoMetadata {
    filtered := msgs[:0:0]
    for _, m := range msgs {
        if m.GetRegion() == region {
            filtered = append(filtered, m)
        }
    }
    sort.Slice(filtered, func(i, j int) bool {
        return filtered[i].GetVelocityScore() > filtered[j].GetVelocityScore()
    })
    if len(filtered) > n {
        filtered = filtered[:n]
    }
    return filtered
}

Three languages, one schema, byte-identical wire format. When we add a field, we regenerate the bindings for all three from the single .proto, and old services keep working because of how proto3 handles unknown fields — which is the next thing to get right.

Schema Evolution Without Breaking the Fleet

This is where teams get burned. You cannot deploy PHP, Python, and Go simultaneously across a running system. There is always a window where an old encoder talks to a new decoder or vice versa. Protobuf handles this gracefully only if you follow the rules:

• Never change a field's number. The number is the identity. The name is cosmetic. Rename freely; renumber never.
• Never change a field's type in an incompatible way. Going uint32 → uint64 is safe (both varints). Going int32 → string is a corruption bug waiting to happen.
• New fields must be optional in spirit. In proto3 every scalar has a default. A new field added by the encoder is simply ignored by an old decoder — it lands in the unknown-fields set and is preserved on re-serialization. That last part matters: an old service that reads and re-writes a record will carry forward fields it does not understand, so it will not strip data added by a newer service.
• Deleting a field means reserving its number. As shown with the GDPR fields above, reserved 12, 13; prevents reuse. Reusing a deleted number is the single most common way to silently corrupt data across a version skew.
• Roll out decoders before encoders. Deploy the services that read the new field first. Once every reader understands field 14, start writing it. Reverse that order and your new field vanishes into unknown-fields on services that have not caught up.

We encode a schema version into a Cloudflare Worker header (X-Meta-Schema: v1) so we can spot version skew in logs during a rollout, but the wire compatibility does not depend on it — that header is an observability aid, not a gate.

What We Measured Afterward

Numbers from our own pipeline after the migration, averaged over a week of production traffic:

• Payload size: 412 B → 119 B per record (−71%), before compression. After gzip the gap narrows but protobuf still wins by ~40% because there is less redundant text for the compressor to exploit.
• Encode/decode CPU: the PHP protobuf C extension encoded roughly 3× faster than json_encode on our record shape, and decoding in Go was not even a measurable line item.
• SQLite write throughput: narrower rows meant fewer page splits; our two-hourly ingest batch finished noticeably quicker under WAL, with less checkpoint pressure.
• Cloudflare Worker egress: the fan-out between services carries a fraction of the bytes, which on a per-request-billed edge platform is a direct cost line.

The non-numeric win is the one I value most: the schema is now a real contract. When someone adds a field, the compiler forces every service to agree on its number and type. The class of bug where PHP sends viewCount and Go expects view_count is simply gone.

When Not to Reach for Protobuf

To be fair to JSON, protobuf is the wrong choice when:

• The payload is consumed by a browser or a public API where human-readability and ubiquitous tooling matter more than bytes.
• The data shape is genuinely dynamic — arbitrary user-defined keys — where a schema fights you instead of helping.
• The volume is low enough that the bytes never show up on any bill or latency graph. Do not add a code-generation step to move a thousand records a day.

For internal, high-volume, fixed-shape machine-to-machine traffic — which is exactly what video metadata fan-out is — the trade lands firmly on protobuf's side.

Conclusion

The migration came down to a few concrete moves: put the hot-path fields on numbers 1–15, turn repeated string codes into enums, right-size the integer types, store the binary blob in SQLite alongside a thin set of denormalized query columns, and treat field numbers as immutable while reserving anything you delete. The 71% payload reduction was the headline, but the durable win was turning a loose JSON convention into a compiler-enforced contract shared by PHP, Python, and Go. If your services are shipping the same field names over and over on every record, that repetition is a bill you are paying — and protobuf is a well-worn way to stop paying it.