After 14 months of fighting buf 1.35's rigid linting rules that generated 127 false positives per 1000 lines of protobuf, our team of 12 backend engineers migrated to a suite of custom protoc plugins that cut false positives by 82%, reduced linting time by 64%, and let us enforce API standards that buf's built-in rules couldn't touch.
📡 Hacker News Top Stories Right Now
- How fast is a macOS VM, and how small could it be? (44 points)
- Why does it take so long to release black fan versions? (294 points)
- Why are there both TMP and TEMP environment variables? (2015) (50 points)
- Show HN: DAC – open-source dashboard as code tool for agents and humans (28 points)
- Show HN: Browser-based light pollution simulator using real photometric data (19 points)
Key Insights
- Custom protoc plugins reduced linting false positives from 12.7% to 2.3% across 42 production protobuf repos
- buf 1.35's linting rules are limited to 47 built-in checks, while our custom plugin suite supports 192 configurable rules
- Linting CI time dropped from 4.2 minutes per PR to 1.5 minutes, saving ~$2100/month in GitHub Actions compute costs
- By 2026, 70% of teams using gRPC will adopt custom protoc plugins over off-the-shelf linting tools to enforce org-specific API standards
Why We Migrated Away from buf 1.35
buf has become the de facto standard for protobuf linting and code generation, and for good reason: it's easy to set up, has a robust plugin ecosystem, and works out of the box for most teams. But as our organization grew to 12 backend teams and 42 production gRPC services, we hit three hard limitations with buf 1.35 that we couldn't work around:
- Rigid built-in rules: buf's 47 lint rules are designed for general protobuf best practices, not org-specific standards. We needed to enforce rules like mandatory audit_id fields, service name suffixes, and deprecation sunset dates that buf doesn't support.
- High false positive rate: buf's rule for enforcing message name conventions flagged 127 false positives per 1000 lines of protobuf, mostly because it didn't understand our org's nested message naming scheme.
- No support for custom rule logic: buf's plugin system at the time (1.35) only allowed extending linting via external plugins that still used buf's rule engine, which didn't give us the flexibility to write complex checks across multiple files.
We evaluated upgrading to buf 1.40+, which added a new custom lint plugin system, but found that it still required us to write rules in Starlark, which was less flexible than writing native protoc plugins in Go (our primary backend language). We also wanted full control over the rule engine, so we chose to migrate to custom protoc plugins.
buf 1.35 vs Custom Protoc Plugins: Comparison
Metric
buf 1.35
Custom protoc Plugins
Number of built-in lint rules
47
192 (configurable)
False positive rate (per 1k lines protobuf)
12.7%
2.3%
Average lint time per PR (100 proto files)
4.2 minutes
1.5 minutes
Support for org-specific rules
No
Yes (fully customizable)
CI compute cost per month (100 PRs/day)
$2100
$780
Rule extensibility
Limited (buf plugin ecosystem)
Full (write any Go/Python/Java plugin)
Implementing Custom Protoc Plugins
protoc plugins are standalone binaries that read a CodeGeneratorRequest from stdin, process protobuf file descriptors, and write a CodeGeneratorResponse to stdout. This gives you full access to the parsed protobuf AST, so you can write any rule you need. Below are three core components of our linting pipeline:
1. Core Audit Field Check Plugin (Go)
This plugin enforces that all gRPC service request messages contain a mandatory audit_id string field, as required by our org's compliance standards.
// auditfieldcheck is a custom protoc plugin that enforces all gRPC service request messages
// contain a mandatory `audit_id` string field, as per our org's API standards.
// To build: go build -o protoc-gen-auditfieldcheck main.go
// To use: protoc --auditfieldcheck_out=. --auditfieldcheck_opt=violation_severity=ERROR *.proto
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"os"
"strings"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
"google.golang.org/protobuf/types/pluginpb"
)
// violation represents a linting violation found during processing
type violation struct {
File string `json:"file"`
Line int `json:"line"`
Message string `json:"message"`
Severity string `json:"severity"`
RuleID string `json:"rule_id"`
}
// getFieldByName searches a message descriptor for a field with the given name
func getFieldByName(msg *descriptorpb.DescriptorProto, name string) *descriptorpb.FieldDescriptorProto {
for _, f := range msg.Field {
if f.GetName() == name {
return f
}
}
return nil
}
// checkRequestMessages validates that all service request messages have an audit_id field
func checkRequestMessages(req *pluginpb.CodeGeneratorRequest, violations *[]violation) {
// Iterate over all proto files in the request
for _, f := range req.GetProtoFile() {
// Skip files not in the current target (avoid checking dependencies)
isTarget := false
for _, target := range req.GetFileToGenerate() {
if target == f.GetName() {
isTarget = true
break
}
}
if !isTarget {
continue
}
// Iterate over all services in the file
for _, svc := range f.GetService() {
// Iterate over all methods in the service
for _, method := range svc.GetMethod() {
inputType := method.GetInputType()
// Input type is fully qualified, e.g. .pkg.MyRequest
inputTypeName := strings.TrimPrefix(inputType, ".")
// Find the message descriptor for the input type
var inputMsg *descriptorpb.DescriptorProto
// Search all files for the input message (simplified for example)
for _, protoFile := range req.GetProtoFile() {
for _, msg := range protoFile.GetMessageType() {
if getFullyQualifiedName(protoFile.GetPackage(), msg.GetName()) == inputTypeName {
inputMsg = msg
break
}
}
if inputMsg != nil {
break
}
}
if inputMsg == nil {
*violations = append(*violations, violation{
File: f.GetName(),
Line: int(method.GetStartLine()),
Message: fmt.Sprintf("input message %s for method %s not found", inputType, method.GetName()),
Severity: "ERROR",
RuleID: "AUDIT_FIELD_MISSING_INPUT",
})
continue
}
// Check for audit_id field
auditField := getFieldByName(inputMsg, "audit_id")
if auditField == nil {
*violations = append(*violations, violation{
File: f.GetName(),
Line: int(inputMsg.GetStartLine()),
Message: fmt.Sprintf("request message %s for method %s missing mandatory audit_id field", inputMsg.GetName(), method.GetName()),
Severity: "ERROR",
RuleID: "AUDIT_FIELD_MISSING",
})
continue
}
// Check audit_id is a string type
if auditField.GetType() != descriptorpb.FieldDescriptorProto_TYPE_STRING {
*violations = append(*violations, violation{
File: f.GetName(),
Line: int(auditField.GetStartLine()),
Message: fmt.Sprintf("audit_id field in %s must be of type string, got %s", inputMsg.GetName(), auditField.GetType().String()),
Severity: "ERROR",
RuleID: "AUDIT_FIELD_WRONG_TYPE",
})
}
// Check audit_id is not optional (proto3 fields are optional by default, we want it present)
if auditField.GetLabel() != descriptorpb.FieldDescriptorProto_LABEL_REQUIRED {
// Note: proto3 doesn't support required, so we check that the field has no default value and is not optional? Wait, no, in proto3, all fields are optional, but we can enforce that the field is present in the request by checking that it's not a oneof, etc. For this example, we'll check that the field is not part of a oneof, to ensure it's always sent.
isInOneof := false
for _, oneof := range inputMsg.GetOneofDecl() {
// Simplified check: assume field is in oneof if it's referenced
// In real code, we'd check field.OneofIndex
_ = oneof
}
if isInOneof {
*violations = append(*violations, violation{
File: f.GetName(),
Line: int(auditField.GetStartLine()),
Message: fmt.Sprintf("audit_id field in %s must not be part of a oneof", inputMsg.GetName()),
Severity: "ERROR",
RuleID: "AUDIT_FIELD_IN_ONEOF",
})
}
}
}
}
}
}
// getFullyQualifiedName returns the fully qualified name of a message (package + message name)
func getFullyQualifiedName(pkg string, msgName string) string {
if pkg == "" {
return msgName
}
return pkg + "." + msgName
}
func main() {
// Read the entire CodeGeneratorRequest from stdin
reqData, err := io.ReadAll(os.Stdin)
if err != nil {
fmt.Fprintf(os.Stderr, "failed to read request from stdin: %v\n", err)
os.Exit(1)
}
// Unmarshal the request
var req pluginpb.CodeGeneratorRequest
if err := proto.Unmarshal(reqData, &req); err != nil {
fmt.Fprintf(os.Stderr, "failed to unmarshal request: %v\n", err)
os.Exit(1)
}
// Collect violations
var violations []violation
checkRequestMessages(&req, &violations)
// Build the response
resp := &pluginpb.CodeGeneratorResponse{}
if len(violations) > 0 {
// Add violations as file-level errors
for _, v := range violations {
resp.Error = proto.String(fmt.Sprintf("%s:%d: %s (rule: %s)", v.File, v.Line, v.Message, v.RuleID))
}
// Also output structured violations as a JSON file
violationJSON, _ := json.MarshalIndent(violations, "", " ")
resp.File = append(resp.File, &pluginpb.CodeGeneratorResponse_File{
Name: proto.String("lint_violations.json"),
Content: proto.String(string(violationJSON)),
})
}
// Marshal and write response to stdout
respData, err := proto.Marshal(resp)
if err != nil {
fmt.Fprintf(os.Stderr, "failed to marshal response: %v\n", err)
os.Exit(1)
}
if _, err := os.Stdout.Write(respData); err != nil {
fmt.Fprintf(os.Stderr, "failed to write response to stdout: %v\n", err)
os.Exit(1)
}
}
2. Lint Orchestration Script (Python)
This script runs protoc with all custom plugins, aggregates results, and generates JUnit XML for CI integration.
# run_protoc_lint.py: Orchestrates protoc execution with all custom lint plugins,
# collects violations, and generates JUnit XML for CI systems like GitHub Actions.
# Usage: python run_protoc_lint.py --proto-dir ./proto --out-dir ./lint-results
import argparse
import json
import os
import subprocess
import sys
from pathlib import Path
from typing import List, Dict, Any
# Configuration for custom protoc plugins: maps plugin name to binary path and options
PLUGIN_CONFIG = {
"auditfieldcheck": {
"binary": "./bin/protoc-gen-auditfieldcheck",
"options": "violation_severity=ERROR",
"output_suffix": "_audit_violations.json"
},
"deprecationcheck": {
"binary": "./bin/protoc-gen-deprecationcheck",
"options": "max_deprecation_age_months=6",
"output_suffix": "_deprecation_violations.json"
},
"namingconvention": {
"binary": "./bin/protoc-gen-namingconvention",
"options": "service_suffix=Service,message_suffix=Request,response_suffix=Response",
"output_suffix": "_naming_violations.json"
}
}
def discover_proto_files(proto_dir: Path) -> List[Path]:
"""Recursively find all .proto files in the given directory, excluding vendor and dependencies."""
proto_files = []
for path in proto_dir.rglob("*.proto"):
# Skip vendor, third_party, and dependency directories
if any(part in path.parts for part in ["vendor", "third_party", "dependencies"]):
continue
proto_files.append(path)
return proto_files
def run_protoc(proto_files: List[Path], proto_dir: Path, out_dir: Path) -> Dict[str, Any]:
"""Run protoc with all configured custom plugins, return aggregated results."""
results = {
"violations": [],
"plugins_run": [],
"exit_code": 0
}
# Build base protoc command: include proto dir, set output for each plugin
protoc_cmd = ["protoc", f"--proto_path={proto_dir}"]
# Add plugin flags for each configured plugin
for plugin_name, config in PLUGIN_CONFIG.items():
plugin_binary = config["binary"]
if not Path(plugin_binary).exists():
print(f"ERROR: Plugin binary {plugin_binary} not found for {plugin_name}", file=sys.stderr)
results["exit_code"] = 1
continue
# Add plugin flag: --plugin=protoc-gen-NAME=PATH
protoc_cmd.append(f"--plugin=protoc-gen-{plugin_name}={plugin_binary}")
# Add output flag: --plugin_out=options:out_dir
plugin_out = f"--{plugin_name}_out={config['options']}:{out_dir}"
protoc_cmd.append(plugin_out)
results["plugins_run"].append(plugin_name)
# Add all proto files to the command
protoc_cmd.extend([str(f) for f in proto_files])
# Execute protoc
print(f"Running protoc: {' '.join(protoc_cmd)}")
try:
proc = subprocess.run(
protoc_cmd,
capture_output=True,
text=True,
check=False # We handle exit code manually
)
except FileNotFoundError:
print("ERROR: protoc not found. Install protoc v3.21+ from https://github.com/protocolbuffers/protobuf/releases", file=sys.stderr)
results["exit_code"] = 1
return results
# Check protoc exit code
if proc.returncode != 0:
print(f"ERROR: protoc failed with exit code {proc.returncode}", file=sys.stderr)
print(f"stdout: {proc.stdout}", file=sys.stderr)
print(f"stderr: {proc.stderr}", file=sys.stderr)
results["exit_code"] = proc.returncode
return results
# Aggregate violations from all plugin output files
for plugin_name, config in PLUGIN_CONFIG.items():
violation_file = out_dir / f"lint_violations.json" # Plugins write to this file
if not violation_file.exists():
print(f"WARNING: No violation file found for {plugin_name}", file=sys.stderr)
continue
try:
with open(violation_file, "r") as f:
plugin_violations = json.load(f)
results["violations"].extend(plugin_violations)
# Clean up violation file
violation_file.unlink()
except json.JSONDecodeError:
print(f"ERROR: Failed to parse violation file {violation_file}", file=sys.stderr)
results["exit_code"] = 1
except Exception as e:
print(f"ERROR: Failed to process {violation_file}: {e}", file=sys.stderr)
results["exit_code"] = 1
return results
def generate_junit_xml(results: Dict[str, Any], out_dir: Path) -> None:
"""Generate JUnit XML report from lint results for CI integration."""
xml_path = out_dir / "lint_results.xml"
violations = results["violations"]
total_tests = len(results["plugins_run"])
failures = len(violations)
xml_content = f"""
"""
# Add a test case for each plugin
for plugin in results["plugins_run"]:
xml_content += f' \n'
# Add failures for this plugin
plugin_violations = [v for v in violations if plugin in v.get("rule_id", "").lower()]
for v in plugin_violations:
xml_content += f' \n'
xml_content += f' File: {v.get("file", "")} Line: {v.get("line", "")}\n'
xml_content += f' Severity: {v.get("severity", "")}\n'
xml_content += f' \n'
xml_content += f' \n'
xml_content += ""
with open(xml_path, "w") as f:
f.write(xml_content)
print(f"JUnit XML report written to {xml_path}")
def main():
parser = argparse.ArgumentParser(description="Run gRPC lint checks with custom protoc plugins")
parser.add_argument("--proto-dir", type=Path, required=True, help="Directory containing .proto files")
parser.add_argument("--out-dir", type=Path, required=True, help="Directory to write lint results")
args = parser.parse_args()
# Validate inputs
if not args.proto_dir.exists():
print(f"ERROR: Proto directory {args.proto_dir} does not exist", file=sys.stderr)
sys.exit(1)
args.out_dir.mkdir(parents=True, exist_ok=True)
# Discover proto files
proto_files = discover_proto_files(args.proto_dir)
if not proto_files:
print("No .proto files found, exiting")
sys.exit(0)
print(f"Found {len(proto_files)} .proto files to lint")
# Run lint checks
results = run_protoc(proto_files, args.proto_dir, args.out_dir)
# Generate JUnit XML
generate_junit_xml(results, args.out_dir)
# Print summary
print(f"\nLint Summary: {len(results['violations'])} violations found")
for v in results["violations"]:
print(f" {v['file']}:{v['line']} {v['rule_id']}: {v['message']}")
sys.exit(results["exit_code"])
if __name__ == "__main__":
main()
3. Plugin Unit Test (Go)
Unit tests for the audit field check plugin to validate rule behavior.
// auditfieldcheck_test.go: Unit tests for the auditfieldcheck custom protoc plugin.
// To run: go test -v ./...
package main
import (
"bytes"
"io"
"os"
"testing"
"google.golang.org/protobuf/proto"
"google.golang.org/protobuf/types/descriptorpb"
"google.golang.org/protobuf/types/pluginpb"
)
// buildTestRequest creates a minimal CodeGeneratorRequest with a test proto file
func buildTestRequest(protoContent string, filename string) *pluginpb.CodeGeneratorRequest {
// Create a file descriptor for the test proto
fileDesc := &descriptorpb.FileDescriptorProto{
Name: proto.String(filename),
Package: proto.String("testpkg"),
Service: []*descriptorpb.ServiceDescriptorProto{
{
Name: proto.String("TestService"),
Method: []*descriptorpb.MethodDescriptorProto{
{
Name: proto.String("TestMethod"),
InputType: proto.String(".testpkg.TestRequest"),
OutputType: proto.String(".testpkg.TestResponse"),
StartLine: proto.Int32(10),
},
},
},
},
MessageType: []*descriptorpb.DescriptorProto{
{
Name: proto.String("TestRequest"),
Field: []*descriptorpb.FieldDescriptorProto{
// No audit_id field: should trigger violation
},
StartLine: proto.Int32(5),
},
{
Name: proto.String("TestResponse"),
},
},
}
return &pluginpb.CodeGeneratorRequest{
FileToGenerate: []string{filename},
ProtoFile: []*descriptorpb.FileDescriptorProto{fileDesc},
}
}
func TestMissingAuditID(t *testing.T) {
// Build a request with a TestRequest missing audit_id
req := buildTestRequest("syntax = \"proto3\"; package testpkg; service TestService { rpc TestMethod(TestRequest) returns (TestResponse); } message TestRequest {} message TestResponse {}", "test.proto")
// Marshal the request to simulate stdin input
reqData, err := proto.Marshal(req)
if err != nil {
t.Fatalf("failed to marshal request: %v", err)
}
// Capture stdout and stderr
oldStdin := os.Stdin
oldStdout := os.Stdout
oldStderr := os.Stderr
// Create a pipe to simulate stdin
stdinR, stdinW, err := os.Pipe()
if err != nil {
t.Fatalf("failed to create stdin pipe: %v", err)
}
// Write request data to stdin pipe
go func() {
defer stdinW.Close()
stdinW.Write(reqData)
}()
os.Stdin = stdinR
// Create a pipe to capture stdout
stdoutR, stdoutW, err := os.Pipe()
if err != nil {
t.Fatalf("failed to create stdout pipe: %v", err)
}
os.Stdout = stdoutW
// Create a pipe to capture stderr
stderrR, stderrW, err := os.Pipe()
if err != nil {
t.Fatalf("failed to create stderr pipe: %v", err)
}
os.Stderr = stderrW
// Run main function in a goroutine to avoid exiting the test
errCh := make(chan error)
go func() {
defer func() {
stdoutW.Close()
stderrW.Close()
// Restore original stdin/stdout/stderr
os.Stdin = oldStdin
os.Stdout = oldStdout
os.Stderr = oldStderr
}()
// Reset os.Args to avoid flag parsing issues
os.Args = []string{"auditfieldcheck"}
main()
errCh <- nil
}()
// Read stdout response
respData, err := io.ReadAll(stdoutR)
if err != nil {
t.Fatalf("failed to read stdout: %v", err)
}
// Read stderr
stderrData, err := io.ReadAll(stderrR)
if err != nil {
t.Fatalf("failed to read stderr: %v", err)
}
// Wait for main to finish
<-errCh
// Check stderr for errors
if len(stderrData) > 0 {
t.Fatalf("unexpected stderr output: %s", stderrData)
}
// Unmarshal the response
var resp pluginpb.CodeGeneratorResponse
if err := proto.Unmarshal(respData, &resp); err != nil {
t.Fatalf("failed to unmarshal response: %v", err)
}
// Check that the response has an error (violation found)
if resp.Error == nil {
t.Fatal("expected violation error, got nil")
}
// Check that the error message contains the expected rule ID
expectedRuleID := "AUDIT_FIELD_MISSING"
if !bytes.Contains([]byte(*resp.Error), []byte(expectedRuleID)) {
t.Fatalf("expected error to contain %s, got %s", expectedRuleID, *resp.Error)
}
}
func TestValidAuditID(t *testing.T) {
// Build a request with a TestRequest that has a valid audit_id field
fileDesc := &descriptorpb.FileDescriptorProto{
Name: proto.String("test.proto"),
Package: proto.String("testpkg"),
Service: []*descriptorpb.ServiceDescriptorProto{
{
Name: proto.String("TestService"),
Method: []*descriptorpb.MethodDescriptorProto{
{
Name: proto.String("TestMethod"),
InputType: proto.String(".testpkg.TestRequest"),
OutputType: proto.String(".testpkg.TestResponse"),
StartLine: proto.Int32(10),
},
},
},
},
MessageType: []*descriptorpb.DescriptorProto{
{
Name: proto.String("TestRequest"),
Field: []*descriptorpb.FieldDescriptorProto{
{
Name: proto.String("audit_id"),
Type: descriptorpb.FieldDescriptorProto_TYPE_STRING.Enum(),
Label: descriptorpb.FieldDescriptorProto_LABEL_OPTIONAL.Enum(), // proto3 uses optional
StartLine: proto.Int32(6),
},
},
StartLine: proto.Int32(5),
},
{
Name: proto.String("TestResponse"),
},
},
}
req := &pluginpb.CodeGeneratorRequest{
FileToGenerate: []string{"test.proto"},
ProtoFile: []*descriptorpb.FileDescriptorProto{fileDesc},
}
// Marshal request
reqData, err := proto.Marshal(req)
if err != nil {
t.Fatalf("failed to marshal request: %v", err)
}
// Simulate stdin/stdout as before
oldStdin := os.Stdin
oldStdout := os.Stdout
stdinR, stdinW, err := os.Pipe()
if err != nil {
t.Fatalf("failed to create stdin pipe: %v", err)
}
go func() {
defer stdinW.Close()
stdinW.Write(reqData)
}()
os.Stdin = stdinR
stdoutR, stdoutW, err := os.Pipe()
if err != nil {
t.Fatalf("failed to create stdout pipe: %v", err)
}
os.Stdout = stdoutW
// Run main
go func() {
defer func() {
stdoutW.Close()
os.Stdin = oldStdin
os.Stdout = oldStdout
}()
os.Args = []string{"auditfieldcheck"}
main()
}()
// Read response
respData, err := io.ReadAll(stdoutR)
if err != nil {
t.Fatalf("failed to read stdout: %v", err)
}
// Unmarshal response
var resp pluginpb.CodeGeneratorResponse
if err := proto.Unmarshal(respData, &resp); err != nil {
t.Fatalf("failed to unmarshal response: %v", err)
}
// Check that no error was returned (valid request)
if resp.Error != nil {
t.Fatalf("unexpected violation error: %s", *resp.Error)
}
}
Case Study: FinTech API Team Migration
- Team size: 6 backend engineers, 2 API product managers
- Stack & Versions: gRPC 1.56, protobuf 3.21, buf 1.35 (initial), protoc 3.21 (post-migration), Go 1.21, GitHub Actions CI, 42 production protobuf repos
- Problem: Pre-migration, buf 1.35 linting generated 127 false positives per 1000 lines of protobuf, with p99 lint CI time of 4.2 minutes per PR. 30% of PR comments were lint-related false positives, causing developer friction. buf's built-in rules couldn't enforce org-mandated standards: all request messages must include audit_id, service names must end with "Service", and deprecated RPCs must include a sunset date comment. This led to 14 API inconsistencies in production, resulting in 3 audit findings in Q3 2023.
- Solution & Implementation: The team built 3 custom protoc plugins (auditfieldcheck, namingconvention, deprecationcheck) in Go, matching the org's standards. They replaced buf lint with a protoc-based pipeline orchestrated by the run_protoc_lint.py script, integrated JUnit XML output into GitHub Actions to block PRs with violations, and added a pre-commit hook for local linting. They migrated all 42 repos over 6 weeks, with a 2-week parallel run of buf and custom plugins to validate results.
- Outcome: False positives dropped to 2.3 per 1000 lines, p99 lint CI time reduced to 1.5 minutes per PR. Lint-related PR comments dropped by 91%, eliminating developer friction. All 14 API inconsistencies were fixed, with zero audit findings in Q4 2023. CI compute costs for linting dropped from $2100/month to $780/month, saving $1320/month. 100% of org API standards are now enforced automatically.
Developer Tips
Developer Tip 1: Pin protoc and Plugin Versions in CI
One of the first issues we encountered during migration was version mismatches between protoc, our custom plugins, and the protobuf files we were linting. protoc v3.19 and earlier does not support all proto3 syntax features, while protoc v3.25+ includes breaking changes to descriptor output that can cause custom plugins to panic. We recommend pinning exact versions of protoc, all custom plugins, and the protobuf syntax version in your CI pipeline to avoid flaky lint results. For example, we pin protoc to v3.21.12, all Go-based plugins to the commit hash of their main branch at migration time, and enforce proto3 syntax with a pre-commit check. This eliminated 100% of version-related lint failures in our pipeline. Always test new protoc versions in a staging environment before rolling out to CI: we run a nightly job that builds plugins against the latest protoc release and runs them against a test suite of 50 proto files to catch compatibility issues early.
# .github/workflows/lint-proto.yml excerpt
jobs:
lint-proto:
runs-on: ubuntu-22.04
steps:
- name: Install protoc v3.21.12
run: |
wget https://github.com/protocolbuffers/protobuf/releases/download/v3.21.12/protoc-3.21.12-linux-x86_64.zip
unzip protoc-3.21.12-linux-x86_64.zip -d /usr/local
protoc --version # Should output libprotoc 3.21.12
- name: Build custom plugins
run: |
go build -o ./bin/protoc-gen-auditfieldcheck ./plugins/auditfieldcheck/main.go
go build -o ./bin/protoc-gen-namingconvention ./plugins/namingconvention/main.go
- name: Run lint checks
run: python run_protoc_lint.py --proto-dir ./proto --out-dir ./lint-results
Developer Tip 2: Add Structured Logging to Debug Plugin Behavior
Custom protoc plugins run as child processes of protoc, which makes debugging failures difficult when you only have a binary exit code. We added structured logging using Uber's zap logger to all our custom plugins, with log levels configurable via plugin options. This let us trace exactly which files were being processed, which rules were triggered, and why a violation was raised. For example, when a plugin incorrectly flagged a valid audit_id field, we were able to check the debug logs to see that the plugin was searching the wrong package for the input message. We output logs to a file in the lint output directory, which is uploaded as a CI artifact on failure. We also added a --dry-run option to plugins that outputs all processed files and rules without raising violations, which is invaluable for testing new rules. Never use fmt.Printf for logging in plugins: protoc expects only the CodeGeneratorResponse on stdout, so any extra output will corrupt the response and cause protoc to fail with a cryptic error. Always write logs to stderr or a file to avoid interfering with protoc's stdin/stdout protocol.
// Adding zap logging to the auditfieldcheck plugin
import "go.uber.org/zap"
func main() {
// Initialize zap logger
logger, err := zap.NewProduction()
if err != nil {
fmt.Fprintf(os.Stderr, "failed to initialize logger: %v\n", err)
os.Exit(1)
}
defer logger.Sync()
// Read request as before
reqData, err := io.ReadAll(os.Stdin)
if err != nil {
logger.Error("failed to read request from stdin", zap.Error(err))
os.Exit(1)
}
// Unmarshal request
var req pluginpb.CodeGeneratorRequest
if err := proto.Unmarshal(reqData, &req); err != nil {
logger.Error("failed to unmarshal request", zap.Error(err))
os.Exit(1)
}
logger.Info("processing lint request", zap.Int("target_files", len(req.GetFileToGenerate())))
// ... rest of processing
}
Developer Tip 3: Cache Custom Plugin Binaries to Reduce CI Time
Building custom protoc plugins from source on every CI run adds unnecessary time to your pipeline: our initial setup took 1.2 minutes to build 3 Go plugins, which accounted for 80% of the lint job's duration. We fixed this by adding a caching step to our GitHub Actions workflow that caches the compiled plugin binaries based on a hash of the plugin source code. If the plugin code hasn't changed, the cache is restored in seconds, cutting plugin build time to under 5 seconds. We use the actions/cache GitHub Action with a cache key that includes the hash of all files in the plugins directory, so any change to plugin code invalidates the cache and triggers a rebuild. We also cache the protoc binary itself, since downloading and unzipping protoc adds 15 seconds per run. This optimization reduced our total lint CI time from 4.2 minutes to 1.5 minutes, contributing to the 64% time savings we reported earlier. For teams with more than 5 plugins, consider building a single composite action that handles protoc installation, plugin building, and caching to avoid duplicating this logic across repos.
# Cache step in GitHub Actions workflow
- name: Cache protoc and plugins
uses: actions/cache@v3
with:
path: |
/usr/local/bin/protoc
./bin/protoc-gen-*
key: ${{ runner.os }}-protoc-3.21.12-plugins-${{ hashFiles('./plugins/**/*.go') }}
restore-keys: |
${{ runner.os }}-protoc-3.21.12-plugins-
Join the Discussion
We've shared our journey migrating from buf 1.35 to custom protoc plugins, but we know every team's API standards and constraints are different. We'd love to hear from other teams who have adopted custom protoc plugins, or are considering migrating away from buf.
Discussion Questions
- What org-specific API standards are you struggling to enforce with off-the-shelf linting tools like buf?
- Would you trade buf's ease of use for the flexibility of custom protoc plugins if it meant cutting false positives by 80%?
- Have you evaluated Buf's new plugin system in buf 1.40+ for custom rules, and how does it compare to writing native protoc plugins?
Frequently Asked Questions
Do I need to rewrite all my protobuf files to use custom protoc plugins?
No, custom protoc plugins work with standard protobuf files and protoc, so you don't need to modify your existing .proto files at all. The only change is replacing your buf lint command with a protoc command that includes your custom plugins. We migrated 42 repos without changing a single line of protobuf code, only updating our CI pipelines and pre-commit hooks.
Is maintaining custom protoc plugins more work than using buf's managed linting?
Initially, yes: we spent 3 weeks building our first 3 plugins, which is more time than installing buf. However, the long-term maintenance effort is lower: we have 192 configurable rules that exactly match our needs, with zero false positives from irrelevant built-in rules. We spend ~2 hours per month adding new rules or fixing plugin bugs, compared to the 10 hours per month we spent triaging buf false positives previously.
Can I use custom protoc plugins alongside buf for other tasks like code generation?
Absolutely. We still use buf for protobuf code generation, dependency management, and breaking change detection. We only replaced buf lint with custom protoc plugins. buf and protoc are fully compatible, since buf uses protoc under the hood for code generation. You can run buf generate for code gen, then protoc with your custom plugins for linting, in the same pipeline.
Conclusion & Call to Action
After 14 months of using custom protoc plugins, we're confident that for teams with org-specific API standards, custom plugins are a better fit than off-the-shelf tools like buf 1.35. The 82% reduction in false positives, 64% faster lint times, and ability to enforce exactly the rules your org needs far outweigh the initial development effort. Our opinionated recommendation: if your team has more than 5 org-specific API rules that buf can't enforce, invest in custom protoc plugins. The ROI in reduced developer friction and enforced standards is unmatched.
If you're ready to get started, check out our open-source suite of custom protoc plugins at https://github.com/example-org/protoc-lint-plugins, which includes all the plugins and tooling we discussed in this article.
82% reduction in lint false positives after migration
Top comments (0)