How to mask PII in Kubernetes before sending logs to Datadog

#datalog #kubernetes #log

The Problem: Datadog Bills and GDPR Nightmares

If you are running applications in Kubernetes and shipping your logs to Datadog, you have probably faced two major headaches:

Cost: Datadog charges by the volume of logs ingested and indexed. Every megabyte counts. And while Datadog offers a built-in Sensitive Data Scanner, it is a premium feature billed separately on top of your base log costs. A free, open-source sidecar lets you bypass the vendor-side premium scrubber entirely.
Compliance: Sending Personally Identifiable Information (PII) like emails, credit card numbers, or API keys to a third-party logging service often violates GDPR and other privacy laws.

The more comprehensive your logs are for debugging, the higher your Datadog bill gets, and the bigger your risk of a privacy breach becomes.

The Standard Approach (And Why It Hurts)

The usual fix is to configure the Datadog Agent to mask or scrub PII before it leaves your cluster. That approach has real drawbacks:

Complexity: Custom parsing rules, regexes, and pipelines in the Agent config are tedious to set up and maintain.
CPU cost at the collector: Running heavy regex over large log volumes inside your shipper burns CPU on every node.
Whack-a-Mole: You keep updating rules as your application output changes.

The Solution: PII-Shield as a Lightweight Sidecar

Instead of loading your cluster-wide log shipper with heavy processing, mask PII before it even leaves the pod.

PII-Shield is a small, dependency-free tool written in Go. It runs as a sidecar container next to your application, reads the app's log stream, redacts secrets and PII using entropy-based detection plus deterministic hashing, and writes the clean stream to its own stdout. By the time the Datadog Agent picks the logs up from the node, they are already sanitized — in-pod, before egress.

Ready-to-Use Pod Configuration

The app writes its logs to a shared volume; PII-Shield watches that file and emits a sanitized stream on its own stdout. The PII-Shield image is a scratch-based static binary, so it is started directly with --watch-file — no shell, no tail.

apiVersion: v1
kind: Pod
metadata:
  name: my-app-with-pii-shield
  annotations:
    # Drop the raw app container's logs to prevent duplicates
    ad.datadoghq.com/my-app.logs: '[{"source": "my-app", "service": "billing", "log_processing_rules": [{"type": "exclude_at_match", "name": "exclude_all", "pattern": ".*"}]}]'
    # Collect only the clean stream from the sidecar
    ad.datadoghq.com/pii-shield-sidecar.logs: '[{"source": "pii-shield", "service": "billing"}]'
spec:
  containers:
    - name: my-app
      image: my-app-image:v1.0.0
      # The app writes to a shared file instead of stdout
      command: ["/bin/sh", "-c"]
      args: ["./my-app-binary > /shared-logs/app.log 2>&1"]
      volumeMounts:
        - name: shared-logs
          mountPath: /shared-logs

    - name: pii-shield-sidecar
      image: thelisdeep/pii-shield:2.1.1
      # Scratch image: run the binary directly, no shell/tail needed
      command: ["/pii-shield"]
      args: ["--watch-file", "/shared-logs/app.log"]
      env:
        - name: PII_SALT
          valueFrom:
            secretKeyRef:
              name: pii-shield-secret
              key: pii-salt
      volumeMounts:
        - name: shared-logs
          mountPath: /shared-logs

  volumes:
    - name: shared-logs
      emptyDir: {}

Note: PII-Shield publishes multi-arch images (amd64 and arm64), so this works as-is on ARM nodes like AWS Graviton.

How does Datadog know what to read?

The main application redirects its output to a file, so its own stdout is empty. The Datadog Agent, which listens to container stdout via Autodiscovery, then picks up only the clean stream from the pii-shield-sidecar. No conflicts, no duplicate logs.

Why this is better

Zero configuration for log shippers: Datadog just receives clean logs — no pipeline rules to maintain.
Bypass premium vendor fees: Datadog's Sensitive Data Scanner is billed on top of your log volume. An open-source in-pod sidecar removes the need for vendor-side scrubbing.
Light footprint on a hot path: the scan loop is a low-allocation Go hot path on a scratch-based image, and the chart ships it under a 30Mi memory / 50m CPU limit — so a sidecar in every pod stays cheap.
Debuggable redaction: with deterministic hashing, user@email.com becomes something like [HIDDEN:a1b2c3]. The same input always maps to the same token for a given salt, so you can still correlate a user across logs without ever seeing the raw value.

By putting the shield where the data is generated, you protect your users' privacy and keep your observability bill in check.

Ready to secure your Kubernetes logs?

PII-Shield is open source (Apache-2.0). Check out the repository on GitHub, try the Helm chart, and if it helps, consider dropping a star.

DEV Community