DEV Community

Tural Muradov
Tural Muradov

Posted on

Why Log Masking Matters in Kubernetes (and How We Enforced PCI Safety with Fluent Bit)

#security #sre #logging #devops

Logging is one of the most underestimated security risks in systems.

Not because engineers are careless, but because logs change faster than security reviews, and production pressure always wins over perfect hygiene.

We will go through:

  • why sensitive data ends up in logs
  • why relying on developers alone is not enough
  • how we enforced PCI-safe logging at the infrastructure level
  • a complete Fluent Bit + Fluentd configuration
  • a production-ready Lua masking script

This solution is used in a real Kubernetes production environment.

The uncomfortable truth about logs

Every experienced engineer has seen logs like these:

POST /pay?pan=4111111111111111&cvv=123
password=secretpass
Authorization failed for user=admin pass=admin123
Enter fullscreen mode Exit fullscreen mode

These usually appear not because someone intended to leak data, but because of:

  • temporary debug logging during incidents
  • forgotten log statements after hotfixes
  • third-party SDKs logging request payloads
  • rushed troubleshooting under pressure
  • team or ownership changes

Why this is dangerous

Logs often:

  • live longer than databases
  • are accessible to more people
  • are copied across multiple systems
  • bypass encryption and access reviews
  • sit outside normal data-protection controls

A single leaked PAN or password in logs can result in PCI DSS violations, GDPR incidents, and audit findings that are hard to fix retroactively.

Why “just don’t log sensitive data” does not work

Even strong engineering teams eventually log sensitive data:

  • humans forget
  • incidents force shortcuts
  • third-party libraries log unexpected payloads
  • reviews do not cover every log line

Security controls must assume mistakes will happen.

That is why we moved responsibility from application code into the logging pipeline.

Enforcing security at the logging layer

We defined one rule:

Sensitive data must never leave the node unmasked, even if an application logs it.

Architecture:

Application
   ↓
Fluent Bit (masking and filtering)
   ↓
Fluentd
   ↓
Elasticsearch / Kibana
Enter fullscreen mode Exit fullscreen mode

Masking happens before logs are forwarded anywhere.

Why Fluent Bit with Lua

Fluent Bit runs:

  • on every node
  • close to the log source
  • before logs leave the host

Lua allows:

  • conditional logic
  • validation (for example, Luhn checks)
  • partial masking instead of blind replacement
  • zero application changes

What we mask and why

Data type Reason
PAN (card number) PCI DSS 3.4
CVV / PIN PCI DSS 3.3
Track1 / Track2 PCI DSS
Passwords PCI DSS 8.x
IBAN GDPR / PII
IP addresses GDPR / PII

Example

Before:

password=secretpass pan=4111111111111111&cvv=123 ip=10.233.105.74
Enter fullscreen mode Exit fullscreen mode

After:

password=****retpass pan=411111******1111&cvv=*** ip=xxx.xxx.xxx.xxx
Enter fullscreen mode Exit fullscreen mode

Fluent Bit configuration (production-safe)

Input: exclude system namespaces early 

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Exclude_Path /var/log/containers/*_kube-system_*.log,                 /var/log/containers/*_logging_*.log,                 /var/log/containers/*_monitoring_*.log,                 /var/log/containers/fluent-bit-*.log,                 /var/log/containers/fluentd-*.log
    Parser cri
    Tag kubernetes.*
    DB /tail-db/tail-containers-state.db
    Mem_Buf_Limit 100MB
Enter fullscreen mode Exit fullscreen mode

This prevents Fluent Bit from ingesting its own logs, kube-system components, and logging feedback loops.

Kubernetes metadata enrichment 

[FILTER]
    Name kubernetes
    Match kubernetes.*
    Merge_Log On
    K8S-Logging.Exclude On
Enter fullscreen mode Exit fullscreen mode

Namespace and container exclusion 

[FILTER]
    Name grep
    Match *
    Exclude kubernetes.namespace_name ^kube-system$
    Exclude kubernetes.namespace_name ^logging$
    Exclude kubernetes.namespace_name ^monitoring$
    Exclude kubernetes.container_name ^fluent-bit$
    Exclude kubernetes.container_name ^fluentd$
Enter fullscreen mode Exit fullscreen mode

Lua masking filter 

[FILTER]
    Name lua
    Match kubernetes.*
    script /fluent-bit/scripts/mask_pci.lua
    call filter
Enter fullscreen mode Exit fullscreen mode

Masking occurs before logs leave the node.

Fluentd output configuration (example) 

<match **>
  @type elasticsearch
  host elasticsearch.logging.svc
  port 9200
  logstash_format true
  include_tag_key true
</match>
Enter fullscreen mode Exit fullscreen mode

At this point, all sensitive data has already been sanitized.

Full Lua masking script 

-- /fluent-bit/scripts/mask_pci.lua
-- Centralized PCI / PII log masking

local function luhn(num)
    local sum, alt = 0, false
    for i = #num, 1, -1 do
        local n = tonumber(num:sub(i, i))
        if alt then
            n = n * 2
            if n > 9 then n = n - 9 end
        end
        sum = sum + n
        alt = not alt
    end
    return sum % 10 == 0
end

function filter(tag, ts, record)
    local msg = record["message"]
    if not msg then return 1, ts, record end

    -- PAN masking
    msg = msg:gsub("(%d%d%d%d%d%d%d%d%d%d%d%d%d?%d?%d?%d?%d?)",
        function(num)
            if #num >= 12 and #num <= 19 and luhn(num) then
                return num:sub(1,6) .. string.rep("*", #num - 10) .. num:sub(-4)
            end
            return num
        end
    )

    -- CVV / PIN
    msg = msg:gsub("(?i)(cvv|cvc|cv2|cvn)=\d{3,4}", "%1=***")
    msg = msg:gsub("(?i)(pin|pinblock)=\d+", "%1=****")

    -- Expiry dates
    msg = msg:gsub("(?i)(exp|expiry|expires)=\d{2}[/\-]\d{2}", "%1=**/**")

    -- Track data
    msg = msg:gsub("(?i)%B%d+%^[^%s]+", "track1=***")
    msg = msg:gsub("(?i);%d+=%d+", "track2=***")

    -- IBAN
    msg = msg:gsub("(?i)\b([A-Z]{2}\d{2}[A-Z0-9]{11,30})\b",
        function(iban)
            return iban:sub(1,6) .. string.rep("*", #iban - 10) .. iban:sub(-4)
        end
    )

    -- Passwords (partial)
    msg = msg:gsub(
        "(?i)\b(pass(word)?|passwd|pwd)\s*([:=])\s*([^\s"&]+)",
        function(key, _, sep, value)
            if #value <= 4 then
                return key .. sep .. string.rep("*", #value)
            end
            return key .. sep .. "****" .. value:sub(5)
        end
    )

    -- IPv4 addresses
    msg = msg:gsub("(%d+%.%d+%.%d+%.%d+)", "xxx.xxx.xxx.xxx")

    record["message"] = msg
    return 1, ts, record
end
Enter fullscreen mode Exit fullscreen mode

PCI DSS alignment

Requirement Enforcement
3.3 CVV and PIN fully masked
3.4 PAN partially masked
3.5 PAN never stored unmasked
8.x Passwords protected
GDPR PII minimized

Lessons learned

  • Developers will eventually log sensitive data
  • Reviews and guidelines are not enough
  • Regex-only masking creates false positives
  • Validation (such as Luhn) is essential
  • The logging pipeline is part of the security perimeter

Final result

  • No PANs in Elasticsearch
  • No credentials in Kibana
  • No application code changes
  • No reliance on human perfection
  • Auditors satisfied

Top comments (0)