DEV Community

Siddharthrane07
Siddharthrane07

Posted on Edited on

Building ScamShield AI: Week 2 — Docker Security, Firewall Bypasses, and Fail-Closed Architecture

#devops #security #python #docker

🛡️ ScamShield AI: Week 2 Build Log

Hardening Container Boundaries: iptables, SSRF Mitigations & Fail-Closed Sandboxes

🚨 The Journey So Far

If you caught Week 1, you know I'm building ScamShield AI — an evidence-grounded threat intelligence platform designed to safely analyze Indian scam SMS messages and WhatsApp forwards (fake KYC alerts, electricity cut-off threats, UPI traps).

To safely "click" a scammer's link without letting them compromise our backend infrastructure, we are building an isolated Docker sandbox using Playwright (Headless Chromium) and Mitmproxy (The Network Guard).

This week was entirely about hardening Stage 1: The Proxy Container.

I learned the hard way that building a Docker container is trivial, but securing it against a determined attacker probing from the inside is a completely different engineering challenge.

Here are my biggest technical takeaways, bugs, and architectural wins from Week 2.

🧠 Lesson 1: iptables Rule Ordering is a Strict Security Boundary

In iptables (the Linux kernel packet filter), rules are evaluated sequentially from top to bottom. The first match wins.

I originally wrote my firewall rules in this intuitive order:

1. Allow loopback (lo)
2. Allow ESTABLISHED, RELATED traffic
3. DROP private IPs (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.169.254)
Enter fullscreen mode Exit fullscreen mode

💥 The Vulnerability:

An attacker exploiting a network connection could potentially match an existing state or craft packets that trigger state-tracking rules prematurely. Because the ESTABLISHED, RELATED rule was placed at #2, any connection matching that condition is accepted immediately. It never reaches the DROP rules sitting at #3.

On a threat-intelligence scanner inspecting hostile URLs, this is a dangerous bypass that risks Server-Side Request Forgery (SSRF) against cloud metadata endpoints (169.254.169.254).

🛠️ The Fix:

We refactored the firewall initialization into 6 strict phases. All prohibited destination drops are enforced in Phase 1, before stateful connection acceptance is even evaluated.

Phase 1: DROP 169.254.169.254/32 (AWS/Cloud Metadata)
         DROP 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 (RFC1918 Private Ranges)
         DROP 100.64.0.0/10 (CGNAT)
...
Phase 4: ACCEPT ESTABLISHED, RELATED
Phase 5: DROP ALL (Default Policy Fallback)
Enter fullscreen mode Exit fullscreen mode

If a packet targets an internal address, the kernel drops it immediately on line 1. No connection tracking rule can resurrect it.

🛑 Lesson 2: Fail-Closed vs. Fail-Open Architecture

My original container entrypoint.sh had logic that looked roughly like this:

if command -v iptables >/dev/null 2>&1; then
    apply_firewall_rules
else
    echo "Warning: iptables not found, continuing anyway..."
fi

exec run_proxy "$@"
Enter fullscreen mode Exit fullscreen mode

If the container was accidentally started without the NET_ADMIN Linux capability, or if a base image update broke iptables, the script simply printed a warning to stdout and happily booted the proxy without a firewall.

This is classic "Fail-Open" design, and in security software, it is fatal. A security boundary that silently degrades to unprotected execution is worse than no boundary at all, because it gives you false confidence.

🛠️ The Refactor:

We rewrote the entire initialization script to be strictly Fail-Closed:

set -euo pipefail

# Fail immediately if NET_ADMIN is missing
if ! iptables -L -n >/dev/null 2>&1; then
    echo "[FATAL] iptables unavailable or insufficient permissions (missing NET_ADMIN?). Aborting." >&2
    exit 1
fi

# Apply rules and verify every single one with iptables -C
apply_firewall_rules || {
    echo "[FATAL] Failed to enforce firewall policy. Aborting startup." >&2
    exit 1
}
Enter fullscreen mode Exit fullscreen mode

Now, if a single firewall rule fails to apply, the container terminates immediately with exit code 1. The sandbox refuses to run without its armor.

🐛 Lesson 3: The Deprecation Bug That Fooled 56 Unit Tests

During a full Docker rebuild, our proxy container suddenly crashed on boot:

mitmdump: error: unrecognized arguments: --termlog-verbosity
Enter fullscreen mode Exit fullscreen mode

We had updated to mitmproxy 10.4.2, which completely deprecated the --termlog-verbosity CLI flag in favor of options syntax: --set termlog_verbosity=info.

Why didn't our 56 passing tests catch this?

Because our test suite was testing startup shell logic using mocked binaries. Our test mock for mitmdump was a dummy script that recorded incoming flags and exited 0 without validating whether those flags were actually accepted by the real mitmproxy CLI!

Our tests were falsely green.

💡 The Takeaway:

  • Unit tests with mocks verify your control flow, but they do not prove third-party tool compatibility.
  • Integration tests with real container binaries are non-negotiable for infrastructure and security harnesses.

We updated the configuration to the modern --set syntax, added negative assertions to the mocks to disallow deprecated arguments, and verified startup against the real binary.

🥷 Lesson 4: The Sneaky "Imposter" User in Debian Images

To adhere to the principle of least privilege, our proxy container must drop root privileges after configuring iptables and run the worker process as an unprivileged user named proxy with explicit UID 1002.

In our Dockerfile, we had:

RUN groupadd -g 1002 proxy || true && \
    useradd -u 1002 -g proxy proxy || true
Enter fullscreen mode Exit fullscreen mode

When checking running processes inside the container with docker top, something looked wrong:

$ docker top scamshield-proxy
UID     PID     PPID    CMD
13      14820   14798   mitmdump -s /app/addon.py ...
Enter fullscreen mode Exit fullscreen mode

The process wasn't running as UID 1002. It was running as UID 13!

🕵️ What happened?

The official python:3.11-slim image is built on Debian. Debian has shipped with a built-in system user named proxy (UID 13) since the 1990s (originally for Squid/inn).

When our Dockerfile executed groupadd ... || true, it failed silently because group proxy already existed, and useradd skipped creation. The container fell back to the system user UID 13.

🛠️ The Fix:

Never rely on implicit user creation in base images. We explicitly modify existing system users or pick a collision-free name:

RUN if id "proxy" >/dev/null 2>&1; then \
        userdel -f proxy && groupdel proxy || true; \
    fi && \
    groupadd -g 1002 proxy && \
    useradd -u 1002 -g proxy -s /usr/sbin/nologin -d /nonexistent proxy
Enter fullscreen mode Exit fullscreen mode

Always inspect docker top and /proc/<pid>/status in the running container to verify that your process is actually running with the exact UID/GID you intended.

📊 Current Project Status

Component Status Details
Regression Test Suite 🟢 Passing 57/57 tests green across entrypoint logic, proxy addons, and preflight gates.
Firewall Startup 🟢 Verified Container boots as root, sets 19 iptables rules, verifies with iptables -C, drops to UID 1002, and starts mitmdump.
CLI Arguments 🟢 Verified Migrated to mitmproxy 10.x options syntax (--set).
Live Network Egress Tests 🟡 In Progress Writing integration tests to confirm traffic to forbidden IPs actually drops at the packet level.

⚠️ Documented Constraints & Next Steps:

  1. IPv6 Hardening: Current rules target IPv4 (iptables). We need matching ip6tables policies to prevent IPv6 leak bypasses.
  2. DNS Egress Restriction: Port 53 UDP/TCP traffic needs to be constrained to trusted resolvers rather than unrestricted outbound access.

🚀 What's Up Next

This week, I am executing our Network Gate Verification Suite. Starting a container cleanly is not proof that packets are blocked; proof requires testing real network requests from inside the sandbox:

  • [ ] Confirm curl -m 2 http://169.254.169.254 drops and times out.
  • [ ] Confirm curl -m 2 http://10.0.0.1 is rejected by the kernel.
  • [ ] Confirm curl https://example.com traverses the proxy cleanly.

Once the network gate passes, we begin Stage 2: The Browser Container (locking down Headless Chromium with Playwright).

💬 Discussion

Have you ever had a unit test suite stay completely green while your production container was fundamentally broken due to mocked tools? How do you structure your egress firewall rules for scraping or threat intelligence workloads?

Drop your thoughts in the comments below! 👇

Follow the project on GitHub: github.com/Siddharthrane07/scamshield

Top comments (0)