DEV Community

khimananda Oli
khimananda Oli

Posted on

Environment Variable Naming Will Break Your App — Here's How Docker, systemd, and Kubernetes Handle It Differently

#kubernetes #devops #linux #docker

A Spring Boot application was crash-looping over 5,000 times on an EC2 instance managed by systemd. The logs showed nothing obvious — just status=1/FAILURE every 30 seconds. No connection errors, no missing JARs, no OOM kills.

The culprit? Hyphens in environment variable names.

PAYMENT-API-PROTOCOL=https
PAYMENT-API-HOST=10.0.1.50:8080
PAYMENT-CRED-USERNAME=svc-account
Enter fullscreen mode Exit fullscreen mode

These were pulled from AWS Secrets Manager and written to a .env file. The systemd service used EnvironmentFile to load them. And systemd was silently ignoring every single one:

Ignoring invalid environment assignment 'PAYMENT-API-PROTOCOL=https'
Ignoring invalid environment assignment 'PAYMENT-CRED-USERNAME=svc-account'
Enter fullscreen mode Exit fullscreen mode

The kicker? The exact same .env file worked perfectly in Docker.

The Rules Are Different Everywhere

Environment variable naming isn't universal. Each runtime has its own rules, and what works in one will silently break in another.

POSIX Standard (What Most Things Follow)

The POSIX specification for environment variables says names must match:

[a-zA-Z_][a-zA-Z0-9_]*
Enter fullscreen mode Exit fullscreen mode

That's letters, digits, and underscores only. No hyphens, no dots, no special characters. The name must start with a letter or underscore.

Most Linux tooling follows this — bash, sh, env, export, systemd, and core utilities all enforce or expect POSIX-compliant names.

Docker: The Permissive One

Docker's --env-file parser is lenient. It accepts hyphens, dots, and other characters in variable names that POSIX wouldn't allow:

# docker-compose.yml or --env-file
PAYMENT-API-PROTOCOL=https    # ✅ works in Docker
my.app.config=value           # ✅ works in Docker
Enter fullscreen mode Exit fullscreen mode

Docker passes these directly to the container's process environment without validation. The container runtime doesn't care — it just injects whatever key-value pairs you give it.

This is why teams that develop exclusively in Docker containers often don't realize they're using non-standard variable names — until they deploy to bare metal or systemd-managed services.

systemd: Strict POSIX

systemd's EnvironmentFile directive follows POSIX rules strictly. If a variable name contains a hyphen, systemd silently drops it with a log warning:

Ignoring invalid environment assignment 'PAYMENT-API-PROTOCOL=https'
Enter fullscreen mode Exit fullscreen mode

The word "silently" is doing a lot of heavy lifting here. The service still starts. The variable just isn't there. Your application boots, can't find the config it needs, and crashes — often with a confusing error that points at dependency injection or missing beans, not at the actual missing variable.

# /etc/systemd/system/myapp.service
[Service]
EnvironmentFile=/opt/app/.env
ExecStart=/usr/bin/java -jar /opt/app/myapp.jar
Enter fullscreen mode Exit fullscreen mode 
# .env file
PAYMENT_API_PROTOCOL=https    # ✅ loaded
PAYMENT-API-PROTOCOL=https    # ❌ silently ignored
Enter fullscreen mode Exit fullscreen mode

Kubernetes: POSIX with DNS Flexibility

Kubernetes ConfigMaps and Secrets follow POSIX for environment variable names injected via envFrom or env. But there's a nuance — ConfigMap keys can contain hyphens and dots because they can also be mounted as files:

# ConfigMap keys can have hyphens
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  PAYMENT_API_PROTOCOL: "https"        # ✅ works as env var
  payment-api-protocol: "https"        # ⚠️ works as file mount, NOT as env var
Enter fullscreen mode Exit fullscreen mode

When you mount a ConfigMap as a volume, each key becomes a filename — and filenames can have hyphens. But if you use envFrom to inject them as environment variables, Kubernetes will skip keys that aren't valid POSIX variable names:

# This will silently skip non-POSIX keys
envFrom:
  - configMapRef:
      name: app-config
Enter fullscreen mode Exit fullscreen mode

Kubernetes logs a warning event on the pod, but unless you're watching events closely, you'll miss it — just like systemd.

The Comparison Table

Feature Docker systemd Kubernetes (env) Kubernetes (volume)
Underscores ✅ ✅ ✅ ✅
Hyphens ✅ ❌ Silent drop ❌ Silent skip ✅ (as filename)
Dots ✅ ❌ Silent drop ❌ Silent skip ✅ (as filename)
Error on invalid No Warning in journal Event on pod N/A
POSIX compliant No Yes Yes N/A

The Real Danger: Silent Failures

The common thread across systemd and Kubernetes is silent failure. The variable isn't there, but nothing crashes at the system level — the process starts, tries to read the missing config, and fails with an application-level error.

In the incident that prompted this article, the Spring Boot error was:

Caused by: java.lang.IllegalArgumentException: 
  Could not resolve placeholder 'PAYMENT_API_HOST' in value "${PAYMENT_API_HOST}"
Enter fullscreen mode Exit fullscreen mode

This looks like a missing property issue. It is — but the root cause was three layers deep: Secrets Manager had hyphens → systemd dropped the variables → Spring couldn't find them.

Best Practices

1. Use Underscores Everywhere

Just use SCREAMING_SNAKE_CASE for all environment variables. It's POSIX-compliant and works everywhere — Docker, systemd, Kubernetes, bash, CI/CD pipelines, Lambda, ECS, everything.

# Do this
PAYMENT_API_PROTOCOL=https
PAYMENT_CRED_USERNAME=myuser

# Not this
PAYMENT-API-PROTOCOL=https
payment.api.protocol=https
Enter fullscreen mode Exit fullscreen mode

2. Validate at Deploy Time

Add a preflight check in your deployment scripts that verifies all required variables are present before restarting the service:

REQUIRED_VARS="DB_HOST DB_PORT API_KEY SECRET_TOKEN"

for var in $REQUIRED_VARS; do
  grep -q "^${var}=" .env || { echo "FATAL: Missing $var in .env"; exit 1; }
done

systemctl restart myapp
Enter fullscreen mode Exit fullscreen mode

This takes 5 lines and prevents crash loops at 3 AM.

3. Transform at the Boundary

If you can't control the source (like a third-party Secrets Manager with legacy keys), transform at the boundary — the point where secrets are pulled into the runtime:

# In deploy.sh — transform hyphens to underscores when pulling from Secrets Manager
aws secretsmanager get-secret-value \
  --secret-id myapp.prod \
  --query SecretString --output text \
  | jq -r 'to_entries|map("\(.key | gsub("-";"_"))=\(.value|tostring)")|.[]' > .env
Enter fullscreen mode Exit fullscreen mode

4. Spring Boot Property Naming

Spring Boot has its own relaxed binding that maps environment variables to properties. Use this to your advantage:

# application.properties
# Spring property with dots — reads from PAYMENT_API_PROTOCOL env var automatically
payment.api.protocol=${PAYMENT_API_PROTOCOL}
Enter fullscreen mode Exit fullscreen mode

But be careful with circular references. If your property name and placeholder are identical:

# ❌ Circular reference — Spring resolves against itself
PAYMENT_API_HOST=${PAYMENT_API_HOST}

# ✅ Different names — env var resolves correctly
payment.api.host=${PAYMENT_API_HOST}
Enter fullscreen mode Exit fullscreen mode

5. Use systemd-analyze to Verify

Before deploying, verify your environment file is valid:

systemd-analyze verify /etc/systemd/system/myapp.service
Enter fullscreen mode Exit fullscreen mode

Or manually test what systemd will actually load:

systemd-run --property=EnvironmentFile=/opt/app/.env /usr/bin/env
Enter fullscreen mode Exit fullscreen mode

Wrapping Up

Environment variable naming is one of those things that "just works" until it doesn't. Docker's permissiveness masks issues that surface the moment you move to systemd or Kubernetes. The fix is simple — stick to POSIX-compliant names (letters, digits, underscores) and validate at deploy time.

The 5,000+ restarts and hours of debugging could have been avoided with two things: underscores instead of hyphens, and a 5-line validation script in the deployment pipeline.

Sometimes the smallest characters cause the biggest outages.

Have you run into similar environment variable gotchas? Drop your war stories in the comments 👇

``

Top comments (0)