Python configuration: how we got here, and why it still hurts

Configuration looks trivial: a .env file and a settings.py. Then the app grows. Secrets leak into logs, port suddenly arrives as the string "abc", and settings get copy-pasted between environments. I went from hardcoded constants to typed schemas, and the main thing I learned is this: configuration is not throwaway glue code. It's part of your infrastructure, and it deserves the same design care as business logic.

I've been writing Python since 2017. One of the first projects I worked on was a 17-year-old Python 2.7 / Django 1.x monolith with settings hardcoded in settings.py, and I assumed the whole problem was "get the values out of the code." Years later I know better.

TL;DR

• Every generation of config tooling (INI, YAML, env vars, typed schemas) fixed one problem and created another.
• 12-Factor's "everything in environment variables" no longer covers secret rotation, dynamic config, or multi-source merging.
• Seven requirements make a useful yardstick for any configuration tool.
• Five anti-patterns survive even in healthy codebases, and no library will cure them for you.

This is the first of three articles. Here I walk through the evolution of configuration approaches in Python: where we came from, where we're headed, what each generation fixed, and which pitfalls are still waiting for us. I won't compare specific libraries yet; that's Part 2.

The evolution

The hardcode era (the 2000s). I'm lucky enough to have missed it, but that 17-year-old project still carried its echoes. Some settings came through os.environ.get; the rest were plain module constants in settings.py:

DATABASE_HOST = "localhost"
DATABASE_PORT = 5432
DEBUG = True

Every change meant a full release. The most painful construct I found there was branching like if env == "production": that swaps hardcoded values. To know what's active right now, you have to read the whole file. When there are dozens of such branches, debugging turns into detective work with a flashlight.

From there, the story forks. The first branch keeps settings in Python but tidies them up: split settings.py into base.py, prod.py, dev.py, and use imports and inheritance. You get a config with types, autocomplete, and computed values. The Django community still lives this way. The downside: the config remains executable code. It's awkward for non-developers to edit, and you can't swap it without a rebuild or redeploy.

The second branch moves values into a declarative file in a separate format, fully decoupled from code. Let's start there.

Config files: settings live apart from code

The first logical step after hardcoding is to move values into a file. In the early 2000s that file was INI, and Python shipped configparser in the standard library:

[database]
host = localhost
port = 5432

import configparser


config = configparser.ConfigParser()
config.read("config.ini")

host = config["database"]["host"]
port = int(config["database"]["port"])  # always a string

Config separated from code. Revolutionary! But the era hit a ceiling fast: everything you pull out of INI is a string. Numbers need manual casting, bool needs a dedicated getboolean(), and lists have to be invented on top of the format. Comma? Semicolon? What if the value contains the delimiter?

Mistype a section name and you get a KeyError somewhere in production. Forget a required key and you find out from the same place. And any attempt at hierarchy runs into the fact that INI knows two levels, section and key, so every project invents its own convention like [database.primary].

XML was popular in enterprise (especially in Java land) but never took root in Python: too verbose, hard to read, painful to edit by hand.

JSON looked ideal: structured, typed, supported everywhere. But the standard has no comments, so you can't document a setting in the file itself. Hand-editing is risky: one missing comma breaks the whole file. And secrets stay in plain text, committed to git alongside the config.

YAML: closer to humans than to machines

In the late 2000s and early 2010s, YAML pulled ahead on the back of the DevOps wave. Ansible and Kubernetes made it the de facto standard for configs.

database:
  host: localhost
  port: 5432
  credentials:
    username: app_user

# Comments work!
logging:
  level: INFO  # and this one can be switched to DEBUG

YAML delivered what JSON lacked: comments, readability, real hierarchy. In exchange it brought its own class of pitfalls. One extra space breaks the whole file; indentation sensitivity is merciless. Magic conversions regularly shoot you in the foot: an unquoted NO becomes False, 010 parses as an octal number (in YAML 1.1), and 1.0 is sometimes a string and sometimes a float. A separate category is YAML bombs: parse an untrusted file without a safe loader and you can take the process down.

Then there's the zoo of implementations. The spec leaves a lot of freedom at the edges, and every parser interprets it differently. For example, PyYAML parses 2023-01-01 as datetime.date by default, while ruamel.yaml gives you a string. An empty key: is None in one parser and an empty string in another. Leading zeros (port: 010) are octal for some parsers, decimal for others, a string for a third. If services in different languages read the same config, the divergences multiply. JSON has some of this too, but its spec is much tighter. In YAML it's a systemic problem.

A separate genre of the same period: Python modules as config, especially popular in Django and Flask.

class Config:
    DATABASE_HOST = "localhost"
    DATABASE_PORT = 5432
    DEBUG = False


class DevelopmentConfig(Config):
    DEBUG = True


class ProductionConfig(Config):
    DATABASE_HOST = os.environ.get("DATABASE_HOST", "prod-db.example.com")

This approach has a lot going for it. It's native Python: the IDE completes fields, mypy catches typos before runtime, class inheritance naturally sorts environments. Values can be anything: read a secret from Vault at startup, assemble a DSN from parts, switch the host by machine name. It's a full-fledged config, just written in an executable language instead of a declarative one.

"Executable" is the controversial part. Importing settings.py runs arbitrary Python, with all that implies: side effects on load, hidden logic in an innocent-looking file. And to answer "where did this value in prod come from?" you sometimes have to walk the call stack.

Overriding a setting without a rebuild is harder than with a declarative format. You can mount a YAML or TOML file into a container as a volume; settings.py is part of the code, not the data (though running Ansible against it on the host is a workable option).

Hot reload is formally possible via importlib.reload(), but treacherous in practice: modules that already imported values by name keep the old references, and part of your app ends up in an inconsistent state. Re-reading a YAML file doesn't have this problem, because the values are data, not names in someone else's namespace.

There's also an organizational flip side. Keeping settings in code works only while config is the sole responsibility of developers: one language, one repo, one set of rules. Once DevOps, SRE, or product start touching it, a Python file becomes a constant source of friction. Not everyone is willing to edit someone else's code to change a value.

The approach is alive and far from fringe. Pulumi is built entirely on this idea: infrastructure and config as code in a real language, as opposed to Terraform's declarative HCL. The "executable vs. declarative config" debate has run for about fifteen years, and there will probably be no winner.

Regardless of format, all files of that generation share the same problems. No typing: is max_connections: "100" a string or a number? And if someone typos "10O", nobody will tell you. Secrets in plain text land in git with the whole commit history. And for different environments you end up breeding config.dev.yaml, config.staging.yaml, config.production.yaml, with no standard way to merge them.

Environment variables: one build for every environment

In 2011, Adam Wiggins of Heroku published The Twelve-Factor App, a manifesto for modern web applications. Factor III said: store config in the environment. It was a different way of thinking.

import os


host = os.environ["DATABASE_HOST"]
port = int(os.environ["DATABASE_PORT"])
debug = os.environ.get("DEBUG", "false").lower() == "true"

With environment variables, secrets stopped landing in the repository, and switching environments came down to swapping variables. Every cloud platform picked it up: Heroku, AWS, GCP, Docker (docker run -e DATABASE_HOST=db), CI/CD systems.

But again, everything is a string. MAX_CONNECTIONS=100 is '100', DEBUG=true is 'true', not True. You rebuild types by hand at every access. There's no hierarchy, so you invent conventions like DATABASE_PRIMARY_HOST. Environment variables are global to the process: one module writes os.environ['TIMEOUT'] = '30', another reads it, and when something goes wrong nobody knows where the value came from.

Complex types are their own pain. There are several ways to pass a list of URLs through an env var, and all of them are crooked:

# JSON in a string (ugly, but works)
export ALLOWED_ORIGINS='["https://app.example.com","https://admin.example.com"]'

# Delimiters (what if a value contains a comma?)
export ALLOWED_ORIGINS=https://app.example.com,https://admin.example.com

# Separate variables (variable explosion)
export ALLOWED_ORIGIN_1=https://app.example.com
export ALLOWED_ORIGIN_2=https://admin.example.com

There's no perfect option. And for local development it turned out that every developer has to set dozens of variables. So the .env file appeared, which is, once again, a configuration file with all its problems.

12-Factor today: what the manifesto is missing

By the late 2010s it was clear the methodology had solved one problem and created another. Nobody disputes the principle of separating config from code. It was right then and it's right now. But the recommendation "everything through environment variables" started cracking at the seams in several scenarios at once.

Clouds learned to store secrets themselves. AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and HashiCorp Vault provide managed storage with rotation, auditing, and granular access control. On the application side, env is still simpler: you get a ready string and move on, while an SDK means an extra dependency and network error handling.

But once you need secret rotation without a restart, on-demand reads, or separate permissions per secret, env hits a ceiling: variables are read once at process start and live until it exits. In those scenarios, calling the secret store through an SDK stops being an unnecessary hop and becomes the only workable option.

Dynamic configuration stopped being exotic. Feature flags, pool sizes, rate limits: you want to turn these knobs without a restart, and the model "an environment variable is set at process start" physically can't do it. Dedicated feature-flag services appeared (Flagr, LaunchDarkly, Unleash) that the app polls at runtime.

Containers brought their own pattern. Docker secrets are files mounted into the container and available via /run/secrets/<name>. Kubernetes secrets are more flexible: they can be attached as files through a volume or as environment variables. The file variant is usually considered safer: the secret doesn't show up in the process env or in environment dumps, and it's easier to rotate without restarting the pod. Either way, this is no longer the "everything through environment variables" model that 12-Factor proposed.

Merging didn't disappear. It moved. Even where an app honestly follows 12-Factor and reads only os.environ, merging multiple sources doesn't go away; it relocates from code into infrastructure. A Helm chart assembles values from values.yaml and values.production.yaml. A ConfigMap turns into pod environment variables, and so does a Secret. A CI/CD pipeline pulls values from Vault and passes them via --env. Before the container starts, all of it is flattened into one set and lands in os.environ.

The application sees the final result but not which layer overrode what. "Where did this value come from?" debugging has moved into Helm, Terraform, or CI secrets, and responsibility for merge correctness is smeared across several teams. The merge is still there. It's just implicit and invisible from the code.

The community noticed. Critics fairly pointed out that 12-Factor reflected Heroku's product particulars more than universal engineering principles: Heroku sold a platform where environment variables were convenient, and the manifesto helped sell it. In November 2024 the methodology was open-sourced for the community to update, which is an admission that it needs refreshing for the modern landscape.

Some teams go the opposite way: they pull merging back out of infrastructure and into the application, so that it's visible and verifiable. The app itself reads several sources and combines them in code: defaults from code, base structure from YAML or TOML, overrides from environment variables, secrets from a secret manager. Managing this zoo by hand is tedious.

os.environ.get(...), yaml.safe_load(...), a separate SDK per store: every source has its own API, its own typing, its own error handling. That's where the demand for aggregator tools came from: tools that read everything at once and fold it into one typed structure.

Structured formats today

TOML became popular once Python adopted it for pyproject.toml (with built-in support since Python 3.11 via tomllib). It deservedly pushed YAML aside as the config format for Python tooling thanks to strict primitive types, readability, and fewer parsing ambiguities. JSON5 went another way, adding comments and trailing commas to JSON. That revived JSON for people who like it but are tired of chasing a parse error on line 47 caused by a stray comma.

In parallel, a whole family of programmable configuration formats grew up, where the file is almost a language rather than a key-value table. A few interesting ones:

• HCL (HashiCorp Configuration Language) is the standard for Terraform, Vault, Consul, and Nomad. It reads better than JSON and supports blocks and interpolation.
• HOCON (Human-Optimized Config Object Notation) came from the Scala/Akka world. It supports composing multiple files via include, reusing values through substitutions (${...}), and referencing environment variables right in the file.
• Jsonnet from Google is JSON with variables, conditionals, and functions. It's widely used in Grafana and Tanka.
• CUE and Dhall go further: they're fully typed languages with validation, where the config and its schema sit side by side. KDL positions itself as a modern TOML replacement with document-like structures. Apple's Pkl (introduced in 2024) offers type-safe configuration with templating and validation in the syntax itself.
• NestedText lives at the opposite pole: a minimalist "everything is a string" format for people tired of YAML's magic conversions.

In Python projects, most of these remain exotic: parsers either don't exist or are poorly maintained. The standard is the same as ten years ago: YAML and environment variables for applications, TOML for tools, JSON for APIs. The rest shows up when needed, usually when a team moves off HCL-based infrastructure and wants to read the same files from the application.

But whichever format you pick, it only solves parsing. The parser finishes the file, and the guarantees end there. port: -1 is perfectly valid, because a format checks syntax, not meaning. And if you want to override defaults with local settings, welcome to a manual dict.update() that wipes out nested keys.

Schema in code: when formats stopped being enough

By the end of the 2010s it was clear that choosing a format is only half the job. The parser returns a dict, and then comes what no format solves: checking that port is a number from 1 to 65535, that required fields are present, and that a typo in a key won't take the service down three days after release. That check gets written in code anyway; the only question is in what form.

Python 3.7 brought dataclasses, and for the first time the config schema got a proper home, next to the types rather than in comments on a dictionary.

from dataclasses import dataclass


@dataclass
class DatabaseConfig:
    host: str
    port: int
    pool_size: int = 10

Dataclasses don't manage configuration on their own; they're just a way to describe the shape of data. Parsing the file, mapping keys to fields, coercing types, merging sources: all of that is still on the developer. But an important shift happens here: the schema separates from loading. Previously the structure lived in the author's head and was checked with if "host" in config["database"]:. Now it lives as types visible to the IDE, mypy, and any third-party library.

That shift opened the road to the next generation of tools: Pydantic, attrs + cattrs, Dynaconf, and others. They take the schema described in code and build everything else around it: parsing, validation, source merging. From this point we stop discussing formats and start building infrastructure around them.

What changed in the last ten years

The evolution of formats and the arrival of loaders is only half the story. On top of it came changes in the environment where applications live, and over the last decade nearly everything has changed. What a single config.ini used to handle has become a full infrastructure problem, driven by several forces at once.

Microservices. When the application is not a monolith but dozens of services, each has its own settings, but there are shared ones too: observability endpoints (Sentry, Prometheus, Jaeger), log levels, feature flags, neighboring services' addresses. How do you avoid duplication? How do you update a setting for everyone at once? The classic "file in the repository" approach stops working.

Many environments. A modern application doesn't live in two worlds (dev and prod) but in a whole ecosystem: local, dev, testing/QA, staging, production, DR (disaster recovery), and sometimes a preview environment per pull request. Each needs its own settings on a common base. Hence the need for config inheritance and for knowing exactly where every value came from.

Secrets. The days when you could keep passwords in a .env file are over (though many still do). The scale of the problem is striking: according to GitGuardian, more than 29 million leaked secrets were detected on public GitHub in 2025 alone. In 2024, JFrog researchers found a token on Docker Hub with admin access to the repositories of Python itself and PyPI, a potentially catastrophic leak that was fixed within 17 minutes. The cost of a mistake is growing faster than the maturity of our practices.

In response, a separate class of tools appeared: HashiCorp Vault, cloud secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault), SOPS for encrypting secrets in git, Sealed Secrets for Kubernetes. Secrets now live in a specialized store rather than in .env, and the application has to know how to work with it.

Secret rotation. The main shift of recent years: secrets must change automatically, without a developer's involvement. The old model, "generate a password for five years, drop it in .env, and forget it," stopped working with modern security requirements and supply-chain attacks. Vault and AWS Secrets Manager support rotation out of the box. For a database it looks like this: every N hours the service creates a new password, updates it in the DB, and hands it to applications. The application has to pick up the new password on the fly, which adds a requirement for config: re-read secrets at runtime without restarting the process.

Through all this, local development must not break. .env is a poor place to store production secrets, but as a local-development tool it works fine: one file, everything at hand, no Vault, no network access to a secret manager. Production, meanwhile, should pull the same values from a protected store. A configuration tool should be able to reproduce both scenarios, and the transition between them should be painless.

Dynamic configuration. The classic approach: change the config, restart the app. But in a world where downtime is costly, you want to turn knobs live: a log level for debugging, a pool size, A/B experiment weights. The configuration system doesn't have to do much: notice the change and hand over the new value.

The real difficulty starts on the application side. Re-reading .env or hitting a store's API is five lines. Carefully reopening a DB pool, resubscribing to a queue, or changing a logger level without losing context is a task of a different order. When we talk about dynamic configuration, we're really talking about the five lines.

All of these challenges moved the threshold. For some teams, config is still "read a file and parse it." For others, it's a full infrastructure layer.

If you have one service, two environments, and secrets in .env, python-dotenv covers you with room to spare, and you can stop reading. But once you add a secret manager, several environments with a shared base of settings, and a wish to understand where a specific value came from, built-in tools run out. That's where the conversation about tools begins.

Requirements for modern configuration

Before picking a tool, it's worth pinning down what a production application (a web service, a data pipeline, a microservice) expects from it. I end up with seven criteria, a ruler you can hold against any solution.

Many sources. Defaults in YAML, overrides in TOML, secrets in environment variables or Docker secrets, local tweaks in .env: you want the tool to read all of this out of the box, without plugins or custom adapters for each format.

Controlled merging. Layering a local config over defaults is the simplest case. But what if, for a list of servers, you need to extend rather than replace? What if two sources setting different values for the same field is an error, not expected behavior? You need conflict-resolution strategies. When two sources set the same field, the tool must know whose value to keep. There are several options:

• Take the value from the last source: overrides win over defaults.
• Take it from the first source: base values are protected from being overwritten.
• Fail with an error, if a conflict means someone misconfigured something.

All of them with the ability to set the rule not globally, but per field.

Type safety. The schema should live in code, not in a developer's head. The IDE suggests fields, mypy catches typos, and the type in a signature documents expectations. Configuration is a contract, and it's best checked at process start, before connections come up and business logic starts running. Better to fail with a clear error in the first second than to get an AttributeError in production an hour later.

Value validation. Types are necessary but not sufficient. A port must be between 1 and 65535. A URL must start with http. A timeout can't be negative. Validation should be part of the schema, not a separate system you can forget to attach to a new field.

Clear errors. When a config is invalid, the developer should see which field, which value, in which file, on which line. Not a stack trace from the depths of a parser, but a human-readable message that points at the problem, so you fix it in a minute instead of half an hour in the logs.

Secrets at the schema level. Passwords, tokens, and API keys should be marked right in the config model with a dedicated type (for example, SecretStr). That's the first line of defense: in repr(), in a config dump at startup, or in an error trace, the secret doesn't appear in plain text. Beyond that it's the job of the logger and the observability stack. All a configuration system owes you is a way to mark a field as sensitive.

Debugging. When production has a different value than you expected, you need to quickly understand where it came from. Which file? An environment variable? The defaults? Without source auditing it turns into a detective story: "let me print(config) and rebuild the image."

These seven are what a tool should solve out of the box, rather than every team solving again. When evaluating configuration solutions, start from this list: which of the seven are critical for your project, which are nice to have, which are optional.

Five configuration anti-patterns I keep running into

Seven criteria are the ideal a tool strives for. But even with a good tool, stubborn anti-patterns settle into the code: solutions that look fine in the short run and turn into bug factories in the long run. I run into these regularly in real projects, even where engineering culture is otherwise healthy. A configuration tool doesn't cure them, so it's important to understand them and consciously avoid them.

A singleton with mutable state. A global config variable initialized once at startup and accessed from anywhere.

# config.py

config = load_config()

# somewhere in a handler

from config import config


if config.feature_flags.new_payment_flow:
    ...

The anti-pattern is that the object is mutable. Someone writes config.feature_flags.new_payment_flow = True in the middle of a handler for debugging, forgets to remove it, and production starts doing things you can never reproduce on staging. Tests suffer first: one test changes a value, another fails, and nobody knows who's to blame. The cheapest insurance is frozen=True on the dataclass: an attempted mutation raises FrozenInstanceError. The more fundamental fix is dependency injection: pass config into a service's constructor instead of importing it globally.

A dictionary as config. config = yaml.safe_load(f), and then accesses like config["database"]["primary"]["host"] are scattered across the code. A dict is mutable, keys are strings, a typo is discovered only at runtime, and IDE and mypy are helpless. This is the same hardcode, hidden one level of abstraction deeper. Developers moved config out of code into a file, but got no guarantees from it.

Config via monkeypatching in tests. Tests replace config values with mock.patch.

@mock.patch("myapp.config.DATABASE_TIMEOUT", 0.1)
def test_timeout(): ...

It works until someone renames a field or changes the config structure. Then the patch silently stops applying, the test keeps passing, and nobody checks real behavior until a timeout blows up in production. DI helps here too: if a service takes config as an argument, the test passes what it needs, with no string-path mocks.

The whole config as a JSON string in one environment variable. This shows up when a team runs into env vars not supporting hierarchy and decides, "let's put the whole hierarchy into one variable." The drawbacks are obvious: changing a value means editing a long JSON string in a terminal, and a typo can't be caught before launch.

APP_CONFIG='{"database":{"host":"localhost","port":5432},"redis":{"url":"redis://..."}}'

The pattern has a legitimate niche: configuration that is inherently tree-shaped while the transport is flat. For example, logging settings, where a logger hierarchy (app.db.pool, app.api.handlers) with levels and handlers decomposes poorly into dozens of LOG_LEVEL_APP_DB_POOL=... variables. The result is worse than one JSON string with a clear structure. A similar story is Lambda with its hard deployment size limit and no way to mount a file.

The rule is simple: if a configuration object is truly hierarchical and there's no other transport, JSON-in-env is acceptable, strictly for that object. If the entire application config is packed this way, it's a sign it's time to mount a file.

Magic environment variables scattered across the code. The logic drifts. One module compares to "production", another to ("staging", "production"), a third checks for inequality. Six months later nobody remembers which behavior is expected in which environment, and when you add a new qa environment, the app behaves differently in different modules.

# somewhere
if os.environ.get("ENV") == "production":
    use_real_payment_gateway = True


# elsewhere, in another module
if os.getenv("ENV") in ("staging", "production"):
    enable_sentry()


# in a third
DEBUG = os.environ.get("ENV") != "production"

The fix is centralization: one place loads the config and normalizes it into typed flags (is_production: bool, enable_payments: bool), which are then handed out via DI. Without that centralization, the pitfalls multiply in proportion to the number of modules.

Conclusion

In 2017 I looked at that Django 1.x project with hardcoded values in settings.py and thought the main problem was "get everything out of the code." Today I know the task has many layers: sources, merging, validation, secrets, rotation, debugging, testing. And I understand that the simplicity left behind in the Django era of one settings.py was also a value, one we'll probably never get back.

What hasn't changed is the principle of separating config from code. It was in the 2011 Heroku manifesto and it's still here in 2026. So is the idea that configuration is a contract between an application and its environment, and a contract has to be checked. And so is the understanding that a config mistake costs more than a code mistake: code fails in tests, config fails in production, at three in the morning, under load.

What changes is the sources, which increasingly move to dedicated services: Vault, AWS Secrets Manager, etcd. The config lifecycle changes too: static config with restarts gives way to dynamic config with live pickup and automatic rotation. Merging of several sources migrated from the application into infrastructure, and from there we want to bring it back under the control of code.

Expectations for error UX have gotten stricter: a stack trace from deep inside a parser is no longer acceptable; we need diagnostics with coordinates. Configuration now needs testing as well, so that config becomes as much an object of control as business logic. Cover the validation schema with unit tests so an invalid port fails on load rather than at runtime; check that secrets don't leak into dumps and logs; build the config for every environment in CI so staging doesn't ship to prod with someone else's values.

I don't recommend choosing a tool by GitHub stars. Take the seven criteria above and decide what is critical in your project, what's nice, and what's superfluous. A microservice in Kubernetes needs one thing, a CLI utility another, an ML pipeline a third. There's no universal answer.

Next up: Part 2 looks at how python-decouple, Dynaconf, Hydra, and pydantic-settings cover these seven requirements in 2026.

Which of the seven hurts most in your project? I'd like to hear about it in the comments.

Useful links

• The Twelve-Factor App manifesto
• Criticism of the 12-Factor App manifesto
• GitGuardian data on leaked secrets
• The admin-level token for Python and PyPI repositories found on Docker Hub