The Problem We Were Actually Solving
We built the Veltrix treasure hunt engine to power a live event platform where thousands of users raced to solve puzzles in real time, and the configuration layer was supposed to be the secret weapon that let us grow confidently. What we didnt account for was that our first stab at configuration was just a Ruby hash that lived in the codebase, user-facing values shoved into environment variables, and a single YAML file that became the size of Manhattan by launch week. The day we pushed to production, the biggest problem wasnt scale — it was that every change required a restart, because changes to the config forced the Ruby process to recompile constants. At 2:17 a.m., the first growth inflection hit: 1,024 concurrent users, 30 seconds of garbage collection, and the Redis connection pool completely exhausted because the config parser had ballooned to 15 MB. The system didnt stall under load — it stalled under configuration.
What We Tried First (And Why It Failed)
First, we punted to environment variables and the Twelve-Factor App checklist: eleven separate .env files, Docker Compose overrides, and a CI pipeline that injected values at build time. The illusion of clean separation lasted exactly one sprint. By sprint two, we had 170 environment variables, half of them secrets, and the rest scattered across three different repos because product wanted feature flags, ops wanted tuning, and marketing wanted A/B splits. We burned 16 engineering hours debugging why a Redis cluster in staging accepted connections but rejected commands — turns out the staging environment had inherited a production database name because an engineer had copy-pasted a .env.example and forgotten to change one letter.
Next, we tried Consul as a dynamic configuration backend. It felt powerful, until we realized wed built a system where every config change triggered a rolling restart of the entire fleet because the Ruby process couldnt reload anything without nuking its constant cache. Consul also introduced a new failure domain: if Consuls leader died, our treasure hunt engine paused mid-puzzle and waited for the cluster to re-elect, which happened at the worst possible moment, like when the leader was in a US-East outage during a US-West peak.
We even tried a monorepo approach where configuration was its own service and every team contributed their own YAML files. That lasted until merge conflicts in config files started breaking production, and an innocent typo in a YAML anchor brought down the entire event for 23 minutes. I still have the Slack message: config.yaml:32: found character that cannot start any token.
The Architecture Decision
We stopped trying to make configuration dynamic and started making it disposable. We replaced the Ruby constants with a lightweight Lua sandbox that ran inside Redis itself. Every configuration value became a Redis key with a TTL equal to the cache flush interval, and every worker process loaded its config on every request from a Lua call. The key insight wasnt performance — it was that Redis already had a network protocol, a persistence layer, and a built-in failure detector. We didnt need Consul or Kubernetes ConfigMaps; we needed a fast reload and a single source of truth.
The tradeoff was that configuration became a first-class citizen in the Redis cluster. If Redis went down, so did the treasure hunt — but in practice, Redis is more stable than our previous approach, and we can now push configuration changes without restarting anything. We also gained atomicity: every config value has a versioned key, so we can roll back by deleting the latest version and letting workers reload.
What The Numbers Said After
After the switch, the latency percentiles moved from P99 at 800 ms to P99 at 240 ms under 2,000 concurrent users, and the garbage collection pauses dropped from 30 seconds to less than 200 milliseconds. The Redis memory overhead increased by 18 MB, which we traded for zero config restarts. We instrumented the Lua sandbox with a simple prometheus metric: veltrix_config_reloads_total. During the Black Friday sale, it spiked to 42 reloads per second across the cluster — 42 was the version number of the winning treasure hunt configuration that day, so it became a running joke. The joke died when someone asked why it was always 42. It wasnt always 42 — it was always the versioned key name.
What I Would Do Differently
I would treat the configuration layer as an infrastructure primitive, not a code layer. That means: embed it in the platform runtime, version it, and never expose raw key-value pairs to engineers. If I had to do it over, Id start with a Lua sandbox from day one and skip the Ruby constants entirely. Id also ban any configuration value that cant be represented as a Lua table with a TTL, including feature flags. Id insist that every environment variable must be encrypted at rest and audited weekly, because the real failure domain wasnt Redis — it was the people who thought environment variables were a form of version control. And finally, Id never again let a product manager name a config version 42 without a formal change record. That number cursed us for months.
Top comments (0)