Don't nuke your test ENVs

NOTE: I know that modifying environment variables in tests is a controversial technique, but we do it on this project, so we have to do it properly.

This week I learned that Application.put_env/4 can cause flaky tests, both when running sync and async.

I was building out a test for a module that pulled data from config.exs. It looked like this:

and I set data as module attributes:

When I started running the tests, I was getting intermittent failures. After some digging, I found that when the tests failed, the returned config list was not what I expected, it only contained a single key/value pair:

I was using a new (to me) mocking library for my tests, so my first thought was that I was somehow overwriting the configs, or that the mocks were using a different set of test-specific configs. I tinkered with that, but it wasn't the case. Then, in the process of trying to narrow down the variables, I changed the test from async: true to async: false. I have had experiences in the past where async tests have data read/write conflicts, which I thought setting my test to async: false would eliminate. But that didn't improve the flakiness, in fact, it made it worse. With no luck fixing the flake, I pinged my team, and got a great piece of advice:

"...any other test does an Application.put_env and you're in for trouble"

Lightbulb moment.

I noticed a private function in another test that was modifying this same config. At first glance, it looked like it reset the config to its original state. The culprit:

To me, what it looked like that private function was doing was:

1. Grab that particular key/value pair
2. Update the value
3. Return that updated key/value
4. After the test completes, reset the key/value to its original state and insert it back into the list of MyApp.Example configs.

But it's a bit deceiving, because the call:

Does not merge into the existing config, it overwrites the entire config with only the key/value passed.

So, the on_exit function reset the entire MyApp.Example config to:

Which meant that if my test ran after the test containing this private function, mine would fail because the necessary config key/values did not exist. And, my debugging attempt to change my test to async: false didn't help, because our non-async tests generally run at the end of the test suite. The fix was to refactor that private function to restore the entirety of the original config in the on_exit block, like this:

After I refactored that private function, I ran a script to execute the test suite 100x, and I saw no failures. Flake fixed!

The lesson learned: If you must modify envs in tests, isolate them carefully or restore the full config explicitly to avoid cross-test pollution.

Comments

[elfenlaid]

Thanks for bringing up Application.put_env peculiarities!

However, there's something is off in your example. Specifically this part: @config Application.compile_env where it grabs the app's config in compile time. It no loger depends on Application env runtime manipulations.

I can't wrap my head around it on my own, could you help me?

[Noel Worden]

That's a great point! You're absolutely correct that @config Application.compile_env(...) sets values as fixed, immutable module attributes at compile time. These module attributes in the test file never changed.

The flakiness was caused by runtime corruption of the global environment.

What I may not have made clear in the post was that the issue came from the functions within the module being tested. Those functions contained logic that called Application.get_env/2 at runtime to retrieve configuration values. The faulty test's cleanup routine had corrupted the global application environment store, causing this runtime lookup to fail (it couldn't find the necessary keys) for subsequent tests.

The fix ensured the full global runtime state was restored in the on_exit hook, resolving the conflict for all dynamic configuration lookups.