Don't nuke your test ENVs

NOTE: I know that modifying environment variables in tests is a controversial technique, but we do it on this project, so we have to do it properly.

This week I learned that Application.put_env/4 can cause flaky tests, both when running sync and async.

I was building out a test for a module that pulled data from config.exs. It looked like this:

and I set data as module attributes:

When I started running the tests, I was getting intermittent failures. After some digging, I found that when the tests failed, the returned config list was not what I expected, it only contained a single key/value pair:

I was using a new (to me) mocking library for my tests, so my first thought was that I was somehow overwriting the configs, or that the mocks were using a different set of test-specific configs. I tinkered with that, but it wasn't the case. Then, in the process of trying to narrow down the variables, I changed the test from async: true to async: false. I have had experiences in the past where async tests have data read/write conflicts, which I thought setting my test to async: false would eliminate. But that didn't improve the flakiness, in fact, it made it worse. With no luck fixing the flake, I pinged my team, and got a great piece of advice:

"...any other test does an Application.put_env and you're in for trouble"

Lightbulb moment.

I noticed a private function in another test that was modifying this same config. At first glance, it looked like it reset the config to its original state. The culprit:

To me, what it looked like that private function was doing was:

1. Grab that particular key/value pair
2. Update the value
3. Return that updated key/value
4. After the test completes, reset the key/value to its original state and insert it back into the list of MyApp.Example configs.

But it's a bit deceiving, because the call:

Does not merge into the existing config, it overwrites the entire config with only the key/value passed.

So, the on_exit function reset the entire MyApp.Example config to:

Which meant that if my test ran after the test containing this private function, mine would fail because the necessary config key/values did not exist. And, my debugging attempt to change my test to async: false didn't help, because our non-async tests generally run at the end of the test suite. The fix was to refactor that private function to restore the entirety of the original config in the on_exit block, like this:

After I refactored that private function, I ran a script to execute the test suite 100x, and I saw no failures. Flake fixed!

The lesson learned: If you must modify envs in tests, isolate them carefully or restore the full config explicitly to avoid cross-test pollution.

Comments

[elfenlaid]

Thanks for bringing up Application.put_env peculiarities!

However, there's something is off in your example. Specifically this part: @config Application.compile_env where it grabs the app's config in compile time. It no loger depends on Application env runtime manipulations.

I can't wrap my head around it on my own, could you help me?