Tricks of declaring dynamic variables in Bash

In any programming languages, it is a common practice to define a variable with the result of some computation and refer to it later when needed. In Bash, it's pretty straight forward to define variables as follows:

> FOO=bar
> echo $FOO
bar

But what if you want to define a variable, of which name is dynamic? This could be useful when the variable name is only known during runtime, as user input. Something like this:

# bash variables names are stored in variables.txt
# as user input
> cat > variables.txt <<EOF
FOO1=bar1
FOO2=bar2
FOO3=bar3
EOF

> do_some_magic

> echo $FOO1
bar1

> echo $FOO2
bar2

> echo $FOO3
bar3

Trick 1 declare dynamic variables

declare is a built function within Bash. It is the magic here to define variables with dynamic names. A peak of declare's man output:

declare: declare [-aAfFgiIlnrtux] [-p] [name[=value] ...]
    Set variable values and attributes.

    Declare variables and give them attributes.  If no NAMEs are given,
    display the attributes and values of all variables.

This is what you need to do:

> var_name=FOO1
> declare $var_name="bar1"
> echo $FOO1
bar1

If you replace the above do_some_magic function with:

while IFS='=' read -ra line; do
  key=${line[0]}
  value=${line[1]}
  declare "$key"="$value"
done < variables.txt 

you will get:

> echo $FOO1
bar1

> echo $FOO2
bar2

> echo $FOO3
bar3

Super cool, right?

But thing gets tricky when you want to modulerize the above while loop as a function for usage in multiple places of your code. The following is what I did:

function define_dynamic_variables() {
  while IFS='=' read -ra line; do
    key=${line[0]}
    value=${line[1]}
    declare "$key"="$value"
  done < variables.txt 
}

But when I run the above function, the result is totally unexpected:

> define_dynamic_variables
> echo $FOO1
bash: FOO1: unbound variable

What's going on?

Trick 2 declare is local when inside function

Why the same piece code works without a function but not when inside a function?

A dive deep into the output of help declare shows:

When used in a function, `declare' makes NAMEs local, as with the `local'
    command.  The `-g' option suppresses this behavior.

The nuance is clearly explained by the help output. It basically says if you need to define a variable that live outside of the function scope, the -g option is needed, otherwise, the variable is defined local to the function scope and hence invisible after the function runs.

After fixing the function to be:

function define_dynamic_variables() {
  while IFS='=' read -ra line; do
    key=${line[0]}
    value=${line[1]}
    declare -g "$key"="$value"
  done < variables.txt 
}

Everything back to normal:

> define_dynamic_variables
> echo $FOO1
bar1

Trick 3 export dynamic variables as environment variable

Sometimes you might want to define the dynamic variable and export it as environment variable at the same time. You might come up with:

declare FOO=bar
export FOO

While it works, a more idiomatic why is to use the -x option:

declare -x FOO=bar  #when define the variable outside function
declare -gx FOO=bar #when define the variable inside function

Summary

Today, we learned 3 tricks related to defining dynamic variables in Bash:

• define dynamic variables with declare
• variable defined by declare is local when inside function
• export dynamic variables as environment variable

Last but not least, as RTFM stands, always refer to manual when something unexpected happens and this sometimes turns out to be more useful than simply googling online.

Comments

[Ben Sinclair]

Given the format you've chosen for storing variables is the same as regular variable assignment in bash, could you explain the technical benefits of doing this beyond sourcing the configuration file with . ./variables.txt ?

I see a couple of practical benefits:

• you don't have to stick to that format
• you can handle validation in your import loop

[Cheng Pan]

several benefits I can think of now:

• file sometimes might not be a desired format, eg when dealing with secrets in CI environment, you want the secrets live in memory instead of stored in files.
• sometimes you might be reading the variable name and value from external sources, eg. Vault. And the set of key/values are unknown ahead of time. In that case, you are not dealing with files directly, even though you can still use it as indirection.
• in my case when I discovered the tricks, I wanted to declare the variables dynamically and export them at the same time. If we source the variables from variables.txt, you still need to export the variables separately