Dynamic injection of secrets into ECS Task Definition with SSM Parameter Store

#aws #ecs #docker #automation

Maybe I have just reinvented the wheel, and some well-known methods of doing this exist, but...

TL;DR;:

SECRET_PATH="/development/api/" && \
aws ssm get-parameters-by-path --path $SECRET_PATH --query "Parameters[*].{name:Name,valueFrom:ARN}"| \
jq --arg replace $SECRET_PATH 'walk(if type == "object" and has("name") then .name |= gsub($replace;"") else . end)'

More details below.

Let's say we have an ECS Service with its Task Definition, and we want to pass some sensitive data for the container(s) described in this Task Definition.
Generally, that would look as follows:


... some other configs here
"portMappings": [
    {
    "hostPort": 0,
    "protocol": "tcp",
    "containerPort": 80
    }
],
"secrets": [
    {
    "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/SOME/PATH/PARAMETER-1",
    "name": "PARAMETER-1"
    },
    {
    "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/SOME/PATH/PARAMETER-2",
    "name": "PARAMETER-2"
    },
    {
    "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/SOME/PATH/PARAMETER-3",
    "name": "PARAMETER-3"
    },
    ... and so on ...
],
"memoryReservation": 128,
... some other configs there

And this is perfectly fine while you have a few services in a single environment with few secrets.

But this becomes a nightmare if the number of secrets changes frequently and you have many environments/workspaces (development/staging/production/QA/etc...) with many services in each.

I tried to develop the way for dynamic injection of all secret variables related to the particular ECS service using Parameters Store and some CLI actions.

Suppose you have a service api in ECS within three environments: development, staging, and production. In such a case, you would probably have the following hierarchy of secrets in the Parameter Store:

/development/api/parameter-1
/development/api/parameter-2
/development/api/parameter-3
... 

/staging/api/parameter-1
/staging/api/parameter-2
/staging/api/parameter-3
...

/production/api/parameter-1
/production/api/parameter-2
/production/api/parameter-3
...

Even now, with manual secretes management in TaskDefinitions, you would have to maintain 3 files (api-dev, api-stage, api-prod) and hardcode the lists for all those secrets.

But here is what can be done to automate the secrets injection into Task Definition:

Get all secrets (parameters) by path without the explicit specification of their names (we just need to inject all that relates to our service)
Format the received JSON according to the syntax of Task Definition
Get valid JSON object that we can simply insert into Task Definition template file with awk or similar

So once again, the oneliner from TLDR section above:

At first, we define the base path for secrets:

SECRET_PATH="/development/api/"

Then we call AWS CLI command to get needed secrets from parameter store, but we don't need all the info about each secret so we use the query option:

aws ssm get-parameters-by-path --path $SECRET_PATH --query "Parameters[*].{name:Name,valueFrom:ARN}"

The output at this stage would look something like this:

[
    {
        "name": "/development/api/parameter-1",
        "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-1"
    },
    {
        "name": "/development/api/parameter-2",
        "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-2"
    },
    {
        "name": "/development/api/parameter-3",
        "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-3"
    }
]

But we need to remove the path from the secret name and leave only the name itself:
we need "name": "parameter-1"
instead of "name": "/development/api/parameter-1"

jq --arg replace $SECRET_PATH 'walk(if type == "object" and has("name") then .name |= gsub($replace;"") else . end)'

So the final output will look like this:

[
  {
    "name": "parameter-1",
    "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-1"
  },
  {
    "name": "parameter-2",
    "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-2"
  },
  {
    "name": "parameter-3",
    "valueFrom": "arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-3"
  }
]

However, to make this work later with awk or sed, it is better to remove all linebreaks so to avoid the dances around linebreaks escaping - simply add -c option to jq and the output will look as follows:

[{"name":"parameter-1","valueFrom":"arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-1"},{"name":"parameter-2","valueFrom":"arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-2"},{"name":"parameter-3","valueFrom":"arn:aws:ssm:us-east-1:123456789123:parameter/development/api/parameter-3"}]

With such an approach we need to maintain the secrets only in Parameter Store (and we don't need to copy/paste their names and ARN's manually) and maintain a single template for a Task Definition per service with some keyword as a value for 'secrets' objects(i.e. "secrets":REPLACE). And we simply replace this keyword by our JSON string using awk or sed later in our CI/CD for registration of the new Task Definition.

Example for awk. Suppose you put the json into 'SECRETS' variable and you need to replace the 'REPLACE' placeholder with its value:

awk -v r="$SECRETS" '{gsub(/REPLACE/,r)}1' td-template.json > td.json

Top comments (3)

Raphael Oliveira • May 18 '21

Nice work!
I would like to know if there is any way to add new parameters to the json file?
I tried like this:

jq --argjson secretsInfo "$ (<file.json)" '.containerDefinitions [1] .secrets + = [$ secretsInfo]' file.tmp

However, the file has an additional error occurring when updating the task in aws.

"secrets": [
[ <--- This
{
"name": "TESTKEY2",
"valueFrom": "arn:aws:ssm:us-west-1:xxxxxx:parameter/TESTKEY"
},
{
"name": "TESTKEY2",
"valueFrom": "arn:aws:ssm:us-west-1:xxxxxx:parameter/TESTKEY2"
}
] <--- This
]

Stan Winsett • Jul 9 • Edited

I know this is old, but in case anyone DOES look up something like this:

This is done, also, via chamber. It uses the path to grab the parameters at the task start time, and inject them as env variables.
raw.githubusercontent.com/SignalMe...

Grab the init.sh, and use that as your entrypoint, with an env variable (SECRET_SERVICE) in your taskdef that matches your path, and you are set. When you add a new parameter, point your app's variable to that parameter (chamber converts it to all upper case), and you are done. Change the value of a parameter? Just restart the task.