If you use Amplify and Git branches, you are going to have a bad time

AWS Amplify is...

A combination of tools that aim to assist with development, deployment, and hosting of your serverless application.

AWS Amplify is three things:

1. It's a cloud service that includes consoles, monitoring tools, build and deploy configurations, and so on.
2. It's a CLI tool that allows you to create Amplify environments, modify them, add resources, and manually publish if you so choose.
3. It's a library for frontend development that simplifies using the resources in your environment like authentication, HTTP calls to your gateways, and more.

You don't have to use the CLI or the library, but you would probably want to since it saves a lot of headaches.

Deploying applications with Amplify

When you create an Amplify app (which contains different environments under it), you can connect it to a branch in your repository. That creates a Hosting Environment. Every hosting environment can be connected to a specific Backend Environment.
When connected to a branch, Amplify will be notified whenever a code was pushed to that branch. At that point, Amplify will rebuild the frontend code in the branch and deploy it.
If connected to a backend environment, Amplify will also redeploy the backend, based on the contents of the branch.

Amplify CLI tries to follow how Git works

After you created a backend environment dev and deployed it, Amplify considers it as the equivalent of Git remote. When you want to work on it, you must pull that environment to your local machine. You can then change it and push your changes back to remote.

When you pull an environment with Amplify CLI, it does two things:

1. Downloads the content of the environment. The code of the lambdas, the paths defined in the gateway, the templates of the resources, etc.
2. Creates/modifies files that say "what is the current environment in this machine". Similarly to Git, where you can checkout multiple branches and your local Git will know which branch you are currently using. For example, a file called aws-exports.js that contains a JSON with a lot of configurations. When your application loads, you need to pass the content of that JSON to the AWS Amplify library so it will know what endpoint to contact when your code performs a HTTP call.

When you push an environment, Amplify CLI will first check against the remote to see if there is any local modifications. If there are, it will compile a .zip with all the files required and upload it to the one currently configured.

That's a lot of information. I thought Amplify was supposed to make my life simpler...

Tools that make your life easier usually do it by abstracting a lot of knowledge and technical understanding. Your Nespresso machine doesn't even show you the coffee grinds. You just put this aluminum pocket into the hole, push the button, and viola!

Amplify definitely will make your life easier. But this will come with a price, and when something breaks, you must understand what it does in order to fix the problem or preemptively avoid the pitfalls.

The reason Git and Amplify don't play well together

Imagine the following scenario:
You have a main Git branch and you have your mainBE backend environment that currently has a lambda function foo.

To develop a new backend service, you branch out of main to a feature branch called feat.

> git checkout main
> git checkout -b feat

You also clone mainBE to a new
environment: featBE

> amplify env checkout mainBE
> amplify add env featBE
> amplify push --y

This action changes team-provider-info.json by introducing a new key to the object: featBE. The value currently (almost) equals that of mainBE. It will now look like this:

"mainBE": {
    "awscloudformation": {
      [...]
      "Region": "us-east-1",
      "AmplifyAppId": "someRandomId"
    },
    "categories": {
      "function": {
        "foo": {
          [...]
          "envVarName": "environment variable value"
        },
    }
},
"featBE": {
    "awscloudformation": {
      [...]
      "Region": "us-east-1",
      "AmplifyAppId": "someRandomId"
    },
    "categories": {
      "function": {
        "foo": {
          [...]
          "envVarName": "environment variable value"
        },
        "bar": {
          [...]
        },
    }
}

You then add the new lambda called bar and push it to your feature environment.

> amplify add function
> amplify push

Now the two entries are different. featBE has details of a new lambda, while mainBE does not.

You develop the new service, test it locally and deployed, and everything is done. You then merge your feature branch back into the main branch.
Problem is, any CI/CD process that tries to deploy the backend will fail. The content of team-provider-info.json under the mainBE key has no definition for the bar lambda.

To successfully deploy mainBE with the new service, you will need to pull featBE, switch to mainBE, and push to it.

> amplify env checkout featBE
> amplify pull --y // to overwrite the current content, assuming this is done on a different machine
> amplify env checkout mainBE
> amplify push

Amplify will see the new content and update mainBE.
But, like before, this will update team-provider-info.json, which means your backend environment doesn't match the code in your source control.

You'll need to push this new difference into your git main branch. This might sound somewhat reasonable, but if your setup uses the Git flow of having dev, testing, and production branches (and maybe more), you'll find that this cycle of having to merge branches, merge backends, then merge again will grow exponentially with every branch in your flow.

For example: let's assume we already performed all the above mentioned, but we also have staging and production environments.

First, we need to merge the new feature to staging:

> git checkout staging
> git merge main

At this point, stagingBE won't build. We need to fix this by taking the backend definition in mainBE and applying them to stagingBE

> amplify env checkout mainBE
> amplify pull --y
> amplify env checkout stagingBE
> amplify push

Now our local branch is ahead of main (and staging)

> git checkout main
> git commit -m "Updated Amplify setup of stagingBE"
> git push

main is updated, but now main and staging are not aligned

> git checkout staging
> git merge main

stagingBE should be ok now, but to make sure that the deployed backend matches 100% to the source code, we will redeploy

> amplify push --y

Now staging environment is ok, but we still have production to deal with

> git checkout production
> git merge staging
> amplify env checkout stagingBE
> amplify pull --y
> amplify env checkout productionBE
> amplify push --y
> git checkout main
> git commit -m "Update Amplify setup of productionBE"
> git push
> git checkout staging
> git merge main
> git checkout production
> git merge staging
> amplify push --y

This is the abbreviated version as most Amplify CLI commands require more information either manually entered or via headless mode.

This whole process can take at best ~15 minutes because Amplify takes time to pull and push. At worst, your CI/CD: tests, automation, deployments, etc will need to execute between after each git push in this process, which will take much much longer.

In the end, no matter what you do, you will end up in a situation where the deployed backend is not based on your source code, which is not a situation you want to be in, certainly not on a regular basis.

By the way, the CI/CD built into the Amplify console won't do all of this. It just takes whatever there is in your source code and tries to deploy.

Finale

All in all, Amplify is a pretty powerful tool but seems to support a common, basic use case. If you need anything more than this, you're probably better off using other tools.

Comments

[Dan Rivett]

Great article, you concisely explained the exact reason why we have always avoided the workflow Amplify advocates under their Team environments page.

We've instead kept the concerns of Amplify environments and git code branches separate to avoid running into the issues you outline. We haven't unfortunately solved the CI/CD dance yet as each developer manages their own Amplify backend, but we've at least avoided the issues you outline above.

I'm not sure why the categories data in team-provider-info.json needs to be stored separately per environment under source control. There must be a good reason, but seems like there's a better way to avoid all this pain.

[Yonatan Korem]

Hi Dan,

First, thanks for the feedback.

We also just can't figure out why the team-provider-info.json works like that. It'll be interesting to hear the actual reasons.

Can you further explain your workflow? Each dev works on their own environment, but how do you merge changes into the main environment?

[Jeff Hachtel]

Amplify helped us get an app up and running in a very short time. We took that route because integrating auth was so easy, then found out how easy everything else was - until it wasn't. We abandoned Amplify and took another route for exactly the reasons you highlight here.

[Yonatan Korem]

Hi Jeff,

That is exactly what happened with us. Authentication was a big selling point for us since we not only set up Cognito with Amplify CLI, but also used the Amplify React library that provides a login screen that fully integrates with Cognito.

It is incredible what you can achieve in one hour with Amplify. But after one month, you'll find it is more of a hinderer.

[Pablo Bermejo]

Great, great, article

[Conor Woods]

I'm not sure i fully understand the issue.
If the amplify/backend/backend-config.json file is saved in git, does this file determine what resources are deployed, not the "categories" portion of the team-provider-info.json file?

[Yonatan Korem]

Hi Conor,

From my understanding, which is based on Amplify failing to deploy and figuring out how to fix it, part of what happens is based on team-provider-info.json file.

If you add an environment variable to a lambda, the files within the lambda directory will say something that translates to "this lambda is expecting to have an environment variable named X".
But, the value of it is defined in the team-provider-info.json file, under the key that matches each environment. That is how Amplify allows you to set different values for the same variable, per environment.

When you add an environment variable, it does not automatically appear in all environments, only in the one you are currently on.
If you commit that file and try to build a different environment based on your git content, it will fail.

Same goes for any resource that you add or remove.

[Conor Woods]

thanks yonathan, the difference in setups must be that I am currently not using env variables.

I got so frustrated with having to set the env variables per lambda function that I now deploy my SSM independently from amplify and share them with all the lambda functions.

[Yonatan Korem]

The same happened when we added lambdas. If your backend definition, which is environment agnostic, defines something which is not included in the relevant environment entry in team-provider-info.json, then when you try to deploy it will fail.

[Conor Woods]

I think my setup might be slightly different as I'm not seeing the same issue.

To compare, here are the steps I take to create and deploy a project:

• I create dev, test, prod envs (in separate AWS accouts) and check in the team-provider-info file to git.
• The CICD (github actions) kicks off when the dev branch is changed and deploys to dev + test
• A merge to main kicks off a prod deploy.
• The CICD checks out the repo, npm i, amplify init (i use an init.sh file for this, see below), amplify push

If i create a new function in dev and merge to main, it creates the new function OK in the prod env, even though the team-provider file remains the same in git.

`#!/bin/bash
set -e
IFS='|'

AWSCLOUDFORMATIONCONFIG="{\
\"configLevel\":\"project\",\
\"useProfile\":false,\
\"profileName\":\"default\",\
\"accessKeyId\":\"${AWS_ACCESS_KEY_ID}\",\
\"secretAccessKey\":\"${AWS_SECRET_ACCESS_KEY}\",\
\"region\":\"us-east-1\"\
}"
AMPLIFY="{\
\"projectName\":\"projname-server\",\
\"envName\":\"${ENV}\",\
\"defaultEditor\":\"code\"\
}"
FRONTEND="{\
\"frontend\":\"ios\"
}"
PROVIDERS="{\
\"awscloudformation\":$AWSCLOUDFORMATIONCONFIG\
}"
CODEGEN="{\
\"generateCode\":true,\
\"codeLanguage\":\"typescript\",\
\"fileNamePattern\":\"graphql/*/.graphql\",\
\"generatedFileName\":\"API.ts\",\
\"generateDocs\":true,\
\"maxDepth\":2\
}"
AUTHCONFIG="{\
\"appleAppId\":\"${AMPLIFY_APPLE_ID}\",\
\"googleAppIdUserPool\":\"${AMPLIFY_GOOGLE_CLIENT_ID}\",\
\"googleClientId\":\"${AMPLIFY_GOOGLE_CLIENT_ID}\",\
\"googleAppSecretUserPool\":\"${AMPLIFY_GOOGLE_CLIENT_SECRET}\"
}"

CATEGORIES="{\
\"auth\":$AUTHCONFIG\
}"
amplify init \
--amplify $AMPLIFY \
--categories $CATEGORIES \
--frontend $FRONTEND \
--providers $PROVIDERS \
--codegen $CODEGEN \
--yes
`

[Yonatan Korem]

First, nice job!

Second, I think what you've made is the custom CI/CD process that solves this problem because you also needed to have different environments on different accounts.

[Conor Woods]

Thank you, it took a lot of pain, frustration and searching through github issues to find that resolution.
I agree with your article: amplify sells a great story if you go down the happy path, but stray slightly, and you're left frustrated.

[Oseh Mathias]

This is how I ended up doing it after 2 years of struggle

Have one branch where you check out to all of your amplify envs and push to them to keep them in sync

For me it's the dev branch

I'll build the dev API and resources, then when I'm happy, while still on the dev branch, amplify env checkout staging, amplify push --y, amplify env checkout main, amplify push --y

Then, you can merge the branches willy nilly and they'll always be in sync