DEV Community: sgrilux

From Cloudformation to Terraform

sgrilux — Thu, 08 Sep 2022 20:59:53 +0000

When it comes to Infrastructure as Code, most of companies choose either AWS Cloudformation or Terraform, but sometimes (often) they start with one and, at some point, decide to use the second one, especially from Cloudformation to Terraform.

So in this post, I'll try to give some tips to start migrating resources from Cloudformation to Terraform.

The migration involves the following steps:

Update Cloudformation stacks
Create Terraform code
Import resources into terraform
Plan and Apply Terraform
Delete Cloudformation stacks

Looks easy right? Yes, it might be, it might not! It all depends on what are you trying to migrate, is it a big project, or it is just a couple of Cloudformation stacks, one region, or multiple regions?

But don't worry I got you covered :)

Let's start!

Update Cloudformation stacks

This is not necessary the first step, it could be done also after step 3 or 4, but for sure it's the most important part and needs to be completed before deleting any stacks.

DeletionPolicy

The first thing we want to make sure is that all resources have the attribute DeletionPolicyset to Retain. This will prevent resources to be deleted when you delete the stack.

Example:

Drift

It is possible that these old Cloudformation stacks have been somehow been abandoned and, because the lack of knowledge, some resources have been modified manually (very bad), so your infrastructure might drifts from what's been deployed through CFN.

You have two options:

Update and redeploy your cfn templates with the changes applied manually
As you are going to delete these task anyway, you can just take a note of the manual changes and add them into terraform (just make sure nobody will use these templates until the migration is completed)

Create Terraform code

Now it's time to write the terraform code that will replace Cloudformation templates. This is the "easy" part. Terraform documentation is your best friend.

For example the instance we have seen earlier can be turned into terraform with something like this:

Import resources into terraform

Once you think everything is coded in terraform you can start importing resources created by Cloudformation into the terraform state file.

To do so you will use the terraform import command.

This part is the most boring part, you need to check your cfn stacks and the AWS console to make sure everything is imported correctly.

Again, here the terraform documentation is a great resource. At the end of each resource documentation there is a little paragraph that explain how to import the resource. Most of the time is just specifying the ID of the resource, but sometime the ID is based on the concatenation of more attributes.

An example is the aws_route53_record resource, which needs the Zone ID + record + record_type, all separated by a _.

$ terraform import aws_route53_record.myrecord Z4KAPRWWNC7JR_dev.example.com_NS

While you are importing resources into your state file, I would suggest to run also some terraform plans, just to make sure you are importing them correctly and your code is aligned.

You can also build few bash scripts (or using you preferred scripting language) to help you with the import.

Plan and Apply terraform

Now that terraform code has been created it's time to run a final plan.

Sometimes is useful to target resources instead showing the full plan.

terraform plan -target "aws_ec2_instance.ec2_instance"

At this point, after the terraform code is created and resources are imported, I'm expecting there is nothing to "apply", at most few intentional changes.

Delete Cloudformation stacks

Finally, your infrastructure is now managed by Terraform and you are free to delete all Cloudformation stacks.

Remember to double check that DeletionPolicy: Retain is in place for all resources.

Some tips

Depending on the dimension of your migration you might be overwhelmed by all resources, import commands, subnets_id, security_groups, plans.

Before starting remember the following:

Don't rush : if it's just few cfn simple templates, the process it's pretty straightforward. However if you are managing a lot of templates, just take your time to think how you want to configure your project: is it multi-region? Do you have more environments? You want to be sure that your new code is easy to manage and to maintain and it's scalable.
Start with basics : this might sound obvious, but try to start with the foundations and then build up on it. This helps you thinking also how you want to structure your code. You can start building first all your network resources, like VPC, Subnets, Nat GW, etc... and then the application stacks that depends on it, EC2, ECS, Lambda. Probably you want to use separate repositories. For instance have the network repository and then application which will use terraform data sources to query data created from the network part. Or use modules, which is my next point.
Use modules : try to use terraform module as much possible. This will help you building a more readable code and also importing and planning your resources with terraform.

Example:

And that's it. Good luck with your migration and thanks for reading this post. If you have any question or need any help, please reach out.

Ciao

Meet Up: Modeling for relationships with DynamoDB

sgrilux — Tue, 06 Sep 2022 12:28:36 +0000

I thought this would be an interesting thing to share.

My company is sponsoring this meet up in Stockholm and it will be hosted by AWS on the 11th October.

Evolate, specialists in Amazon Web Services, invite you to an interactive and unforgettable evening with AWS Hero and DynamoDB specialist Alex DeBrie. He is the author of The DynamoDB Book, the comprehensive guide to data modeling with DynamoDB, and a world-renowned expert on data modeling, serverless architectures, and general AWS usage. Alex has helped thousands of developers learn DynamoDB and worked with a variety of impressive clients, including government agencies and publicly-traded enterprises. He had the second-most viewed session at AWS re:Invent, and has helped write some of the official guides for AWS database services. Alex is joining us to discuss the basics of DynamoDB and the various mechanisms to handle relationships in your DynamoDB data modeling.

Please join us, we have limited spots available for this, so sign up now.

https://www.google.com/maps/search/?api=1&query=59.333157%2C%2018.066673

ECS Container credentials

sgrilux — Sun, 08 May 2022 11:20:01 +0000

I've recently come across an issue assuming a role from an ECS container.

The task role of my ECS task had the policy to assume the role in the destination account and I also followed steps and troubleshootings from AWS documentation (See here) However I was still unable to assume the role.

My main problem was that the process wasn't able to see the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable so it couldn't get the credentials from the role.

As per AWS doc:

The environment variable AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is available only to PID 1 processes within a container. If the container is running multiple processes or init processes (such as wrapper script, start script, or supervisord), the environment variable is unavailable to non-PID 1 processes.

To set your environment variable so that it's available to non-PID 1 processes, export the environment variable in the .profile file. For example, run the following command to export the variable in the Dockerfile for your container image:

RUN echo 'export $(strings /proc/1/environ | grep AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)' >> /root/.profile

Now additional processes can access the environment variable.

As suggested I tried add the command in the Dockerfile but .... didn't help. I also tried to add the same in the docker entypoint script ... the same. I tried different things but still nothing.

My container was running as part of a gitlab pipeline (see the Gitlab Runner Job) and the only way I made it working was to export AWS_CONTAINER_CREDENTIALS_RELATIVE_URI in the pipeline job.

For ex.

test-job1:
  stage: test
  script:
    - export $(strings /proc/1/environ | grep AWS_CONTAINER_CREDENTIALS_RELATIVE_URI)
    - aws s3 ls --profile cross-account-role

At the moment this is the only way I could get the credentials from the task role. It's not elegant but it's working.

If anyone hade the same issue and managed to find a better solution I'll be happy to hear from you and if I helped even a bit to fix your problem then I am even happier.

CIAO!

Serverless Gitlab Runner #Part1

sgrilux — Sat, 30 Apr 2022 14:47:49 +0000

I come from a sysadmin background and Ive spent many years supporting infrastructures and applications during nights and weekends. Nowadays with Cloud you can easily build solutions that requires little or zero maintenance.

So today I want to talk about how we can implement a Serverless Gitlab runner solution on a AWS ECS Cluster using Fargate. This is the first part where I explain the concept and a diagram that shows how to implement it. Next I will show some code I will post on Github as soon as its ready.

What?

Lets start with a couple of questions: What is a gitlab runner? And What is Fargate?

A Gitlab runner is nothing more that an application that execute Gitlab jobs in pipelines. It gets the code from gitlab and the configuration from .gitlab-ci.yml and execute it. It can be a virtual machine or a docker container. You can use Shared runners provided by Gitlab or host your own runners. Im not here to go deep and explain how gitlab pipeline works and how to configure .gitlab-ci.yml, if you are here you probably know already and probably better than me :)

Fargate is an AWS managed service where you can run your ECS or EKS cluster. Being s managed service means that you dont need to do anything, AWS manages it for you, configuration and patches. In case of ECS what you need is create configurations that are needed by your application to run on the cluster, like memory, cpu, network and the docker image. These configurations are called task definition. Again, you all know how ECS works, right? :)

Why?

After the What? there is always the Why?. So why we want to build such solution and why on ECS and not on EKS? Good question

Well for start we do it for fun, why not! And second ECS is a bit easier than EKS. ECS is the AWS container orchestrator, it's a proprietary solution. EKS is basically kubernetes on AWS. You don't need to install manually k8s on EC2 instances.

With ECS you just need to configure a task definition and its done. It requires less expertise to setup a cluster and run your application.

On the other hand Fargate is the AWS managed compute engine. You don't need to provision any instance, AWS manages it for you. That means less operational overhead of scaling, patching, securing, and managing servers. Fargate is compatible with both ECS and EKS.

Here you can find also how to run runners on EKS.

HOW?

Now comes the fun question: how can we build this?

Heres a diagram

As you can see everything is mainly managed by few lambda functions. But let me explain how the flow works.

What we need is a Docker image for the Gitlab runner executor with Fargate drivers and some other images for the runner jobs which are then executed by the executor.

First you need to create a webhook in your repository which endpoint is a lambda, passing the token that will be used by the runner to register automatically to gitlab. When you configure the token you pass also the secret that will be used for authentication. Depends on your needs you can pass more parameters in the query strings.
The webhook lambda authenticates the call against a predefined secrete we have storer in SSM Parameter Store and put an event to EventBridge.
An EventBridge rule filters the event and call another lambda that will start a runner with Fargate drivers.
The next lambda get the event from EventBridge, read some parameters from SSM (like the runner token) and check if there are other runners running. If all is good it starts the gitlab runner executor passing the token and few more information.
The runner register against Gitlab and it's now ready to accept jobs.
When a new pipeline is triggered the runner starts new jobs in separate containers.
An event on EventBridge is also scheduled to run every tot time to check if the runner is doing something (if there are jobs running), and if not it will stop the runner to save money.

Well go more into the details on part 2 when Ill show you the code.

The problem

This solution is quite nice, although if you have some complex pipelines it's not perfect as there is a limitation in the taskDefinition that won't allow to override the job image parameter in a pipeline. That means you need bigger Docker images to manage different cases, which is not ideal.

References

https://docs.gitlab.com/runner/configuration/runner\_autoscale\_aws\_fargate/

Conclusion

I know this solution is not ideal, but I like experimenting and even improve what's already there. I'm sure both Gitlab and AWS will solve the problem of the image soon.

I hope you liked this article and gave you some inspiration and hopefully I can push some code in Github soon to share with you in the next part.

For now thanks for stopping by and reading it and I'm happy to "hear" your thought especially if you have already implemented it.

Ciao!

Tagging with terraform

sgrilux — Fri, 22 Apr 2022 19:25:33 +0000

We all know that when we share a resource in AWS with a different account, tags are not shared alongside the resource. Tags are account based so you need to assign tags to the other account as well.

Terraform comes to help with aws_ec2_tag which allows us to tag indivvidual resources that are created outside Terraform.

So lets jump directly to an example.

You have a networking account with a VPC that you are sharing with your production account. You want to Name your VPC with the same name that has been given in the networking account.

first you need to collect VPC data (using a provider that connects to the networking account)

data "aws_vpc" "selected" {
  filter {
    name = "tag:Environment"
    values = ["production"]
  }

  provider = aws.central-networking
}

then, with a production provider, you can tag your VPC

resource "aws_ec2_tag" "my_prod_vpc" {
  resource_id = data.aws_vpc.selected.id
  key = "Name"
  value = data.aws_vpc.selected.tags.Name

  provider = aws.production
}

You can also use for_each to assign multiple tags to the same resource:

resource "aws_ec2_tag" "my_prod_vpc" {
  for_each = local.tags

  resource_id = data.aws_vpc.selected.id
  key = each.key
  value = each.value
}

Thats it, easy peasy!

I hope you have enjoyed itsee you on the next post.