In this blog, I will cover some key concepts and architecture of Pulumi. We will be building and provisioning the AWS landing zone.
Platform Engineering with Pulumi- Episode 1: Building the AWS Landing Zone with Pulumi
I have been learning Terraform, Ansible, Vagrant, etc to step into IaC, Writing Infrastructure code declaratively, and creating landing zone infrastructure, with a single click, is magic.
However, I wonder, Is Terraform the ultimate tech for IaC?? Coming with an application developer background, and entering the full-stack cloud developer — learning another language and syntax is painful.
GitOps & Configuration Management: I see more and more we are treating infrastructure code as application code, where we have a CI/CD pipeline, with continuous testing requirements. We need strict configuration management, and change management governance on IaC (sometimes stricter than Application Code to avoid Configuration Drift (I explained this problem in my other blog here)).
Logic: Sometimes I want to use conditional statements, control flows (such as loops) while writing IaC, which I could do easily with application programming languages. I had to learn YAML, JSON, and remember the complex semantics, to make sure I indent and define the objects properly. I don’t have sophisticated IDEs (like what I would get with application programming languages) to validate the syntax or semantics
Learning a new language: Terraform is a DSL (domain-specific language) and requires significant effort to learn and master.
How would it be, if I don’t have to learn any new language, use the existing IDEs and still write IaC…that is a dream come true for a full stack developer like me. And that is where I was super impressed with what Pulumi had to offer. It provides a framework for me to build Infrastructure code in any of the popular languages, that I already know!
To get a deeper understanding of this amazing tool. I am writing this series of blogs. In this “Platform Engineering with Pulumi” blog series, I will be building a landing zone on AWS to deploy my Node.js application and automate the deployments with GitOps. The series has these 3 blogs:
Episode 1: Building the AWS Landing Zone with Pulumi (current blog): In this blog, I will cover some key concepts and architecture of Pulumi. We will be building and provisioning the landing zone.
Episode 2: Building and Deploying the nodejs application: In this episode we will be building a simple ContactList application in nodejs/HTML and deploying it on the EC2 instance and store all the contacts in DynamoDB
Episode 3: Platform & Application deployment with GitOps automation: In this episode we will be automating the inftrastructure and application code deployment and changes using GitOps principles. We will be using GitHub Actions to continously deploy infrastrcture code and using AWS CodeDeploy to deploy application code
In this blog series, we will be building the following infrastructure on AWS and will run our applications in the EC2 instance. You guys might be wondering why EC2? If I were to do this, I would have done it on ECS/OpenShift/K8s, which is more modern. But then, there is no challenge. I selected this (legacy) architecture, to explore more concepts, so bear with me :-D and play along
Step 1: Install Pulumi
I am using macOS, it's very convenient for me to install using the following command. Please refer to the Pulumi documentation for instructions to install Pulumi for your target OS.
brew install pulumi
Step 2: Setup AWS CLI
Please refer to the AWS documentation to install and set up. In my case, I had already installed the CLI, created a credential on AWS IAM, and used aws configureto configure the CLI with the API key and secret access key, that I created in IAM.
We also need to export 2 environment variables, so that Pulumi can use them to log in to our AWS account.
export AWS_ACCESS_KEY_ID= && export AWS_SECRET_ACCESS_KEY=
Now we have aws CLI and Pulumi CLI ready to connect to our aws account. Let's now start writing some infrastructure code.
Step 3: Generate Pulumi project — boilerplate code for AWS in Python
Python?? yes that's the best part about Pulumi, I can write my infrastructure code in my desired language. Pulumi supports the following languages when I was writing this blog for AWS.
To generate the boilerplate code for AWS in Python language run the following command:
pulumi new aws-python
This will prompt us to log in to Pulumi, I used my GitHub as a sign-in. Once login is successful the Pulumi CLI generates the boilerplate code.
The following screenshot shows the generated files. Let's understand what is generated:
venv: is the Python virtual environment, which will help us work in a sandbox, as we install all the dependencies.
main.py: This is the main python code file, where we will be writing the code
Pulumi.yaml: This file has the project metadata, that we have provided while generating the boilerplate.
Pulumi.dev.yaml: This file has the configuration values for the stack, we can define different environments as stacks.
requirements.txt: This has all the python dependencies. we will be installing all the dependencies in the next step
Let's understand Pulumi architecture. The following picture shows a high-level architecture of Pulumi:
Project: Pulumi project is a collection of the files, of a module. Each project has a Pulumi.yaml file which has the project configuration. a new project can be created using Pulumi new command. Refer to Pulumi documentation for more details.
Program: Program is the actual infrastructure code, This is written in high-level languages such as python, javascript, etc.
Stack: Stack is a very important concept. This allows us to define various configurations for each environment and allows us to deploy and manage individual environments. typically we would have dev, test, prod environments. For each stack, we will have a Pulumi..yaml file. Refer to Pulumi documentation for more details
CLI: Pulumi CLI provides a command-line interface to build, run and manage the Pulumi runtime/deployment engine.
State: State is a very important component of the architecture, which stores the current state of the infrastructure, and makes sure, the configuration is synced up with the actual infrastructure on the cloud. By default the state is stored in the Pulumi backend, however, we can configure some other object storage (such as S3) to store the state. Please refer to Pulumi documentation for more details.
Hyperscaler integration: Pulimi internally uses hyperscaler APIs or Terraform modules to provision and manage the infrastructure.
Step 4: Activate virtual environment and install dependencies
Let's activate the venv using:
source ./venv/bin/activate
And install the dependencies with the following command:
pip3 install -r requirements.txt
You will see a bunch of dependencies installed in the Python environment. Now we are ready to write some infra code.
Step 5: Set up public/private key infra
When we provision EC2 instance, we will need a public/private key infrastructure to be programmatically created (if we had created the instance from the AWS console, we would have downloaded it manually.) Since we are automating the provisioning, we will create our public/private keys, and configure the EC2 to use our created key-pair
ssh-keygen -t rsa -f rsa -b 4096 -m PEM
This command creates rsa, rsa.pub
Step 6: Write Infra code
Here is the complete code, in snippets and explanation:
Line 1–7: import the dependent modules
pulumi is the core module.
pulumi_aws module has the aws objects, that we will be using.
provisioners module is an implementation of Terraform provisioner in Pulumi, which allows us to copy files, run commands remotely on the EC2 instance. Refer to documentation.
base64 has the library to encode and decode base64.
json module is required to serialize json configurations.
Line #9–13: We are initializing the config object, and retrieving the configuration from the Pulumi.dev.yaml file, where we have set the key name and public key. To create this secret configuration, we can use
pulumi config set --secret privateKey
pulumi config set --secret privateKeyPassphrase
the publicKey is not a secret is a direct copy-paste from the public key
Here is a screenshot:
Line #15–25: helper function to encode base64
Line #27-28: getting the secret configs from the respective Pulumi stack config file (since we are running in dev stack, it will read from Pulumi.dev.yaml)
Line #31–33: We are creating a keypair with our keys on AWS. Here is a screenshot after we run this code.
Now let's set up the VPC:
Line #37–66: In the above code, we are creating the VPC and an internet gateway. We are then creating the public subnet, in which we will be running the EC2 instance. We also need a route to the internet gateway, to access the public internet.
Line #69–95: We are creating a security group with 2 ingresses (One on port 80 to access the website, that we are going to build, and one on port 22, to SSH to the EC2 instance) and 1 egress for the EC2 instance to access the internet.
Line #98–124: We are creating a DynamoDB table with 2 fields — ContactName and ContactNumber.
Line #133–136: We are creating a VPC endpoint, to access this DynamoDB directly from the VPC, let's create a VPC endpoint (instead of going through the public internet).
Line #143–158: We are creating an EC2role that we will need when we set up the GitOps to deploy the applications using CodeDeploy (which I will be covering in my next blog).
Line #160–165: We are looking up the right AMI, using a filter.
Line #167–182: We are providing the user data to set up the EC2 instance when we start it for the first time, which does the typical yum update, installs cURL, Node.js, Yarn, Ruby, CodeDeploy (which we will need to deploy the applications using GitOps, I will be covering this in the next blog).
Line #183: We are creating an IAM instance profile and connecting that to the EC2role that we created, which we will need when we do the Code Deploy.
Line #184–195: We are creating an instance of the AMI, that we looked up, and creating a t2.micro instance, and passing the user data, IAM instance profile, public subnet, and the security group we created. We are also passing the keypair that we created so that we can connect to this EC2 instance using the key pair that we created.
Lets now run this code, and see the output, by running pulumi up
The following screenshot shows Pulumi comparing the state and providing the status of what resources will be created. I will discuss state management in a separate blog.
Once, we verify, we can access it to go ahead. The following screenshot shows the output of all the resources created.
Let's now connect to the EC2 instance and check by running:
ssh -i rsa ec2-user@3.239.26.182
Now that we have created the landing zone successfully using Pulumi, let's build and deployNode.js application and run it. Since it's already a long blog…we will continue in Episode 2.
Hope this was helpful, let's meet in the next blog…until then stay safe, and have fun, take care.
You can access the complete source code in my GitHub here.
References:
Top comments (0)