DEV Community: Sean Ziegler

Learning AWS on your own? 10 services you need to know

Sean Ziegler — Wed, 02 Sep 2020 13:01:31 +0000

As of January 2020, there are approximately 190 AWS services. Even if you study full time, that's a lot if you're trying to learn AWS on your own. Fortunately, you can build full-featured applications after learning a small set of the core AWS services. These core services will form the foundation of the vast majority of applications built on AWS.

Can you learn AWS on your own?

Yes, absolutely, you don't need any formal training or even an online course to learn AWS. How hard will it be? Well, that depends on your past experience in the IT / software industry. If you consider yourself a pro, you might not struggle very much. For IT beginners, AWS might not be the best starting point. Ultimately, it is possible to learn AWS on your own and you have to decide for yourself (based on your goals and experience) what path is best for you.

Study the most important AWS services first

Before you get caught up in the AWS machine learning section, consider getting a solid grasp of the fundamental services. It will make you a better engineer in the long run, I promise. I consider these the ten most important AWS services. Obviously, this is just my opinion. This list will change depending on who you ask and what they do in the cloud.

Start with this list and once you have a basic understanding of AWS, branch out to the services specific to your needs.

I highly recommend reading the Getting Started guide and FAQ for each service. After, research the key topics to deepen your understanding. If you can do that, you're well on your way to a solid AWS foundation and maybe even an entry level certification.

Amazon EC2 - (Elastic Cloud Compute)

Amazon EC2

What is EC2? | FAQ

Amazon EC2 provides virtual servers, called instances, in the cloud. These are the cloud replacement for server racks in a data center. You can provision as many instances as you need and only pay for what you use. EC2 instances are available on-demand and can automatically scale up or down depending on your needs.

Key topics to understand

Instance types
Regions & Availability zones
User-data
Instance store volumes (EBS)
Autoscaling & Scaling policies
Security groups
SSH keypairs
Public / Private / Elastic IPs

AWS IAM - (Identity and Access Management)

AWS IAM

What is IAM? | FAQ

AWS IAM allows you to restrict access to your cloud resources. With IAM, you can specify exactly who (or what) can interact with each of your AWS resources. IAM enables you to give the right people access to the right areas of your AWS account and enforce security policies as you see fit.

Key topics to understand

Users and groups
Service roles
Trust relationships
Identity federation
Managed and inline policies
Security tokens (STS)

Amazon S3 - (Simple Storage Service)

AWS S3

What is S3? | FAQ

S3 is low-cost highly scalable data storage for the cloud. S3 enables you to store your data in folders called buckets that are cost efficient, high performing, and reliable. You can upload and retrieve objects via the AWS SDK, CLI, or management interface. S3 is one of AWS's oldest and most popular services because it is so versatile.

Key topics to understand

Storage classes and retrieval times
Lifecycle mangement
Multi-part uploads
Data consistency model
Cross region replication
CORS
Static website hosting
Pre-signed URLs

Amazon RDS - (Relational Database Service)

Amazon RDS

What is RDS? | FAQ

RDS provides databases in the cloud. RDS supports most of the major database engines and can scale up or down depending on the load. It automates many of the common tasks associated with database management such as software patching, backups, and recovery. RDS is a transactional database, in contrast DynamoDB is Amazon's serverless NoSQL offering.

Key topics to understand

Database engine options
Database instance types
Multi-AZ deployments
Read replicas
Auto-scaling
Encryption options
Backups and restores

Amazon Route53

Amazon Route 53

What is Route53? | FAQ

Route53 manages DNS within the AWS ecosystem. Route53 serves three major functions: registering domains, DNS routing, and health checking. It is highly available by default and can support high traffic out of the box. You can use any DNS registrar with AWS, but there are some advantages to staying within the AWS eco-system (health checks and alias records to name a few).

Key topics to understand

Domain concepts - domain name, registrars, TLDs
DNS record types
Health checks
Routing policies
DNS failover

Amazon ELB - (Elastic Load Balancer)

Amazon ELB

What is ELB? | FAQ

ELB provides load balancers which distribute incoming traffic across multiple resources inside of AWS, like EC2 servers or ECS nodes. If your traffic is too much for one server or you want to make your application highly available, you need a load balancer. ELB increases your applications reliability by handling SSL, health checks, and traffic routing so your application servers can focus on their primary task.

Key topics to understand

Health checks
Stickiness
CLB vs. NLB vs. ALB
Cross AZ load balancing
Connection draining
SSL Certificates

Amazon CloudFront

Amazon CloudFront

What is CloudFront? | FAQ

CloudFront is Amazon's CDN offering. CloudFront serves static files like HTML, CSS, JS, and images through a worldwide network of edge locations. Using CloudFront edge locations serves your website's static assets from locations that are physically closer to the user. CloudFront adds speed and reliability while reducing costs and taking some work off more expensive compute resources.

Key topics to understand

Edge locations
Origins
Signed URLs
CORS
Origin Access Identities

Amazon VPC - (Virtual Private Cloud)

Amazon VPC

What is Amazon VPC? | FAQ

VPCs are a tricky topic. With a VPC, you can define the network that your resources use to communicate. Strictly, a VPC is not a service, but it's something that you need to understand because it forms the backbone of networking in AWS. VPCs form the backbone of AWS solutions and can even link-up on-premises and cloud resources.

Key topics to understand

Public vs. private IPs
Subnets
Route tables
Access control lists and security groups
Internet & NAT gateways
VPC Peering
VPC Endpoints

AWS Lambda

AWS Lambda

What is AWS Lambda? | FAQ

AWS Lambda is one of my favorite AWS services. Lambda functions are small functions that only run while they are being executed. Lambda functions are serverless and scale easily, which means you only pay for what you use and can handle heavy traffic out of the box. If no one is using your website or application, you don't pay.

Key topics to understand

Supported runtimes
Function execution limits
Triggers and integrations
Lambda@Edge

Amazon API Gateway

API Gateway

What is API Gateway? | FAQ

Amazon designed API Gateway for building APIs at scale. API Gateway enables you to build APIs that directly return responses or act as a proxy for other AWS services. API Gateway includes features for managing API caching, API keys and usage plans, request / response transformation and much more. Serverless applications commonly rely on an API gateway proxying requests to an array of Lambda functions.

Key topics to understand

REST vs. HTTP APIs
Resources and methods
Triggers and integrations
Stages, versioning, and deployments
API keys and usage plans
Lambda custom authentication

Bonus: AWS CLI and SDK

Usually, people start learning AWS using the console, but you'll quickly realize you also need a method of interacting with the services through code and the command line. Many amateurs who learn AWS on their own work solely with the web interface, but most professionals primarily use the CLI. I encourage you to research the AWS CLI and the AWS SDK that corresponds to the language you're most comfortable with.

AWS CLI

What is the AWS CLI? | CLI Reference

The AWS CLI (Command Line Interface) is exactly what it sounds like. Instead of clicking buttons on a GUI in a web browser, you can install the AWS CLI and manage all your AWS resources from a terminal. Many professionals prefer the AWS CLI because it is more powerful, and personally, I agree with them.

AWS SDK

The AWS SDKs (Software Development Kits) are Amazon's solution for interacting with AWS resources inside of code. If you want to store images in S3 with a Python script, you'll need the Python SDK. Similarly, if you want to store records in an RDS database with PHP, you'd need the PHP SDK. I suggest getting familiar with at least one of these, you'll probably be needing it.

C++
Go
Java
JavaScript
.NET
Node.js
PHP
Python
Ruby

Where to go next

After you've learned the first 10 services, where do you go next? You should decide what you want to build and focus on the services that will enable you to do that. Learning an AWS specialty on your own can take your career to an even higher level.

Serverless

Lambda
Step Functions
SQS
SNS

DevOps

CloudPipeline
CloudFormation
ECS
EKS

Monitoring and administration

CloudWatch
CloudTrail
Systems Manager

Data workloads

Athena
EMR
Kinesis
Elastic search

Machine learning

Sagemaker
Lex
Polly
Textract
Transcribe

Hopefully, this article helped you decide where to start learning AWS. Learning AWS on your own can be a very challenging task, but I promise it's well worth it!

How to build an API with AWS Lambda and the Serverless Application Model (SAM)

Sean Ziegler — Sun, 05 Jul 2020 11:16:39 +0000

Designing an API on AWS is hard. Designing a reliable API on AWS is even harder. The AWS Serverless Application Model (SAM) is an open source framework that makes deploying serverless resources much easier. SAM extends CloudFormation, so you can use all the usual CloudFormation resources inside of a SAM template. SAM can deploy serverless functions, databases, and APIs.

In this post I’ll be covering the basics of using the AWS SAM template language, the AWS SAM CLI, and some advanced features like usage plans and API keys.

The SAM template language and the SAM CLI

SAM is composed of two parts, the SAM template language and the SAM CLI. The SAM template language provides a method of describing serverless resources in JSON or YAML blocks. The SAM template language provides seven resource types.

AWS::Serverless::Api
AWS::Serverless::Application
AWS::Serverless::Function
AWS::Serverless::HttpApi
AWS::Serverless::LayerVersion
AWS::Serverless::SimpleTable
AWS::Serverless::StateMachine

For example, I can define a Lambda function using an AWS::Serverless::Function block. Here I’ve created a Lambda function using python3.6 and SAM will store the code in an S3 bucket.

Type: AWS::Serverless::Function
Properties:
  Handler: index.handler
  Runtime: python3.6
  CodeUri: s3://bucket/key

The SAM CLI provides a command-line interface for building, testing, and deploying the serverless resources described by your SAM templates. The SAM CLI has eight commands.

sam build - Processes your template file, prepares code and dependencies for deployment
sam deploy - Deploys resources described in SAM template
sam init - Initializes a new SAM project
sam local - Creates local version of serverless resources for easier testing and debugging
sam logs - Show your resources's logs
sam package - Creates a zip and uploads an artifact to S3
sam validate - Validates template file is valid CloudFormation syntax
sam publish - Publishes an application to the Serverless Application Repository

Start a SAM project with the SAM CLI

Let's get started building a simple API that takes two URL parameters, a and b, and returns their product.

AWS SAM has a CLI that makes creating a new project simple. Install the AWS SAM CLI and then use sam init to start a new project. I like to use the quick start templates to get a project going quickly, but you can also select 2 for a Custom Template Location and give it a filepath or URL.

sam init

Which template source would you like to use?
    1 - AWS Quick Start Templates
    2 - Custom Template Location

Choice: 1

Choose a language for your project's runtime. I'll be using python3.6.

Which runtime would you like to use?
    1 - nodejs12.x
    2 - python3.8
    3 - ruby2.7
    4 - go1.x
    5 - java11
    6 - dotnetcore3.1
    7 - nodejs10.x
    8 - python3.7
    9 - python3.6
    10 - python2.7
    11 - ruby2.5
    12 - java8
    13 - dotnetcore2.1
    14 - dotnetcore2.0
    15 - dotnetcore1.0
Runtime: 9

Name the project and select the Hello World template. This will generate a sample project with a Lambda function and a template for an API Gateway resource.

Project name [sam-app]: SAMdemo

Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git

AWS quick start application templates:
    1 - Hello World Example
    2 - EventBridge Hello World
    3 - EventBridge App from scratch (100+ Event Schemas)
    4 - Step Functions Sample App (Stock Trader)
    5 - Elastic File System Sample App
Template selection: 1

-----------------------
Generating application:
-----------------------
Name: SAMdemo
Runtime: python3.6
Dependency Manager: pip
Application Template: hello-world
Output Directory: .

Next steps can be found in the README file at ./SAMdemo/README.md

SAM generates a bunch of boilerplate code and a few folders for you. There are four main parts to know.

template.yaml - Defines your SAM resources
hello_world/ - Directory for Lambda function code
tests/ - Directory for unit tests for your function, run using Pytest
events/ - Mocked events for testing functions

Create a Lambda function

We need to create a function that can take in two URL parameters and return a product. I will reuse the hello_world function the SAM CLI generated by renaming the hello_world folder multiply and editing the app.py file inside it.

mv hello_world/ multiply/
vim multiply/app.py

The app.py file defines the Lambda function. This function will take in two integers, a and b, as URL parameters and return their product. We can access the URL parameters through the event variable. The event variable contains data about the API Gateway event that triggered this Lambda function.

import json

def lambda_handler(event, context):
    '''<snip>'''

    a = event["queryStringParameters"]['a']
    b = event["queryStringParameters"]['b']

    product = int(a) * int(b)

    return {
        "statusCode": 200,
        "body": json.dumps({
            "product": product,
        }),
    }

Lambda functions must return a JSON response, so I've chosen to just dump the product result into the body of the response.

Define an API Gateway using a SAM template

SAM also generated a file called template.yaml. I will tell SAM that I want to deploy a Lambda function by including an AWS::Serverless:Function block inside the SAM template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Function and API for multiplying two numbers

Resources:
  MultiplyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: multiply/
      Handler: app.lambda_handler
      Runtime: python3.6
      Events:
         Multiply:
           Type: Api
           Properties:
             Path: /multiply
             Method: POST
             RestApiId:
               Ref: MultiplyApi
Outputs:
  MultiplyApi:
    Description: "API Gateway endpoint URL for Prod stage for Multiply function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/multiply/"
  MultiplyFunction:
    Description: "Multiply Lambda Function ARN"
    Value: !GetAtt MultiplyFunction.Arn
  MultiplyFunctionIamRole:
    Description: "Implicit IAM Role created for Multiply function"
    Value: !GetAtt MultiplyFunctionRole.Arn

This template will deploy a Lambda function backed by the code in the multiply folder. After the deployment, the Outputs section will return an API Gateway Endpoint, a Lambda function ARN, and an IAM role for the function.

Deploy the SAM template with the SAM CLI

It is time to deploy the SAM template. sam deploy --guided starts an interactive session which guides you through the deployment process.

~/SAMdemo ❯ sam deploy --guided                                                         

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Not found

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sam-app]: SAMDemo
    AWS Region [us-east-1]: 
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [y/N]: N
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: Y
    MultiplyFunction may not have authorization defined, Is this okay? [y/N]: Y
    Save arguments to samconfig.toml [Y/n]: Y

After you finish the guided process, SAM will generate a change set (Just like CloudFormation) and deploy your resources. You will see the outputs from template.yaml printed onto the screen.

CloudFormation outputs from deployed stack
-------------------------------------------------------------------------------------------------
Outputs
-------------------------------------------------------------------------------------------------
Key                 MultiplyFunctionIamRole
Description         Implicit IAM Role created for Multiply function
Value               arn:aws:iam::<snip>

Key                 MultiplyFunction
Description         Multiply Lambda Function ARN
Value               arn:aws:lambda:us-east-1:<snip>
Key                 MultiplyApi
Description         API Gateway endpoint URL for Prod stage for Multiply function
Value               https://<snip>.execute-api.us-east-1.amazonaws.com/Prod/multiply/
-------------------------------------------------------------------------------------------------

Successfully created/updated stack - SAMDemo in us-east-1

Testing the API with generated events

SAM provides a method for testing your API locally prior to deployment. First, let's generate a mock API Gateway event. The command sam local generate-event apigateway aws-proxy > events/multiply.json will generate a fake API event and save it to a JSON file. Change the queryStringParameters in multiply.json to some integers for a and b.

"queryStringParameters": {
    "foo": "bar"
  },

  "queryStringParameters": {
    "a": "5",
    "b": "3"
  }

Now lets use sam local invoke to invoke the Lambda function and provide it the multiply event.

sam local invoke -e multiply.json

Invoking app.lambda_handler (python3.8)

Fetching lambci/lambda:python3.8 Docker container image......
Mounting /Users/seanziegler/Coding/SAMdemo/multiply as /var/task:ro,delegated inside runtime container
START RequestId: a17fdb0c-15bc-1cb0-c8a5-836299856b0c Version: $LATEST
END RequestId: a17fdb0c-15bc-1cb0-c8a5-836299856b0c
REPORT RequestId: a17fdb0c-15bc-1cb0-c8a5-836299856b0c  Init Duration: 93.96 ms Duration: 2.85 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 25 MB

{"statusCode":200,"body":"{\"product\": 15}"}

The function returned product: 15 which is what we expected. You can use sam local generate event to generate mock events for many services including S3, APIGateway, DynamoDB, SQS, and more. Generating events and testing locally is a great pattern for ensuring your serverless functions are reliable.

Configuring an API Key and usage plan

Adding API keys and a usage plan to an API is a straightforward process. It's possible to set up both using the Auth object on AWS::Serverless::Api.

On the Multiply route I will require an API key, limit requests to 500 per day, and limit requests to 5 requests per second.

MultiplyApi:
     Type: AWS::Serverless::Api
     Properties:
       StageName: Prod
       Auth:
         ApiKeyRequired: true
         UsagePlan:
           CreateUsagePlan: PER_API
           Quota:
             Limit: 500
             Period: DAY
           Throttle:
             RateLimit: 5

Let's make a request and see what happens.

curl -X POST https://<snip>.execute-api.us-east-1.amazonaws.com/Prod/multiply?a=3&b=1

{"message": "Missing Authentication Token"}

Okay, that didn't work because I didn't supply an API key.

Let's try it again with an API key I generated for myself earlier.

curl -X POST -H "x-api-key:<API KEY>" https://<snip>.execute-api.us-east-1.amazonaws.com/Prod/multiply?a=5&b=3

{"product": 15}

Conclusion

That's all it takes to build a small API using SAM. You can extend this API by adding more routes and functions as resource declarations in the template.yaml file. The SAM template language reference and the SAM CLI reference are your friends.

A complete guide to using the AWS Systems Manager Parameter Store in your cloud applications

Sean Ziegler — Sun, 24 May 2020 13:06:56 +0000

Environment variables are used by nearly every developer. They are great for setting parameters that change based on the environment of the system. Unfortunately, they don't scale well. Sure, I agree, there are ways to do it, but it isn't a fun process. AWS offers a product I think can take the place of environment variables (or at least reduce your reliance on them). The AWS Systems Manager Parameter Store is a perfect solution for building modern cloud applications.

What is AWS Systems Manager Parameter Store?

AWS Systems Manager is a product designed to help you manage large groups of servers deployed into the cloud. For instance, it provides a remote connection to systems, security and patch updates, remote command execution, and other administration tasks at scale.

It also provides a feature called the Parameter Store. The parameter store is a superb place to store centralized data like API keys, database strings, passwords, and other configuration data.

Why use the AWS Systems Manager Parameter Store?

The Parameter Store is a great way to make your application less stateful and improve your ability to deploy across several environments. The parameter store has a few advantages over other methods of managing variables:

Easy to update from a central interface
Hierarchy structure
Supports encryption to store secrets like passwords
Supports versioning and roll back of parameters
Allows access control, both for IAM users and roles
Ability to audit parameter access using CloudTrail
Supports throughput of 1,000 transactions per second (must be increased in your settings)

I choose parameters over environment variables because I can update the parameter in one location and the changes are instantly available to any code using the parameter.

Hierarchy Structure

Perhaps the most interesting thing about the Parameter Store is the hierarchy structure. Hierarchies are parameters that start with a slash. They are a great way to organize parameters in a manageable fashion. I often create parameters for dev, test, and prod.

/dev/API_KEY
/dev/DB_STRING
/test/API_KEY
/test/DB_STRING
/prod/API_KEY
/prod/DB_STRING

This is a painless way to separate and manage parameters even when you have thousands of them.

Even with multiple environments, each part of your application can request the data it needs using the hierarchy structure.

Parameter Types

A parameter is a piece of data stored within AWS Systems Manager Parameter Store. AWS provides no validation on any parameters (with one exception covered later).

There are three types of Parameter Store parameters (and a fourth kinda-weird bonus type).

String
StringList
SecureString

String

Strings are exactly what you expect. Strings are any block of text such as Hello World, test, or wow this is a great blog post.

StringList

StringList is, again, rather intuitive. A StringList is a collection of strings separated by a comma. For example, Cat,Dog,Rabbit and Mercury,Mars,Melons are two examples of string lists.

SecureString

SecureString is used for sensitive data like passwords and API Keys. Data stored in a SecureString parameter are encrypted using keys managed by the AWS Key Management Service. You should know that these parameters are free to use, but AWS will charge you for the Key Management Service as usual.

Subtype - AMI Datatype

There is one strange "bonus" type you should know. When using a string attribute, you can use an additional parameter --data-type and then specify an Amazon machine image resource number.

aws ssm put-parameter \
    --name "\amis\linux\golden-ami" \
    --type "String" \
    --data-type "aws:ec2:image" \
    --value "ami-12345abcdeEXAMPLE"

The parameter store will validate that the AMI image is valid, then you'll be able to use the AMI in other services by referencing the parameter.

Parameter Tiers

There are two types of parameters: standard parameters and advanced parameters. Advanced parameters support parameter policies which can set parameter expiration, notify you if parameters expire, and let you know if a parameter hasn't changed in a while.

You can upgrade parameters to advanced parameters, but you can never downgrade to a standard parameter. There's really no reason to use an advanced parameter unless you run up against one of the limits below or you need the advanced policies they offer for notifications.

Standard Parameters

Free
Max of 10,000 parameters (per region)
4KB max size

Advanced Parameters

Paid
Max of 100,000 parameters (per region)
8KB max size
Supports parameter policies

Intelligent Tiering

This option is a blend of the two standard options. When you select intelligent tiering, the parameter store will inspect each parameter to see if requires advanced features. If it does, the store automatically upgrades the parameter to the advanced tier.

Intelligent tiering helps control cost and prevent failures because you hit the limit on standard parameters or tried to store a key larger than 4KB. If you don't mind spending the money on advanced parameters, it's worth considering.

Managing Parameters using the AWS CLI

You'll need to install and configure the AWS CLI if you haven't already.

Creating parameters is very easy. There's a built in command to create parameters.

aws ssm put-parameter --name "/test/email/liscence" --type "String" --value "XM56HE1I9M2AC9W30UM1"

To create a SecureString, add a --Key-Id and specify a KMS Key ARN.

aws ssm put-parameter --name "/test/email/password" --value "4G00DPA$$W0RD" --type "SecureString" --key-id "arn:aws:kms:us-east-2:123456789012:key/1a2b3c4d-1a2b-1a2b-1a2b-1a2b3c4d5e"

Getting parameters is even more fun. To get a parameter by name, use get-parameters.

aws ssm get-parameters --names "/dev/API_KEY" "/test/API_KEY" "/prod/API_KEY"

SecureString parameters require a --with-decryption flag.

aws ssm get-parameters --names "/prod/API_KEY" --with-decryption

You can get all the parameters in hierarchy

aws ssm get-parameters-by-path --path "/dev"

and use describe-parameters to query parameters by type.

aws ssm describe-parameters --filters "Key=Type,Values=StringList"

Parameter Versioning

Versioning is another great feature of the parameter store. If you overwrite a parameter that already exists, the parameter's version will increment.

aws ssm put-parameter --name "/test/email/liscence" --type "String" --value "XM56HE1I9M2AC9W30UM1"

aws ssm put-parameter --name "/test/email/liscence" --type "String" --value "XM56HE1I000000000000" --overwrite

Lets inspect the parameter's history.

aws ssm get-parameter-history --name "/test/email/liscence"

{
    "Parameters": &#91;
        {
            "Name": "/test/email/liscence",
            "Type": "String",
            "LastModifiedDate": "2020-05-22T16:25:04.303000-04:00",
            "LastModifiedUser": "arn:aws:iam::&lt;redacted>",
            "Value": "XM56HE1I9M2AC9W30UM1",
            "Version": 1,
            "Labels": &#91;],
            "Tier": "Standard",
            "Policies": &#91;]
        },
        {
            "Name": "/test/email/liscence",
            "Type": "String",
            "LastModifiedDate": "2020-05-22T16:25:38.281000-04:00",
            "LastModifiedUser": "arn:aws:iam::&lt;redacted>",
            "Value": "XM56HE1I000000000000",
            "Version": 2,
            "Labels": &#91;],
            "Tier": "Standard",
            "Policies": &#91;]
        }
    ]
}

Parameter Policies

Parameter polices allow you to set expirations for parameters, get notified when a parameter expires, and also get notified if a parameter hasn't changed in a while. Don't ask me why policies are only good for these three things, but that's how it works. Maybe AWS will add more options.

You can set an expiration time with policies.

{
   "Type":"Expiration",
   "Version":"1.0",
   "Attributes":{
      "Timestamp":"2018-12-02T21:34:33.000Z"
   }
}

You can set up notifications if a parameter is expiring.

{
   "Type":"ExpirationNotification",
   "Version":"1.0",
   "Attributes":{
      "Before":"15",
      "Unit":"Days"
   }
}

And finally, you can set up notifications if a parameter has not changed in a set time period.

{
   "Type":"NoChangeNotification",
   "Version":"1.0",
   "Attributes":{
      "After":"20",
      "Unit":"Days"
   }
}

Assigning a parameter policy with the AWS CLI is relatively straightforward.

aws ssm put-parameter   
    --name "/dev/apikey" \
    --value "XM56HE1I9M2AC9W30UM1" \
    --type "String" \
    --overwrite \    
    --policies "&#91;{policies-enclosed-in-brackets-and-curly-braces}]"

Setting up service roles for EC2

If you want to use the Parameter Store with other services (you probably do), you'll need to grant that service access via a service role.

I've included a role here which will give you access to the Parameter Store for whatever service assumes the role (Just don't forget to attach this to your service role).

{
    "Version": "2012-10-17",
    "Statement": &#91;
        {
            "Sid": "AllowSSMAccess",
            "Effect": "Allow",
            "Action": &#91;
                "ssm:PutParameter",
                "ssm:DeleteParameter",
                "ssm:GetParameterHistory",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:DescribeParameters",
                "ssm:DeleteParameters"
            ],
            "Resource": "arn:aws:ssm:*:*:parameter/*"
        },
    ]
}

If you require more strict access control, you can limit access to read-only or only allow access to certain parameters.

Using Boto3 to interact with the Parameter Store

Obviously, no service would be complete with a way to interact with it from code and the Parameter Store is no exception. The AWS SDK, in my case Boto3 since I use python, offers a straightforward way to interface with the Parameter Store.

For example, in the code below I access three different prefixes based on a single environment variable set on the host (you can do this with EC2 user data or a Dockerfile) and then my application knows which set of parameters to retrieve. I use it to a variety of things including API keys, log file locations, ports, debug status, and more.

## PIP
import boto3

ssm = boto3.client('ssm', region_name='us-east-1')
env = os.environ.get('env')

# Select SSM parameters based on the 'env' environment variable
if env == 'DEV':
    prefix = '/&lt;project>-dev/'

elif env == 'TEST':
    prefix = '/&lt;project>-test/'

elif env == 'PROD':
    prefix = '/&lt;project>-prod/'

# If env is not set, raise error to stop server from starting
else:
    raise AttributeError('No value set for environment type (env)')

secrets = {
    'ENV' : env,
    'DEBUG' : ssm.get_parameter(Name= prefix + 'DEBUG')&#91;'Parameter']&#91;'Value'],
    'PORT' : ssm.get_parameter(Name= prefix + 'PORT')&#91;'Parameter']&#91;'Value'],
    'FLASK_SECRET_KEY' : ssm.get_parameter(Name= prefix + 'FLASK_SECRET_KEY', WithDecryption=True)&#91;'Parameter']&#91;'Value'],

    'COGNITO_ID' : ssm.get_parameter(Name= prefix + 'COGNITO_ID', WithDecryption=True)&#91;'Parameter']&#91;'Value'],
    'CLIENT_ID' : ssm.get_parameter(Name= prefix + 'CLIENT_ID', WithDecryption=True)&#91;'Parameter']&#91;'Value'],
    'INVOKE_API' : ssm.get_parameter(Name= prefix + 'INVOKE_API', WithDecryption=True)&#91;'Parameter']&#91;'Value'],
    'API_GATEWAY_KEY' : ssm.get_parameter(Name= prefix + 'API_GATEWAY_KEY', WithDecryption=True)&#91;'Parameter']&#91;'Value'],
    'SNS_CONTACT_ARN' : ssm.get_parameter(Name= prefix + 'SNS_CONTACT_ARN')&#91;'Parameter']&#91;'Value'],

    'RAINFOREST_API_KEY' : ssm.get_parameter(Name= prefix + 'RAINFOREST_API_KEY', WithDecryption=True)&#91;'Parameter']&#91;'Value'],

    'LOG_FILE' : ssm.get_parameter(Name= 'LOG_FILE')&#91;'Parameter']&#91;'Value']
}

Don't forget to add the service role for any machine that will run code that accesses the Parameter Store!

Conclusion

That's everything that you need to know about how to integrate the AWS Systems Manager Parameter Store. Hopefully, you learned how to use the AWS Parameter Store a little better and can incorporate it into your future work.

If you enjoyed this content, follow me on Twitter to see more like this!

Follow @Sean_Ziegler

How to build a free static resume site with AWS S3, Cloudfront, and Route 53

Sean Ziegler — Thu, 14 May 2020 20:59:11 +0000

Forrest Brazeal, an AWS Serverless Hero who I follow on Twitter, recently tweeted about a #CloudResumeChallenge to build a resume site using AWS. In return, Forrest would use his network to help everyone who completed the challenge land a job. An awesome deal! I already have IT experience (which disqualifies me) but I thought it would be fun to build the site anyway and write up a post about how I did it.

He tweeted about it here

The challenge set some rules for the website:

Resume must be written in HTML and styled with CSS
Deployed as an Amazon S3 static website
Should be accessible via HTTPS
Have a custom domain name
Include a Javascript visitor counter
Visitor count stored on DynamoDB
The site must communicate with DynamoDB via Lambda and API Gateway
Lambda and API Gateway should be deployed with AWS SAM CLI
Website automatically updates on push to GitHub

Setting up the site content

All right, let's start with the content. Since I'm not in this for the networking, I didn't think it made much sense to develop another personal site for this challenge. I did it on Alan Turing instead. Why? Because he's cool, and none of us would be looking at this without him.

Anyway, create an empty Git repo.

git clone git@github.com:seanziegler/AlanTuringResume.git

I hate HTML and CSS so I stole a nice little resume template from this CSS-Tricks article.

Originally, it was based on Cthulu which.... was a little strange, but okay. After some Wikipedia research and a little front-end stumbling, I had a clean one-page resume site about Alan Turing developed.

Well, this makes my resume look weak.

Create an S3 Static Site

Okay, on to the more interesting part, the back-end. The first requirement of the site is it should be a static site hosted by AWS S3.

First thing we need is an S3 bucket. You'll need to configure the AWS CLI if you haven't done so already.

aws s3 mb s3://turingresume

Copy all the files for your website into the bucket. The root page of your website (often index.html) should be at the root level of the S3 bucket.

aws s3 cp <project_dir> s3://turingresume --recursive

Last, tell S3 that this should be a static website, not a normal bucket.

aws s3 website s3://turingresume --index index.html

Check to see if your website is available. You can access it at http://"bucketname".s3-website-"bucketregion".amazonaws.com/

If you can see your website, you've successfully deployed the static site using S3. Let's work on making it easier to reach your site since that S3 website URL isn't exactly memorable.

Purchase a domain

We're off to Route 53 where you can purchase a domain from inside the AWS console. Unfortunately, AlanTuring.com wasn't available, so I went with the much friendlier heyitsalan.com. I realized later that it looks a lot like "hey its Satan" which is unfortunate, but I already paid for the domain and I'm not made of money.

All right, it's not the best domain, but it works.

Ok, that's all we can do in Route53 for now.

Create an SSL certificate in ACM

Since we will serve the site over HTTPS, we will need an SSL certificate. Fortunately, the AWS Certificate Manager makes it simple to generate a certificate.

Make sure you request a certificate for *.yourdomain.com so it covers all sub-domains (especially the www and non-www versions of your URL).

You'll need to add a TXT record to the DNS records of your domain to prove that you own it.

DNS Validation proves you own the domain covered by the SSL certificate.

If you bought your domain on Route53 like I did, you can just click the "Add DNS Records to Route53" button. If you're on a different host, you'll have to look at their documentation.

Once AWS Certificate Manager sees the DNS record and knows you own the domain, it will issue your SSL certificate.

Create a CloudFront Distribution

We're ready to tie everything together and get this domain name tied up. All we need to do is create a HTTPS CloudFront Distribution and with our S3 website as its origin. After that, we'll point the domain name to the CloudFront distribution to finish. Start by creating a CloudFront distribution.

You'll also need to add your domain as an alternate CNAME in the CloudFront distribution and import the SSL certificate you generated earlier.

Don't forget to add * in front of your domain so all sub-domains redirect here.

Fire up Route53 and head to the DNS records for the domain you registered. We will add CNAME alias record, which points to the domain name for the CloudFront distribution created earlier.

Now, you can test your domain and see if it works (although you might need to wait a few minutes for the site to propagate through the CloudFront edge locations).

Another requirement down.

Deploy a DynamoDB table with CloudFormation

Since our problem statement calls for a DynamoDB table to hold the visitor count, I will use CloudFormation. CloudFormation is an excellent product for deploying AWS resources and makes life so much easier when you need to update or manage your resources later.

This is a super simple DynamoDB table. In all honestly, I'm not convinced this is a good use of DynamoDB, but the problem statement calls for it, so who am I to argue?

AWSTemplateFormatVersion: "2010-09-09"
Resources: 
  TuringResumeCounterDynamodb: 
    Type: AWS::DynamoDB::Table
    Properties: 
      AttributeDefinitions: 
        - 
          AttributeName: "Site"
          AttributeType: "N"
      KeySchema: 
        - 
          AttributeName: "Site"
          KeyType: "HASH"
      ProvisionedThroughput: 
        ReadCapacityUnits: "1"
        WriteCapacityUnits: "1"
      TableName: "turingresumecounter"

Deploying this is a single AWS CLI command (I love CloudFormation!).

aws cloudformation deploy --template-file dynamodb.yml --stack-name turingresumecounter

I will create a single attribute to hold the visitor count.

aws dynamodb put-item --table-name turingresumecounter --item '{"Site":{"N":"0"}, "Visits": {"N": "0"}}'

Great, now I can read and write from this attribute to keep track of the visitor count. I don't think this is a good solution if you are running in a production setup, so I might re-factor this later.

Deploy a serverless API with Lambda and API Gateway

The site still needs some interface to communicate with DynamoDB. I'm going to implement this with Lambda and API Gateway since they are the de facto standard for serverless APIs on AWS. I'm a big fan of API Gateway because it makes it a breeze to set up rate limits, throttling, and other usage plan metrics for your API.

CloudFormation is great, but it can be a bear to use when deploying serverless resources. AWS makes a companion product called the SAM CLI. The SAM CLI designed to make deploying serverless assets easier and is perfect for our use case.

The first step is to create another YAML file that describes the deployment.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  Counter:
    Type: AWS::Serverless::Function
    Properties:
      Handler: function.handler
      Runtime: python3.8
      Policies: AmazonDynamoDBFullAccess
      Events:
        HttpGet:
          Type: Api
          Properties:
            Path: '/counter'
            Method: get

The Handler: function.handler line above tells SAM that we will provide the lambda function in a file named function.py with a function named handler.

The handler function uses the AWS SDK for Python called boto3.

UPDATE: After talking to Forrest, we decided it would be best if future participants had to design these parts on their own. Good luck :)

Great, now when we send a GET request to the API Gateway URL we will see the visitor count. Now, let's work that into the site itself.

Wow, someone call Google, this API is HOT!

Add the visitor counter to the site

The site needs an element to display the visitor counter. I'm going to create some HTML to display the visitor count and add it to the bottom of the page.

<div>
    <p style = "text-align: center;">
    Visits: <span id="visits">0</span>
    </p>
</div>

Yes, I know inline CSS is bad. Sue me. Let's add some Javascript magic to the footer. This function sends a GET request to the API gateway endpoint every time the page is refreshed. Technically, this is a counter of page views not visitors, but I digress.

UPDATE: After talking to Forrest, we decided it would be best if future participants had to design these parts on their own. Good luck :)

Cool, let's check the page.

Looking good.

Create a pipeline for automated deployments

We are nearing the finish line! All that's left is creating a pipeline in CodePipeline to deploy our code to the S3 bucket when we push to Git. I already wrote about how to deploy from GitHub to EC2, but since we're going to S3 this time, let's do a quick overview.

Setup a pipeline.

I usually allow AWS to create the service role - much easier that way.

Make the Git repo your source.

Setup a deployment stage which pushes the files to your S3 bucket.

Finally done.

That's all there is to do! You've got a fully functional static S3 that updates on Git push and is served over HTTPs, all for essentially free.

Final product doesn't look too bad for a back-end engineer!

If you enjoyed this content, follow me on Twitter to see more stuff like this!

How to manage multiple RaspberryPi's easily with Ansible

Sean Ziegler — Thu, 30 Apr 2020 20:43:54 +0000

I plan to run a bunch of services on my home network. Right now, my plan is to host most of the services on a small RaspberryPi cluster in a C4 Labs Cloudlet Case. All of these Pi's run Raspbian and each one will run a few services that I like having hosted on my home network.

Although 3 RPis isn't exactly impossible to configure manually; why would I want to do that? Plus, the Cloudlet Case accepts up to 8 RPis which would be a lot to manage manually.

The Plan

For now, I'm going to configure all the following things with Ansible:

.bashrc - This enables me to use the same bash aliases across all nodes
Changing the default password of every pi
Installing Vim
.vimrc - Configures Vim settings like the color scheme and line numbers across all nodes
MOTD - Just for fun, I'm going to install the fortune | cowsay packages and broadcast funny cow sayings on log in
Prometheus Exporters - Eventually we will deploy some Prometheus exporters to remotely view each node's status
Mounting a network share on boot

In the future I'll also be deploying some Docker containers via Ansible but first we need to configure Ansible itself.

Installing Ansible on the Master Node

Since Raspbian is a Debian derivative, we will need to add the Ansible PPA to the /etc/apt/sources file.

cd /etc/apt
sudo vim sources

The Ubuntu PPA works fine for Raspbian, at the top of the file add:

deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main

Update apt and then install with:

sudo apt update 
sudo apt install ansible

Alright, let's define the hosts on the network. Right now, there are three. One will function as a master node that runs Ansible and the others will be slaves that Ansible controls via SSH.

[master]

pi@192.168.0.101

&#91;slave]

pi@192.168.0.102
pi@192.168.0.103

I could have used Ansible's built-in expansion syntax and written the slaves as 192.168.0.[102:103] but I like the readability of listing them explicitly.

You need SSH keys configured for Ansible to work without password prompts. I already have SSH setup on all these nodes but I'll be writing a post on setting up headless ssh on a RPi in the future.

Let's check if Ansible can talk to our nodes:

ansible all -m ping

192.168.0.101 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
192.168.0.103 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
192.168.0.102 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

Great! We can talk to everything on to writing playbooks.

Writing a playbook

Okay, first let's add a very basic Ansible playbook to install Vim to all three nodes. First, move to the Ansible directory and create a folder for playbooks and the playbook itself:

cd /etc/ansible
mkdir playbooks
touch install_vim.yml

Inside the playbook, define a task that installs the latest version of Vim on all the hosts. I'll also add configuration to use sudo to install Vim.

- hosts: all
    remote_user: pi
    tasks:
      - name: Install Vim
        become: true
        become_method: sudo
        apt:
          name: vim
          state: latest
          install_recommends: true

Save the file and run the following command:

ansible-playbook install_vim.yml

Your output should be all ok.

PLAY [all] **************************************************************************

TASK &#91;Gathering Facts] **************************************************************
ok: &#91;pi@192.168.0.101]
ok: &#91;pi@192.168.0.103]
ok: &#91;pi@192.168.0.102]

TASK &#91;Install Vim] ******************************************************************
ok: &#91;pi@192.168.0.101]
ok: &#91;pi@192.168.0.103]
ok: &#91;pi@192.168.0.102]

PLAY RECAP **************************************************************************
pi@192.168.0.101           : ok=2    changed=0    unreachable=0    failed=0
pi@192.168.0.102           : ok=2    changed=0    unreachable=0    failed=0
pi@192.168.0.103           : ok=2    changed=0    unreachable=0    failed=0

Great, now that Vim is installed let's advertise some Vim settings (I like the desert color scheme and line numbers) to all 3 machines.

Let's make a new folder to hold files we want to advertise to other nodes and then create the .vimrc file:

cd ..
mkdir repo
cd repo
touch .vimrc

Inside the .vimrc file add the following lines to enable a desert color scheme and line numbers:

color desert
set number

Now let's define a new task in the Ansible recipe:

- name: Advertise .vimrc
   become: true
   become_method: sudo
   copy:
    src: ../repo/.vimrc
    dest: /etc/vim/vimrc.local

Alright, re-run the ansible-playbook install_vim.yml command and you should see all the same output plus a few new lines:

[Advertise .vimrc] *************************************************************
changed: &#91;pi@192.168.0.101]
changed: &#91;pi@192.168.0.103]
changed: &#91;pi@192.168.0.102]

PLAY RECAP **************************************************************************
pi@192.168.0.101           : ok=3    changed=1    unreachable=0    failed=0
pi@192.168.0.102           : ok=3    changed=1    unreachable=0    failed=0
pi@192.168.0.103           : ok=3    changed=1    unreachable=0    failed=0

Vim is now installed on every node and my personal .vimrc file is also being used. The beauty of this system is adding new nodes is as simple as adding it to the Ansible inventory file and then will have all these new settings pushed to them immediately.

/etc/profile

What better way to use our new Ansible system than syncing some bash aliases to all three nodes. I really like using bash aliases but if they aren't perfectly synced across every system they can quickly become a pain. Let's make a new Ansible playbook for this task.

cd /etc/ansible/playbooks
touch etc_profile.yml
vim etc_profile.yml

Now let's define two tasks. One that adds the aliases to the /etc/profile file and one that sources the file so the aliases become available to us in the bash shell.

- hosts: all
 remote_user: pi
  tasks:
  - name: Add aliases to global profile
    become: true
    become_method: sudo
    blockinfile:
     path: /etc/profile
     insertafter: EOF
     block: |
       alias 'll=ls -al'
       alias '..=cd ..'
       alias '...=cd .. &amp;&amp; cd..'
       alias 'mvansible=cd /etc/ansible'

  - name: Source profile
    become: true
    become_method: sudo
    shell: . /etc/profile
    args:
    executable: /bin/bash

This playbook will allow us to add bash aliases via the master node and sync them to all other nodes in the network with a single Ansible command!

Let's try it:

ansible-playbook etc_profile.yml

PLAY &#91;all] *********************************************************                                                                     ************

TASK &#91;Gathering Facts] *********************************************                                                                     ************
ok: &#91;pi@192.168.0.101]
ok: &#91;pi@192.168.0.103]
ok: &#91;pi@192.168.0.102]

TASK &#91;Add aliases to global profile] *******************************                                                                     ************
ok: &#91;pi@192.168.0.101]
ok: &#91;pi@192.168.0.103]
ok: &#91;pi@192.168.0.102]

TASK &#91;Source profile] **********************************************                                                                     ************
changed: &#91;pi@192.168.0.101]
changed: &#91;pi@192.168.0.103]
changed: &#91;pi@192.168.0.102]

PLAY RECAP *********************************************************                                                                     ************
pi@192.168.0.101           : ok=3    changed=1    unreachable=0    f                                                                     ailed=0
pi@192.168.0.102           : ok=3    changed=1    unreachable=0    f                                                                     ailed=0
pi@192.168.0.103           : ok=3    changed=1    unreachable=0    f                                                                     ailed=0

Awesome! It looks like everything worked. Just for fun lets try it on a slave node:

pi@raspberrypi3BP-1:~ $ ll

total 56K
drwxr-xr-x 7 pi   4.0K Oct 12 00:21 .
drwxr-xr-x 3 root 4.0K Sep 15 23:54 ..
drwx------ 3 pi   4.0K Oct 11 21:38 .ansible
-rw------- 1 pi   6.5K Oct 11 23:10 .bash_history
-rw-r--r-- 1 pi    220 Jul 10 01:07 .bash_logout
-rw-r--r-- 1 pi   3.5K Sep 15 23:27 .bashrc
drwx------ 3 pi   4.0K Sep  7 19:12 .config
drwx------ 3 pi   4.0K Jul 10 01:30 .gnupg
drwxr-xr-x 3 pi   4.0K Sep  7 18:24 .local
-rw-r--r-- 1 pi    807 Jul 10 01:07 .profile
drwxr-xr-x 2 pi   4.0K Sep  7 20:02 .ssh
-rw------- 1 pi   1.6K Oct 12 00:21 .viminfo
-rw-r--r-- 1 pi    165 Sep  7 18:53 .wget-hsts

Looks like our aliases were correctly synced.

Ansible is now configured. We have a huge amount of potential for installing other applications and advertising configurations. Expect to see some more Ansible posts in the future!

Deploying code from GitHub to AWS EC2 with CodePipeline

Sean Ziegler — Thu, 30 Apr 2020 11:32:41 +0000

I’m not that old, but when I built my first website I manually copied code to an Apache server with FTP. That server was also running in my closet. As you can imagine, this deployment process was not exactly fullproof and I experienced frequent outages on my website. Deployments have come a long way in the 10 years since my first website and the general goal today is to automate code deployments as much as possible. Let’s explore how we can deploy code from GitHub to EC2 using CodePipeline.

Why automate?

Put simply, the more repeatable your deployments are, the less downtime you will have. Automated deployments are easier to test, easier to troubleshoot, lend themselves to scalability, can perform rollbacks on failure, support blue/green and canary deployments. The list goes on.

Ideally, your resources should be treated like cattle, not like pets. If it gets sick, you shoot it and replace it with another one. Automated deployments make that easy.

First, a Git repo

Before we do anything else, we will need a Git repository to house the code. I’ll clone down a empty repo and create a demo file.

git clone git@github.com:seanziegler/AWSDeployDemo.git
touch demo.txt
echo "This is prod" > demo.txt
git add . 
git commit -m "Init commit"
git push origin master

I’m going to create a seperate branch so I can push seperately to test and master.

git branch test
git checkout test
echo "This is test" > demo.txt 
git add .
git commit -m "Create test branch"
git push origin test

Obviously, you’d normally be deploying an application, not just a text file.

Creating deployment targets

Now that we have repositories set up, some instances are needed. I’m going to use the management interface since this is just a demo but you can also use the awscli or Terraform/Cloudformation.

It’s good practice to create a new subnet inside your VPC for each application you deploy.

Okay, next up are the EC2 instances themselves. I’m going to make two: one for test and one for production. Obviously this is just a dummy deployment but CodePipeline is versatile enough to allow deployments across hundreds of resources if needed.

I’ll also add some tags to identify the production vs test instance. I’ll be using these later to tell CodePipeline which one is which.

Okay, we have what we need for instances.

Installing the CodeDeploy agent

Before we can build a pipeline, we need to configure CodeDeploy and it’s agent. CodeDeploy requires a small agent daemon installed on every machine that is a deployment target.

You can check if you have the CodeDepoy agent installed by connecting to your instance and running (on Amazon Linux or RHEL):

sudo service codedeploy-agent status

You’ll either get an error or a message that the service is stopped. If the service is stopped, simply run:

sudo service codedeploy-agent start

If you get an error, the codedeploy-agent isn’t installed. So it becomes a bit more involved. Here’s the docs page on how to install the CodeDeploy agent. Essentially it boils down to the following commands:

sudo yum update -y
sudo yum install ruby wget -y
cd /home/ec2-user
wget https://bucket-name.s3.region-identifier.amazonaws.com/latest/install
chmod +x ./install
sudo ./install auto

In the wget command above, you’ll need to replace both bucket-name and region-identifier with the values that corresponds to your region.

Region Name	bucket-name	region-identifier
US East (Ohio)	aws-codedeploy-us-east-2	us-east-2
US East (N. Virginia)	aws-codedeploy-us-east-1	us-east-1
US West (N. California)	aws-codedeploy-us-west-1	us-west-1
US West (Oregon)	aws-codedeploy-us-west-2	us-west-2
Canada (Central)	aws-codedeploy-ca-central-1	ca-central-1
Europe (Ireland)	aws-codedeploy-eu-west-1	eu-west-1
Europe (London)	aws-codedeploy-eu-west-2	eu-west-2
Europe (Paris)	aws-codedeploy-eu-west-3	eu-west-3

Once installed, check one last time to make sure the agent is running.

sudo service codedeploy-agent status

CodeDeploy Role

In order for CodeDeploy to work, we need to assign it a service role with the correct permissions. Let’s create a role with the AWSCodeDeployRole policy attached to it.

Now that the instance roles are setup, it’s time to turn to CodeDeploy.

Creating an Application and Deployment Group

Creating an application in CodeDeploy is straightforward. Just give it a name and tell it that you plan on deploying code to EC2 instances.

Creating a deployment group is a bit more complicated but not difficult. Give it a name and assign the CodeDeploy service role you created previously. This will allow CodeDeploy to access the resources specified in this deployment group.

Next, choose EC2 instances for deployment. You can also deploy to autoscaling groups but for now I’ll deploy to a single instance. Don’t forget to add the env:test tag. You should see one matched instance that corresponds to the test instance created earlier.

We don’t need a special deployment setting so just leave the default AllAtOnce setting in place. Uncheck Enable load balancing since we haven’t set that up.

Repeat this process again but target the instances with the env:prod tag.

appspec.yml

CodePipeline needs to know what to do with the files in your Git repository when you deploy. A file called appspec.yml is CodePipeline’s way of defining the tasks you want to run when deploying code. There’s too much to cover here, but AWS has examples of how to build out an appspec.yml file.

For this project, I’m just going to copy in some text files from the Git repo to test that the pipeline is working. You’ll need to do your own research on how to build an appspec file to suit your deployment. If you want to follow along with my dummy deployment, put this in a file at the root of your git repo. Don’t forget to name it appspec.yml!

version: 0.0
os: linux
files:
  - source: ./demo.txt
    destination: /

Building a pipeline

We need to build two pipelines for this to work. One pipeline that deploys the test branch to test instances and one pipeline that deploys the master branch to production instances.

Creating a pipeline is a straightforward process. I suggest letting CodePipeline create your servie role for you. It’s just easier that way.

Create a stage that uses your Github repository and the test branch as the source.

Skip the build stage for now. If you want, you can come back later and setup CodeBuild to do integration tests or build binaries if you need that sort of thing.

Select the CodeDeploy provider and choose the application and deployment group for your test instances.

Click next and deploy the pipeline. You’ll see it try to download your source and deploy it to your test instances.

Great, we deployed to our test instances. But what about production? Click “Edit” at the top of the pipeline page. And then click “Edit” on each stage of the pipeline. From here, you can add another source and deploy action for the master branch and production deployment group.

When you click save, your pipeline will deploy the test branch to the test deployment group and the master branch to the production deployment group.

Now, let’s just check and see if the deployments worked. First, let’s check the test instance.

ssh ec2-user@test-instance

Last login: Sun Apr 26 16:00:02 2020 from <IP>

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

[ec2-user@test-instance ~]$ cat /demo.txt
This is test

Next, check the production instance.

ssh ec2-user@prod-instance

Last login: Sun Apr 26 16:00:02 2020 from <IP>

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/

[ec2-user@prod-instance ~]$ cat /demo.txt
This is prod

Awesome! Just what was expected. Your pipeline is setup and ready to go.

Customising the deployment.

There are several ways you can customize your pipeline.

appspec.yml

For one thing, you’ll want to start with building out an appspec.yml file that reflects the structure of your application. Copy over all the files you need, run scripts to setup dependencies, etc.

AMI

Make a custom AMI for the instances in each of your group. Include any dependencies your code needs to run (and the codedeploy-agent) in the AMI rather than installing it every deployment. This will increase the relability and speed of your deployments.

Autoscaling groups

Instead of creating deployment groups with specific EC2 instances identified, consider deploying to autoscaling groups instead so you can apply scale-in and scale-out rules.