DEV Community: James MIllar

Eat Sleep Cloud Repeat - The Podcast

James MIllar — Thu, 28 Jan 2021 17:35:21 +0000

Hi there - I'm James Millar. If you don't already know me, I'm a freelance software developer and Pluralsight author from the UK. I specialise in cloud technologies, mainly Azure and AWS. I write extensively about both platforms and I'm passionate about teaching people about the cloud.

So I decided to launch a new podcast - Eat Sleep Cloud Repeat.

So what’s it all about? Do we really need another podcast about the cloud?

Well in this podcast I want to put something together for beginners. Those people that are new to the cloud. Sure there’s going to be useful stuff here for everyone but I’m going to assume that you know nothing about cloud computing at all. Maybe you run a small business or perhaps you lead a team of software developers, there’s going to be something here for you.

In the first few episodes, we’re going to be focusing on what exactly the cloud is. Why you might want to embrace it and how you can move your existing code into the cloud.

We’re then going to look at how you can use the cloud to drive your business forward. How you can innovate and get the edge on your competitors.

Some of it’s going to be business focused and some will be more technical. I’m even hoping to get a few guests on to talk about their experiences so if you're interested in getting involved then let me know.

Find out more at james-millar.co.uk or listen now on Apple Podcasts or Spotify

Getting to Grips with CAP Theorem

James MIllar — Thu, 28 Jan 2021 17:28:40 +0000

If you’re a software developer working on any kind of cloud platform it’s highly likely that someone has already mentioned the CAP theorem. If they haven’t already then, they will do soon for sure.

CAP theorem is also known as Brewers theorem after the computer scientist Eric Brewer.

Essentially CAP theorem is a law within computer science that states that it’s impossible for a distributed data store to simultaneously provide more than two of the following three guarantees:

· Consistency

· Availability

· Partition tolerance

Now a lot has been written about CAP theorem and we’re not going to go into too much depth but it’s important to have an understanding of what this theorem is as it will allow you to recognize some of the constraints and tradeoffs that you’re going to encounter when designing your cloud system.

So let’s break down those three guarantees.

Consistency

So the C in C-A-P stands for Consistency. By consistency, we mean that every read operation on the data store will return the most recent data.

Imagine an airline application where a user checks in for their flight and changes their seat. In a consistent system, as soon as any other user requests the seat map details, even a few milliseconds later, they're going to see that seat as unavailable, they will see the most recent data.

Now if you're familiar with ACID transactions in a relational database, there's some cross over here with this idea of consistency, though they're not exactly the same thing. But indeed, we might use database transactions to support the C in CAP theorem.

Availability

The A in CAP stands for availability. When we’re talking about availability within CAP theorem, we say that every request must receive a response.

Now we know we can never achieve perfect availability, but if we follow best practice then we can build availability into our system from the start and then we might hope to achieve a system where, for example, 99. 9% of the requests will receive a response.

Now with data stores in the cloud such as Azure SQL and Cosmos DB, high availability is built right in but the design of your architecture is still important. What happens if a data center goes offline, is that data replicated to another region?

Partition Tolerance

The P in CAP stands for partition tolerance. Now you see the word partition crop up a lot in computer science. We’re not talking about a data partition here or a disk partition. In CAP Theorem the partition is a network partition. This is a partition between 2 parts of a system. Something that splits the network and causes parts of the system to become unavailable. In some ways, this isn’t dissimilar to the idea of graceful degradation. Expecting a system to remain responsive even when features we'd hope for aren't there.

Imagine we have a system that’s composed of 4 components. A,B C and D.

Let’s say that nodes A and B can communicate, and nodes C and D can communicate but there is a problem which is stopping A and B from talking to C and D. Well, if we are partition tolerant then we will still allow A and B and C and D to continue to operate and do their work as normal.

While CAP theorem is specifically talking about network partitions, we’re not just talking about network-level errors here. We’re talking about any kind of failure that takes a component offline such as hardware failures or software maintenance.

Ok now what?

So now we know what the C A and P refer too but that’s only half of the story.

You see, Eric Brewer, the computer scientist who came up with this, stated that a distributed shared data system can only have two of the three guarantees shown here.

So the point of this C-A-P is that we're saying a system cannot be available and consistent and partition tolerant all of the time and that in the design of a distributed system, you need to find an intersection of two of these three qualities, the one that you're most comfortable with.

Now just to be clear, this doesn't mean we just give one of these features up, because if things are running successfully, we can continue to do this and have no issues. We’re more interested in what we do when there is a problem, what will we prioritize?

Now technically one intersection between the two would be the one between consistency and availability, but this is an unlikely spot for cloud systems because a lot of what we've been doing is introducing partition tolerance, we've been working with redundancy and queueing and caching and other techniques.

So let's make things a little easier, because most typically with this CAP theorem and with cloud architectures, you will begin by assuming that partition tolerance is something that you want and that if you need to pick two things, partition tolerance will be one of them, then you just need to prioritize either consistency or availability.

So, if we decide that two of the three things that we're interested in, the intersection between partition tolerance and consistency is important to us, that says we want to maintain consistency and have partition tolerance, even if this means sacrificing availability. So, if there is a network partition, an issue with the network, then I am willing for the system, or at least parts of it, to stop being available and stop processing requests to make sure that the system stays in a consistent state.

But then there's also another choice, the intersection of partition tolerance and availability, where we might want our system to be highly available and resilient to network issues, and we'd be willing to sacrifice consistency to get there. It's not that we give up consistency altogether. We can assume that most systems are eventually consistent.

But as an example of this, if I update my profile on a social media site, maybe other users on other continents won't see my latest updates for seconds or minutes, or even hours. These systems are not immediately consistent. Now the system will probably be eventually consistent, and other users will eventually see my updates, but this might happen, for example, because we prioritize the system to be available and partition tolerant, and we're using geo-replication in our storage, but we decide not to use distributed transactions to make all updates instantly pushed out around the world, because distributed transactions can be expensive, and they could also make the system unavailable if there are any issues during a network partition.

So, with that kind of situation, we might have a database that will asynchronously replicate data across all the different regions, and eventually, all that data will be consistent.

But we don't worry about up-to-the-second consistency as long as the different pieces of the system stay up and available. Now obviously the choice that you make here depends on your business. Does it need to prioritize consistency or does it prioritize availability?

So, where an airline booking system might value consistency, a social media site might value availability. Now it's true this isn't always a simple choice, and it's also true it's not one choice for an entire system. You might have pieces of the system that are A-P, and other pieces that are C-P. If you have financial transactions in the system, you might need them to be consistent and partition tolerant, whereas the user profile part of the system could be available and partition tolerant. The CAP theorem just says that what you won't be able to do is build the entire system in a way that makes all three guarantees for availability and consistency and partition tolerant all of the time.

AWS CloudFormation Templates: Getting Started. Available Now

James MIllar — Thu, 28 Jan 2021 17:23:44 +0000

I'm excited to announce a new course is available now in the Pluralsight library,

The AWS CloudFormation Templates: Getting Started course shows you step by step how you can use CloudFormation Templates to codify your infrastructure, providing you with a single source of truth and allowing you to automate the deployment of your infrastructure. You will explore how to create resources, discover how to use parameters and outputs and how to define mappings. When you’re finished with this course, you’ll have the skills and knowledge of CloudFormation templates needed to create templates for your own infrastructure.

Find it now only on LinkPluralsight.

AWS CloudFormation Templates: Getting Started. Available Now

James MIllar — Thu, 28 Jan 2021 17:21:38 +0000

I'm excited to announce a new course is available now in the Pluralsight library,

Find it now only on Pluralsight

Creating EC2 Instance with Cloud Formation

James MIllar — Thu, 28 Jan 2021 17:18:27 +0000

So in this post, we’re going to take a look at how you can create an AWS EC2 Instance using a CloudFormation template. We’re going to create a few different resources in our template and then talk briefly about how to use the template to create a stack in the AWS portal.

So first up we need to create a new YAML file for our template. I use VSCode for this but you can use any text editor you like.

Your basic template should look like this:

AWSTemplateFormatVersion: 2010-09-09
Description: >- Creating an EC2 Instance
Resources:

Now we can create our resources in any order we like so let's start by adding a SecurityGroup. we can use this to allow traffic to our instance on port 80 and port 22. We need access to port 22 for SSH access. That way we can connect remotely to our instance.

The definition of our security group resources looks like this and you can see it defines two ingress rules, one for port 80 and one for port 22.

WebServerSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: >-
        Enable HTTP access via port 80  + SSH access
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: 0.0.0.0/0

Next up we need to define our EC2 resource. Now in this example, I’m setting the ImageId, InstanceType and KeyName properties directly here, in a future post we will look at how we can bind these values to input parameters.

WebServer:
    Type: 'AWS::EC2::Instance'
    Properties:
        ImageId: 'ami-01419b804382064e4'
        InstanceType: 't2.small'
        SecurityGroups:
          - !Ref WebServerSecurityGroup
        KeyName: 'myEC2KeyPair'

Now there are a couple of things to point out here. Firstly notice how we reference our security group using the inbuilt Ref function.

Secondly, for the key name property, I’m supplying the name of an existing EC2 KeyPair. If this KeyPair doesn't already exist then template creation will fail.

Finally, let’s add an ElasticIP address and associate it with our EC2 instance. To do this we’re going to create two resources. The ElasticIP address itself and an IPAssociation resource to tie the IP address to the instance.

IPAddress:
    Type: 'AWS::EC2::EIP'
  IPAssoc:
    Type: 'AWS::EC2::EIPAssociation'
    Properties:
      InstanceId: !Ref WebServer
      EIP: !Ref IPAddress

Again, notice how we can reference other resources within our template using the Ref function.

So our finished template now looks like this:

AWSTemplateFormatVersion: 2010-09-09
Description: >- Creating an EC2 Instance
Resources:
  WebServerSecurityGroup:
    Type: 'AWS::EC2::SecurityGroup'
    Properties:
      GroupDescription: >-
        Enable HTTP access via port 80  + SSH access
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: '80'
          ToPort: '80'
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: '22'
          ToPort: '22'
          CidrIp: 0.0.0.0/0
  WebServer:
    Type: 'AWS::EC2::Instance'
    Properties:
        ImageId: 'ami-01419b804382064e4'
        InstanceType: 't2.small'
        SecurityGroups:
          - !Ref WebServerSecurityGroup
        KeyName: 'myEC2KeyPair'
  IPAddress:
    Type: 'AWS::EC2::EIP'
  IPAssoc:
    Type: 'AWS::EC2::EIPAssociation'
    Properties:
      InstanceId: !Ref WebServer
      EIP: !Ref IPAddress

Creating a Stack

Now you have your template you probably want to use it to create some resources.

There are several ways you can do this but the easiest, especially when starting out is to use the AWS Portal.

Simply log into AWS, select CloudFormation from the Services menu (under Management & Governance) and click “Create a Stack”

Simply upload your template and follow the prompts and after a few minutes, your resources will be created.

Introducing CloudFormation Templates

James MIllar — Thu, 28 Jan 2021 17:15:19 +0000

This is the first in a series of posts where we’re going to be exploring CloudFormation templates.

If you’re using Amazon Web Services then you should have already heard of CloudFormation. Ideally, you’re already using it in which case you can stop reading now! If not then read on…

CloudFormation is Amazon’s resource automation stack, It allows you to define your infrastructure in code, inside simple YAML or JSON templates. That’s great I hear you say but why would I want to do that?

Well by codifying your infrastructure, that is describing it within a file, you have a way of including it within our source control solution. We can check the files into GitHub or whatever source control solution we’re using. This makes maintaining the infrastructure much easier as you have the full power of a source control system to manage changes to the files.

It also makes deployments much easier. Because the infrastructure is defined within a file, you can reliably and repeatably deploy it over and over again. You can even include it in your build and release pipeline so as your application is deployed to the cloud, any changes required to the infrastructure can also be deployed at the same time.

Infrastructure as code also helps prevent what’s known as environmental. Drift Before Infrastructure as code, developers had to maintain many different development environments, replicating changes across environments manually. This invariable didn’t happen in a consistent way and you end up with differences between one environment and another.

In addition to all of this, your test teams get early access to production-like test environments and It’s possible to spin up an entirely new test environment that’s identical to your production environment very quickly. This might be for load testing or penetration testing for example and once finished the environment can be torn down and discarded.

So let's start by looking at some of the key concepts here. Firstly, we define our infrastructure in what’s known as a template. This is a simple text file in either JSON or YAML format. In order to actually create the resources though we need to create a stack, essentialy we tell AWS to read the template and then create all of the resources that it defines. Stacks can be created within the AWS portal or through the CLI.

So let's take a look at possibly the simplest of templates.

This template creates an Amazon S3 bucket called MyBucket

Resources:
    MyBucket:
        Type:  AWS::S3::Bucket

Now creating buckets is easy as Amazon can create them with default values but what if we want to override or configure those properties. Well, we can do that too. In this example, we’re setting the Access Control property to Public Read.

Resources:
    MyBucket:
        Type:  AWS::S3::Bucket
        Properties:
            AccessControl: PublicRead

Now there are over 500 different resource types that we can create with CloudFormation and we will look at more of those over the coming weeks. We will also look at how you can inject values into and output values from your templates. We will also look at how you can make your templates highly customisable.

In the next post, I’m going to walk you through a simple scenario and show you how to create and execute a complete CloudFormation Template that uses multiple resources.