DEV Community: Christopher Miles

PICO-8 Animation and Collision Detection

Christopher Miles — Mon, 16 Jan 2023 02:58:43 +0000

In my last article we spent some time getting our game shelled out, throwing a sprite for the player on the screen and moving it around a bit. What would really snazz that up, I think, would be a little animation of the player and ending the game if the player hits a hazard of some kind.

In this article we'll add some animation to snazz up our player, introduce a hazard and then add some collision detection so that we can end the game if the player happens to smack into the hazard. Not a challenging game but it does have a beginning, middle and an end!

Animating the Player

What we would like is for the player to animate while it is moving around the screen. We'd like one set of animations while it's moving along the "x" axis and another set while it moves along the "y" axis. To keep things simple we will only have two frames for each animation.

To make this work we're going to setup the simplest possible animation system. We'll have one variable that keeps track of each "tick" of the game (each cycle through our game loop); every time we complete a game loop we'll increment the tick counter. When we reach 16 ticks we'll start back at 0 and start all over again.

This simple system will give us 16 possible frames of animation, one per game loop cycle. That's more than we need and it gives us a little room to grow if we decide to get clever.

Create Our Tick Counter

We're going to build on the code from the previous article, either catch up by reading through it or just copy the relevant code out so that we're all on the same page.

Let's start by adding two variables to our init_actors function: one for the current tick and one for our maximum number of ticks. The code below will get this done.

function init_actors()
  atk=0 atx=15
  ply1={x=8,y=8,spr=0}
end

The atk variable stands for "animation tick" and starts off at zero. The atx variable stores our maximum number of ticks and we set it to fifteen, giving us sixteen ticks in total.

Next we need to increment and reset our tick counter. Let's add a function to manage our animation counters.

function update_anim()
  atk+=1
  if atk>atx then
    atk=0
  end
end

Simple, right? We update our tick counter and if we hit our maximum number of ticks then we reset it back to zero.

Creating More Player Sprites

Next we need more sprites for our player, we want two to indicate movement horizontally and two more for vertical animation. Use the first four cells of sprites. You may also copy your sprite with the dotted square looking tool and paste it into the next cell to save time.

I've altered my sprites to change the direction the player is looking, I also change the fringe at the bottom of the ghost to give the impression of movement. The idea here is that the changes between the sprites don't necessarily need to be dramatic to provide the illusion of motion.

Now we can associate our new sprites with our player. After thinking it over a bit, I decided to add two arrays to the player data: one with the horizontal frames and another with the vertical. Doing it this way, we could have different animations for the various characters in the game.

Update the init_actors() function to match the excerpt below.

function init_actors()
  atk=0 atx=15 
  ply1={x=8,y=8,spr=0,
    alr={0,1},aud={2,3}}
end

We've added two more variables to the ply1 table, alr and aud. Both of these contain an array with the index of the sprites that make up the animation. The alr variable (animation left/right) references the first two sprites (0 and 1). The aud (animation up/down) variable points to the third and fourth sprites (2 and 3).

We are now tracking the animation frames (ticks) and we have references to the sprites we want to use for the animation. The only bit remaining is swapping the sprites for the player as it moves around, we'll do this by manipulating the ply1.spr variable that indicates which sprite to draw for the player.

Swapping the Player's Sprite

Since we need to know which button the player is pressing to figure out which direction to move the player, we can save a little time by weaving our animation code into our button detection and movement code. We check if the left or right, or if the up or down buttons are being pressed, then we update the player's position and alternate the player's sprite.

function update_ply1()
  if btn(0) or btn(1) then
    if atk > 8 then
      ply1.spr=ply1.alr[1]
    else
      ply1.spr=ply1.alr[2]
    end

    if btn(0) then
      ply1.x -= 2
    end
    if btn(1) then
      ply1.x += 2
    end
  end

  if btn(2) or btn(3) then
    if atk > 8 then
      ply1.spr=ply1.aud[1]
    else
      ply1.spr=ply1.aud[2]
    end

        if btn(2) then
      ply1.y -= 2
    end
    if btn(3) then
      ply1.y += 2
    end
  end
end

One game cycle goes by too quickly to really be visible, instead we change the sprite of the player every eight ticks. If we're on a tick less then eight then we draw the first sprite otherwise we draw the second. You can also see in the code above that we use the ply.alr sprites if buttons 0 or 1 are pressed (left or right) and we use the ply.aud sprites if buttons 2 or 3 (up or down) are pressed.

That's it! Run the game and give it a try, you should see your player 8-bit-ly animated as you move it around the screen. 😎

Hazards and Collisions

Now that we have our animated player moving around the screen, the next thing we need to be able to do is detect collisions between our player and the various hazards of the game. For that we need at least one hazard!

Adding a Hazard

Switch over to the sprite editor and add one more sprite, this one will be the thing that ends the game if the player were to touch it. It could be a hole or a bomb or an alligator, whatever you'd like. With that done, let's add our hazard to the game. Let's update the init_actors function to track our hazard and if the game is over (the player has collided with the hazard).

function init_actors()
  atk=0 atx=15
  ply1={x=8,y=8,spr=0,
    alr={0,1},aud={2,3}}
  hzd={x=64,y=64,spr=4}
  over=false
end

The hzd table has the "x" and "y" positions of our hazard along with the index of the sprite used to draw it on-screen. The over variable lets us keep track of the status of the game, we start out with it not being over.

Our hazard also needs to be drawn on screen, add this function right after draw_ply1.

function draw_hzd()
  spr(hzd.spr,hzd.x,hzd.y)
end

Simple right? We just draw the hazard on the screen. Update the _draw function to call this one as well.

function _draw()
  cls()
  draw_ply1()
  draw_hzd()
end

Now we'll draw the hazard every loop of the game, just like the player.

Detecting Collision

Detecting a collision is really checking the distance between the player and the hazard, if they intersect then there's our collision.

function collision(a,b)
  return (abs(a.x-b.x)+
    abs(a.y-b.y)) <= 8
end

The abs function returns the "absolute value", the positive or negative information is discarded. For instance, 0 - 8 = -8 but with abs we just get back 8. For our purposes we want the distance between the two sprites, we don't really care if it's a positive or negative number.

Once we calculate the "x" distance and the "y" we add them up and check to see if it's less than or equal to the width or height of our sprites, which is eight. If they are eight pixels apart or less then they've collided.

We can add our collision logic way down to the end of our update_ply1 function. Go ahead and add the code below directly before the end of the function.

  if collision(ply1,hzd) then
    over=true
  end

We check to see if the player has collided with our hazard and, if they have, we set the over flag to "true". This indicates the game has ended.

Ending the Game

Now we need to let the player know that the game has ended! Let's add a new function (right after draw_hzd) that displays "Game Over" on the screen.

function draw_over()
  if over then
    print("game over", 2, 2)
  end
end

As advertised, it checks to see if the game is over and, if it is, we display the game over text in the upper left of the screen.

The only thing we're missing is and end to the action when the game ends. Right now, the player could keep moving around. Add the snippet of code below to the very beginning of the update_ply1 function.

if over then return end

If the game is over, we won't update the player's state. They'll be frozen wherever they were when they collided with the hazard.

Whew! We didn't add much code but we've definitely added a lot of functionality! Our player animates as they move and we introduced a hazard into our game. We added some collision detection, ending the game if the player collides with the hazard.

This might be enough to get you going on your own game. 😉

Getting Started with PICO-8

Christopher Miles — Sun, 08 Jan 2023 18:27:44 +0000

I've been interested in fantasy consoles for a long time now but my interest has been limited to reading a blog article here and there and thinking something like "Man, that looks like fun!" After literal years of doing other things I've finally decided to put my own 8-bit arcade game together. I took a look at the field, tried the Pixel Vision 8 and then saw the project archived, regrouped and decided to go with the PICO-8!

At first I started out working with the Pixel Vision 8 project but I found some bugs that were frustrating me and while I was working on fixing those bugs, I found some more bugs. While I was thinking about fixing those bugs the project was archived, there's currently no maintainer. Becoming the maintainer of one of these projects is out-of-scope, I'm more interested in getting a game coded up, so I decided to pony up the $15.00 and purchase a PICO-8 license. I recommend you do the same, its not that much money and it helps keep the project active.

It's a pretty nice product and reasonably polished. It has handy tools for writing code, drawing sprites and tile maps, and editing music. While you could use external tools and pull in files, it's nice to be able to do everything in one place. Plus it looks pretty dang awesome on my snazzy retro looking PC.

This post will cover getting the PICO-8 application installed, drawing your first sprite and moving that sprite around on screen. My plan is to add more articles as I get the game built up. I'm warning you now that this could take a while, I have a day job as well. 😉

Install the PICO-8 Application

Check all of the pockets of your coats and look under the sofa cushions until you've found $15 and then head over to the PICO-8 page. Once you fill in your e-mail address and hand over your dollars, you'll end up with a license through the Humble website. You can then download the distribution for your operating system (or a ZIP of the distribution if you're on Linux).

For those of you on Windows or MacOS, there's not much to the installation; double-click the Windows installer or copy over the MacOS application from the disk image. Those of us on Linux will have to put the distribution somewhere and either start from the command line or create a "desktop" file to launch the application... Unless you are on Arch Linux, there is an AUR package for PICO-8 that's pretty easy to get working.

With all of that work out of the way, you are ready to get started! Launch the PICO-8 application and you will be greeted with the welcome screen.

They're definitely going for the old 8-bit home computer vibe, this reminds me of the Atari 800 or Commodore 64 where you'd get dropped at a BASIC interpreter. This is a bit better as the PICO-8 speaks Lua which is quite a bit more fun to work with.

What you are looking at right now is the interactive console, press the "escape" key on your keyboard to switch over to "editor" mode. It should look like the image below.

The icons in the upper-right hand corner represent the different editors: code, sprite, map, sound effects and music; this image is of the code editor. More information about all of these tools and how they work can be found in the PICO-8 manual.

Draw a Sprite

For now, switch over the the sprite tool (the second one from the left). For those of you not in the know, a sprite is a bitmap graphic that can easily be moved around on screen or stacked on top of or below other bitmaps. We will use this sprite to represent the player.

Each sprite on the PICO-8 is 8x8 pixels in size, these sprites are displayed in rows beneath the sprite editor area. There's one sprite there already, it kind of looks like a white star. Click on that sprite to bring it into focus and display it in the editor area.

The sprites don't have names but they each have a number: the first one (the one you're editing) is 0, the one to the right of that would be 1 and the one to the right of that is 2 and so on.

There are several tools to make editing your sprite easier, feel free to explore them. Go ahead and edit the sprite until you have something you are somewhat satisfied with, after all, you can always go back and make changes later. This sprite will represent the player in your game.

Displaying Your Sprite

The next step is to write enough code to get your sprite to display when your game is run.

The Game Loop

The PICO-8 lets you write your game code with Lua. It's pretty concise and, in my opinion, pretty easy to pick up from scratch; don't worry if you don't know much about Lua. You can read up on the language later, this tutorial will give you enough knowledge just-in-time style to get by.

Make sure you are in "editing" mode, press the "escape" key if you are not. Pick the code editor from the set of icons in the upper-right, it's the first one and kind of looks like two parenthesis "()". You will be welcomed by an entirely empty page of code, go ahead and bang in the code listed below.



function _init()

end

function _update()

end

function _draw()

end

This won't actually do anything but it does provide a short game that does nothing! Press the "escape" key to switch back to console mode and type "run" (the run function will start your game) at the prompt. You should receive no errors and see nothing at all happen. When you grow tired of all the nothing, press the "escape" key to stop your game from running.

These three functions are common to every game, they are called by the PICO-8 to perform the following...

Setup your game
Update the current state of your game (i.e. move the player, the bad guys, etc.)
Draw the current state of the game on screen

While the setup function ("_init") is only called once the other two are called over and over. When the "_update" function is called it can do things like see if any buttons are pressed and move the player to a new location. After that, the "_draw" function will be called to draw the current state of the game to the display. This is called "the game loop", it's a common pattern in a lot of different kinds of video games.

Draw the Player

While we could jam all of our code into those three functions, this style is frowned upon by sane people everywhere. Instead we'll add some new functions to the top of our file and then call those functions where appropriate. First up is keeping track of the state of our player, add the following code to the top of your file.



function init_actors()
  ply1={x=8,y=8,spr=0}
end

This code creates a new variable named ply1 that keeps track of all the data about our player: their coordinates on screen (x and y) and the index of the sprite we're using to represent them (spr). The data we've assigned to the player is called a "table", it stores any number of keys and values and makes it easy to get at their values and update them.

Now we need to call our function to setuip our player along with the rest of the game. Edit your "_init" function to match the listing below.



function _init()
  init_actors()
end

Next up we want to draw the player. This will be pretty simple; we need to position the sprite in the player's current location and then draw the sprite. Type in the code below right underneath your "initActors" function.



function draw_ply1()
  spr(ply1.spr,ply1.x,ply1.y)
end

The spr function is provided by the PICO-8's sprite library, it handles all of the nitty-gritty of drawing the sprite to the display. It takes three arguments, conveniently all of the data it needs is available from our player data. We provide the name (numeric index) of our sprite, the coordinate on the "x" axis and then the coordinate of the player along the "y" axis. With that information in hand, our sprite will be drawn.

The last step is to update our "_draw" function to actually draw the player's sprite. Edit your "_draw" function to look like the code listed below.



function _draw()
  draw_ply1()
end

Your entire file of code should look something like this...

Go ahead and save the code file and run your game. You should see your sprite on the screen. 😎

Move the Player

The last step is to let the player move around by pressing the arrow keys on their keyboard. There's only a little bit of logic to handle, we'll add it to a new function named "update_ply1".



function update_ply1()
  if btn(0) then
    ply1.x -= 2
  end
  if btn(1) then
    ply1.x += 2
  end
  if btn(2) then
    ply1.y -= 2
  end
  if btn(3) then
    ply1.y += 2
  end
end

The PICO-8 has a btn function is doing most of the work here. This function accepts the index of button type (player 0 up, player 0 down, etc.) and returns true if that button is being pressed. We then check each button we're interested in and then (if it's being pressed) we update the state of the player to match.

To make this work, add a call to this function to our "_update" function.



function _update()
  update_ply1()
end

Try it out! You'll notice it doesn't work exactly as expected. The problem is that we aren't clearing the display before we draw the sprite in the next location. The cls function will clear the screen, if we add that right before we draw the items on screen, we'll be all set.



function _draw
  cls()
  draw_ply1
end

Woot! If you are interested in going the extra mile, how would we change the "update_ply1" code to let the player appear on one side of the screen when they reach the other, like in Pac Man? What would we do if we wanted the boundaries of the display to act as a wall? What might we change to make the player move faster or slower? There are many variations even for simple games. 🤯

Build Your Game

The last thing we'll cover is building a "cartridge" that contains your game, this provides you one with one file that you can share with friends. You simply drag the cartridge file onto the PICO-8 (or executable) to load the game. With the PICO-8 we can also save the cartridge as an actual PNG image that you can view in an image viewer or web browser. It will have a screenshot as well as a little description of your game.

Switch to the code editor, we're going to add a couple of lines to the top of your game's code. We'll add two comments; these won't change how your game works but they will be used to describe your game and will be part of the cartridge. Two dashes in a row ("--") start a comment, the comments are displayed at the bottom of your cartridge image. Add one comment with the name of your game and another with your name. Make sure these are the first two lines of code in the editor!



--move the player
--cmiles74

Next, switch back to the console and type "run" to run the game. Move the player around a little bit and then press the "control" key along with the "7" key (control-7). You should see the PICO-8 say "captured label image" down at the bottom of the screen. You've just taken the screenshot that will decorate your cartridge!

Now you can save your cartridge to disk. Type the following at the console to save your game.



save move-the-player.p8.png

At this point your game is built! You can switch to your operating system's file browser so that you can navigate to your cartridge directory. This will vary based on your operating system.

Operating System	Cartridge Path
MacOS	~/Library/Application Support/pico-8/carts
Linux	~/.lexaloffle/pico-8/carts
Windows	%AppData%/Roaming/pico-8/carts

When you visit this directory you should see your cartridge listed. If you drag it over to your web browser, you'll see what the image looks like.

You can load your cartridge by double-clicking it or dragging it onto your PICO-8 window. You can also email this file to friend or post it somewhere on the internet. Anyone with the PICO-8 application installed can run your game or play around with it, perhaps even making their own changes and sending you an updated cartridge.

Conclusion

It's quite an achievement to get this much done so quickly, the tools provided by the PICO-8 make it easy to get straight to game building. Looking at your code, there's not so much that it's hard to read, hopefully it's also pretty clear how things fit together.

Good luck on your next video game! If you are looking for inspiration, type "splore" at the PICO-8 console and check out some of the games people have written. Don't forget to check back here for more articles, I plan to get a couple more written.

Getting Started with Pixel Vision 8

Christopher Miles — Fri, 09 Dec 2022 20:15:45 +0000

I've been interested in fantasy console for a long time now, but my interest has been limited to reading a blog article here and there and thinking something like "Man, that looks like fun!" After literal years of doing other things, I've finally decided to put my own 8-bit arcade game together. After taking a look at the field, I decided to go with the Pixel Vision 8! My plan is to try an get some of the people at work interested so it's important that the console be free. 😉

It's a pretty nice product and reasonably polished. It has handy tools for writing code, drawing sprites and tile maps, and editing music. It's also pretty modular, each specialized function is managed by a "chip" (color, tile, sprite, etc.) that can be customized or even replaced with your own code. Plus, it looks pretty dang awesome on my snazzy retro looking PC.

This post will cover getting the Pixel Vision 8 application installed, drawing your first sprite and moving that sprite around on screen. If people are interested and if I have time, I may write another post that covers getting an actual game coded up.

Install the Pixel Vision 8 Application

Head over to the Pixel Vision 8 home page and click on the link for the "Latest Build". This will take you to the page with the latest release on the project's GitHub page. Scroll down to the "Assets" section and then click on the archive for your operating system. For instance, if you were on MacOS and the latest release was 1.0.18, you'd click on the link that reads "PixelVision8-v1.0.18-macOS.zip".

Once you have the release downloaded, go ahead and unzip it. This is kind of on you, as every platform is a bit different. Good luck!

With the distribution unpacked, open up the directory and take a look at the contents. There's a lot of stuff in there, you want to double-click the executable named "Pixel Vision 8". The machine will boot up, load it's operating system and present you with a welcome screen.

Pretty neat, eh? It kind of reminds me of the Amiga Workbench. Go ahead and click the "Yes" button to create a new project. You will be dropped at the "Workspace Explorer" where you can browse all of the files in your starter template project.

Draw a Sprite

First things first: we need a sprite to display on screen. For those of you not in the know, a sprite is a bitmap graphic that can easily be moved around on screen or stacked on top of or below other bitmaps. We will use this sprite to represent the player.

Sprites in Pixel Vision 8 are all 8x8 pixels in size, when your game starts up the file "sprites.png" is loaded and chopped up in to 8x8 images. Each of these images is then loaded into the memory of the sprite chip until it runs out of space. Each sprite is given an index number at load time (starting from 0) and you can use that index to fetch a particular sprite.

Double-click the folder labeled "Sprites" to open it up, then choose "New Sprite" from the "Workspace Explorer" menu (the menu is under an icon that looks like three colored slashes, "///"). The "New Sprite" dialog will appear, asking you to name the file; change the name to "sprites" (plural).

At this point we run into a minor bug in the Pixel Vision 8 Workspace Explorer: we need the "sprites" file in the root of our project, next to our "code" file. Unfortunately we can only create a new sprite file when we're in the "Sprites" directory. Lucky for us this is easy to fix, simply drag the "sprites" file to the folder named "..", the one with the green up-arrow icon. You will be asked to confirm the move, go ahead and say "yes". You may now double-click the ".." folder to move up a level and you will see the files in your project.

Now that you have a file of sprite data, double-click the "sprites" file to open it. The sprite editor will open and you will be asked if you want to use the Pixel Vision 8 color palette, go ahead and say yes. You should now see one lone sprite in the upper-left hand corner of your canvas.

We want edit that little sprite, but it's so small! In the lower right-hand corner of the editor is a large yellow button; click that button to zoom in and make the sprite as large as you can (if you click while holding down the shift key then everything will get smaller 😉). Now that it's nice and big, edit that sprite until you think it looks like your player; you can select the draw color by clicking on it, the pencil and line tools may be used to draw your spite. The magenta background will be ignored, at game time it will be replaced with whatever is behind your sprite.

When you like the look of your sprite, choose "Save" from under the "///" menu to save your sprite and then choose "Quit" to close the sprite editor.

Displaying Your Sprite

The next step is to write enough code to get your sprite to display when your game is run. You should be back at the "Workspace Explorer", looking at the contents of the "sprites" directory. Double-click the "code" file to edit your game's code in the editor.

The Game Loop

The Pixel Vision 8 lets you choose between using Lua or C#, for this project we'll be using Lua. It's pretty concise and, in my opinion, easier to pick up from scratch than C#. You can read up on the language later, this tutorial will give you enough knowledge just-in-time style to get by.

You will see a bunch of code in the file already, this is the sample code that comes with each new workspace. In my opinion it's not all that helpful; press "Control"+"A" to select all of the text and then hit the backspace or delete key to remove all of it. Next, type in the text below to code up the skeleton of your game.

function Init()

end

function Update(timeDelta)

end

function Draw()

end

This won't actually do anything but it does provide a short game that presents an entirely black screen! Choose "Save" from under the "///" menu and then choose "Run Game" to try it out. Your game should load and you should see a screen of black. When you are done exploring this barren landscape, press the "Escape" key to quit your game and return to the editor.

These three functions are common to every game, they are called by the Pixel Vision 8 to perform the following...

Setup your game
Update the current state of your game (i.e. move the player)
Draw the current state of the game on screen

While the setup function ("Init") is only called once the other two are called over and over. The "Update" function will be called, it can do things like see if any buttons are pressed and move the player to a new location. After that, the "Draw" function will be called to update what we see on the display. This is called "the game loop", it's a common pattern in a lot of different kinds of video games.

Draw the Player

While we could jam all of our code into those three functions, this style is frowned upon by sane people everywhere. Instead we'll add some new functions to the top of our file and then call those functions where appropriate. First up is keep track of the state of our player, add the following code to the top of your file.

function InitActors()
  player = {posX = 8, posY = 8, sprite = 0}
end

This code creates a new variable named player that keeps track of all the data about our player: their coordinates on screen (posX and posY) and the index of the sprite we're using to represent them (sprite). The data we've assigned to the player is called a "table", it stores any number of keys and values and makes it easy to get at their values and update them.

Now we need to call our function to initialize our player when our game initializes. Edit your "Init" function to match the listing below.

function Init()
  InitActors()
end

function DrawPlayer()
  DrawSprite(player.sprite, player.posX, player.posY)
end

The DrawSprite function is provided by the Pixel Vision 8 (via the sprite chip), it handles all of the nitty-gritty of drawing the sprite to the display. It takes three arguments, conveniently all of the data it needs is available from our player data. We provide the sprite, the coordinate on the "x" axis and then the coordinate of the player along the "y" axis. With that information in hand, our sprite will be drawn.

The last step is to update our "Draw" function to actually draw the player's sprite. Edit your "Draw" function to look like the code listed below.

function Draw()
  DrawPlayer()
end

Your entire file of code should look something like this...

Go ahead and save the code file and run your game. You should see your sprite on the screen. 😎

Move the Player

The last step is to let the player move around by pressing the arrow keys on their keyboard. There's only a little bit of logic to handle, we'll add it to a new function named "UpdatePlayer".

function UpdatePlayer()
  if (Button(Buttons.Left, InputState.Down, 0)) then
    player.posX -= 2
  end
  if (Button(Buttons.Right, InputState.Down, 0)) then
    player.posX += 2
  end
  if (Button(Buttons.Down, InputState.Down, 0)) then
    player.posY += 2
  end
  if (Button(Buttons.Up, InputState.Down, 0)) then
    player.posY -= 2
  end
end

The Pixel Vision 8 Button function is doing most of the work here. This function accepts a button type (up, down, left, right, etc.), a state for the button, and a controller index. The state of the button is either "Down" (pressed) or "Up" (not pressed) and there are only two controllers (0 for player 1, 1 for player 2). When you call the "Button" function it takes this information and compares it to the actual state of the controller, if the controller is in that state then the function returns "true".

When this function is called we check each button and, if it's pressed, we update the current location of the player. Simple, right?

To make this work, add a call to this function to our "Update" function.

function Update(timeDelta)
  UpdatePlayer()
end

Try it out! You'll notice it doesn't work exactly as expected. The problem is that we aren't clearing the display before we draw the sprite in the next location. The Clear function will clear the screen, if we add that right before we draw the items on screen, we'll be all set.

function Draw()
  Clear()
  DrawPlayer()
end

Woot! If you are interested in going the extra mile, how would we change the "UpdatePlayer" code to let the player appear on one side of the screen when they reach the other, like in Pac Man? What would we do if we wanted the boundaries of the display to act as a wall? What might we change to make the player move faster or slower?

Build Your Game

The last thing we'll cover is building a "disk" that contains your game, this provides you one with one file that you can share with friends. You simply drag the disk file onto the Pixel Vision 8 window (or executable) to run the game.

Go ahead and quit the code editor, bringing you back to the Workspace Explorer and viewing the contents of your game's directory. First we want to change the name of your game and it's description; double click on the "Info" file or edit your game's information.

With this tool you may edit the name and description of your game, along with a couple other items. Go ahead and change the name and description, then choose "Save" from the "///" menu.

Next, click the "Build" button to the right of your game's name. You will be asked if you really want to build the game's disk, go ahead and click the "Build" button. Your game will be built and then you will be asked if you want to visit the new build directory, go ahead and click the "No" button and close the editor.

At this point your game is built! You can switch to your operating system's file browser, there should be a "PixelVision8" directory present. When you open up that directory, you should see another named "Workspace", this contains all of the files you were looking at with the Workspace Explorer. Open the "MyFirstGame" directory to see your game's files, then navigate through the "Builds" and "Build" directories. You should now see the disk file for your game.

Drag the disk to the Pixel Vision 8 window. The disk with your game should be visible on the desktop of the Workspace Explorer and the files that make up your game should be visible. Now open the disk by double-clicking it and then pick "Run" from the "///" window: you will be able to run your game!

You can email this file to friend or post it somewhere on the internet. Anyone with the Pixel Vision 8 application installed can run your game or play around with it, perhaps even making their own changes.

Conclusion

It's quite an achievement to get this much done so quickly, the tools provided by the Pixel Vision 8 make it easy to get straight to game building. Looking at your code, there's not so much that it's hard to read, hopefully it's also pretty clear how things fit together.

Good luck on your next video game! If time allows, I'll post another article that gets a little closer to a playable game.

Really Using Visual Studio Code Development Containers

Christopher Miles — Sun, 26 Sep 2021 16:45:43 +0000

Here's a hypothetical: you have to on-board another developer and, on top weathering their entirely reasonable questions about why this or that tool isn't being used, you have to help them get their development environment setup. In addition to explaining that, yes, this other tool would be great and improve the project but time is kind of short right now and there are features that need to be developed you suddenly notice that they are not using Linux. All of your suggestions work, but not quite, and it is truly frustrating.

Development Containers

At this point, like all of us, you wonder "is there a better way?" There's always a better way and the idea of using Docker to support your development environment is nothing new. The interesting bit is that Visual Studio Code is bringing some tooling to the table that makes this much easier.

A spec for Development containers or "devcontainers" to specify how to build your development container.
An extension called "Remote Containers" to make it easier to develop in these containers locally.
GitHub Codespace will run your development container in the cloud and connect remotely from Code or through their website.

Before we go further, let me clarify: a development container isn't just one container. You can also provide a Docker compose file where you can describe all of the other containers your need to work on your project (a database server, web server, etc.)

Put this all together and every new developer on your project is literally one click away from being productive.

Fastest: They can click the link on GitHub and use the web-based version of Code and get to work within minutes.
Fast: They can click one more button to spin up the required containers on GitHub and then connect from their own, locally installed and customized copy of Code.
Somewhat Quickly: They can clone the project and open it in their local copy of Code, letting Code provision the containers on their local machine
Not quick at all: For the truly picky developers, they can do whatever they like on their own time, we've already provided three working options!

In my opinion, they real pain-point we're avoiding here is the morass of confusion and mediocre performance that is Docker Desktop. I could detail my complaints about the product but the fact is that if you're reading this article, we're all on the same page already.

Getting It Setup

If you're on the fence about Visual Studio Code, then just put your concerns aside and trust me; it's worth it. It's free, it's popular, it's easy to install and there are extensions available for nearly every feature it lacks. It may not be every developer's preference but it's easily good enough to get the job done.

Next install (if you haven't already)...

Visual Studio Code
Git
Docker (if you are on Linux) or Docker Desktop (if you're on MacOS or Windows)

After that, the first step is to get the development container setup and working. You can start from scratch by creating a directory for your sample project (maybe called "new-project", avoid spaces in the directory name) and then a directory inside of that called .devcontainer with a file called devcontainer.json inside of that. The simplest file will contain only a pointer to a Docker image. Using Code, create and save the following devcontainer.json file.



{
  "image": "mcr.microsoft.com/vscode/devcontainers/php:8-bullseye"
}

That will spin up a development container based on Microsoft's PHP image, which in turn is built on Debian Bullseye. To try it out, choose "Command Palette..." from under the "View" menu and start typing "Developer: Reload Window" to choose that command.

Code will close and re-open the sample project and then it will prompt you to install the recommended extensions. Click on the "Install" button."

Now that the extension is installed, Code will suggest that you re-open the project one more time but, this time, open it in a container.

This time Code will open up the project and then spin up a new Docker container using the image we specified in the .devcontainer.json file. It will then install it's own set of tools inside that container to make it easier to edit files, run code, etc.

When it's all done you'll be back at the a regular Code window, with your project files on the left-hand side. But if you select "New Terminal..." from under the "Terminal" window, you'll see that the terminal has opened a session inside of the container!

Customize the Development Container

This is the simplest scenario where we just plop a development container definition into our project and Code spins it up. But what if we needed to add some of our own tools? Perhaps our team relies heavily on Flyway for database migrations and HTTPie for testing our RESTful endpoints. This can easily be done by writing our own Docker image definition. First remove the "image" attribute from your devcontainer.json file and replace it with a "dockerFile" attribute that points to our soon-to-be-created Dockerfile.



{
    "dockerFile": "Dockerfile"
}

Next, create a new Dockerfile next to your devcontainer.json file and add the following recipe to it.



FROM mcr.microsoft.com/vscode/devcontainers/php:8-bullseye

# update apt
RUN apt update

# install httpie
RUN apt install -y httpie

# install flyway
WORKDIR /usr/local
RUN wget -qO- https://repo1.maven.org/maven2/org/flywaydb/flyway-commandline/8.0.0-beta1/flyway-commandline-8.0.0-beta1-linux-x64.tar.gz \
| tar xvz && sudo ln -s `pwd`/flyway-8.0.0-beta1/flyway /usr/local/bin

Instead of using the PHP image directly we use it as a starting point for our own container image. We run apt update to pull down the latest list of packages for the distribution and then we use apt to install HTTPie. With that done, we follow the instructions from the Flyway website to get that tool installed as well.

Save the file and then reload the window (by choosing the option from the command palette). Code will close and then open the project and prompt you to open the project yet again in the container, go ahead and do so. While it's doing that it will notice that the development container definition has changed and ask you if you'd like to rebuild it, do that as well.

Code will delete the Docker image, rebuild the image using our Docker file and then drop us at the project window. You can open up a new terminal panel to verify that our will truly has been done and that our tools are now available.

At this point you should start to get excited, thoughts of time saved and frustrations avoided racing through your mind. But wait, there's more!

Add More Containers to the Development Environment

If we have anything at all in common, your next though is something along the lines of "how do I get more containers in there, for things like a database or a web server?" In the same way we added a Dockerfile we can also add a Docker Compose definition. When Code starts up the project it will bring up all of the containers in the compose definition and then connect to the development container. Let's add a database server to our devcontainer.

Open up your devcontainer.js file, remove the dockerFile attribute and replace it with a dockerComposeFile entry.



{
  "dockerComposeFile": "docker-compose.yml",
  "service": "development",
  "workspaceFolder": "/workspace"
}

We point our devcontainer definition at our Docker Compose file, with the service attribute we let Code know which container we'd like it to connect to when that stack for the project is brought up. Lastly, we add a workspaceFolder attribute and provide the complete path where we want our project files mounted.

We also need to create that compose file. Create a new file called docker-compose.yml right next to our devcontainer definition and add the following.



services:
  development:
    build: 
      context: .
      dockerfile: Dockerfile
    volumes:
      - ..:/workspace
    stdin_open: true

In the stanza above, we describe the container that we'll be using for our development environment. We point it at our existing Dockerfile and we map in the volume that will hold our project files (the directory above the one holding our devcontainer definition, mounted at "/workspace"). The context attribute tells compose to look for files in the current directory, stdin_open indicates that we'll be connecting with an interactive session.

That takes care of our development container, now we'll add a database server. Since we're using PHP for this project it's clear that the database needs to be MySQL. Before we add the database service to our compose file, let's setup the credentials we'd like to use. Docker Compose will read a file named .env when it starts assembling the stack and it will make the variables in that file available to our script. Create a new file named .env next to your docker-compose.yml file and add the text below.



DATABASE_HOST=mysql_db
DATABASE_USER=example_account
DATABASE_PASSWORD=applesauce
DATABASE_DB=example

The database credentials will now be in environment variables that we can use in our compose file. Add the following service you your docker-compose.yml file to get our database server setup.



  mysql_db:
    hostname: mysql_db
    image: mysql/mysql-server:latest
    environment:
      - MYSQL_ROOT_PASSWORD=${DATABASE_PASSWORD}
      - MYSQL_DATABASE=${DATABASE_DB}
      - MYSQL_USER=${DATABASE_USER}
      - MYSQL_PASSWORD=${DATABASE_PASSWORD}

Before we spin this up, it would be nice to have an convenient way to interact with our database server from inside Code...

Add Visual Studio Code Extensions

In fact, it would also handy to have Code extensions that we've agreed to use in our project installed along with everything else. Let's add the PHP extension pack and then a MySQL client. Open up the your devcontainer.json file and add the extensons attribute listed below.



{
    "dockerComposeFile": "docker-compose.yml",
    "service": "development",
    "workspaceFolder": "/workspace",
    "extensions": [
        "felixfbecker.php-pack",
        "cweijan.vscode-mysql-client2"
    ]
}

Open the command palette and ask Code to reload the folder. Code will prompt you to rebuild the devcontainer as the configuration files have changed; agree and let it bring up the new environment. When it's finished you will be back at your Code window, the new extensions will be listed in the "Extensions" panel under "Dev Container - Installed".

You can click on the icon for our database client and type in the connection information (remember the "hostname" attribute in our compose file?) to connect to our new MySQL database.

Now we're starting to get somewhere! We have a container we can use for development, with all of our tools and Code extensions installed and another container running our database server. From here we could add a third container to run our web server and serve our application. We could even add migrations to get our application in a usable state, ready for developers to start working on new features.

This article is already getting pretty lengthy, we won't detail getting migrations or a web server working here. You can take a stab at it on your own or you can checkout the full example project on GitHub.

Complete example project with web server and migrations

GitHub Codespace

The last thing I wanted to mention is that we have everything we need to host our development environment on GitHub Codespace. If we publish this on GitHub, our devcontainer definition will get picked up and (if it's enabled for your account) the "Codespace" tab with a "New codespace" button will appear for anyone who stumbles upon the project. Anyone who would like to contribute can simply click the button and start work in seconds. Codespace will use the their GitHub credentials to handle pushing, pulling and committing code.

Summary

I encourage you to checkout the example project on GitHub, you should try it out in Codespace as well as launching the stack in Codespace and then connecting with your own, local version of Visual Studio Code. In my opinion this latter option is pretty compelling: you can customize Code the way you like it (including extensions that you prefer) and then connect to a set of running containers hosted at GitHub.

This is especially compelling if you are on a platform that only provides Docker through Docker Desktop (MacOS or Windows). While Docker Desktop is fine for small projects, anything of significant size will bring Docker Desktop to it's knees. The real pain point here is sharing files from the host machine to the virtual machine that's running Linux (and Docker) and then sharing those files once more with the actual containers running in Docker.

I hope this article has given you an idea of how Code development containers work and how they might fit into your own workflow. In my opinion, it's an exciting feature and really gets us to a place where using Docker to containerize our development environment starts to become accessible to everyone, regardless of their level of experience with Docker and the related tooling. While Codespace might feel a bit gimmicky, it does allow anyone who may want to contribute to your project a quick and easy way to get started and to submit that pull request.

Credits

The banner image is from this website.

Getting Started with Clojure, Even on Windows

Christopher Miles — Thu, 29 Apr 2021 20:23:31 +0000

When you find yourself looking to flex your language learning skills, functional languages often end up on your short list. After reading this past year's "State of the Clojure Community" survey results, I couldn't help but notice how few developers who run Windows are choosing to learn Clojure. For sure, it could be the shining allure of F#. But maybe it's also that many tutorials focus on Linux or MacOS and often skip over the installation of the Java development kit. In any case, this article is going to cover getting up-and-running with Clojure, specifically when using Windows and we won't be using the Windows subsystem for Linux (WSL).

In my opinion, Clojure has a lot to offer anyone working on software development regardless of the operating system we have on our primary machine. 😉

Aside from being a functional language, Clojure has some other interesting features: there is a very real stress on immutability and that makes some other things, like concurrent or parallel code, easier to reason about. The language continues to evolve and improve and the community is reasonably friendly and accessible.

Tools

This article assumes that you have little or no experience with Clojure and that you haven't really done much in the way of setup ahead of time. To that end we've chosen some tools for you.

Visual Studio Code for editing source code
Calva will be the VS Code plugin we use to make working with Clojure straightforward
Clojure runs on the Java virtual machine, we'll get our JVM from AdoptOpenJDK
There are a couple different tools for managing Clojure projects, we'll use Leiningen

All of these are quick and painless to install, even under Windows.

Install Your Tools

We're going to start at the bottom of our list and work our way up, the first thing we will install is...

Java

Open up your preferred web browser and navigate to the AdoptOpenJDK website. Whatever they reccomend is fine, leave the radio buttons as-is and click on the "Latest Release" button to download the Java Development Kit installer. Double-click the installer to run, accept the license agreement, and install the software.

Leiningen

Clojure projects are typically managed by a tool, we'll be using Leiningen. It will be responsible for downloading and managing libraries that we use in our project and building our code during development and deployment.

Where Do You Keep Your Scripts?

Leiningen is distributed as a batch file that you place in your path. This is pretty common and if you have a location for storing files like this, you can skip this part. If you don't have a location, read on!

I recommend that you store your loose executables and scripts in a directory named "bin" in your home directory. We will now create that folder and then add it to your path, making these scripts available to you from wherever you happen to be on your machine.

Open up a File Explorer and then select the directory with your name from the right-hand side to view the contents of your "home" directory ("C:\Users\YOUR_NAME"). Create a new directory for your loose scripts and call it "bin" (C:\Users\YOUR_NAME\bin"), you can double-click it to view it's contents and then click on the path at the top of the window and copy it to the clipboard.

Next, right click on the icon for "This PC" and choose the "Properties" option; the details for your computer will appear. Click on the "Advanced System Settings" item in the right-hand menu, then click on the "Environment Variables" button in the lower right-hand corner. The "Environment Variables" window will appear with the entries specific to your account at the top. Highlight the entry with the value "Path" in the "Variable" column and then press the "Edit..." button. You will be present with a list of all the directories added to your path, click the "New" button to add a new entry and then paste in (or type) the path to your new "bin" directory ("C:\Users\YOUR_NAME\bin"). Press the "OK" button to make the change permanent and then again to dismiss the "Environment Variables" window and then again to close the "System Properties" window.

Install Leiningen

Installing Leiningen is straightforward: we download the script and then place it in the same directory where we keep all of our loose tools and scripts (i.e. "C:\Users\YOUR_NAME\bin" or any location on your executable path). Open your web browser and navigate to the Leiningen website and scroll down to the "Install" section, you will see the instructions and links to the two scripts (one for Linux and one for Windows). Right click on the script for Windows ("lein.bat") and choose the "Save Link As..." item to download a copy to your machine. The last step is to place the file in your "bin" directory ("C:\Users\YOUR_NAME\bin").

Now the Leiningen is available, we can tell it to download its own dependencies so that it's ready to run. Open up a Powershell window and then run the following:

PS > lein self-install

The self-install should go pretty quickly, typically the only output you'll see is...

Downloading Leiningen now...

Just to test it out, you can ask Leiningen what version it is.

PS> lein -version
Leiningen 2.9.6 on Java 11.0.11 OpenJDK 64-Bit Client VM

Visual Studio Code

You probably have Visual Studio Code installed and running already, but for the sake of completion we'll provide a brief overview of that here. If you're all set with Code, you can definitely skip this step.

When you visit the Visual Studio Code download page, you can click on the big blue "Windows" download box. That will get you the "User Installer" for your machine. This version will install everything in your current account profile and, in general, is much easier to keep up-to-date.

Once downloaded, you can double-click to start the installation. After you accept the license and choose the install location, you will be presented with a couple other options. All things being equal, I recommend that you check the boxes to add an "Open with Code" action to Windows Explorer files and directories.

Install the Calva Clojure Plugin

Calva is a plugin for Visual Studio Code that works with Leiningen (and other tools) to provide an integrated development environment where you can work on your Clojure code and get real work done. Since we already have our Java development environment and Leiningen installed, this will be very easy!

Open up Visual Studio Code and then click on the "View" menu and choose the "Extensions" item. The "Extensions" panel will load in the left hand side of the editor window and a list of popular extensions will be displayed. Click in the search box at the top of the panel and type in "Clojure". You will see that there are a variety of plugins for Code that support Clojure, "Calva" should be listed towards the top. Go ahead and click on "Calva" to select it, information about the plugin should appear in the main body of the editor window. Click on the "Install" button at the bottom of the Calva header panel to install it.

Create Your Project

Now that we have all of the tools installed, we are ready to create our project! Close the editor panel for Calva and then click on the "View" menu and choose the "Explorer" option. Next, click on the "File" menu and choose "Open Folder..." A new open dialog will appear, navigate to where you would like to store your code and then click the "New Folder" button towards the top of the window to create a new folder, call it something like "tutorial". Next, click on the new folder to select it and click on the "Select Folder" button at the bottom of the window.

Create the Project File

Every project needs a project file where we can store information about our project, like it's name, version and any libraries that it depends on. Click on the "File" menu and choose "New File" to create a new file. Before we start adding content to the file, choose "Save As..." from under the "File" menu and name your file "project.clj", you want to save it right into the root of your project directory.

We'll fill this file in with our project definition. You can model on the very simple one listed below (or copy it directly).

(defproject tutorial "0.1"
  :dependencies [[org.clojure/clojure "1.10.3"]]
  :main tutorial)

Here we set the name of our project, it's version and a description. Next we add the version of Clojure we're working with as a dependency (1.10.3 is the current version at the time of this writing). On the last line we point out which file should be the entry point to the project. You can name it anything, in this case we called it "tutorial" meaning that Leiningen should look for a file called "tutorial.clj" in our source code directory.

Add a Source File

Now we can add a file of source code. Leiningen will look for a source code in the "src" directory; in the "Explorer" panel on the left-hand side of your Code window, you will see some icons appear at the top when the mouse enters the panel or it gains focus. Click on the icon that looks like a folder to create a new directory and call it "src". Click on that new directory and then click on the icon that looks like a file to create a new file inside that "src" directory and call it "tutorial.clj". Add the text below to your "tutorial.clj" file and then save it.

(ns tutorial)

(defn -main
  [& args]
  (println "Hello!"))

The "-main" function represents the entry point to our application, if we were to compile our project and run it on it's own, the "-main" function is called with any parameters we pass in on the command line.

Start An Interactive Session (REPL)

Select the "Command Palette" item from under the "View" menu, the command window will appear at the center top of your Code window, it will have a text area at the top where you can type in text and a list of matching commands in the body of the window. Type in "jack" and you will see the that the list narrows by a lot, you will see the "Calva: Start a Project REPL and Connect (aka Jack-in)" item in that list, to the right of it will be the command shortcut to invoke the task (Control-Alt-C and then Control-Alt-J). Note the shortcut as it's easier to type then navigating the command palette and then select the "Calva: Start a Project REPL..." item.

Calva will now ask you what kind of REPL session you'd like to start. Go ahead and pick "Leiningen" (the first item) to start a new REPL with the Leingingen tool. The panel at the bottom of your window will show you details as Calva sets up and connects to your REPL and then a new editor pane will appear with the contents of your session (typically called "output.calva-repl").

Go ahead and click into the REPL pane to make it active. You'll see some introductory text from Calva and then the prompt (clj:tutorial:>). Type the following after the prompt and press return.

(-main)

This will call the main function and you should see output similar to what is pictured below.

; Jack-in done.
clj꞉tutorial꞉> 
(-main)
Hello!
nil
clj꞉tutorial꞉>

Congratulations! You have installed all of the tools and started a new Clojure project. 🏆

Working with Calva

There's a good amount of documentation on the Calva website, you should definitely check it out. There is a "try first" tutorial that goes over compiling your whole file or just the form you are working on. From there you can explore the rest of the documentation to get productive in no time.

Build a Self Contained File

Now that we have some code that knows how to say "hi", the last thing we'll do is build one Java archive (JAR) file that contains our application. We can then distribute that archive to anyone else who would like to run our code.

First we need to compile a class with our entry point so that a regular Java runtime knows how to start it up. Open up the "src\tutorial.clj" file and add (:gen-class) to the namespace declaration at the top, like so:

(ns tutorial
  (:gen-class))

(defn -main
  [& args]
  (println "Hello!"))

Next we'll ask Leiningen to compile one archive for our project. If we had included any libraries in our project, Leiningen would add those to the archive as well. Click on the "View" menu and choose the "Terminal" item, a new terminal pane will open at the bottom of your code window. Type in the following and press return to build a self contained JAR file for your project.

C:\> lein uberjar

Leiningen will display some output and build your archive. You may run it like so...

C:\> java -jar target\tutorial-0.1-standalone.jar

You should see the output with the text "Hello!" At this point you should know enough to be dangerous. Good luck! 😏

Provision Pragmatically and Programmatically with CloudFormation: Part 3, Backup Jobs and Wrap-Up

Christopher Miles — Sun, 21 Apr 2019 23:31:20 +0000

This is the third and final part in a three part series where we build up a CloudFormation tempate for what I like to think of as a pretty typical environment: one virtual private cloud broken into two subnets and including two instances. So far we have covered...

In this final part we put our backup jobs in place and wrap-up the template. We have covered a lot of CloudFormation syntax and have realy exercised the tool, after this last part in the series we'll have everything you need to support your next project. If you want to browse through the complete template, it is available on GitHub.

Setting Up Lambda for Snapshotting (and Retention)

What, when and how to backup your instances and data is a complicated subject and covers a lot of ground. For this project we're only going to deal with the bare minimum: performing nightly snapshots of your images and retaining those snapshots according to a schedule. This will get a reliable image of your server's storage at backup time but it may not guarantee that your database files are in a consistent state. We don't walk through the process here, but my advice is to setup a job on your database server that writes a backup of the database to disk before the snapshot job runs. If you ever restore a snapshot you will have a consistent backup of the database right there along with it.

We will use the Amazon Lambda service to perform the snapshot and cull old snapshot images. Since the Lambda script runs independently of our instances we can be sure it will always run on-time and we can monitor the success or failure of the backup job on it's own: we don't need to provide administrative access to our instances in order to perform or monitor the backup process. It's also really nice that we can get the backup job writtent and start it running all from within our CloudFormation script.

Daily Snapshot Job

The first thing we need to do is to setup a new role for our backup job, this role will have the ability to manage snapshots and write logs to CloudFormation (so we can monitor those backup jobs).

  TutorialLambdaBackupRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: TutorialLambdaBackupRolePolicy
          PolicyDocument:
           Statement:
             - Effect: Allow
               Action:
                 - ec2:CreateTags
                 - ec2:CreateSnapshot
                 - ec2:DeleteSnapshot
                 - ec2:Describe*
                 - ec2:ModifySnapshotAttribute
                 - ec2:ResetSnapshotAttribute
                 - xray:PutTraceSegments
                 - xray:PutTelemetryRecords
                 - xray:GetSamplingRules
                 - xray:GetSamplingTargets
                 - xray:GetSamplingStatisticSummaries
                 - logs:*
               Resource: "*"

We've seen all of this stuff before: we create a new role and we associate the AWS service (in this case Lambda) and the AssumeRole action, letting whoever we assign the role the ability to "assume" it. We then create a new policy for the role that allows whoever assumes this role access to manage snapshots for our account as well as the ability to write to the CloudFormation logs and the Amazon X-Ray (tracing) service.

Next we create our Lambda process, I'm going to leave out the code for now and we'll go over that next. There are better ways to get your backup script into your Lambda but they all involve adding a lot of complexity to this article. When you have the time and energy you can look into this on your own (or maybe I'll write another article). In any case, having the script inline works for now.

  TutorialCreateBackupLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: ebs-snapshots-create
      Description: Create EBS Snapshots
      Handler: index.handler
      Role: !GetAtt TutorialLambdaBackupRole.Arn
      Environment:
        Variables:
          REGIONS: !Ref AWS::Region
      Runtime: python3.6
      Timeout: 90
      TracingConfig:
        Mode: Active
      Code:
        ZipFile: |
        ...
        code ellided for clarity
        ...

We use the AWS::Lambda::Function resource to provision our new Lambda function ad we assign it a name, description and tell it how to invoke itself. We're going to write the backup job in Python because it keeps the code simple and nearly everyone knows Python. ;- And next, the Python code itself.

import os
from datetime import datetime, timedelta
import boto3

def handler(event, context):

  ec2_client = boto3.client("ec2")
  env_regions = os.getenv("REGIONS", None)
  today = datetime.today()
  total_created = 0

  if env_regions:
    regions = ec2_client.describe_regions(RegionNames = env_regions.split(","))
  else:
    return total_created

  # loop through all of our regions
  for region in regions.get("Regions", []):
    regionName = region["RegionName"]
    print("Checking for volumes in region %s" % (regionName))
    ec2_client = boto3.client("ec2", region_name = region["RegionName"])

    # query for volumes with matching tags that are in use
    result = ec2_client.describe_volumes(Filters = [
      {"Name": "status", "Values": ["in-use"]}
    ])

    if not result:
      print("No matching EBS volumes with tag 'Client' = '%s' and tag 'Project' = '%s' in region %s" % (clientTag, projectTag, regionName))
      return total_created

    for volume in result["Volumes"]:
      volume_name = volume["VolumeId"]
      volume_description = "Created by ebs-snapshots-create"

      # check the volume for matching tags
      if "Tags" in volume:
        for tag in volume["Tags"]:
          if tag["Key"] == "Name":
            volume_name = tag["Value"]
          elif tag["Key"] == "Description":
            volume_description = tag["Value"]

      # daily
      print("Creating snapshot for EBS volume %s" % (volume["VolumeId"]))
      result = ec2_client.create_snapshot(
        VolumeId = volume["VolumeId"],
        Description = volume_description
      )
      ec2_resource = boto3.resource("ec2", region_name = region["RegionName"])
      snapshot_resource = ec2_resource.Snapshot(result["SnapshotId"])
      snapshot_resource.create_tags(Tags = [
        {"Key": "Name", "Value": volume_name},
        {"Key": "Description", "Value": volume_description},
        {"Key": "Interval", "Value": "Daily"}
      ])
      total_created += 1

      # quarterly
      if today.day == 1 and today.month % 4 == 0:
        print("Creating quarterly snapshot for EBS volume %s" % (volume["VolumeId"]))
        result = ec2_client.create_snapshot(
        VolumeId = volume["VolumeId"],
        Description = volume_description
        )
        ec2_resource = boto3.resource("ec2", region_name = region["RegionName"])
        snapshot_resource = ec2_resource.Snapshot(result["SnapshotId"])
        snapshot_resource.create_tags(Tags = [
          {"Key": "Name", "Value": volume_name},
          {"Key": "Description", "Value": volume_description},
          {"Key": "Interval", "Value": "Quarterly"}
        ])
        total_created += 1

      # annual
      if today.day == 31 and today.month == 12:
        print("Creating annual snapshot for EBS volume %s" % (volume["VolumeId"]))
        result = ec2_client.create_snapshot(
        VolumeId = volume["VolumeId"],
        Description = volume_description
        )
        ec2_resource = boto3.resource("ec2", region_name = region["RegionName"])
        snapshot_resource = ec2_resource.Snapshot(result["SnapshotId"])
        snapshot_resource.create_tags(Tags = [
          {"Key": "Name", "Value": volume_name},
          {"Key": "Description", "Value": volume_description},
          {"Key": "Interval", "Value": "Annual"},
        ])
        total_created += 1

  return total_created

You only have so many characters available for an inline Lambda script so I've kept the comments to a minimum. The script is straightforward and the code (for the sake of clarity) doesn't do anything tricky; it should be easy to follow. Basically what it does is check all of the regions we provided in the REGIONS environment variable and then it checks that region for instances. When it finds one it gets a handle on all of it's volumes and then snapshots each one.

First the daily snapshot is created, the script always takes this snapshot
If this is the first day of the first month of the quarter then a quarterly snapshot is taken as well
If it's the last day of the year then we take the annual snapshot

For each snapshot created our script sets the snapshot name to match the volume name and the description of "Created by ebs-snapshot-create" (the name of our Lambda function). It also sets the "Client" and "Project" tags on each snapshot to match the environment variable values.

The script keeps track of how many snapshot it's created and returns that number when the function ends. Along the way we also print data to standard out: the standard out and the return value will all be gathered and logged when the script runs.

Daily Snapshot Retention Job

We don't want to accrue snapshots indefininitely, let's not forget that we get charged for their storage. ;-) This next job will enforce our retention policy.

  TutorialDeleteBackupLambda:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: ebs-snapshots-delete
      Description: Delete Old EBS Snapshots
      Handler: index.handler
      Role: !GetAtt TutorialLambdaBackupRole.Arn
      Environment:
        Variables:
          REGIONS: !Ref AWS::Region
      Runtime: python3.6
      Timeout: 90
      TracingConfig:
        Mode: Active
      Code:
        ZipFile: |
        ...
        code elided for clarity
        ...

It's almost exactly like the last job, aside from it's name and description. The real difference is in the script for the Lambda function. The script calculates the cutoff date for our daily snapshots: seven days old. As the script checks through the snapshots for the volumes it will check the tags that we placed on the snapshots in the backup job, specificaly the "Interval" tag. Daily snapshots older than our cutoff date will be removed leaving only the seven momst recent daily snapshots.

We do the same for the quartlerly snapshots; we calculate the cutoff data and then check through all of the snapshots tagged with an "Interval" value of "Quarterly". All of the snapshots past the cutoff are deleted leaving us with the four most recent quartlery snapshots.

The annual snapshots we leave as is, we will accrue those at the rate of once a year forever. You never know when someone will want to dig up old, historic data.

import os
from datetime import datetime, timedelta
from dateutil import relativedelta
import boto3

def handler(event, context):

  ec2_client = boto3.client("ec2")
  date_format = "%Y-%m-%dT%H:%M:%S.%fZ"
  daily_cutoff = (datetime.utcnow() - timedelta(days = 7)).strftime(date_format)
  env_regions = os.getenv("REGIONS", None)
  today = datetime.today()
  total_deleted = 0

  if env_regions:
    regions = ec2_client.describe_regions(RegionNames = env_regions.split(","))
  else:
    return total_created

  for region in regions.get("Regions", []):
    regionName = region["RegionName"]
    print("Checking for snapshots in region %s" % (regionName))

    # query for Daily snapshots with matching tags
    result = ec2_client.describe_snapshots(Filters = [
      {"Name": "tag:Interval", "Values": ["Daily"]}
    ])

    if not result:
      print("No matching Daily snapshots with tag 'Client' = '%s' and tag 'Project' = '%s' in region %s" % (clientTag, projectTag, regionName))
      return total_deleted

    for snapshot in result["Snapshots"]:
      snapshot_time = snapshot["StartTime"].strftime(date_format)
      snapshot_id = snapshot["SnapshotId"]

      if daily_cutoff > snapshot_time:
        snapshot_name = ""
        if "Tags" in snapshot:
          for tag in snapshot["Tags"]:
              if tag["Key"] == "Name":
                  snapshot_name = tag["Value"]

        print("Deleting Daily EBS snapshot ID = %s, Name = %s" % (snapshot_id, snapshot_name))
        ec2_client.delete_snapshot(SnapshotId = snapshot_id)
        total_deleted += 1

    # query for quarterly snapshots with matching tags
    if today.day == 1 and today.month % 4 == 1:
      quarterly_cutoff_raw = datetime.utcnow() - relativedelta.relativedelta(months = 15) # keep five quarters
      quarterly_cutoff = quarterly_cutoff_raw.strftime(date_format)
      result = ec2_client.describe_snapshots(Filters = [
        {"Name": "tag:Interval", "Values": ["Quarterly"]}
      ])

      if not result:
        print("No matching Quarterly snapshots with tag 'Client' = '%s' and tag 'Project' = '%s' in region %s" % (clientTag, projectTag, regionName))
        return total_deleted

      for snapshot in result["Snapshots"]:
        snapshot_time = snapshot["StartTime"].strftime(date_format)
        snapshot_id = snapshot["SnapshotId"]

        if quarterly_cutoff > snapshot_time:
          snapshot_name = ""
          if "Tags" in snapshot:
            for tag in snapshot["Tags"]:
                if tag["Key"] == "Name":
                    snapshot_name = tag["Value"]

          print("Deleting Quarterly EBS snapshot ID = %s, Name = %s" % (snapshot_id, snapshot_name))
          ec2_client.delete_snapshot(SnapshotId = snapshot_id)
          total_deleted += 1

  return total_deleted

With these two functions added to our CloudFormation template we are nearly ready to go. All we need to do is call these jobs every night.

Rules to Invoke our Jobs

We want to run our jobs every night, first the backup job and then the retention job that culls out old snapshots. Lucky for us CloudWatch will let you create a "rule" that is triggered on a schedule, we can create our own rule that will start our backup jobs. The AWS::Events::Rule resource can be used to do exactly this.

  TutorialPerformBackupRule:
    Type: AWS::Events::Rule
    Properties:
      Description: Triggers our backup lambda script
      ScheduleExpression: cron(30 4 * * ? *)
      State: ENABLED
      Targets:
        - Arn: !GetAtt TutorialCreateBackupLambda.Arn
          Id: TutorialBackupRule

  TutorialDeleteBackupRule:
    Type: AWS::Events::Rule
    Properties:
      Description: Triggers our backup delete lambda script
      ScheduleExpression: cron(45 4 * * ? *)
      State: ENABLED
      Targets:
        - Arn: !GetAtt TutorialDeleteBackupLambda.Arn
          Id: TutorialBackupRule

These two jobs use the familiar cron style syntax, we backup at 4:00AM and then delete old snapshots at 4:45AM every day. Each rule has a pointer to the ARN of the Lambda function it will invokce.

With our jobs all setup, we need to provide our rules with permissions to invoke our Lambda functions with the AWS::Lambda::Permission resource.

  TutorialPermissionCreate:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt TutorialCreateBackupLambda.Arn
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt TutorialPerformBackupRule.Arn

  TutorialPermissionDelete:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt TutorialDeleteBackupLambda.Arn
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt TutorialDeleteBackupRule.Arn

We add the InvokeFunction permission to our two CloudFront triggers so that they are allowed to invoke our Lambda functions. With that our backups are all taken care of.

Template Wrap-Up: Output

The last bit of any CloudFront template is provided output data, this data will be provided as a JSON object. You could provide thiss data to another process that double-checks that the resources have been created or performs additional tasks. Here's our output section...

Outputs:
  WebServer:
    Description: A reference to the web server
    Value: !Ref TutorialWebServer

  DatabaseServer:
    Description: A reference to the database server
    Value: !Ref TutorialDatabaseServer

  BackupS3Bucket:
    Description: S3 Bucket used to backup data
    Value: !Sub |
      https://${TutorialBackupS3Bucket.DomainName}\

Here we return references to our two server instances and our S3 bucket. There are certainly more items we could list in this section, perhaps our reference to our VPC. For now we'll leave it short and to the point.

Whew! That is the last piece in the series! You can model on the template we've put together here and start provisioning resources on CloudFormation with a slightly more rapidity and confidence. ;-D

Provision Programmatically with CloudFormation: Part 2, Finally Some Instances!

Christopher Miles — Wed, 06 Feb 2019 17:37:24 +0000

This is the second part in a three part series where we build up a CloudFormation template for what I think of as a pretty typical environment: one virtual private cloud broken into two subnets and including two instances. If you haven't read the first part in the series, I encourage you to check it out now!

Part 1: we setup of the VPC and subnets

Setup an "Admininstrators" Group

For this project we're creating a general "administrators" group that has access to the backup bucket with our database backups. I generally don't allow anyone direct access to the EC2 console unless they really need it, so I don't address that in this template (in addition, not every EC2 web console action can be managed this way). My thinking here is that we typically give "administrators" or "developers" accounts on the individual instances, they don't really need the EC2 web console in order to get their work done.

  TutorialAdminGroup:
    Type: AWS::IAM::Group
    Properties:
      Policies:
        - PolicyName: TutorialAdminPolicy
          PolicyDocument:
            Statement:
            - Effect: Allow
              Action:
                - s3:ListBucket
                - s3:ListBucketByTags
                - s3:ListBucketMultipartUploads
                - s3:ListBucketVersions
              Resource: !Sub ${TutorialBackupS3Bucket.Arn}
            - Effect: Allow
              Action:
                - s3:PutObject
                - s3:DeleteObject
                - s3:GetObject
              Resource: !Sub ${TutorialBackupS3Bucket.Arn}/*

We use the IAM Group resource to create our new group. The only property that we provide is the Policies property that contains a list of PolicyName and PolicyDocument pairs, in this case we define only one that we have named "TutorialAdminPolicy".

A PolicyDocumentcontains a Statement that holds a list of Effect instances; each of those in turn contains an Action property with a list of permissions. We use the Resource property to tie in a reference to our bucket, in this case the backup bucket's ARN. If we take a look at the first Effect, you'll see that we've assigned four "ListBucket..." permissions for our backup bucket. The second effect applied three more actions to every file in the backup bucket, that's indicated by the /* at the end of the bucket's ARN.

Any account that we add to this group will be able to download or delete any of the files in the bucket (as well as upload). In this case when we say "files" we really mean database dumps for this project. They will also be able to pull up the S3 console for the bucket but they will need a link that is directly to the bucket. They will not be able to log into S3 and browse the list of all buckets.

One last note: since we are creating an IAM role with our template, we need to let CloudFormation know that this is okay. From here on out we need to add the --capabilities flag to our aws commands.

aws cloudformation create-stack \
  --stack-name tutorial \
  --template-body file://template.yaml \
  --parameters ParameterKey=KeyPairName,ParameterValue=cloudfront-tutorial \
  --tags Key=Project,Value=cf-tutorial \
  --capabilities CAPABILITY_IAM

If you don't include the tag, CloudFormation will exit with an error. The same flag can be used with "update" commands.

Setup a Role for Our Instances

Next we need to setup the role that our instances can assume to get access to resources (for now, just the backup bucket).

  TutorialInstanceRole:
    Type: AWS::IAM::Role
    Properties:
      ManagedPolicyArns:
        - arn:aws-us-gov:iam::aws:policy/CloudWatchAgentServerPolicy
        - arn:aws-us-gov:iam::aws:policy/service-role/AmazonEC2RoleforSSM
        - arn:aws-us-gov:iam::aws:policy/AmazonSSMReadOnlyAccess
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: sts:AssumeRole

We use the Role resource to create our new instance role and we assign three canned policies...

the first lets the instance run the CloudWatch agent and submit events to the CloudWatch service
the next lets the instance interact with Systems Manager, letting us manage the instances as a group
The last provides read-only access to the Systems Manager parameter store.

Amazon's CloudWatch service will aggregate data submitted by your instances and let you setup reports, dashboards and alerts based on this data (i.e. when CPU usage gets too high or available disk space is too low). Systems Manager provides some tools for managing your instances, like installing patches or a software package. It lets you perform these actions on several instances as a group which can be handy. I'm not going to go over these services in-depth here but I encourage you to spend some time checking them out if you haven't done so already.

  TutorialInstanceRolePolicy:
    Type: AWS::IAM::Policy
    Properties:
      PolicyName: TutorialInstanceRolePolicy
      Roles:
        - Ref: TutorialInstanceRole
      PolicyDocument:
        Statement:
          - Effect: Allow
            Action:
              - s3:ListBucket
              - s3:ListBucketByTags
              - s3:ListBucketMultipartUploads
              - s3:ListBucketVersions
            Resource: !Sub ${TutorialBackupS3Bucket.Arn}
          - Effect: Allow
            Action:
              - s3:PutObject
              - s3:DeleteObject
              - s3:GetObject
            Resource: !Sub ${TutorialBackupS3Bucket.Arn}/*

We are creating a new Policy resource that we will assign to our instances. We link the role we just created and then add a new policy that provides access to the backup bucket. As you can see, it is exactly the same as the role that we created for our administrators group.

The last piece of this puzzle is an "instance profile" that we can assign to our instances.

  TutorialInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Roles:
        - !Ref TutorialInstanceRole

With all of the other work done, there's not much to see here. We create a new InstanceProfile resource and link it to our role. When we provision instances we can assign them this profile, they will then have the ability to write to our backup bucket, submit performance and event data to CloudWatch and accept remote commands from System Manager.

Provision CloudWatch Log Groups

The very last thing we need before provisioning are a couple CloudWatch log groups. We can instruct our instances to ship some of their log files out to these groups and then view the logs through the CloudWatch console.

  TutorialBootLog:
    Type: AWS::Logs::LogGroup

  TutorialKernelLog:
    Type: AWS::Logs::LogGroup

We use the LogGroup resource to create two new log groups. These groups will be names after the name we provide in the template with some random characters tacked on at the end. We're creating the groups now because we're going to reference them directly when we provision the instances.

Provision the Database Server

I know what you are thinking: all that work and we're only now provisioning our instances! It's true, there was a lot of preparation and things to think about but we are well on our way to having a template that will provision our servers in a repeatable and reasonably safe manner. Just think of all the templates you will be writing!

We're going to provision the database server first. This one is a little bit involved so I'm going to break it up into pieces. First we provision a new instance with the Instance resource...

  TutorialDatabaseServer:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          files:
            /etc/profile.d/cloudformation-init.sh:
              content: !Sub |
                export BACKUP_S3_BUCKET="${TutorialBackupS3Bucket}"

Note that in the code stanza above I stop before the properties, we'll go over the properties next.

The first thing we do is setup the "metadata" for the CloudFormation initialization script (we will call that at the very end) with the CouldFormation Init type. This is were we tell the initialization script what we would like it to do when it runs, in this case we add a new environment variable to a new file in /etc/profile.d called cloudformation-init.sh. Whenever someone logs into the machine this script (along with everything else in the directory) will be evaluated, the end result will be that the name of our backup bucket will be available through the BACKUP_S3_BUCKET environment variable. Keep in mind that this metadata doesn't do anything on it's own: the initialization script will use it when it runs and we'll do that at the end of our instance definition.

We also need to provide the configuration file for the CloudWatch agent. This agent will collect performance data and log files and ship them back to the CloudWatch service.

            /etc/amazon/amazon-cloudwatch-agent.json:
              content: !Sub |
                {
                  "metrics": {
                    "append_dimensions": {
                      "AutoScalingGroupName": "${!aws:AutoScalingGroupName}",
                      "ImageId": "${!aws:ImageId}",
                      "InstanceId": "${!aws:InstanceId}",
                      "InstanceType": "${!aws:InstanceType}"
                    },
                    "metrics_collected": {
                      "cpu": {
                        "measurement": [
                          "cpu_usage_idle",
                          "cpu_usage_iowait",
                          "cpu_usage_user",
                          "cpu_usage_system"
                        ],
                        "metrics_collection_interval": 30,
                        "totalcpu": false
                      },
                      "disk": {
                        "measurement": [
                          "used_percent",
                          "inodes_free"
                        ],
                        "metrics_collection_interval": 30,
                        "resources": [
                          "*"
                        ]
                      },
                      "diskio": {
                        "measurement": [
                          "io_time"
                        ],
                        "metrics_collection_interval": 30,
                        "resources": [
                          "*"
                        ]
                      },
                      "mem": {
                        "measurement": [
                          "mem_used_percent"
                        ],
                        "metrics_collection_interval": 30
                      },
                      "statsd": {
                        "metrics_aggregation_interval": 30,
                        "metrics_collection_interval": 10,
                        "service_address": ":8125"
                      },
                      "swap": {
                        "measurement": [
                          "swap_used_percent"
                        ],
                        "metrics_collection_interval": 30
                      }
                    }
                  },
                  "logs": {
                    "logs_collected": {
                      "files": {
                        "collect_list": [
                          {
                            "log_group_name": "${TutorialBootLog}",
                            "file_path": "/var/log/boot.log"
                          },
                          {
                            "log_group_name": "${TutorialKernelLog}",
                            "file_path": "/var/log/messages"
                          }
                        ]
                      }
                    }
                  }
                }

This configuration file tells the CloudWatch agent to ship back data on CPU, disk, memory and swap file usage back to CloudWatch as well as two log files: the boot and kernel logs.

Next we'll set the properties for our instance...

    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyPairName
      NetworkInterfaces:
        - SubnetId: !Ref TutorialPrivateSubnet
          DeviceIndex: 0
          GroupSet:
            - !Ref TutorialPrivateSecurityGroup
      BlockDeviceMappings:
        - DeviceName: !Sub "/dev/${RootDevice}"
          Ebs:
            VolumeSize: !Ref VolumeSize
            VolumeType: gp2
      IamInstanceProfile: !Ref TutorialInstanceProfile
      Tags:
        - Key: "Name"
          Value: "Tutorial Database Server"

Way back in part 1 we create these parameters and provided default values for them, now they are finally coming into play. If you look back at the top of the template, you'll see that we set parameters for the image, instance type, key pair name and root volume. While our template requires that a key pair name be provided the other values are populated with default values. My expectation is that the default values will normally be used, most of the reason they are at the top in the parameter section is to make them easy to get at.

The first thing we do is choose the image we'd like for this instance, in the code above we have a reference to the parameter, the default value is for the Amazon Linux 2 image. There are other Linux images out there, I picked this one mostly because it already has the AWS tools and CloudFormation stuff installed, making things that much easier. Also kind of interesting, if you develop locally you can work with an Amazon Linux 2 Docker Container on your workstation.

Next we use our instance type parameter, the default values is for t2.micro because this is a tutorial and I don't want to cost you money; this type is eligible for use under Amazon's free tier. If you haven't used up your 750 hours of free tier usage for the month you shouldn't be charged for provisioning these instances. Setting cost aside, a micro instance is likely enough to host a low traffic website, like your personal blog.

We set the name of the key pair we want to use to provision our instance, note that the value we reference in the KeyName property is the one parameter we set for this script back in part 1. You will need to have this key pair handy in order to SSH to the instance.

We'd like the database server to be on our private subnet and we take care of that when we specify the NetworkInterfaces property. Here we pass in a reference to our private subnet, we then set the security group to our "private" security group with the GroupSet property of the network interface.

Every instance needs disk space and we can customize the Elastic Block Store (EBS) volumes for our instance with the BlockDeviceMappings property. We map in one volume of 250GB (referencing our parameter) to the root device by using the substitution function with our parameter to set the value to /dev/xvda, we chose the "general purpose" (gp2) volume type.

The default volume type is "gp2" and it's a reasonable choice, more information about the various types may be found in the EBS volume type documentation. Also note that the root device that the instance boots from may vary from one Linux distribution to the other. For instance, under Ubuntu the root volume needs to be mapped to /dev/sda1; if the instance can't find the volume you will see it start up in the EC2 console but it will stop in just a couple of minutes.

Instead of managing credentials for our EC2 instances we thoughtfully created a profile! We set the profile for our instance with the IamInstanceProfile property and pass in a reference to that profile.

Lastly we set tags on our instance, in this case we set the "Name" tag.

With that out of the way the only thing left to do is to invoke the CloudFormation initialization script when the instance starts up. To get this done we write a little script that calls cfn-init at start time.

      UserData: 
        Fn::Base64: 
          !Sub |
            #!/bin/bash -xe

            # cloudformation initialize
            /opt/aws/bin/cfn-init -v -s ${AWS::StackName} --region ${AWS::Region} -r TutorialDatabaseServer

            # download and install cloudwatch agent
            wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
            yum -y install amazon-cloudwatch-agent.rpm
            mv /etc/amazon/amazon-cloudwatch-agent.json /etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json
            sudo systemctl enable amazon-cloudwatch-agent
            sudo systemctl start amazon-cloudwatch-agent

Documentation for cfn-init is available on the CloudFormation site. You can see in the example above that we call the script with the name of our stack, the stack's deployment region and the name of our server. When the script runs it will inspect the Metadata property that we set at the the beginning of our instance stanza and will carry out those tasks. For this instance, the initialization script will add that new file to /etc/profile.d with our custom environment variables.

The cfn-init script handles getting our configuration files written to disk, the other thing we need to do is get the CloudWatch agent installed and running. The next bit of script downloads the current version, installs it, moves our configuration file into place and then enables and starts the service so that the agent is started at boot time.

And that's it... For our database server. We need to do pretty much the same thing for our web server.

  TutorialWebServer:
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        config:
          files:
            /etc/profile.d/cloudformation-init.sh:
              content: !Sub |
                export BACKUP_S3_BUCKET="${TutorialBackupS3Bucket}"
                export DATABASE_SERVER="${TutorialDatabaseServer.PrivateIp}"

Mostly everything is exactly the same but there are a couple small differences. When we setup the Metadata for the initialization script we added another environment variable that contains the private IP address of the database server, in this way we can deploy this template more than once and the web server in each stack will know where to find it's matching database server.

I don't include it in this article, but we provide the exact same configuration file for the CloudWatch agent that we used for the web server. If you are writing the template along with this article, take a minute to copy and paste that stanza into the file now.

    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      KeyName: !Ref KeyPairName
      NetworkInterfaces:
        - SubnetId: !Ref TutorialPublicSubnet
          DeviceIndex: 0
          GroupSet:
            - !Ref TutorialPublicSecurityGroup
      BlockDeviceMappings:
        - DeviceName: !Sub "/dev/${RootDevice}"
          Ebs:
            VolumeSize: !Ref VolumeSize
            VolumeType: gp2
      IamInstanceProfile: !Ref TutorialInstanceProfile
      Tags:
        - Key: "Name"
          Value: "Tutorial Web Server"
      UserData: 
        Fn::Base64: 
          !Sub |
            #!/bin/bash -xe

            # cloudformation initialize
            /opt/aws/bin/cfn-init -v -s ${AWS::StackName} --region ${AWS::Region} -r TutorialDatabaseServer

            # download and install cloudwatch agent
            wget https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
            yum -y install amazon-cloudwatch-agent.rpm
            mv /etc/amazon/amazon-cloudwatch-agent.json /etc/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.json
            sudo systemctl enable amazon-cloudwatch-agent
            sudo systemctl start amazon-cloudwatch-agent

In terms of the properties, the only difference is that we have placed our web server in the public subnet so that it can communicate with the public internet (as web servers so often need to do).

At this point we have covered the first four goals that we laid out for ourselves in part 1. With the template as it stands right now, we can...

Provision resources for a project
Retire resources for a project
Isolate resources for a project
Report on resources by project

I think this is a pretty significant milestone! From here we can tweak the template to match a particular project and with just one command setup the environment. If we have a client that wants a "test" and a "production" environment, all we need to do is provision the template twice.

Reporting on Resources

This is a little outside the realm of CloudFormation, but I thought we'd take a short break from the template and take a look at some ways we can use the tags we have so diligently been placing on our resources. Take a look at the billing console for your account. You will be dropped at the dashboard and some summary data for your account will be visible.

If you haven't done so already, click on the "Cost Explorer" link from the left-hand navigation bar and enable it for your account. It's a handy tool that lets you do some simple querying and reporting on your resources and their costs.

Next click on the "Cost Allocation Tags" link, also on the left-hand navigation bar. At the top you can see that there are some cost allocation tags that AWS can generate on it's own, go ahead and click the "Activate" button to turn those on. Lower down on the page will be a list of all of the tags that we have defined (as well as any tags you had already setup). You can choose which tags you want to make available in the Cost Explorer but my advice is just to click the top-most checkbox and make them all active. You never know when a tag will come in handy! Click the "Activate" button and confirm that, yes, you want the tags to be "active".

Now click back to the "Cost Explorer" link and click on "Launch Cost Explorer". If you just activated the cost explorer now, you will probably have to wait until tomorrow; keep it in mind and try to come back to this section.

If the Cost Explorer has data on hand then you will be presented with another dashboard attempting to summarize your costs on amazon. Click on the magnifying glass icon on the left-hand navigation bar and choose "Cost and Usage", the report builder will appear with your last six months of spending. Click on the "Last 6 Months" pull-down and choose "1M" from the "Historical" section at the bottom, this will show your last month of spending.

On the right-hand side is a list of filters and this is where the tags will come in handy. Click on the "Tag" link in the filter list, a list of your tags will appear; click on "Project" and a list of all your values for the "Project" tag will be listed. In this tutorial we've been putting "cf-tutorial" in the "Project" tags, check the box for "cf-tutorial" and then press the "Apply filters" button to update the report.

What you are looking at now is likely a very dull report, because we've been using low-cost free-tier instances. But, still, what we have is a report on just the resources that belong to this project. If you were to place a "Client" tag in your templates you could report on the entire cost of a client (maybe you could use that to figure out how to bill) and you can break down a client's costs by particular projects. It is a valuable tool and definitely worth putting some time in exploring your options.

Set Alarms on Instance Usage

I think of this as related to our reporting goals: we'd like to set some alarms that monitor the data we're sending to CloudWatch to let us know if our servers start to go off the rails. We'll set alarms for when CPU usage gets high or if we start to run out of disk space but I bet you can think of lots of other things you'd like to monitor.

We're going to use the Simple Notification Service to alert us when an alarm is hit. The way this will work is that when CloudWatch sees an alarm trigger, it will send a message to the SNS "topic". It will be our responsibility to log into the SNS console and add ourselves to the notification list (by email or text message), we aren't going to populate the list of subscribers.

  TutorialAlarmTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: TutorialAlarms

We use the Topic resource to create our new alarm target topic. With the SNS topic created we can now create our first alarm.

  TutorialDatabaseCPUAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: Database Server CPU Usage High
      AlarmActions:
        - !Ref TutorialAlarmTopic
      MetricName: CPUUtilization
      Namespace: AWS/EC2
      Statistic: Average
      Period: 60
      EvaluationPeriods: 3
      Threshold: 89
      ComparisonOperator: GreaterThanThreshold
      Dimensions:
        - Name: InstanceId
          Value: !Ref TutorialDatabaseServer

With the Alarm resource we create an alarm for CPU usage on the database server, if the server is using more than 89% of the CPU for more than 3 minutes then the alarm will trigger and send a message to our SNS topic.

  TutorialDatabaseDiskAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: Database Server Disk Usage High
      AlarmActions:
        - !Ref TutorialAlarmTopic
      MetricName: disk_used_percent
      Namespace: CWAgent
      Statistic: Average
      Period: 60
      EvaluationPeriods: 3
      Threshold: 89
      ComparisonOperator: GreaterThanThreshold
      Dimensions:
        - Name: InstanceId
          Value: !Ref TutorialDatabaseServer
        - Name: ImageId
          Value: !Ref ImageId
        - Name: InstanceType
          Value: !Ref InstanceType
        - Name: path
          Value: /
        - Name: device
          Value: !Sub ${RootDevice}1
        - Name: fstype
          Value: !Ref RootFsType

This next alarm is for the amount of disk used by the database server, here we set an alarm to trigger if more than 89% of the disk is in use. This one will also wait for the disk to be in this state for 3 minutes before triggering the alarm.

Provisioning this alarm is a bit more work, you can see that we had to add several Dimensions to get the alarm setup. This is because when the CloudWatch agent reports this data it references all of these dimensions and we need to match them all in order to have a working alarm. If we had left one of these out then the alarm wouldn't match any data and would never trigger.

The only bit left is the backup jobs but we'll leave that for part 3. ;-) Congratulations on working your way through all of this material! While it can be pretty dry reading you can really get a lot of mileage building templates for projects that you know you will be deploying over and over.

Part 3: Backup Jobs and Wrap-Up

Provision Pragmatically and Programmatically with CloudFormation, Part 1

Christopher Miles — Fri, 25 Jan 2019 18:37:18 +0000

How many times have you gazed upon the list of services that Amazon is providing through their Amazon Web Services (AWS) family of products and felt an unpleasant mixture of anxiety and confusion? If you're anything like me, it would be literally every time you visited their website. While I was one of the first in my circle of software developer friends to get on board the Elastic Compute Cloud (EC2) train, I haven't been great at keeping up with new developments. This year, I decided, would be the year that I get back on track and sort out this alphabet soup of cloud virtualization technology.

After clicking around at random and reading an unwholesome amount of bland introductory pages, I finally started to get an idea of how I should try to put all of these pieces together. The first thing I figured out:

It is way, way too hard to get everything up and running by clicking a bunch of buttons.

What are your goals? For me, it's to be able to deploy a web-based application for one of my clients. As a corollary, I want to be able to entirely scrap a web-based application for one of my clients (maybe it's been replaced, maybe they've hired another consultant). In effect, that's two goals. Plus I would like to keep all of my clients separate, I don't want changes to a firewall rule to effect all of my clients, just the one client who's willing to take the risk. If you think about it, keeping things isolated will also make it easier to figure out who we can charge for what services at the end of the month; we should also tag these resources in a way that makes sense. That's four goals and that's only what we've thought of so far. And we should snapshot on a daily basis and maybe not keep all of the snapshots. Oh, backing up the database is important, too!

Provision resources for a project
Retire resources for a project
Isolate resources for a project
Report on resources by project
Snapshot instance volumes and delete old backups on a schedule
Backup the databases on a schedule and delete old backups on a schedule

Looking at these goals, the number of services, the way many of them build on each other... It all becomes clear that the answer is automation. Of course, Amazon provides an automation tool: CloudFormation.

This is a first in a series of articles that will cover all of these goals. We'll start out with an empty file and we'll gradually build that file up, adding more and more resources to that file as we go along. I am planning three articles all together, this on covers a brief introduction to the tools and gets as far as setting up a virtual private cloud for the project, splitting it into two networks and putting security groups in place. So exciting!

I encourage you to follow along as we build up the template but you can always checkout the finished product over at GitHub. After slogging through this article you can follow along with second and third parts of this series. At then end of the series we'll have one template that literally does it all.

Install the AWS Command Line Tools

Amazon provides a goofy drag-and-drop interface for CloudFormation but that runs kind of counter to the whole point of this article. You can also write your CloudFormation template and then upload it via their web interface, I think that's also a little clunky. My recommendation is that you install the AWS Command Line Tools and use those tools to deploy. Amazon also provides packages for Windows, but if you're working from within the Windows environment you might be more interested in the AWS Tools for PowerShell which do much the same thing but, you know, with PowerShell.

This is kind of a power tip, but if you're managing more than one Amazon account or dealing with more than one region, you can configure profiles to make that process easier.

With one of these tools installed and properly configured, you can invoke CloudFormation directly from your console window! :-)

Start Our New CloudFormation Template

In the world of CloudFormation, the document you are writing is called a "template". I believe it's referred to as a template because one template can be used to create many different sets of resources. For instance in this tutorial we're going to be writing a template that will provision a web server and a database server; we can use the template to provision as many of these web-and-database-server pairs as we might like, all independent of one another.

You can write your template in YAML or JSON, if you're writing it by hand (as we are right now) then I strongly recommend that you write it in YAML. I know, I know, YAML isn't everyone's favorite but getting all of the ending braces and brackets lined up in a JSON document is it's own kind of mental torture. You deserve better, trust me: don't do that to yourself.

Anyway, we're going to start off with a description and a parameter. A parameter, as you may have suspected, is a value that we can supply when we are actually using our template to provision something. Open up a new file in your favorite editor, type in the text below and save it as template.yaml.

Description: Provisions a simple cluster with a web and a database server

Parameters:
  KeyPairName:
    Type: String
    Description: Name of keypair used when provisioning instances

Throughout this article we'll continue to build up this template but if you need to jump ahead or need to refer to the entire thing, you can browse through it in the GitHub project.

Some decisions are best postponed until the absolute last minute and parameters are here when you have decisions to postpone. When we decide to provision our resources, we can provide the name of the EC2 key pair to use at that time.

I'm not going to go into a big discussion about how to manage your EC2 key pairs. Maybe you have one per person who has access to provision instances. Maybe you have one per client. However you manage it, you can pass it in when you provision resources.

Next we're going to add some more parameters but we'll be providing default values for all of them. The thinking here is two-fold: first, we want the flexibility to be able to change some of the features at provisioning time. Second, we can reference these parameters throughout the template and this keeps us from sprinkling these identical values throughout the template.

  InstanceType:
    Type: String
    Description: Instance type to provision
    Default: t2.micro
  ImageId:
    Type: String
    Description: Image to use when provisioning instances
    Default: ami-035be7bafff33b6b6  # Amazon Linux 2
  RootDevice:
    Type: String
    Description: Name of the root device on provisioned instances
    Default: xvda  # varies based on Linux type (Ubunut /dev/sda1)
  RootFsType:
    Type: String
    Description: File system type for root device
    Default: xfs

You can see that these all relate to the instances that we will be provisioning. We call out the instance type, the image we'll use to create the instances and the name of the root volume and it's file type. Note that the root volume and file type aren't something we can really choose, they are dictated by the image (in this case, Amazon Linux 2).

Provision a New Virtual Private Cloud

We need at least one resource in order to provision our template. For this project we're going to create our own Virtual Private Cloud (VPC) to hold our resources. This will keep things isolated from everything else you might have deployed to AWS.

Resources:

  TutorialVPC:                   # name of the resource
    Type: AWS::EC2::VPC          # type of resource
    Properties:                  # properties of the resource
      CidrBlock: "10.6.0.0/24"
      EnableDnsSupport: true
      EnableDnsHostnames: true
      Tags:
        - Key: "Name"
          Value: "Tutorial VPC"

The Resources section of the template will contain a stanza for each of the resources we provision, this will be the bulk of the template. Each stanza in this section will start with the name of the resource (TutorialVPC), we can use this name when we need to reference the resource while creating other resources (for instance, when we add subnets to this VPC). After the name we need to declare the Type of resource: every CloudFormation resource has a type, they are all clearly documented in the CloudFormation API Documentation. In the example above we are creating a new VPC instance. Next we set the Properties for our resource, the values provided here will be particular to each resource. Most resources accept a Tags property but not all, if you set Tags on a resource that doesn't support that property CloudFormation will throw an error.

Every VPC needs a block of IP addresses, we set this with the CidrBlock property and provide a non-routable network with 254 available addresses. I chose a network that won't conflict with the address range I use at home or at work, this is important if you ever want to use a Virtual Private Network (VPN) to connect your Amazon VPC to your own network, perhaps to make access to these resources easier.

We'd like Amazon to continue to assign names to our instances, so we set the EnableDnsSupport and EnableDnsHostnames properties to true. Lastly we set some tags on our resources to remind ourselves why we provisioned them in the first place.

Provision Resources With Our Template

With our template set, we're ready to use it to provision some resources! When we provide the template to CloudFormation, we also have to provide our parameter values and we have the opportunity to tag all of the provisioned resources (maybe by client or project or both).

Go ahead and run the command below to begin provisioning. When CloudFormation sets up all of the resources it links them together into a "stack". Once complete, you can deal with the stack as one unit. The stack name provides a human-friendly handle for all of the related resources.

aws cloudformation create-stack \
  --stack-name tutorial \
  --template-body file://template.yaml \
  --parameters ParameterKey=KeyPairName,ParameterValue=cloudfront-tutorial \
  --tags Key=Project,Value=cf-tutorial

Remember to provide your own key pair name where we have used cloudfront-tutorial.

As soon as CloudFormation receives the template it will return a new StackId with the Amazon Resource Name (ARN) for the stack, you will have to wait a bit for the stack to finish being provisioned. You can check on your stack through the CloudFormation web console and see how things are going. Some resources take longer than others, CloudFormation will ensure they are created in the right order and work out the dependencies. This template should move along pretty quickly as we aren't provisioning much.

I broke the command over several lines to try and make it easier to follow. In the create-stack command above we...

Ask the AWS tools to call out to CloudFormation with the "create-stack" command
Set the name of the stack we are creating to "tutorial"
Provide our template to CloudFormation with the file:// URL
We set the one template parameter, KeyPairName (note the weird way we have to set parameters)
We provided a one tag key and value that CloudFormation will use to tag all of the provisioned resources

You may separate multiple parameters with a space, multiple tags are separated with a comma. It's weird, I don't know why they aren't uniform.

When provisioning is complete, you can head over to the VPC web console to inspect your new virtual cloud. There's not a lot to see, but if you choose "Yourq VPCs" from the left-hand navigation bar and then select your "Tutorial VPC" from the list, you can click the "Tags" tab and look at the tags associated with the resource. You'll see the tag we specified in the template (Name) and the tag we passed into the create-stack command (Project). CloudFormation also set several tags of it's own, linking the resource to the stack.

As we work through the template we might update the stack or delete it entirely. The command to update the stack is almost exactly the same.

aws cloudformation update-stack \
  --stack-name tutorial \
  --template-body file://template.yaml \
  --parameters ParameterKey=KeyPairName,ParameterValue=cloudfront-tutorial \
  --tags Key=Project,Value=cf-tutorial

If CloudFormation runs into any problems while creating or updating your stack, it will roll back to the last valid state of the stack. If this happens during the creation of the stack you'll end up with no resources at all, if it happens during an update then it will be as if your update attempt never occurred.

Deleting the stack is quite a bit shorter.

aws cloudformation delete-stack --stack-name tutorial

Go ahead and run that command now to delete the stack. The AWS tool command won't return any results but if you switch to the CloudFormation console you might catch a glimpse of the stack before it's entirely deleted. If you switch over to the VPC console you'll see the new VPC is gone, it was deleted along with the stack.

In my opinion, this is a really nice way to work. You can edit your template and work on getting all of the bits and pieces of your stack created and linked together in a way that makes sense (provisioned in the VPC, in the correct subnet, with the correct security group and policies, etc.) When you feel pretty good about your work you can go ahead and provision. If there's a problem or something isn't lining up the way it should you can simply delete the stack and continue to work on your template. For me, this is far more convenient then clicking through the various web consoles and trying to link everything up by filling out fields or applying actions to all of the various pieces.

Retaining Resources through Stack Updates and Even Deletion

As much as CloudFormation can, it will try to preserve resources between updates. On the other hand there are some changes that can't be made after provisioning: in those cases CloudFormation will delete the resource and create a replacement. For many things this doesn't really matter, but for things like instances or Elastic IP addresses you may really need to retain those resources. The CloudFormation Documentation is pretty clear on what resource updates can be performed without "service interruption", keep your eyes open for those warnings.

You can also set a DeletionPolicy for the resources in your template. This lets you signal to CloudFormation that, for certain resources, you would like to keep the resource (even if a replacement is required for provisioning) or (if the resource supports it) perform a "snapshot" backup before deleting the resource (i.e., EC2 volumes or RDS instances). That said, even with the deletion policy set for your critical resources, you should think hard about updating them after provisioning the stack.

My recommendation is that you leave the deletion policies out of your template while you are working on it. Wait until you feel pretty good about the template, maybe until the point where you're ready to deploy your application or project. When you reach that point, go back and set the deletion policy to Retain on those things that are truly critical (volumes with data on them, instances you've customized heavily, etc.) and then update your stack.

Now that we have the basic ideas and the machinery down, we can move a little more quickly.

Provision an Internet Gateway

For this project We're going to divide our virtual private cloud into two subnets: one will be able to communicate with the public internet and will house our web server. The other subnet will be able to talk to the first but will not face the internet, it will house our database server. In this way we can ensure that machines on the public internet cannot communicate directly with our database server.

In order to get our external subnet facing the internet, we need to provision an Internet Gateway and then setup routes so that instances in our virtual cloud can send and receive traffic over the public internet.

  TutorialInternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: "Name"
          Value: "Internet Gateway"

There isn't a lot to configure for the InternetGateway resource. We give it a name and add a tag. Next we need to attach it to our virtual cloud.

  TutorialVPCGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref TutorialVPC
      InternetGatewayId: !Ref TutorialInternetGateway

There's not a lot to configuring the VPCGatewayAttachment either. The interesting bit is where we tell it which VPC to which we'd like it attached. We use the !Ref function to indicate that we are providing a reference to another resource in the template and then we provide the name of that resource.

Setup Our Subnets

With that internet stuff out of the way, we can create our two subnets. First, our public subnet.

  TutorialPublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref TutorialVPC
      CidrBlock: "10.6.0.0/25"
      MapPublicIpOnLaunch: true
      Tags:
        - Key: "Name"
          Value: "Public Subnet"

We use the Subnet resource to split off a chunk of our VPC with half of our address space, we use the VpcId property to link this subnet declaration to our VPC. We set the MapPublicIpOnLaunch property to true so that each instance gets a public IP when launched.

  TutorialPrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref TutorialVPC
      CidrBlock: "10.6.0.128/25"
      MapPublicIpOnLaunch: false
      Tags:
        - Key: "Name"
          Value: "Private Subnet"

In the stanza above, we take the rest of our address space and allocate that to our private subnet, again we use the VpcId property to link this subnet declaration to our VPC. We don't want instances in this network to have public IP addresses and we indicate that by setting the MapPublicIpOnLaunch property to false.

Setup Our Routing

We always have a default route that lets the addresses in our VPC communicate with each other but we don't start out with any routes out to our internet gateway and the public internet. In order to setup our routes we first need to setup our routing tables.

  TutorialPublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref TutorialVPC
      Tags:
        - Key: "Name"
          Value: "Public Internet Route Table"

  TutorialPrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref TutorialVPC
      Tags:
        - Key: "Name"
          Value: "Private Route Table"

There's not much to the RouteTable resource, we indicate that we're going to be managing routes for our VPC with the VpcId property and set a tag.

Next we're going setup the actual routes.

  TutorialInternetRoute:
    Type: AWS::EC2::Route
    DependsOn: VPCGatewayAttachment
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref TutorialInternetGateway
      RouteTableId: !Ref TutorialPublicRouteTable

The Route resource adds a new route to a routing table. Here we add a route to our public table to route traffic for addresses on the public internet through our internet gateway. We let CloudFormation know that this resource shouldn't be created unless the VPCGatewayAttachment is already setup with the DependsOn attribute. Like the Retain attribute, you can place this on any resource.

  TutorialPublicSubnetPublicRoute:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref TutorialPublicRouteTable
      SubnetId: !Ref TutorialPublicSubnet

With our route to the public internet setup, we can use the SubnetRouteTableAssociation resource to link our routing table to our public subnet.

Next we need to do something similar for our private subnet. The big difference here is that our private subnet doesn't connect directly to the internet. Instead we will use a Network Address Translation (NAT) gateway to let instances on our private subnet connect to the internet without allowing machines on the public internet to initiate connections with instances in our private subnet.

  TutorialNatGatewaySubnetRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref TutorialVPC
      Tags:
        - Key: "Name"
          Value: "NAT Gateway Subnet Route Table"

We create a new routing table and associate it to our VPC.

  TutorialNatGatewayInternetRoute:
    Type: AWS::EC2::Route
    DependsOn: TutorialVPCGatewayAttachment
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref TutorialInternetGateway
      RouteTableId: !Ref TutorialNatGatewaySubnetRouteTable

And then we create a route to the public internet through our internet gateway.

The next step is to create our NAT gateway, but first we need to get a publicly accessible IP address that will be assigned to our NAT gateway. We can use the EIP (Elastic IP) Resource to get a handle on a new Elastic IP address.

  TutorialNatGatewayElasticIP:
    Type: AWS::EC2::EIP
    Properties:
      Domain: vpc

We set the domain value to vpc, indicating that we want to use this address from our VPC. Kind of funny, that's the only valid value for that property.

Now we can setup our NAT gateway...

  TutorialNatGateway:
    Type: AWS::EC2::NatGateway
    DependsOn: VPCGatewayAttachment
    Properties:
      AllocationId: !Sub ${TutorialNatGatewayElasticIP.AllocationId}
      SubnetId: !Ref TutorialPublicSubnet
      Tags:
        - Key: "Name"
          Value: "NAT Gateway"

We use the NatGateway resource to create our new NAT gateway and indicate that we need our VPCGatewayAttachment resource provisioned before this is created. We set the SubnetId property to put it in our public subnet so that it can communicate with the internet.

While it would make a lot of sense to provide the Elastic IP we provisioned directly to the gateway, we instead need to provide the AllocationId associated with that Elastic IP instead. Here we use the !Sub function to get the AllocationId of our NatGatewayElasticIP and provide that value to the AllocationId property.

Now we can setup our route for the private subnet to get to the internet via our NAT gateway.

  TutorialPrivateSubnetNatRoute:
    Type: AWS::EC2::Route
    DependsOn: TutorialNatGateway
    Properties:
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref TutorialNatGateway
      RouteTableId: !Ref TutorialPrivateRouteTable

We create our route, using the DependsOn attribute to indicate that we need the NatGateway provisioned before this route is created. Then we use the NatGateway property to link in our gateway and the RouteTableId to tie the route to our routing table.

  TutorialPublicSubnetRouteAssoc:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref TutorialPublicRouteTable
      SubnetId: !Ref TutorialPublicSubnet

  TutorialPrivateSubnetRouteAssoc:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref TutorialPrivateRouteTable
      SubnetId: !Ref TutorialPrivateSubnet

The final step is to associate our routing table with our subnets! We use the SubnetRouteTableAssociation to attach our table of routes to the matching subnet.

Setup Our Security Groups

Before we can provision our instances we need to get our security groups set. We will have one for each subnet and these groups will specify what traffic we will allow in and out of each of our subnets. We will allow web traffic in and out of our public subnets, in our private subnet we're only going to allow web traffic out and nothing inbound.

  TutorialPublicSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Description: "Web Server Security Group"
    Properties:
      VpcId: !Ref TutorialVPC
      GroupDescription: Web Server Security Group
      SecurityGroupIngress:
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 80              # HTTP 
          ToPort: 80
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 443             # HTTPS
          ToPort: 443
        - CidrIp: 0.0.0.0/0         # Replace with your IP Range
          IpProtocol: tcp
          FromPort: 22              # SSH
          ToPort: 22
        - CidrIp: 10.6.0.0/24       # Inside the VPC
          IpProtocol: -1            # All traffic

We use the SecurityGroup resource to create our new group for the public subnet. The Description we provide is visible in the web console and then we move on to the properties. We use the VpcId property to indicate that this group is for our VPC and then provide another, very similar description, for the GroupDescription (this property is required).

With that out of the way, we move on to the in-bound (ingress) rules. Each rule is provided by a Security Group Rule property that is made up of an IP address range, the protocol and then a range of ports. We have created rules for HTTP, HTTPS, and SSH and traffic inside of our VPC. If you were provisioning a Windows instance, don't forget to add port 3389 for Remote Desktop access.

In the example above we have opened up the SSH port to the entire internet: this is not a great idea. Since these ports are used almost exclusively for management of the instances, a better choice would be to open these ports up to the address range that is used by your office. If you don't have a fixed range of addresses you can use then I encourage you to look into other solutions, perhaps using a VPN.

      SecurityGroupEgress:
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 80              # HTTP
          ToPort: 80
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 443             # HTTPS
          ToPort: 443
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: 587             # SSL SMTP
          ToPort: 587
        - CidrIp: 10.6.0.0/24       # Inside the VPC
          IpProtocol: -1            # All traffic
      Tags:
        - Key: "Name"
          Value: "Public Subnet"

Here we setup the rules for outbound traffic: HTTP, HTTPS, and SMTP over SSL. We also have a rule that allows all traffic inside our VPC. If you are more security conscious, you can nail down specific types of traffic inside your VPC (i.e., just web, SSH and database traffic).

The security group for our private subnet is much simpler.

  TutorialPrivateSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Description: "Private Security Group"
    Properties:
      VpcId: !Ref TutorialVPC
      GroupDescription: Private Subnet Security Group
      SecurityGroupIngress:
        - CidrIp: 10.6.0.0/24       # Inside the VPC
          IpProtocol: -1            # All traffic
      SecurityGroupEgress:
        - CidrIp: 10.6.0.0/24       # Inside the VPC
          IpProtocol: -1            # All traffic
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: '80'            # HTTP
          ToPort: '80'
        - CidrIp: 0.0.0.0/0         # Public Internet
          IpProtocol: tcp
          FromPort: '443'           # HTTPS
          ToPort: '443'
      Tags:
        - Key: "Name"
          Value: "Private Subnet"

We are only allowing incoming traffic from inside our VPC into the private subnet. Outbound, we allow all traffic destined for other addresses in our VPC and we're allowing only HTTP and HTTPS to the public internet. Traffic from addresses in our private subnet is already being routed through our NAT gateway, we're likely only going to use the internet to download software updates.

Provision Other Resources

There are a lot of other resource that we might provision, aside from instances. You might need queues, notification topics, etc. For this project we're going to setup an S3 bucket to hold dump files from our database server.

The nice thing about setting these up before the instances is that we can tell the instances about these resources and set policies so that our instances can access these resources without providing them with a specific set of credentials.

  TutorialBackupS3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      AccessControl: Private
      Tags:
        - Key: "Name"
          Value: "Backup Bucket"

Simple enough! We use the Bucket resource to create our bucket and we set that bucket to be private with the AccessControl property.

It was a long walk but we have accomplished a lot. Go ahead and provision your template and wait for CloudFormation to finish getting everything setup. From the CloudFormation web console, select your stack and click on the "Resources" tab to see all of the items that you have provisioned. Such progress!

Until Next Time

That is all we are going to cover in this article, we went over a lot of material. In the next piece in the series we'll setup some groups and roles and actually provision some instances. Once you've recovered from all of this YAML, check out part 2!

Part 2: we setup some buckets, groups and privileges, and our instances

Thank you for reading!