DEV Community: Christopher Vensand

Deploying Your First Kubernetes Cluster on AWS Using EKS

Christopher Vensand — Mon, 06 Jan 2025 05:25:01 +0000

Welcome to the second post in my series, Building Internet Scale Services with Kubernetes and AWS. If you're new to Kubernetes or AWS, I recommend going back and reading the first post for foundational knowledge about these technologies.

In this post, I'll walk you through creating your first Kubernetes cluster in AWS using EKS (Elastic Kubernetes Service). We’ll use Terraform to provision the infrastructure, ensuring we can easily modify or recreate our setup whenever needed.

Why Use Terraform?

Terraform allows you to define your infrastructure in code and then provision it across different platforms. For example, you might want a Kubernetes cluster in AWS for your core applications, a database in GCP for specialized features, and an object store in Azure. Using Terraform, you can define all of these resources in a unified set of files making multi-cloud architectures simpler to manage.

If you relied entirely on the AWS Console (the web interface) to create your infrastructure, you would have to click the same sequence of buttons every time you wanted to recreate your setup. With Terraform, you can simply run your code again. This “infrastructure as code” approach is a powerful way to maintain and scale your infrastructure.

Setting Up an AWS Account

Before provisioning any infrastructure, you’ll need an AWS account. The easiest way to do this is through the AWS Console. Head over to the AWS sign-up page and create your account.

Creating an IAM User

Once your AWS account is created, you'll need to create an IAM (Identity and Access Management) user whose credentials Terraform will use to provision your infrastructure.

In the AWS Console, type IAM in the search bar at the top and select the IAM service.
Click on Users under Access management on the left panel.
Since you likely won't have any users yet, click Create user.
Choose any username you like.
Under Set permissions, click Attach policies directly and select AdministratorAccess.
- Note: In a production environment, you should grant only the minimum permissions needed rather than full administrative privileges. However, for simplicity in this tutorial, we’ll use AdministratorAccess.
Review your settings and click Create user.

Generating AWS Access Keys

Next, you’ll need to create an Access Key for this IAM user. Terraform will use these credentials to communicate with your AWS account.

Click on your newly created user.
Go to the Security credentials tab.
Under Access keys, click Create access key.
For Use case, choose Command Line Interface (CLI).
Follow the prompts, then click Create access key.
Important: Copy or download the Access Key and Secret Access Key. You will not be able to view the Secret Access Key again once you close this window. If lost, you’ll have to create a new key.

That’s all you need from the AWS Console for now!

Setting Up Your Terminal

Now that you’ve created your IAM user and obtained your AWS access keys, it’s time to install the command-line tools needed to provision your Kubernetes cluster. You’ll need to install and configure two essential CLIs: the AWS CLI and Terraform.

macOS

Install Homebrew and run the following commands in your terminal.

brew install awscli terraform
aws --version
terraform --version

Other Platforms

If you’re using Windows or Linux, refer to the official documentation for installation details:

Windows
- AWS CLI Installation Docs
- Terraform Download Page
Linux
- AWS CLI Installation Docs
- Terraform Download Page

Configuring the AWS CLI

Once both CLIs are installed, configure the AWS CLI to use the credentials you obtained earlier by running the following command:

aws configure

When prompted, enter:

AWS Access Key ID: Your IAM user’s access key.
AWS Secret Access Key: Your IAM user’s secret key.
Default region name (e.g., us-east-1, us-west-2): Your preferred AWS region.
Default output format (e.g., json): The output format for CLI commands.

Your system is now ready to interface directly with AWS through the command line!

Understanding the Terraform Code

With your environment set up, let’s clone the repository and review how the Terraform files work together to provision your Kubernetes cluster:

git clone https://github.com/chrisvensand/terraform-aws-eks
cd terraform-aws-eks

Inside this repository, you’ll find three main Terraform files—main.tf, provider.tf, and versions.tf—that define your AWS and EKS resources. Below is a breakdown of what each file does:

main.tf

This file contains the core configurations for your VPC (networking) and EKS cluster. It references publicly available Terraform modules to simplify the setup:

VPC Module

   module "vpc" {
     source  = "terraform-aws-modules/vpc/aws"
     name = "my-eks-cluster-vpc"
     cidr = "10.0.0.0/16"

     azs             = slice(data.aws_availability_zones.available.names, 0, 2)
     public_subnets  = ["10.0.1.0/24", "10.0.2.0/24"]
     private_subnets = ["10.0.3.0/24", "10.0.4.0/24"]

     enable_nat_gateway    = true
     single_nat_gateway    = true
     enable_dns_hostnames  = true
     enable_dns_support    = true
   }

source: Uses the terraform-aws-modules/vpc/aws module, which wraps all the AWS VPC resources (VPC, subnets, NAT gateways, etc.) in an easy-to-use package.
CIDR, public_subnets, private_subnets: Define the IP address ranges for your VPC. The public subnets are accessible from the internet, while the private subnets are reserved for internal traffic (where your EKS worker nodes will typically reside).
enable_nat_gateway and single_nat_gateway: Provision a NAT gateway for secure outbound internet traffic from private subnets.
enable_dns_hostnames and enable_dns_support: Enable DNS for resources in your VPC, allowing you to map IP addresses to DNS names.

EKS Module

   module "eks" {
     source  = "terraform-aws-modules/eks/aws"
     version = "~> 20.31"

     cluster_name    = "my-eks-cluster"
     cluster_version = "1.31"

     enable_cluster_creator_admin_permissions = true

     cluster_compute_config = {
       enabled    = true
       node_pools = ["general-purpose"]
     }

     vpc_id     = module.vpc.vpc_id
     subnet_ids = module.vpc.private_subnets

     tags = {
       Environment = "dev"
       Terraform   = "true"
     }
   }

source: Leverages the terraform-aws-modules/eks/aws module. This module takes care of creating the EKS control plane, node groups, and associated IAM roles.
cluster_name and cluster_version: Specify the name and Kubernetes version for your cluster.
enable_cluster_creator_admin_permissions: Grants the user deploying the cluster (i.e., your AWS account credentials) full administrative access to the cluster.
cluster_compute_config: Configures how node groups are created, naming them and selecting instance types or node pool strategies.
vpc_id and subnet_ids: Tie the EKS cluster into the VPC created in the vpc module, ensuring the cluster resides in private subnets.
tags: Attach key-value tags to your EKS resources for easy tracking and organization.

Data Source: aws_availability_zones

   data "aws_availability_zones" "available" {}

Fetches the list of available AWS Availability Zones in your chosen region. The VPC module references this data to build out the subnets in two of those zones (ensuring high availability).

provider.tf

provider "aws" {
  region  = "us-west-2"
  profile = "default"
}

region: Specifies the default AWS region (in this case, us-west-2) where Terraform will provision your resources.
profile: Defines the AWS CLI profile to use. If you’ve already configured your ~/.aws/credentials, specifying default means Terraform will use that profile’s credentials.

versions.tf

terraform {
  required_version = ">= 1.3.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
    }
  }
}

required_version: Ensures you’re running Terraform v1.3.0 or later.
required_providers: Declares which providers are needed (in this case, aws) and where Terraform should download them from (the HashiCorp registry).

Finally, Deploying Your EKS Cluster

All set! You’ve done a lot of preparation, but for a fully managed, infinitely scalable Kubernetes cluster, the setup is actually pretty streamlined. Here’s how to deploy:

Initialize Terraform

   terraform init

This downloads any required plugins or modules.

Apply your infrastructure

   terraform apply

Terraform will display a plan, detailing every resource it intends to create. Review and confirm to kick off the deployment.

Note: This can take around 10 minutes, so be patient. Terraform will keep you updated on progress.

Once the deployment finishes, head to the EKS page in the AWS Console. You should see your brand-new cluster ready to roll! Give yourself a well-deserved pat on the back.

Scalability Note: An EKS cluster can scale to 1,000 nodes (and even beyond, if you request a limit increase from AWS). This should be more than enough capacity for most use cases. If you truly need global presence, you can reuse these same Terraform configurations in additional AWS regions worldwide—just change the region in your provider settings.

What’s Next?

In the next blog post, we’ll build a simplified version of “Netflix” on our new Kubernetes cluster, exploring how to run and manage a more complex, microservices-style application. With your EKS cluster in place, the sky’s the limit!

Feel free to drop any questions or comments below. I hope this tutorial empowers you to spin up scalable, resilient Kubernetes clusters on AWS with Terraform—happy building!

About Me

I previously worked at Riot Games as a software engineer on the infrastructure team. While there, I helped the company transition from on-premises infrastructure to AWS and optimized backend services for games like Valorant and League of Legends, running on Kubernetes clusters worldwide.

Building Internet Scale Services with Kubernetes and AWS

Christopher Vensand — Mon, 30 Dec 2024 03:11:17 +0000

The services we enjoy everyday—like YouTube, Netflix, and online games—operate at an incredible scale, handling millions (or even billions) of users seamlessly. Behind the scenes, technologies like Kubernetes and AWS make this possible by enabling developers to deploy, manage, and scale applications reliably across the globe.

This blog series will walk you through what Kubernetes and Amazon Web Services (AWS) are, how they complement each other, and how you can use them to build internet-scale applications. Part 0 (this post) will introduce the basics. Future posts will include boilerplate code to help you set up your own Kubernetes cluster in AWS and practical examples of building scalable services. Onward!

What Do We Mean by Internet-Scale?

When I say “internet scale,” I’m referring to services accessed by millions—or even billions—of people every month. Think YouTube, Netflix, Reddit, or Wikipedia. These platforms operate at a scale that’s hard to imagine.

To put it in perspective:

Netflix accounted for roughly 15% of all internet traffic in 2022.
In the United States alone, Netflix viewers spent 16.3 billion hours watching content. That’s the equivalent of 23,254 human lifetimes!

Building infrastructure capable of supporting this scale requires advanced tools—and that’s where Kubernetes and AWS shine.

Why Kubernetes and AWS?

So how do Kubernetes and AWS make internet-scale services possible? Services that handle traffic at this scale can’t rely on just one server to deliver the smooth experiences we’re all used to. A single server would be overwhelmed, and even multiple servers wouldn’t suffice if they aren’t distributed globally. Without global distribution, users far from the servers would experience lag, downtime, or other issues.

This is where Kubernetes and AWS step in. Kubernetes ensures your applications run efficiently across thousands of servers, while AWS provides the global infrastructure and managed services needed to host and scale these applications seamlessly.

Kubernetes at a Glance

Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management.

But what does that really mean?

Let’s break it down: Kubernetes automates many of the hard parts of running software on thousands of servers. It’s called a "container orchestration system" because it expects your software to be packaged into containers. A container is a lightweight bundle that includes your software and everything it needs to run—such as its operating system dependencies and environment variables.

Imagine you’re running multiple restaurants, and each one has different menus, staff, and hours of operation. Kubernetes acts like a central manager, making sure each restaurant (server) runs efficiently and has exactly what it needs. It assigns tasks, scales resources up or down depending on demand, and ensures nothing falls through the cracks.

A group of servers managed by Kubernetes is called a cluster. These clusters allow you to run applications reliably at massive scale. For example, the largest reported Kubernetes cluster belongs to JD.com, a major Chinese retailer, and it runs on tens of thousands of servers!

While Kubernetes handles the management of your applications, it still needs the underlying infrastructure to run on. This is where AWS comes in.

AWS at a Glance

AWS (Amazon Web Services) is one of the leading cloud computing platforms, providing on-demand resources like compute power, storage, and networking. It’s designed to support everything from small personal projects to large-scale, highly available enterprise applications.

But what does this actually mean? Remember the servers we’ve been talking about? These are physical machines that need to be housed somewhere, maintained, and regularly upgraded. Managing these machines yourself would require a massive investment in data centers, IT staff, and maintenance schedules—not to mention ensuring reliability and uptime. AWS simplifies this process by managing the servers for you and charging you only for what you use.

Some of the most relevant AWS services include:

Amazon EC2 (Elastic Compute Cloud): Virtual machines in the cloud that scale up or down depending on demand.
Amazon S3 (Simple Storage Service): A highly scalable and durable storage service for any type of data.
Amazon EKS (Elastic Kubernetes Service): A managed Kubernetes service that takes care of provisioning, scaling, and maintaining Kubernetes clusters on AWS.

For this blog series, we’ll focus on EKS, as it’s the service that bridges Kubernetes and AWS. EKS automates much of the work required to set up and manage Kubernetes clusters, making it easier to deploy and scale your applications globally. With AWS’s global infrastructure, you can run your applications closer to users around the world, ensuring faster response times and a better user experience.

Next Steps

Kubernetes and AWS are a powerful combination for building internet-scale services. Kubernetes provides the tools to manage applications efficiently across thousands of servers, while AWS offers the underlying infrastructure to support these applications globally.

In the next post, we’ll dive into some Terraform code for setting up your own Kubernetes cluster on AWS!

Part 1: Deploying Your First Kubernetes Cluster on AWS Using EKS