DEV Community: Bibin Wilson

How to Setup a AWS EKS Anywhere Cluster

Bibin Wilson — Thu, 07 Jul 2022 05:58:48 +0000

In this blog, you will learn what EKS anywhere is and how to set up an EKS Anywhere development cluster and register it to the AWS EKS console using the EKS connector.

What is EKS Anywhere?

EKS Anywhere is an AWS feature to run and manage EKS clusters in on-premises environments. It simplifies the on-premise Kubernetes management and enables a consistent Kubernetes experience in a multi-cloud Kubernetes deployment. Also, you will have full control over the control plane and the worker nodes.

EKS anywhere uses Amazon EKS Distro (EKS-D), a Kubernetes distribution customized and open-sourced by AWS. It is the same distro that powers the AWS-managed EKS. This means that when you install EKS anywhere, it comes with parameters and configurations optimized for AWS.

Also, you can register the EKS anywhere clusters to the AWS EKS console using the EKS connector.Once the cluster is registered, you can visualize all the anywhere cluster components in the AWS EKS console.

EKS connector is a Statefulset that runs the AWS System Manager Agent in your cluster. It is responsible for maintaining the connection between EKS anywhere cluster and AWS.

Following the key use cases of EKS anywhere:

Hybrid cloud consistency: EKS anywhere enables operational consistency across your on-premise and cloud Kubernetes clusters.
Disconnected environment: You can run EKS anywhere cluster without internet connection. At the same time, your disconnected clusters will have the same features as the cloud EKS with EKS-D distro.
Application modernization: With EKS-D, the administrative overhead of patching and version upgrades are reduced. So you can focus more on the applications.

EKS Anywhere Development Cluster Setup

You can setup EKS anywhere development clusters using Docker on a MAC or Ubuntu systems.

Follow the steps given below to setup EKS anywhere development cluster.

Step 1: Install Docker Desktop

Go to Docker downloads and install the Docker desktop on MAC. For Ubuntu use this link to download the deb package

Mac users, open the following file and change deprecatedCgroupv1 to true. And then restart Docker desktop to apply the changes.

vi ~/Library/Group\ Containers/group.com.docker/settings.json

Or else you will get the following error.

Error: failed to validate docker desktop: EKS Anywhere requires Docker desktop to be configured to use CGroups v1. Please  set `deprecatedCgroupv1:true` in your `~/Library/Group\ Containers/group.com.docker/settings.json` file

Step 2: Install eksctl-anywhere

brew install eks-anywhere

Step 3: Generate the cluster configuration

Generete the EKS anywhere cluster configuration using the following command.

CLUSTER_NAME=dev-eks-cluster
eksctl anywhere generate clusterconfig $CLUSTER_NAME \
   --provider docker > $CLUSTER_NAME.yaml

You cluster configuration would look like the following.

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: dev-eks-cluster
spec:
  clusterNetwork:
    cniConfig:
      cilium: {}
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
  controlPlaneConfiguration:
    count: 1
  datacenterRef:
    kind: DockerDatacenterConfig
    name: dev-eks-cluster
  externalEtcdConfiguration:
    count: 1
  kubernetesVersion: "1.22"
  managementCluster:
    name: dev-eks-cluster
  workerNodeGroupConfigurations:
  - count: 1
    name: md-0

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: DockerDatacenterConfig
metadata:
  name: dev-eks-cluster
spec: {}

---

Step 4: Deploy the cluster

Deploy the cluster using eksctl. It will take a while for the cluster to be provisioned.

eksctl anywhere create cluster -f dev-eks-cluster.yaml

You will get the following output after a succesful cluster deployment.

Step 5: Validate the cluster

Once the cluster is created you can see a folder named dev-eks-cluster with the kubeconfig file. Lets export the kubeconfig file.

Replace /path/to with the abosolute path of the dev-eks-cluster folder location. Or checkout the Kubeconfig file guide to know more about kubeconfig file practical usage.

KUBECONFIG=/path/to/dev-eks-cluster/dev-eks-cluster-eks-a-cluster.kubeconfig

Now, lets list the pods in the kube-system namespace to validate the cluster. You should see all the pods in running state as shown below.

Step 6: Generate & Deploy EKS Connector Configs

Now that we have a running EKS anywhere development clusters, we will go ahead and register the cluster using EKS connector.

Note: Before you register the cluster, ensure you have a valid AWS cli configuration with admin privileges in your system. Refer the AWS CLI configuration guide for more details.

Here is the command to generate EKS connector config YAMLs. Replace region value if you are using a different region.

eksctl register cluster --name dev-eks-cluster --provider EKS_ANYWHERE --region us-west-2

The above command creates the following three YAML files. Where eks-connector.yaml is the EKS connector agent statefulset YAML.

eks-connector-clusterrole.yaml
eks-connector-console-dashboard-full-access-group.yaml
eks-connector.yaml

Lets deploy all the three YAMLs.

kubectl apply -f eks-connector-clusterrole.yaml
kubectl apply -f eks-connector-console-dashboard-full-access-group.yaml
kubectl apply -f eks-connector.yaml

Step 7: Validate EKS Anywhere Cluster Registration On EKS console

If you head over to the AWS EKS console, you should see the newly registered cluster as shown below.

If you click the cluster name, you can view all the information and objects of the EKS anywhere cluster running in you local workstation.

Here is the EKS connector demo I have added in Youtube

Conclusion

In this guide we looked at EKS anywhere Kubernetes development cluster setup. It is pretty easy to get started. Do give it a try.

Also if you are looking for light weight Kubernetes development clusters, checkout my minikube tutorial.

If you are learning Kubernetes, checkout 30+ Kubernetes beginners tutorials.

3 Ways to Automate AWS EBS Snapshot Lifecycle Management

Bibin Wilson — Mon, 06 Jul 2020 06:11:12 +0000

In 2013 when I started with AWS, I had to write custom scripts with AWS CLI to manage EBS snapshots. Later in 2015, I switched to Lambda functions. Today there are managed options from AWS to manage the Snapshot lifecycle.

In this article, I have covered three ways to automate EBS snapshot creation with its key features and some best practices.

When it comes to AWS, snapshots are essential for backing up root volumes and data volumes.

Right from dev till production, snapshots lifecycle management should be set up to avoid accidental data loss and data availability.

It is essential to have well-managed snapshot automation to ensure the data is getting backed up every day.

Read the full article here https://devopscube.com/automate-ebs-snapshot-creation-deletion/

How to Become A DevOps Engineer: A Comprehensive Guide

Bibin Wilson — Mon, 11 May 2020 14:05:34 +0000

As of the current IT market, the DevOps domain is one of the best options for IT folks in terms of salary and career growth. One common question I get quite often is "How to become a DevOps engineer?"

In this blog, I will try to answer this with my own experiences in practicing DevOps in different organizations.

Many people argue (including me) that there is nothing like a "DevOps Engineer" or a "DevOps Team" because it is not a thing. However everyone in the industry now got used to the term "DevOps engineer" and as long as you understand the DevOps philosophy, these titles don't matter much.

Having said that, there are few misconceptions about what DevOps really means. One such misconception is "Automation is DevOps". Developing skills related to automation is not enough to become a DevOps Engineer.

Wikipedia says,

DevOps (a clipped compound of development and operations) is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes.

From the above definition, it is clear that DevOps is not about any tools or technologies. It is a philosophy for making different IT teams work together to deliver better and fast results through continuous feedback.

Read this article to understand DevOps in a better way. What does DevOps really mean?

Here is an interesting trend graph shoring DevOps popularity in last 5 years.

Organizations trying to achieve DevOps requires people with collaborative skills, willing to change and adopt new technologies, a good understanding of systems, automation tools, CI tools, Version control systems, networking, experience in using project management tools, etc. that are required for getting an app into production without much delay.

Also, the design or the pipeline designed by the team should be able to deliver small updates or releases without much manual intervention. This could happen only if there is a cultural shift in the way teams work.

Skillsets To Become a DevOps Engineer

You must understand the fact that DevOps is not specific to developers or system engineers. It's for anyone who is passionate about evolving practices, technologies, and willing to work in a collaborative environment where everything is automated to make everyone's life so easy.

In this article, I will explain how you should prepare yourself for tools and technologies to adapt and work in DevOps culture.

Note: In this article, I have covered many verticals. It is not possible for a beginner to be a master of everything. However, having a fair amount of knowledge in these areas will help you if you are pursuing a career in DevOps

Understanding DevOps Culture

The first and foremost thing is to understand the DevOps culture. It is all about bringing people together to work towards a common goal in an efficient way. One thing IT managers should do before getting into DevOps toolsets is that every member of the team should be mentored on how DevOps work and its cultural aspects. It avoids lots of confusion in the team.

People will stop pointing fingers for various issues once they understand the fact that when there is delay or issue in project delivery, everyone involved in the project is equally responsible.

Once you practice DevOps culture, you will stop saying that "CI/CD and automation is DevOps"

Resources

DevOps Culture and Mindset [Coursera]
The Phoenix Project [Recomended eBook]

Learn about *nix Systems

We are in an era where we cannot live without Linux/Unix systems. You should get a better understanding and working knowledge of various Linux distributions that are highly used by organizations (RHEL, Centos, Ubuntu, CoreOS, etc).

As per The Linux foundation case study, 90% of the public cloud workload runs on Linux

Here is another interesting study from Redhat, which shows the different Linux distros being used in the public cloud.

Now you have enough reasons on why you should focus on Linux.

When it comes to Linux, its all terminal, GUI is less preferred in *nix world. Get your hands dirty with terminals of these systems.

You can use a Virtual box or AWS/GCP/Azure to spin up Linux servers.

You can start with the following.

Understand the Linux booting process
Install and Configure web servers (Apache, Nginx, Tomcat, etc..)
Learn how Linux processes work.
Learn how SSH works.
Learn about different file systems.
Learn about system logging, monitoring, and troubleshooting.
Learn about important protocols (SSL, TLS, TCP, UDP, FTP, SFTP, SCP, SSH)
Learn to manage services and try to create a service on your own (Initd, Systemd)
Host static/Dynamic websites on web servers.
Setup Load balancers & Reverse Proxys (Nginx, HA proxy, etc)
Break something and learn to troubleshoot.

Resources

Introduction to Linux [edX]
Learn Linux in 5 days [Udemy]

Understand How Infrastructure Components Work

The basic building block of any organization is its Infrastructure. It could be on the cloud or on-premise Data Center. An overall understanding of Infrastructure components is a must for a person who wants to practice or work in a DevOps environment. You should mainly have a basic understanding of the following.

Networking

Subnets
Public network
Private network
CIDR Notations
Static/Dynamic IP's
Firewall
Proxy
NAT
Public & Private DNS
Troubleshooting
VPN

High Availability

Clusters
Fail Over Mechanisms
Disaster Recovery

Security

PKI Infrastructure
SSL certificates

Storage

SAN
Backups
NFS

Single Sign On

Active Directory/LDAP

Load Balancers

L5 Load Balancers
L7 Load Balancers
Load balancing algorithms
Reverse Proxy

VPN

Site to Site VPN
Client to site VPN

There could be more things but I have highlighted the key components in an IT Infrastructure.

Get Certified On Cloud

When I say "Get Certified", please do not use the exam dumps just to pass the certification. It adds very less value to you. May be its good for the organization to show the clients that they have certified cloud engineers.

Most of the public cloud market share is currently owned by AWS. Here is the report from Businesswire.

Pick any one public cloud, preferably AWS, and learn about all its core infrastructure services. Do hands-on on all the core services and understand how it works.

Watch AWS re-invent videos and understand how other organizations are using AWS services for hosting their applications. Trust me, you will learn a lot from these videos and no online training will provide that much information on how to run production workloads on AWS.

If you are planning to get certified GCP, watch their Google Next videos.

Use the certification to gauge yourself on the respective platform.

Resources:

Learn to Automate

Automation has become an important aspect of every organization. We no more create servers manually we just automate it.

As per a report from Redhat, many organizations are investing in their automation initiatives. Check out this data.

From provisioning servers, application configuration, deployment, everything should be automated. You can learn any of the following devops toolsets that fit your needs.

For Dev Environment

Vagrant
Docker Desktop
Minikube
Minishift

For infrastructure provisioning

Terraform
CLIs (of respective cloud provider)

For Configuration Management

Ansible
Chef
Puppet
Saltstack

VM image management

Packer

Resources:

Containers, Distributed Systems & Service Mesh

Container adoption is increasing day by day. The organization you work for might not be using containers now, however, it is best to have a hands-on knowledge working with container technology like Docker. It will gain you some competitive edge among your peers.

Once you understand docker, you can try out its clustering and orchestration tools like Kubernetes, Docker Swarm, etc.

These platforms are best suited for microservices-based architecture.

Here is an interesting Kubernetes usage trend by Datadog.

Here is the five year increasing search trends for Kubernetes.

A service mesh is an advanced topic when it comes to distributed systems. If you are a beginner to container toolsets, you can learn this after gaining a good knowledge of microservices-based architecture.

Resources

Logging & Monitoring

Logging and monitoring are very important aspects of an infrastructure.

Most of the apps deployed in the infrastructure will produce logs. Based on architecture and design, logs will be pushed and stored in a logging infrastructure.

Every company will have a logging infrastructure. Commonly used stacks are Splunk and ELK. Also, there are few SaaS companies like Loggly which provides logging infrastructure.

Logging systems will be used by developers, operations team and security teams to monitor, troubleshoot, and audit applications and infrastructure.

In every organization, mission-critical applications will be monitored 24/7. There will be monitoring dashboards. Generally, dashboards are created from logging sources, or metrics generated by the application.

Also, there will be alerting systems. Based on the rules configured in the monitoring systems, alerts will be triggered.

For example, an alert could be triggered as a slack notification, Jira ticket, email alert, ServiceNow incident ticket, or xMatters phone call. Alerting workflows differ from organization to organization.

As a DevOps engineer, you should be able to query logs and troubleshoot issues in non-prod and prod environments. Understand regular expressions is very important to query logs in any logging tool.

Resources

Understand Security Best Practices (DevSecOps)

DevSecOps is another area dealing with integrating security practices in each stage of devops.

Wikipedia says,

DevSecOps is an augmentation of DevOps to allow for security practices to be integrated into the DevOps approach. The traditional centralised security team model must adopt a federated model allowing each delivery team the ability to factor in the correct security controls into their DevOps practices.

Checkpoints 2020 security survey shows different cyber attacks by regions.

In cloud environments, crypto mining is a common attack. This mostly happens when the cloud access secrets are maintained poorly so that hackers get access to it.

When it comes to DevOps, secret management for applciations and infrastructure componets should follow standard security practices.

Following image shows the key DevSecOps standard practices published by Redhat.

Source: Redhat.com

Hashicorp Vault is a great secret management tool you can look at. There are many workflows available to manage environment secrets.

Resources:

Learn Coding & Scripting

In today's world, we treat everything as code. Even though there are enough tools to automate everything, you might need custom functionality that a tool may not offer. In such cases, coding/scripting comes in handy to achieve those functionalities.

For example, Jenkins pipeline as code requires an understanding of groovy, Ansible custom module requires understanding on python, Writing Kubernetes operator requires Golang experience.

You can learn the following commonly used scripting languages.

Bash/Shell
Python
Golang

Golang is really getting popular in the DevOps domain. Lots of DevOps tooling are done using Golang nowadays. In fact tools like Kubernetes, terraform are written in go.

A survey was done by JFrog for Golang adoption during GopherCon and 18% of the respondents said they use Golang for DevOps related work.

Resources

Learn Git, Learn to Document, Learn about GitOps

It is very important to version control everything you do (except passwords and secrets :P). Git is the best version control tool. There are plenty of tutorials available on git and it will not take much time to learn important git operations.

You can start with Github or Bitbucket as your remote code repository.

Once you understand Git, learn about GitOps.

So what is this GitOps anyway? here is what gitops.tech explains GitOps

GitOps is a way of implementing Continuous Deployment for cloud native applications. It focuses on a developer-centric experience when operating infrastructure, by using tools developers are already familiar with, including Git and Continuous Deployment tools.

Next important thing is to document every important thing you do. Every repository must have a README file which should explain your code in a better way. Good documentation will not only help you but also someone who tries to use your code.

Resources:

Understand End To End Application Delivery Lifecycle

When it comes to application delivery lifecycle, there are three important concepts you need to be aware of.

Continuous Integration
Continuous Delivery
Continuous Deployment

Read this release process management article to understand how a typical application development, build, testing, deployment, approval process, and validation work.

Learn to use CI/CD tools like Jenkins, Travis CI, GoCD, etc

Here is a good pictorial representation on CI/CD process by bmc.

source: bmc

DevOps vs SRE

SRE is another evolving topic in DevOps community.

SRE is set of practices and philosophies emerged from google.

Here is what google says about DevOps and SRE

DevOps and SRE are not two competing methods for software development and operations, but rather close friends designed to break down organizational barriers to deliver better software faster.

I recommend these official documents from Google to understand more about SRE.

Read Read and Read

Nothing will gain knowledge like reading. Read at least one DevOps tech blog related to engineering. Follow all the engineering blogs like Netflix, Twitter, Google, etc. Learn how they are using the right toolsets, their deployment strategies and their latest open source projects.

Follow like-minded people on LinkedIn, Reddit, Medium, Quora etc.

Resources

List of Best DevOps Blogs & Resources

Write a Blog

It's good to share with others about your experiences and learning. You can publish tutorials, learnings, and your experiences on your personal blog. It will help others and it will create a personal brand for yourself. It takes less than 30 minutes to set up a WordPress blog or a Medium blog. If you want help to start your blog, drop a message to us at contact@devopscube.com

Whenever you learn something new, you can write about it. It will be a reference to you as well as others. You can share it on Linked in groups, Dzone, etc..

Conclusion

The tools and processes involved in DevOps are not limited to what is mentioned in this article. However, these are commonly used opensource tools and technologies you can start with to become a DevOps engineer.

Now I’d like to hear your thoughts:

What’s your key takeaway from this?

Or maybe you have a question about different verticals explained.

Either way, leave a comment below right now.