DEV Community: Dmitrii Kotov

Why Idempotence Matters in CI/CD Pipeline Build Steps

Dmitrii Kotov — Tue, 26 Nov 2024 13:07:07 +0000

Recently, I was caught off guard by a question: why should the steps of a build script in a pipeline be idempotent? Why can't we build and push the container every time? Why is idempotence so important?

Idempotence in pipeline build steps is important for many reasons.

Predictability and Stability:

When a step in a pipeline is idempotent, it means that running it multiple times will always give the same result. This is very important for CI/CD pipelines because the same build might be triggered more than once. For example, if there is a small error, or someone makes changes to the code, or during testing, you might need to run the pipeline again. If the steps are idempotent, you don’t have to worry about unexpected results. Everything will work the same way every time.

Avoiding Wasting Resources:

If you build and push a container every time, even when nothing has changed, it can waste resources. Each time you push, it creates a new version of the container in the registry, even if it’s exactly the same as the previous one. This takes up storage space and can make things harder to manage. It can also cost more money if you’re using cloud services to store these containers.

Preventing Errors:

When steps are not idempotent, they might behave differently each time you run them. This can cause bugs that are difficult to understand and fix. For example, if a build step depends on a file that changes randomly or a tool that behaves differently depending on the environment, the pipeline can fail in unpredictable ways. Idempotent steps make the pipeline more reliable.

Saving Time with Caching:

Idempotent steps allow you to use caching. This means that if something has already been built or processed, it doesn’t need to be done again. For example, if your container image has several layers and only one of them has changed, you can reuse the unchanged layers. This can save a lot of time during the build process.

Easier Debugging:

When your pipeline is predictable, it’s much easier to find out what went wrong if something fails. You can check each step and know that it works the same way every time. If the pipeline behaves differently on every run, debugging becomes a nightmare.

Why shouldn’t we rebuild the container every time?

If the code or dependencies haven’t changed, rebuilding the container is unnecessary. It doesn’t add any value but uses extra time, CPU, and storage. Every new build creates a new version of the container image, which can make it hard to manage the versions and figure out which one you actually need. It’s better to reuse existing builds whenever possible.

In summary, idempotence is important because it makes the pipeline stable, saves resources, prevents errors, and makes everything faster and easier to manage. It’s like creating a system where you know exactly what will happen, no matter how many times you use it.

Kubernetes Interview Questions: Kubernetes Architecture: Node

Dmitrii Kotov — Mon, 18 Dec 2023 09:00:00 +0000

What are Nodes in Kubernetes?

Nodes are where Kubernetes places containers into Pods to run. A node may be a virtual or physical machine, depending on the cluster.

Who manages each node in a Kubernetes cluster?

Each node is managed by the control plane and contains the services necessary to run Pods.

What are the key components on a node in Kubernetes?

The components on a node include the kubelet, a container runtime, and the kube-proxy.

What are the two main ways to add Nodes to the API server in Kubernetes?

The two main ways are: 1) The kubelet on a node self-registers to the control plane, and 2) You (or another human user) manually add a Node object.

What happens after a Node object is created or a kubelet self-registers in Kubernetes?

After a Node object is created, or a kubelet on a node self-registers, the control plane checks whether the new Node object is valid.

When is a node considered eligible to run a Pod in Kubernetes?

A node is eligible to run a Pod if it is healthy, meaning all necessary services are running.

What happens if a node is not healthy in Kubernetes?

If the node is not healthy, it is ignored for any cluster activity until it becomes healthy. Kubernetes keeps the object for the invalid Node and continues checking to see whether it becomes healthy.

How can you stop health checking on an invalid Node in Kubernetes?

You, or a controller, must explicitly delete the Node object to stop that health checking.

What is the requirement for the name of a Node object in Kubernetes?

The name of a Node object must be a valid DNS subdomain name.

What issues can arise if a Node instance is modified without changing its name?

This may lead to inconsistencies if an instance was modified without changing its name.

What should be done if a Node needs to be replaced or updated significantly in Kubernetes?

If the Node needs to be replaced or updated significantly, the existing Node object needs to be removed from the API server first and re-added after the update.

What is the preferred pattern for the registration of Nodes in Kubernetes?

The preferred pattern, used by most distros, is self-registration of Nodes.

What is a good practice when Node configuration needs to be updated in Kubernetes?

It is a good practice to re-register the node with the API server when Node configuration needs to be updated.

What issues can arise if the Node configuration is changed on kubelet restart while Pods are already scheduled on the Node?

Pods already scheduled on the Node may misbehave or cause issues if the Node configuration is changed on kubelet restart.

Why is Node re-registration important in the context of updating Node configuration?

Node re-registration ensures all Pods will be drained and properly re-scheduled, maintaining the integrity of the cluster’s operations.

How can you create and modify Node objects in Kubernetes?

You can create and modify Node objects using kubectl.

What should you set the kubelet flag to when creating Node objects manually?

When you want to create Node objects manually, set the kubelet flag --register-node=false.

What does marking a node as unschedulable do?

Marking a node as unschedulable prevents the scheduler from placing new pods onto that Node but does not affect existing Pods on the Node. This is useful as a preparatory step before a node reboot or other maintenance.

How do you mark a Node as unschedulable in Kubernetes?

To mark a Node unschedulable, you run the command kubectl cordon $NODENAME.

What is a special case where Pods will still run on an unschedulable Node?

Pods that are part of a DaemonSet tolerate being run on an unschedulable Node. DaemonSets typically provide node-local services that should run on the Node even if it is being drained of workload applications.

What information does a Node's status contain in Kubernetes?

A Node's status contains the following information: Addresses, Conditions, Capacity and Allocatable, and Info.

How can you view a Node's status and other details in Kubernetes?

You can view a Node's status and other details using the command kubectl describe node <insert-node-name-here>.

What is the purpose of heartbeats sent by Kubernetes nodes?

Heartbeats sent by Kubernetes nodes help the cluster determine the availability of each node, and to take action when failures are detected.

What are the two forms of heartbeats for nodes in Kubernetes?

The two forms of heartbeats for nodes are: 1) Updates to the .status of a Node, and 2) Lease objects within the kube-node-lease namespace.

What is the role of the node controller in Kubernetes?

The node controller is a Kubernetes control plane component that manages various aspects of nodes.

What is the first role of the node controller in a node's life?

The first role is assigning a CIDR block to the node when it is registered, if CIDR assignment is turned on.

What happens if the node controller finds that the VM for an unhealthy node is not available?

If the VM for an unhealthy node is not available, the node controller deletes the node from its list of nodes.

What is one of the responsibilities of the node controller regarding node health?

The node controller is responsible for updating the Ready condition in the Node's .status field if a node becomes unreachable, setting it to Unknown.

What action does the node controller take if a node remains unreachable?

If a node remains unreachable, the node controller triggers API-initiated eviction for all of the Pods on the unreachable node.

How long does the node controller wait before submitting the first eviction request for an unreachable node?

By default, the node controller waits 5 minutes between marking the node as Unknown and submitting the first eviction request.

How often does the node controller check the state of each node by default?

By default, the node controller checks the state of each node every 5 seconds.

Can the period for checking the state of each node by the node controller be configured?

Yes, this period can be configured using the --node-monitor-period flag on the kube-controller-manager component.

What is the default eviction rate limit set by the node controller?

The node controller limits the eviction rate to --node-eviction-rate (default 0.1) per second, meaning it won't evict pods from more than 1 node per 10 seconds.

How does the node eviction behavior change when a node in an availability zone becomes unhealthy?

When a node in an availability zone becomes unhealthy, the node controller checks the percentage of unhealthy nodes and may reduce the eviction rate if a certain threshold of unhealthy nodes is reached.

What happens to the eviction rate if the fraction of unhealthy nodes is at least `--unhealthy-zone-threshold`?

If the fraction of unhealthy nodes is at least --unhealthy-zone-threshold (default 0.55), then the eviction rate is reduced.

What is the node eviction policy in small clusters?

In small clusters (less than or equal to --large-cluster-size-threshold nodes, default 50), evictions are stopped.

How is the eviction rate adjusted in larger clusters with unhealthy nodes?

In larger clusters with unhealthy nodes, the eviction rate is reduced to --secondary-node-eviction-rate (default 0.01) per second.

Does the node controller consider per-zone unavailability if the cluster does not span multiple cloud provider availability zones?

If the cluster does not span multiple cloud provider availability zones, then the eviction mechanism does not take per-zone unavailability into account.

What happens to the eviction rate if all nodes in a zone are unhealthy?

If all nodes in a zone are unhealthy, the node controller evicts at the normal rate of --node-eviction-rate.

What is the node controller's policy for evictions in cases where all zones are completely unhealthy?

If all zones are completely unhealthy, the node controller assumes a connectivity issue and does not perform any evictions.

What type of information about resource capacity do Node objects track in Kubernetes?

Node objects track information about the Node's resource capacity, such as the amount of memory available and the number of CPUs.

How do Nodes that self-register report their capacity?

Nodes that self-register report their capacity during the registration process.

What does the Kubernetes scheduler ensure regarding resources on a Node?

The Kubernetes scheduler ensures that there are enough resources for all the Pods on a Node.

How does the scheduler determine if a Node has enough resources?

The scheduler checks that the sum of the requests of containers on the node is no greater than the node's capacity.

What is excluded from the sum of requests when the scheduler checks a Node's capacity?

The sum of requests excludes any containers started directly by the container runtime and any processes running outside of the kubelet's control.

Where can you find information about reserving resources for non-Pod processes?

Information about explicitly reserving resources for non-Pod processes can be found in the section "reserve resources for system daemons."

What does the kubelet do during a node system shutdown in Kubernetes?

The kubelet attempts to detect node system shutdown and terminates pods running on the node.

How does the kubelet handle pods during a node shutdown?

During a node shutdown, the kubelet ensures that pods follow the normal pod termination process and does not accept new Pods.

What does the Graceful node shutdown feature depend on?

The Graceful node shutdown feature depends on systemd, using systemd inhibitor locks to delay the node shutdown.

Is the GracefulNodeShutdown feature gate enabled by default in Kubernetes?

Yes, the GracefulNodeShutdown feature gate is enabled by default in Kubernetes.

How is the node marked during a shutdown, and how does the kube-scheduler respond?

Once systemd detects a node shutdown, the kubelet sets a NotReady condition on the Node with the reason "node is shutting down". The kube-scheduler honors this condition and does not schedule any new Pods onto the affected node.

How are pods terminated by the kubelet during a graceful shutdown?

During a graceful shutdown, the kubelet terminates pods in two phases: first, it terminates regular pods running on the node, and then it terminates critical pods.

What happens to the Node and Pods if a node termination is cancelled?

If node termination is cancelled, the Node returns to the Ready state, but Pods that started the termination process will not be restored and need to be re-scheduled.

How are pods marked and shown in `kubectl` when evicted during a graceful node shutdown?

Pods evicted during a graceful node shutdown are marked as shutdown, with their status shown as Terminated in kubectl get pods, and kubectl describe pod indicating that the pod was terminated due to imminent node shutdown.

What are the two phases of pod shutdown in the Graceful Node Shutdown feature?

The Graceful Node Shutdown feature shuts down pods in two phases: non-critical pods, followed by critical pods.

What happens to pods that are part of a StatefulSet during an undetected node shutdown?

During an undetected node shutdown, pods that are part of a StatefulSet get stuck in terminating status on the shutdown node and cannot move to a new running node.

Why can't the StatefulSet create a new pod with the same name during an undetected node shutdown?

The StatefulSet cannot create a new pod with the same name because the kubelet on the shutdown node is not available to delete the pods.

What happens to the volumes used by pods during an undetected node shutdown?

If there are volumes used by the pods, the VolumeAttachments will not be deleted from the original shutdown node, so the volumes used by these pods cannot be attached to a new running node.

What are the two phases of pod termination during a non-graceful shutdown?

The two phases are: 1) Force delete the Pods that do not have matching out-of-service tolerations, and 2) Immediately perform detach volume operations for such pods.

What is a potential warning regarding the use of swap memory in Kubernetes?

When the memory swap feature is turned on, there is a risk that Kubernetes data such as the content of Secret objects written to tmpfs could be swapped to disk.

How can a user configure how a node uses swap memory?

A user can configure the node's use of swap memory by setting memorySwap.swapBehavior, for example, to UnlimitedSwap or LimitedSwap.

What is the support status of swap with different cgroup versions?

Swap is supported only with cgroup v2, and cgroup v1 is not supported.

What does the process of safely draining a node involve?

Safely draining a node involves using kubectl drain to safely evict all pods from a node before performing maintenance, optionally respecting the PodDisruptionBudget defined.

What does `kubectl drain` do?

kubectl drain safely evicts all pods from a node, respecting the PodDisruptionBudgets specified, in preparation for maintenance like kernel upgrades or hardware maintenance.

How do you drain a node with DaemonSet pods?

When draining a node with DaemonSet pods, use kubectl drain --ignore-daemonsets <node name> as the DaemonSet controller immediately replaces missing Pods with new ones.

What should you do with the node during maintenance and after it's completed?

During maintenance, power down the node or delete its VM. After maintenance, if the node remains in the cluster, use kubectl uncordon <node name> to resume scheduling new pods onto the node.

Can you drain multiple nodes in parallel?

Yes, you can run multiple kubectl drain commands for different nodes in parallel, and they will still respect the PodDisruptionBudget specified.

What alternative is there to using `kubectl drain` for evictions?

As an alternative to kubectl drain, you can programmatically cause evictions using the eviction API for finer control over the pod eviction process.

What information does a Node's status contain in Kubernetes?

A Node's status contains Addresses, Conditions, Capacity and Allocatable, and Info.

What are the different types of Addresses found in a Node's status?

The types of Addresses include HostName, ExternalIP, and InternalIP, which vary depending on the cloud provider or bare metal configuration.

How are taints related to node conditions in Kubernetes?

When problems occur on nodes, Kubernetes automatically creates taints matching the conditions affecting the node, like node.kubernetes.io/unreachable or node.kubernetes.io/not-ready.

Kubernetes Interview Questions: Kubernetes Components

Dmitrii Kotov — Sun, 03 Dec 2023 07:37:59 +0000

What does a Kubernetes cluster consist of?

A Kubernetes cluster consists of a set of worker machines, called nodes, that run containerized applications.

What is the minimum number of worker nodes a Kubernetes cluster can have?

Every cluster has at least one worker node.

What do the worker nodes in a Kubernetes cluster host?

The worker node(s) host the pods that are the components of the application workload.

Who manages the worker nodes and the Pods in a Kubernetes cluster?

The control plane manages the worker nodes and the Pods in the cluster.

Can control plane components be run on any machine in the cluster?

Yes, control plane components can be run on any machine in the cluster.

Are user containers run on the machine hosting the control plane components?

No, user containers are not run on the machine hosting the control plane components.

What are the components of a control plane?

kube-apiserver, etcd, kube-scheduler, kube-controller-manager, cloud-controller-manager,

What is the kube-apiserver?

The kube-apiserver is the main implementation of a Kubernetes API server, which is a component of the Kubernetes control plane.

What role does the API server play in the Kubernetes control plane?

The API server is the front end for the Kubernetes control plane and exposes the Kubernetes API.

Can you run multiple instances of kube-apiserver?

Yes, you can run several instances of kube-apiserver and balance traffic between those instances.

What is etcd in the context of Kubernetes?

etcd is a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data.

What is the kube-scheduler?

kube-scheduler is a control plane component that watches for newly created Pods with no assigned node, and selects a node for them to run on.

What factors are considered in the kube-scheduler's scheduling decisions?

Factors considered include individual and collective resource requirements, hardware/software/policy constraints, affinity and anti-affinity specifications, data locality, inter-workload interference, and deadlines.

What is the kube-controller-manager?

kube-controller-manager is a control plane component that runs controller processes.

How are the controllers in kube-controller-manager implemented to reduce complexity

To reduce complexity, all the controllers are compiled into a single binary and run in a single process.

What is the responsibility of the Node controller in kube-controller-manager?

The Node controller is responsible for noticing and responding when nodes go down.

What does the Job controller do?

The Job controller watches for Job objects that represent one-off tasks, then creates Pods to run those tasks to completion.

What is the role of the EndpointSlice controller?

The EndpointSlice controller populates EndpointSlice objects, which provide a link between Services and Pods.

What function does the ServiceAccount controller perform?

The ServiceAccount controller creates default ServiceAccounts for new namespaces.

What are the components of a Worker Node?

kubelet, kube-proxy, container runtime

What is the kubelet?

The kubelet is an agent that runs on each node in the cluster and ensures that containers are running in a Pod.

What is the primary function of the kubelet?

The primary function of the kubelet is to take a set of PodSpecs, provided through various mechanisms, and ensure that the containers described in those PodSpecs are running and healthy.

Does the kubelet manage all containers on a node?

No, the kubelet does not manage containers that were not created by Kubernetes.

What is kube-proxy?

kube-proxy is a network proxy that runs on each node in your cluster, implementing part of the Kubernetes Service concept.

What is the primary role of kube-proxy in a Kubernetes cluster?

The primary role of kube-proxy is to maintain network rules on nodes.

How does kube-proxy manage network traffic?

kube-proxy uses the operating system packet filtering layer if it is available. If not, kube-proxy forwards the traffic itself.

What is a container runtime in the context of Kubernetes?

A container runtime is a fundamental component that empowers Kubernetes to run containers effectively.

What are the responsibilities of a container runtime within the Kubernetes environment?

It is responsible for managing the execution and lifecycle of containers.

Does Kubernetes support multiple container runtimes?

Yes, Kubernetes supports container runtimes such as containerd, CRI-O, and any other implementation of the Kubernetes CRI (Container Runtime Interface).

What are addons in Kubernetes?

Addons use Kubernetes resources (DaemonSet, Deployment, etc) to implement cluster features.

Where should namespaced resources for addons be located?

Namespaced resources for addons belong within the kube-system namespace as they provide cluster-level features.

How can one find more information about available addons?

For an extended list of available addons, you can refer to the section titled "Addons" in the relevant Kubernetes documentation.

Terraform Expertise: Valuable Takeaways from Years in Production

Dmitrii Kotov — Mon, 27 Nov 2023 07:34:37 +0000

In the article, I want to share my insights on the most important lessons learned from years of using Terraform in production.

Use versioning in remote states

Everyone makes mistakes. Some of them may break a Terraform state, and in such situations, availability of a previous state version can help you to avoid completely reimporting the whole infrastructure. You simply roll back the state to the previous version and resolve all problems in the code or in the infrastructure manually. Often, those issues will be resolved after applying the previous state of the code to this version of state. Moreover, it can help you avoid situations where the state is updated not because of changes but due to the major version update of Terraform, and your pipelines or providers aren't ready yet, or there's a bug, leading you to roll back the state instead of editing it directly. You might have noticed this if you have gone through the process of updating the Terraform version from 0.12.x to 1.2.x

Backup your state

Even if you have state versioning, it's better to have a backup. For example, if you are using an AWS S3 backend, you can replicate the entire bucket containing your state or states to another S3 bucket in a different region. Even with the protection that AWS S3 provides, it's better to have a couple of separate copies of the state for disaster situations.

Don't create big states

Large states are very inconvenient to use:

Building a plan takes too long.
Applying changes takes too long.
Simultaneous work is difficult:
- Many people want to make changes in one state and have to wait for other people's state plan and apply.
- A broken state paralyzes work with infrastructure as code.

It's better to use multiple small states with small amounts of resources. For example:

State for VPC (Amazon Virtual Private Cloud)
State for EKS (Amazon Elastic Kubernetes Service)
State for RDS (Amazon Relational Database Service) instance and infrastructure for it ()
State for SES (Simple Email Services in AWS)
etc.

If a state becomes broken, the blast radius will be minimal. Each folder containing Terraform configurations can be tracked separately in pipelines and executed only when there are changes.
This also allows people to avoid using the --target option during terraform apply. People tend to use this option when they know about the problems in the state. They would rather not spend much time waiting for the pipeline, especially not wanting to see errors caused by resources they haven't touched.

Always run Terraform in pipelines, store changes in a version control system

Try to avoid situations where you need to make changes locally and run Terraform on your workstation. Always use pipelines. This approach ensures that all changes (plan/apply outputs) are visible, and if you need to ask for help, it will be best to have all the necessary information in one place.

Don't use destroy

The opposite of the apply command in Terraform is destroy. Supporting the destroy command in pipelines is a difficult task. This command is rarely used. Instead, when you want to delete the entire infrastructure, particularly with small Terraform states that contain only a few items, simply delete everything that describes resources or calls modules, and leave the .tf files empty. After git commit and reviewing the plan to verify everything, you can just use the same apply pipeline. Don't worry about leaving an empty state stored in an S3 bucket; it doesn't take up much space, but it simplifies your workflow and allows you to look back at the history of changes if needed.

Don't be afraid to use third-party modules

Use already existing Terraform modules rather than writing your own:

It takes less time; you just need to understand how to use third-party modules, as almost all of them provide examples.
You don't need to maintain your modules, write tests, or manage the right versioning to enable rolling updates in your infrastructure.
Using third-party modules and blueprints, you can get help from people who have already encountered problems with them and found solutions.
Additionally, most modules navigate potential pitfalls and strive for universality; using these modules can offer flexibility in implementing changes or adopting new features.

Automated checks for new versions of modules, providers, and Terraform itself using Renovate

As the modern world moves faster and faster, you need to stay up-to-date with your modules, providers, and Terraform versions. This is not only because new features appear and bugs are fixed, but also because many Terraform providers change their APIs, remove deprecated features, and so on. Therefore, you have to be prepared, especially for certain Terraform codes that you run only a few times a year (e.g., for VPC).
In that case, Renovate can help you get information about new module and provider updates. For example, I had a case where Terraform stopped working because the AWS provider was deprecated; we were using Terraform version 0.12.21, and the AWS provider version was 2.20.0, with the state containing everything in the AWS region. Consequently, I had to update Terraform to version 1.1.7 and the AWS provider to 4.6.0. Therefore, pay attention to warnings about deprecation; you wouldn't want to find out that you can't use Terraform, especially at the most crucial moment

There is a good video about using renovate for that purpose:

Always check the migration guide for modules and providers when updating to a major version.

You can avoid many problems by reading migration guides, which are provided by the developers of modules and providers. Even if you are not concerned about infrastructure and recreating resources doesn’t frighten you, the process can halt in the middle of an update, leaving you with inconsistent state and infrastructure. Spending five minutes reading a migration guide can help you avoid hours spent resolving issues.

There is another fundamental way to do it: delete all resources (refer to the section where I discuss not using 'destroy') and the infrastructure, and then recreate it. This approach will work if you:

Are not concerned about maintaining your existing infrastructure.
Seek an easier way to update across more than one major version of a module or provider.

GitHub and Issue Terraform module's and provider's Issue tracker are your best friends

Whether or not you use third-party modules, the issue trackers of providers and modules are the second-best source of information after official documentation. They have often helped me solve issues. This has replaced the need for Googling and reading articles, which usually just provide straightforward solutions. Simply search for your error or issue in the search box, and you'll often find that someone else has already encountered this problem. Even if the issue persists, you might find workarounds that help solve the problem, or gather ideas on what to do next.
Even if you don’t use third-party modules, the source code can still be useful as a source of standard solutions or ideas for writing your own. Checking it can save you a lot of time, but be mindful of code licenses when copying code.

Maintain an infrastructure managed by Terraform that is as close as possible to the production infrastructure

It's not always possible, but it's better to have a place where you can test your changes. Sometimes, even if your changes seem safe and the Terraform plan appears unsuspicious, they can still cause difficulties. At times, the provider may have a bug, and it's better to discover this by applying changes to a less critical infrastructure rather than risking damage to the production one

Practical Approaches for Navigating Unclear Documentation in Terraform

Sometimes the documentation for a module or provider is not obvious, and it's unclear how to achieve specific goals. In that case, you can apply changes manually (remember the previous advice about having a separate infrastructure for tests), and then simply review the Terraform plan. In most cases, you just need to add parameters that are changed in the plan to your Terraform file, and everything will work. However, occasionally, you may need to use the import command. Both scenarios help you to practically understand what exactly happened and how to handle it.

Conclusion

In the ever-evolving landscape of infrastructure management, the lessons I learn from experience are invaluable. Throughout this article, we've navigated through a series of crucial practices for using Terraform effectively in production environments. From the importance of versioning and backing up your state, to the wisdom of avoiding large states and incorporating third-party modules, each point underscores a broader principle: the need for vigilance, foresight, and adaptability in managing our digital infrastructure.

As Terraform continues to evolve, so too must our strategies for leveraging its power. Remember, the goal isn't just to avoid errors but to create a robust, efficient, and scalable infrastructure that can withstand the tests of time and change. Embracing tools like Renovate for staying updated, adhering to migration guides, and keeping an eye on issue trackers are not just best practices, but essential habits for any infrastructure manager seeking excellence.

Finally, it's important to note that while Terraform provides us with a powerful toolset, the real strength lies in the community and the shared knowledge we gain from each other. Articles, forums, and discussions like this are not just for solving today's challenges but are stepping stones towards a more resilient and dynamic future in infrastructure management.

As we continue our journey with Terraform, let's carry these lessons forward, always looking for new ways to refine our approach, adapt to new challenges, and share our insights with the community. After all, the most powerful tool in our arsenal is the collective wisdom and experience we share.

Preparing for a DevOps Engineer Interview: A Comprehensive Guide

Dmitrii Kotov — Sat, 06 May 2023 16:43:09 +0000

Introduction

Landing a job as a DevOps engineer can be a rewarding and fulfilling experience. However, acing the interview requires thorough preparation, a deep understanding of DevOps principles, and the ability to demonstrate your technical and interpersonal skills. In this guide, we will walk you through the essential steps to prepare for a DevOps engineer interview. We will cover essential topics, provide links to resources, and share tips to help you succeed in your job search.

1. Understand the DevOps Principles and Practices

First and foremost, you must have a solid understanding of the core principles and practices that underlie the DevOps philosophy. Familiarize yourself with the following concepts:

Continuous Integration (CI)
Continuous Deployment (CD)
Infrastructure as Code (IAC)
Configuration Management
Monitoring and Logging
Containerization and Orchestration
Cloud Computing Platforms (AWS, Azure, GCP)
Microservices Architecture

Resources to learn DevOps principles and practices:

The DevOps Handbook (https://www.amazon.com/DevOps-Handbook-World-Class-Reliability-Organizations/dp/1942788002)
The Phoenix Project (https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592)
Introduction to DevOps: Transforming and Improving Operations (https://www.edx.org/course/introduction-to-devops-transforming-and-improving-operations): This free online course is offered by the Linux Foundation through edX. It covers the fundamentals of DevOps, including continuous integration, continuous deployment, and infrastructure as code. The course is self-paced and provides a solid foundation in DevOps principles.

2. Master Relevant Tools and Technologies

As a DevOps engineer, you'll be expected to work with a variety of tools and technologies. Here are some popular ones that you should be familiar with:

Version Control: Git, GitHub, GitLab, Bitbucket
CI/CD Tools: Jenkins, Travis CI, CircleCI, GitLab CI/CD, AWS CodePipeline
Configuration Management: Ansible, Puppet, Chef, SaltStack
Infrastructure as Code: Terraform, AWS CloudFormation, Azure Resource Manager
Containerization: Docker, containerd
Container Orchestration: Kubernetes, Docker Swarm, Amazon ECS
Monitoring and Logging: Prometheus, Grafana, ELK Stack (Elasticsearch, Logstash, Kibana), Splunk
Cloud Platforms: Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP)

Resources to learn tools and technologies:

Git and GitHub (https://www.codecademy.com/learn/learn-git)
Docker Mastery (https://www.udemy.com/course/docker-mastery/)
Kubernetes (https://kubernetes.io/docs/tutorials/kubernetes-basics/)
Ansible for the Absolute Beginner (https://www.udemy.com/course/learn-ansible/)
Terraform Getting Started (https://learn.hashicorp.com/collections/terraform/aws-get-started)

3. Brush Up on Programming and Scripting Languages

As a DevOps engineer, you'll need to be proficient in at least one scripting language for automation and configuration tasks. Common languages used in DevOps include Python, Ruby, Bash, and PowerShell. Familiarity with a programming language such as Go or Java can also be advantageous.

Resources to learn programming and scripting languages:

Learn Python (https://www.codecademy.com/learn/learn-python)
Learn Go (https://www.codecademy.com/learn/learn-go)
Bash Scripting Tutorial (https://linuxconfig.org/bash-scripting-tutorial-for-beginners)
PowerShell (https://docs.microsoft.com/en-us/powershell/scripting/learn/)

4. Study System Design and Networking Concepts

A strong understanding of system design and networking concepts is essential for a DevOps engineer. Be prepared to discuss topics such as load balancing, horizontal scaling, caching strategies, fault tolerance, and network protocols during the interview.

Resources to learn system design and networking concepts:

System Design Primer (https://github.com/donnemartin/system-design-primer)
Scalability, Availability, and Stability Patterns (http://www.slideshare.net/jboner/scalability-availability-stability-patterns)
High Scalability Blog (http://highscalability.com/)
Computer Networking: A Top-Down Approach (https://www.amazon.com/Computer-Networking-Top-Down-Approach-7th/dp/0133594149)

5. Prepare for Behavioral Questions

In addition to technical questions, you should also be prepared for behavioral questions that assess your ability to work in a team, handle challenging situations, and adapt to change. Use the STAR (Situation, Task, Action, and Result) method to structure your answers and provide concrete examples from your past experiences.

Resources for behavioral questions:

25 Behavioral Interview Questions (https://www.amtec.us.com/blog/behavioral-interview-questions)
How to Answer Behavioral Interview Questions (https://www.indeed.com/career-advice/interviewing/how-to-answer-behavioral-interview-questions)

6. Review Sample DevOps Interview Questions

Familiarize yourself with common DevOps interview questions to help you practice and refine your answers.

Resources for sample DevOps interview questions:

Top DevOps Interview Questions (https://www.edureka.co/blog/interview-questions/top-devops-interview-questions/)
DevOps Interview Questions and Answers (https://www.simplilearn.com/tutorials/devops-tutorial/devops-interview-questions)
DevOps Interview Questions (https://www.interviewbit.com/devops-interview-questions/)
Top 50+ DevOps Interview Questions in 2023 (https://intellipaat.com/blog/interview-question/devops-interview-questions/?US)

7. Practice Coding and Problem Solving

Although DevOps interviews may not focus heavily on coding and algorithms, you may still be asked to solve problems or write code during the interview. Practice your problem-solving skills using platforms like LeetCode, HackerRank, and Codewars.

Resources for practicing coding and problem-solving:

LeetCode (https://leetcode.com/)
HackerRank (https://www.hackerrank.com/)
Codewars (https://www.codewars.com/)

8. Participate in Mock Interviews

Mock interviews are an excellent way to build confidence and improve your interview skills. Enlist the help of friends, colleagues, or online platforms like Pramp to practice answering questions in a realistic interview setting.

Resources for mock interviews:

Pramp (https://www.pramp.com/)
Gainlo (http://www.gainlo.co/)

Conclusion:

Preparing for a DevOps engineer interview takes time and dedication. By following this comprehensive guide, you'll be well-equipped to tackle the technical and behavioral aspects of the interview process. Remember to review the key DevOps principles, master relevant tools and technologies, and practice your problem-solving and coding skills. Finally, be prepared to demonstrate your knowledge of system design, networking concepts, and Amazon Leadership Principles during the interview. Good luck!

Migrate Molecule for Ansible pipeline from Travis.ci to Github Actions

Dmitrii Kotov — Sun, 07 Feb 2021 13:09:58 +0000

I decided to migrate from Travis.ci to GitHub actions because I had spent all credits and couldn't renew them.

Before migration my ci/cd configuration look like:

---
language: python
python: '3.6'

env:
  global:
    - MOLECULEW_USE_SYSTEM=true
  matrix:
    # Spin off separate builds for each of the following versions of Ansible
    - MOLECULEW_ANSIBLE=2.8.18
    - MOLECULEW_ANSIBLE=2.9.16
    - MOLECULEW_ANSIBLE=2.10.4

# Require Ubuntu 20.04
dist: focal

# Require Docker
services:
  - docker

install:
  # Install dependencies
  - ./moleculew wrapper-install

  # Display versions
  - ./moleculew wrapper-versions

script:
  - ./moleculew test

branches:
  only:
    - master
    - /^(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)([\.\-].*)?$/

notifications:
  webhooks: https://galaxy.ansible.com/api/v1/notifications/

And the molecule configuration:

---
dependency:
  name: galaxy

driver:
  name: docker

lint: |
  set -e
  yamllint .
  ansible-lint
  flake8
platforms:
  - name: ansible-role-oh-my-zsh-debian-9
    image: debian:9
  - name: ansible-role-oh-my-zsh-debian-10
    image: debian:10
  - name: ansible-role-oh-my-zsh-ubuntu-16.04
    image: ubuntu:16.04
  - name: ansible-role-oh-my-zsh-ubuntu-18.04
    image: ubuntu:18.04
  - name: ansible-role-oh-my-zsh-ubuntu-20.04
    image: ubuntu:20.04
  - name: ansible-role-oh-my-zsh-centos-7
    image: centos:7
  - name: ansible-role-oh-my-zsh-centos-8
    image: centos:8
  - name: ansible-role-oh-my-zsh-fedora-32
    image: fedora:32
  - name: ansible-role-oh-my-zsh-fedora-33
    image: fedora:33
  - name: ansible-role-oh-my-zsh-opensuse-15.1
    image: opensuse/leap:15.1
  - name: ansible-role-oh-my-zsh-opensuse-15.2
    image: opensuse/leap:15.2

provisioner:
  name: ansible

verifier:
  name: testinfra

These files available in my repository and actual for release version 2.4.1

Following GitHub Actions setup guide, I've created a file structure for my future pipeline

  .github/
    workflows/
      ci.yml

And take like the example one's of geerlingguy repository I've made a action ci file:

---
name: CI
'on':
  pull_request:
  push:
    branches:
      - master
jobs:
  test:
    name: Molecule
    runs-on: ubuntu-latest
    strategy:
      matrix:
        ansible:
          - 2.8.18
          - 2.9.16
          - 2.10.4
    steps:
      - name: Check out the codebase.
        uses: actions/checkout@v2
      - name: Set up Python 3.
        uses: actions/setup-python@v2
        with:
          python-version: '3.x'
      - name: Install test dependencies.
        run: pip3 install ansible==${{ matrix.ansible }}  molecule[docker] docker testinfra yamllint ansible-lint flake8
      - name: Run Molecule tests.
        run: molecule test
        env:
          PY_COLORS: '1'
          ANSIBLE_FORCE_COLOR: '1'

Let's look closer at each part of this file.

First is the configuration that describes when we will be run our pipeline. It'll be run on all pull requests or push to master branch

---
name: CI
'on':
  pull_request:
  push:
    branches:
      - master

Next, is the jobs description. In my case, there is only one test. I gave the name for this job and setup os version ubuntu-latest. And the matrix part for the strategy is similar to the Travis.ci syntax.

jobs:
  test:
    name: Molecule
    runs-on: ubuntu-latest
    strategy:
      matrix:
        ansible:
          - 2.8.18
          - 2.9.16
          - 2.10.4

Steps:

    steps:
      - name: Check out the codebase.
        uses: actions/checkout@v2
      - name: Set up Python 3.
        uses: actions/setup-python@v2
        with:
          python-version: '3.x'
      - name: Install test dependencies.
        run: pip3 install ansible==${{ matrix.ansible }}  molecule[docker] docker testinfra yamllint ansible-lint flake8
      - name: Run Molecule tests.
        run: molecule test
        env:
          PY_COLORS: '1'
          ANSIBLE_FORCE_COLOR: '1'

This step executes the Github action that checkout repository.

      - name: Check out the codebase.
        uses: actions/checkout@v2

The next one installs ansible.

      - name: Set up Python 3.
        uses: actions/setup-python@v2

And step that installs dependencies.

      - name: Install test dependencies.
        run: pip3 install ansible==${{ matrix.ansible }}  molecule[docker] docker testinfra yamllint ansible-lint flake8

ansible==${{ matrix.ansible }} - installs different ansible version for each environment.

The final step runs molecule

      - name: Run Molecule tests.
        run: molecule test
        env:
          PY_COLORS: '1'
          ANSIBLE_FORCE_COLOR: '1'

PY_COLORS: '1': This forces Molecule to use the colorized output in the CI environment. Without it, all the output would be white text on a black background.
ANSIBLE_FORCE_COLOR: '1': This does the same thing as PY_COLORS, but for Ansible’s playbook output.

DEV Community: Dmitrii Kotov

Why Idempotence Matters in CI/CD Pipeline Build Steps

Predictability and Stability:

Avoiding Wasting Resources:

Preventing Errors:

Saving Time with Caching:

Easier Debugging:

Why shouldn’t we rebuild the container every time?

Kubernetes Interview Questions: Kubernetes Architecture: Node

What are Nodes in Kubernetes?

Who manages each node in a Kubernetes cluster?

What are the key components on a node in Kubernetes?

What are the two main ways to add Nodes to the API server in Kubernetes?

What happens after a Node object is created or a kubelet self-registers in Kubernetes?

When is a node considered eligible to run a Pod in Kubernetes?

What happens if a node is not healthy in Kubernetes?

How can you stop health checking on an invalid Node in Kubernetes?

What is the requirement for the name of a Node object in Kubernetes?

What issues can arise if a Node instance is modified without changing its name?

What should be done if a Node needs to be replaced or updated significantly in Kubernetes?

What is the preferred pattern for the registration of Nodes in Kubernetes?

What is a good practice when Node configuration needs to be updated in Kubernetes?

What issues can arise if the Node configuration is changed on kubelet restart while Pods are already scheduled on the Node?

Why is Node re-registration important in the context of updating Node configuration?

How can you create and modify Node objects in Kubernetes?

What should you set the kubelet flag to when creating Node objects manually?

What does marking a node as unschedulable do?

How do you mark a Node as unschedulable in Kubernetes?

What is a special case where Pods will still run on an unschedulable Node?

What information does a Node's status contain in Kubernetes?

How can you view a Node's status and other details in Kubernetes?

What is the purpose of heartbeats sent by Kubernetes nodes?

What are the two forms of heartbeats for nodes in Kubernetes?

What is the role of the node controller in Kubernetes?

What is the first role of the node controller in a node's life?

What happens if the node controller finds that the VM for an unhealthy node is not available?

What is one of the responsibilities of the node controller regarding node health?

What action does the node controller take if a node remains unreachable?

How long does the node controller wait before submitting the first eviction request for an unreachable node?

How often does the node controller check the state of each node by default?

Can the period for checking the state of each node by the node controller be configured?

What is the default eviction rate limit set by the node controller?

How does the node eviction behavior change when a node in an availability zone becomes unhealthy?

What happens to the eviction rate if the fraction of unhealthy nodes is at least --unhealthy-zone-threshold?

What is the node eviction policy in small clusters?

How is the eviction rate adjusted in larger clusters with unhealthy nodes?

Does the node controller consider per-zone unavailability if the cluster does not span multiple cloud provider availability zones?

What happens to the eviction rate if all nodes in a zone are unhealthy?

What is the node controller's policy for evictions in cases where all zones are completely unhealthy?

What type of information about resource capacity do Node objects track in Kubernetes?

How do Nodes that self-register report their capacity?

What does the Kubernetes scheduler ensure regarding resources on a Node?

How does the scheduler determine if a Node has enough resources?

What is excluded from the sum of requests when the scheduler checks a Node's capacity?

Where can you find information about reserving resources for non-Pod processes?

What does the kubelet do during a node system shutdown in Kubernetes?

How does the kubelet handle pods during a node shutdown?

What does the Graceful node shutdown feature depend on?

Is the GracefulNodeShutdown feature gate enabled by default in Kubernetes?

How is the node marked during a shutdown, and how does the kube-scheduler respond?

How are pods terminated by the kubelet during a graceful shutdown?

What happens to the Node and Pods if a node termination is cancelled?

How are pods marked and shown in kubectl when evicted during a graceful node shutdown?

What are the two phases of pod shutdown in the Graceful Node Shutdown feature?

What happens to pods that are part of a StatefulSet during an undetected node shutdown?

Why can't the StatefulSet create a new pod with the same name during an undetected node shutdown?

What happens to the volumes used by pods during an undetected node shutdown?

What are the two phases of pod termination during a non-graceful shutdown?

What is a potential warning regarding the use of swap memory in Kubernetes?

How can a user configure how a node uses swap memory?

What is the support status of swap with different cgroup versions?

What does the process of safely draining a node involve?

What does kubectl drain do?

How do you drain a node with DaemonSet pods?

What should you do with the node during maintenance and after it's completed?

Can you drain multiple nodes in parallel?

What alternative is there to using kubectl drain for evictions?

What information does a Node's status contain in Kubernetes?

What are the different types of Addresses found in a Node's status?

How are taints related to node conditions in Kubernetes?

What happens to the eviction rate if the fraction of unhealthy nodes is at least `--unhealthy-zone-threshold`?

How are pods marked and shown in `kubectl` when evicted during a graceful node shutdown?

What does `kubectl drain` do?

What alternative is there to using `kubectl drain` for evictions?