DEV Community: dejanualex

kubectl context like a pro

dejanualex — Mon, 23 Mar 2026 20:42:58 +0000

If your work involves working with multiple Kubernetes clusters, then this might be for you.

☸️ A file that is used to configure access to a cluster is called kubeconfig(usually placed at ~/.kube/config), but you can easily override the location by using --kubeconfig=<path_to_config> flag or using KUBECONFIG environment variable export KUBECONFIG=<path/to/your_kubeconfig>. A Kubernetes config file describes clusters, users, and contexts. You can use multiple contexts to target different Kubernetes clusters.

apiVersion: v1
kind: Config
preferences: {}

clusters:
- cluster:
  name: <cluster_name>
...

users:
- name: <user_name>
...

contexts:
- context:
  name: <context_name>

You can render your current config: kubectl config view.

Without further ado, let's take the example of merging two config files(one for each cluster):

# create backup for current config
cp ~/.kube/config ~/.kube/config.bak

# merge the 2 configs
KUBECONFIG=~/.kube/config:/path/to/new/config kubectl config view --flatten > ~/intermediate_config

# replace the current config with the intermediate_config
mv ~/intermediate_config ~/.kube/config

A context is a combination of a cluster and user credentials.You can:

List contexts: kubectl config get-contexts
Check the current context: kubectl config current-context
Switch to desired context: kubectl config use-context <context_name>
Delete context: kubectl config delete-context <context_name>

⚠️ Running aws eks update-kubeconfig --region <us-east-1> --name <cluster_name> pulls your cluster's API endpoint and CA data to create/update a context in your local kubeconfig. This automatically sets that cluster as your active target, so any immediate kubectl commands will execute against it.

$ aws eks update-kubeconfig --region us-east-1 --name demo-eks
...
$ kubectl config current-context
arn:aws:eks:us-east-1:255656399702:cluster/demo-eks

From Zero to EKS in Minutes

dejanualex — Mon, 09 Mar 2026 21:51:59 +0000

First, you're going to use the following CLIs as local tooling:

eksctl: to create EKS clusters in the AWS cloud or on-premises (with EKS Anywhere), as well as modifying and deleting those clusters
aws: to interact with AWS services
kubectl: manage resources within your Kubernetes cluster.

⚠️ Before creating the cluster, be sure you have an IAM identity set up in place, i.e.:

IAM User specifically for running eksctl commands.With the needed policies (AmazonEKSClusterPolicy needed for EKS cluster management, AmazonEKSWorkerNodePolicy needed for node group operations, AmazonEC2FullAccessRequired for VPC, subnets, SGs, EC2 instances, AWSCloudFormationFullAccess required by eksctl, etc.). i.e. eksctl-manager

⚠️ The user is overly broad for production, since it grants full IAM control over your entire account. For least privilege, you'd scope it down to only the actions eksctl needs.

IAM role (with AmazonEKSClusterPolicy) that AWS's EKS control plane assumes to manage AWS resources inside your account on your behalf.

For the IAM User, create an access key (select Command Line Interface (CLI) as the use case), copy the Access key and Secret access key.

On the local machine, do aws configure to set AWS Access Key ID and AWS Secret Access Key (of home user), add a section to ~/.aws/config. Now you can use those on the local machine to configure a new profile, i.e., eks-manager:

aws configure --profile eks-manager

AWS Access Key ID [None]: ....
AWS Secret Access Key [None]: ...
Default region name [None]: us-east-1

aws configure set region us-east-1 --profile eks-manager

You need a VPC before creating the EKS cluster, but one of the advantages of using eksctl is that it automatically creates a VPC (if you do not provide one).

Create demo-eks cluster in us-east-1 running Kubernetes 1.33, with:

a single managed node group,
autoscaling enabled,
OIDC configured (this enables IAM Roles for Service Accounts (IRSA), allowing individual pods to have their own IAM permissions rather than using the node's broad permissions,
ALB ingress support: It attaches the necessary IAM policies to your worker nodes to enable them to interact with an Application Load Balancer. This flag alone does not install the controller.
No SSH access to worker nodes.

eksctl create cluster \
  --profile eks-manager \
  --name demo-eks \
  --region us-east-1 \
  --version 1.33 \
  --managed \
  --nodegroup-name ng-general \
  --node-type t3.medium \
  --nodes 2 \
  --nodes-min 2 \
  --nodes-max 4 \
  --with-oidc \
  --alb-ingress-access \
  --ssh-access=false

From the networking perspective, the following resources have been created:

A VPC in the desired region us-east-1
Two subnets in each availability zone, one public subnet (for NAT GW, Internet GW, and LB) and one private (for worker nodes).
InternetGateway allows inbound and outbound traffic for resources in public subnets (VPC Edge, which provides the route for external traffic to enter the VPC)
NATGateway enabling egress traffic from private subnets (it translates the private IP of your worker node into its own public IP to reach the internet)

Finally, add the cluster to your kubeconfig: aws eks update-kubeconfig --region us-east-1 --name demo-eks --profile eks-manager

k8s debug pod

dejanualex — Tue, 20 Jan 2026 15:18:57 +0000

At various points in an Amazon EKS cluster’s lifecycle, direct access to a worker node may be required. This can be done using kubectl debug to open an interactive shell on the target node.

# start debug container
kubectl debug nodes/<nodename> --profile=sysadmin -it --image=<image>

The root filesystem of the Node will be mounted at /host.
The container runs in the host IPC, Network, and PID namespaces, although the pod isn’t privileged, so reading some process information may fail, and chroot /host may fail.
If you need a privileged pod, create it manually or use the --profile=sysadmin flag.

The sysadmin profile typically sets up the pod with:

privileged: true
hostPID: true (for node networking)
hostNetwork: true (for process visibility)
host filesystem mounted at /host

To help ensure no vulnerabilities are introduced into the cluster, an easy approach is to use something like Docker Hardened Images(since DHI repositories require authentication, I’ve mirrored Alpine in my own repository) i.e. kubectl debug nodes/node01 --profile=sysadmin -it --image=dejanualex/alpine:3.23

Next to “switch” from the container filesystem you need to chroot /host And from there you are effectively in the node’s userland, and you can use the node’s binaries:

kubectl debug creates a debug pod with a name derived from the node name, so remember to delete the pod once you’re done debugging.

Kubernetes operator AWS ECR ecr-creds-refresher

dejanualex — Sun, 21 Dec 2025 19:42:27 +0000

dejanualex for AWS Community Builders

Dec 21 '25

ECR-creds-refresher

#aws #kubernetes #elasticcontainerregistry #containers

Comments

2 min read

ECR-creds-refresher

dejanualex — Sun, 21 Dec 2025 19:40:07 +0000

Generally, the following are the main methods to authenticate to a private registry in order to be able to pull images from it:

Configuring the container runtime on each node, i.e. k3s checks if /etc/rancher/k3s/registries.yaml file exists, to retrieve the registry configuration when generating the containerd configuration to authenticate to the private registry.
Using a kubelet credential provider plugin to dynamically fetch credentials for private registries. (configure the kubelet to invoke a plugin binary to dynamically fetch registry credentials).

In this particular case, the motivation was that I had to authenticate to AWS Elastic Container Registry private repositories, and couldn’t modify the aforementioned cluster configurations. Therefore ecr-creds-refresher operator was the natural workaround.

Pre-requisites

Before obtaining the ECR authentication authorization token ( aws ecr get-login-password --region REGION | docker login --username AWS --password-stdin AWS_ACCOUNT_ID.dkr.ecr.REGION.amazonaws.com), ensure that you have an AWS user/role with the required ECR permissions and valid AWS credentials (such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or temporary credentials when assuming a role).

Demo

When attempting to spin up a pod based on an image from the private ECR repo:

kubectl run test \
--image=255656399702.dkr.ecr.us-east-1.amazonaws.com/os/alpine:latest \
--image-pull-policy=Always -- sleep 5

The pod will fail with ImagePullBackOff:

To fix this, we need:

A secret that holds the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
A mechanism to obtain the authentication token. ⚠️ Important note: An authentication token is used to access any Amazon ECR registry that your IAM principal has access to and is valid for 12 hours.

The second point is exactly what the ecr-creds-refresher will do for us, more concretely:

Upon startup, it will read the AWS credentials from the configured secret (which can be in any namespace).
Watches for ECRPullSecret custom resources (CRs), on CR Creation/Update/Resume, will fetch the ECR token from AWS, update the secret that holds the token, and patch the default ServiceAccount in the desired namespaces.
Last but not least, the operator does a periodic refresh of the ECR token and updates all secrets in the desired namespaces.

By desired namespace, I mean the namespaces in which we want to run pods that use images from private ECR repositories (this is easily configurable using the operator CustomResource).

🔄 Operator 👉 demo and repo 👉 ecr-creds-refresher.

Not All OSS Is Created Equal

dejanualex — Wed, 08 Oct 2025 10:10:32 +0000

"Open-source software is a type of computer software in which source code is released under a license in which the copyright holder grants users the rights to study, change, and distribute the software to anyone and for any purpose."

Open source doesn't just mean public access to a codebase; it indicates how the code can be used and distributed afterward. It's about opening up a living project for participation from anyone who wants to get involved. One of the key reasons companies open-source projects is that they want the community to get involved.

There are 3 dimensions to engaging with open-source software:

Consumers who study or use the repositories of others.
Contributors who are actively involved in the improvement of the repositories of others.
Producers/Maintainers who build and maintain their own repositories that are open to others.

The truth is that many projects aren't destined for open-source greatness.

While your criteria may vary based on your company's goals and process level, here are some things to consider before open-sourcing a project:

Does your project contain intellectual property that you want to protect? If so, then opening its source would give away its value. Don't open-source those kinds of projects unless you feel the benefits outweigh the risks.
Is the project in a stable state with good code quality? The project doesn't have to be perfect, but potential contributors may walk away if the project is in terrible shape to begin with.
Is your project useful to people outside of your company? If not, then you probably aren't getting any participation.
Are people outside your company able to contribute? They need access to all project dependencies, build processes, and whatever else is needed to run the project. If they can't run it, then they can't contribute.
Does your team have the bandwidth to support an open-source program? If you open-source a project and then don't support it, you might lose your opportunity to build a trusting community.

Can you change the code? Can you share it or sell it? Can you use it commercially?

A license agreement comes with the source code and specifies what can and cannot be done. The license spectrum impose different obligations from permissive to restrictive.

If you're contributing to an existing project, it's almost always easiest to continue using that project's license. To find its license, look for a file called LICENSE or COPYING, and skim the project's README.

Permissive licenses:

"Do whatever you want, just give credit"…carry minimal restrictions, and allow practically every type of use, modification, and redistribution, the only restriction is that the original attribution to the authors remains included in the source code or as part of the downstream use of the new software.

Open Source Initiative defines it as "a non-copyleft license that guarantees the freedom to use, modify, and redistribute", basically as a free software license e.g. BSD-like, MIT, Apache.

GitHub's choosealicense website describes the permissive MIT license as "[letting] people do anything they want with your code as long as they provide attribution back to you and don't hold you liable."

Restrictive licenses:

"If you modify and distribute, you must share your changes under the same license"… considered protective, as it grants use rights, and forbid proprietization, requiring that all modified/extended versions also be free and released under the same terms and conditions.

The Copyleft licenses are rather "viral" as it spreads to the entire project, Richard Stallman (GPL creator) prefers terms like "protective".

One GPL dependency could force to open-source the entire product
You find a MIT-licensed library, use it in your proprietary app, sell the app as closed-source, perfectly legal MIT doesn't care.
You find a GPL-licensed library, use it in your proprietary app, want to sell the app as closed-source 🚫not allowed, your entire app must be GPL and open-source.

GPL has a loophole, running modified GPL software on your servers, results in no obligation to share, but AGPL(GNU Affero General Public License) close this loophole. If users access your software over a network, you must share the source code,

That's why many companies avoid AGPL like the plague and often prefer MIT/Apache licenses.

GPL: "Distribute it, and you must share the source".
AGPL: "Distribute it, or use it over a network, and you must share the source

List of OSI Approved Licenses:

S̶u̶c̶c̶e̶s̶s̶f̶u̶l̶ stories: It's all about distribution

Keep in mind that about 80% of a software solution relies on existing components maintained outside of the project, and not all open-source licenses play nicely together.

⚠️ Combining Permissive + Copyleft → Copyleft wins, everything becomes copyleft.
📌 Derivative works, or future versions, of permissively licensed software can be released as proprietary software.
💸 For most licenses, obligations trigger when you distribute the software (i.e. selling the SW to customers, giving away binaries, providing downloadable apps).

AWS Spot Instances: Business Case Essentials

dejanualex — Wed, 16 Jul 2025 11:11:39 +0000

TL;DR Convince your client, organization to migrate to spot instances.

Assuming that you don't have access to tools like AWS Cost Explorer, AWS Cost and Usage Reports, or AWS Cloud Intelligence Dashboard to examine your workloads and to retrieve estimated cost and usage information.

Don't forget costs are driven by 3 factors: compute, storage, and networking (ingress is often free or at very low cost). You're going to focus on the compute side of things as being a cost driver.

Step 1 Ask for a bill or estimated bill summary.

Step 2 Identify the pricing plan (Free Tier, On-Demand, Reserved Instances, or Savings Plans). Rule of thumb use on-demand (pay-as-you-go) for flexibility, reserved instances for predictable workloads, and free tier for eligible services.

Step 3 Do a cost estimation using AWS Pricing calculator to better understand the bill. AWS Pricing Calculator does not include any taxes and the prices for AWS services vary between Regions.

Step 4 Examine the compute usage and identify suitable services and workloads. Services like EKS, OpenSearch, CloudWatch, Kinesis, and Firehose suggest stateless/fault-tolerant/bath-oriented workloads suitable for Spot Instances. Therefore EKS worker nodes, data processing jobs, CI/CD workloads or OpenSearch indexing tasks can be migrated to Spot.

Step 5 Why Spot Instances? Spot Instances enable access to unused EC2 capacity at discounts of up to 90% compared to On-Demand pricing. To improve application availability on Spot be as diverse as possible in your instance type and Availability Zone selection.

Show potential savings i.e. average interruption frequency and savings over on-demand rates over last 30 days for m6i.xlarge instances.

Important note, prepare for Spot Instance Interruption. When Amazon EC2 interrupts (reclaims) a Spot Instance by default it will terminate the instance, unless you specify a different interruption behaviour, such as stop or hibernate.

Step 6 End you business case with confidence:
"Preliminary analysis indicates that the solution could benefit from Spot Instances as a substantial opportunity for cost optimization...The absence of Spot usage reflects untapped savings potential".

Related read 👉 The art of guesstimating for estimation tips.

The art of guesstimating

dejanualex — Thu, 10 Jul 2025 22:40:01 +0000

⚠️ Disclaimer: The following post may contain biased opinions.

At some point in your career, you may be required to provide an estimate related either to cost or capacity, without having the complete view of the solution's architecture and/or infrastructure.

My simple approach to guestimates relies on the four elements:

1) Do not be afraid to make assumptions...or educated guesses
Rarely is the sweet spot (costs vs. capacity) hit; as a result, architectures tend to be overprovisioned (most of the time) or underprovisioned (typically reveals itself through system instability).

2) Pick your battle
Costs are driven by 3 factors: storage, networking (ingress is often free or at very low cost), and compute. Choose one and try to address it as best as you can.

3) Scaling is the solution
Whether using HPA, VPA, or Cluster Autoscaler, the objective is to scale down when resource utilization drops.

4) Follow the money
There’s often a tendency to overcommit (exceed allocatable resources), resulting in excessive reservations, underutilized capacity, and ultimately, cost overruns. Make use of tools whenever necessary:

Goldilocks: Provides recommendations for workload's resource requests and limits.

OpenCost: Vendor-neutral project that helps reduce cost overruns by monitoring cloud infrastructure and container costs in real time.

Karpenter: Open-source node lifecycle management project that automates provisioning and deprovisioning of nodes based on the specific scheduling needs of pods.

Keda: Scaling workloads based on events (message queues, databases, or APIs).

KubeGreen: Simple Kubernetes addon that automatically shuts down (some of) your resources when you don't need them.

💡 As a final thought, remember:

Overcommitment is not a bad thing in itself, but it works on the assumption that not all the pods will claim all of their usable resources at the same time.
Horizontal scaling is best suited for stateless workloads, and vertical scaling for stateful workloads.
HPA requires at least 1 Pod to be running at all times, so that it can collect the metrics used to inform future scale-up decisions.
Memory resource units Mi is mebibytes and M is megabytes (computers use the binary system, therefore Mi usage is preferred).
CPU limits are enforced by CPU throttling, and memory limits are enforced by the kernel with OOM kills.
Allocatable resources hold greater significance than capacity when it comes to workload placement.

TL;DR YAML anchors

dejanualex — Wed, 09 Jul 2025 09:03:56 +0000

Two main components: Anchor & which defines a chunk of configuration and Alias * used to refer to that chunk elsewhere.

Simple example, create dwarfs.yaml:

cat<<EOF>dwarfs.yaml
- &dwarf Gimli
- Dain
- Thorin
- *dwarf 
- Balin
- *dwarf 
EOF

When an YAML parser reads dwarfs.yaml the alias will render the value identified by the anchor.

An anchor can be referenced by multiple aliases, and also we can merge maps (to add more values, or override existing ones) i.e.

cat<<EOF>dwarfs.yaml
# anchor
Gloin: &dwarf_basics
  role: Member
  hood_color: white
  skills: &skills_base
    - battle
    - mining

# alias and merge map
Oin:
  <<: *dwarf_basics
  role: Leader
EOF

In dwarfs.yaml file, Gloin is being used as base definition. Oin uses << as key, which indicates that key-values from another mapping should be merged. Important note role value will be override in Oin.

[Boost]

dejanualex — Sun, 16 Feb 2025 19:53:46 +0000

The Scoop On OpenSearch sizing

dejanualex for AWS Community Builders ・ Sep 2 '24

#aws #opensearch #tutorial #elasticsearch

AWS Academy by PartyRock

dejanualex — Fri, 14 Feb 2025 23:54:05 +0000

TL;DR: PartyRock is Amazon Bedrock's playground where anyone can create generative AI-powered applications simply by describing the app they want to build, without needing to write any code.

One of the announcement from re:Invent 2024 was PartyRock's Free daily usage. Previously, PartyRock offered a free trial period for a limited time. Starting in 2025, all users will have a recurring free daily usage granted, with no credit card required.

For both newcomers or even experienced users, the vast array of services and offerings in AWS can feel overwhelming. I’ve always sought a quick entry point that provides a top-down approach to understanding and learning a new service or cloud concept.

So, why not use PartyRock to fulfil this goal? 🤔...One prompt, 4 widgets🔥🔥🔥🔥

Want to learn about EKS, its prerequisites, and installation methods? Just check app 👉 AWS-Academy

The Scoop On OpenSearch sizing

dejanualex — Mon, 02 Sep 2024 19:15:15 +0000

Search and analytics means you can search and analyze your data once it has been ingested into OpenSearch.

Getting acquainted with the terminology is mandatory when working with OpenSearch, therefore you should check OpenSearch for Humans - a friendly guide. Additionally, a simple yet concrete OpenSearch Calculator implementation.

Index size? Number of shards per index?

OpenSearch splits indices into shards. Each shard stores a subset of all documents in an index. A shard can be either primary (used for WRITE operations e.g. index, re-index, delete) or replica (used for HA and READ operations e.g. searches). OpenSearch defaults to one primary and one replica shard, for a total of two shards per index.

⚠️ Primary shards cannot be on the same node as the replica.

A shard is a piece of an OpenSearch index, each shard is a full Lucene Index, and each instance of Lucene is a running process that consumes CPU and Memory.

Avoid Oversharding and Node hot spotting

When a shard is involved in an indexing or search request, it uses the CPU to process the request. Each shard you add to an index distributes the processing of requests for that index across an additional CPU. The number of active shards that your domain can support depends on the number of CPUs in the cluster.

Node hot spotting occurs when resource utilizations are unevenly distributed across nodes, e.g. uneven JVM heap size usage. To quickly detect node hot spotting use OpenSearch API:

GET _cat/nodes?v=true&h=name,heap.current,heap.percent

As a rule of thumb, the allocated heap size should be based on the available RAM: set Xms and Xmx to the same value, and no more than 50% of your total memory. A larger Java heap size is useful for indexing, but as memory usage increases, garbage collection becomes more frequent and takes longer.

Shard Count

Shard count is secondary to shard size. Shard size matters because it impacts both search latency and write performance.
A small set of large shards uses fewer resources than many small shards (too many small shards will exhaust the memory - JVM Heap), however, on the other side, too few large shards prevent OpenSearch from properly distributing requests.
For fast indexing (ingestion), you need as many shards as possible; for fast searching, it is better to have as few shards as possible

When an index is created, the number of shards must be specified and cannot be changed later without reindexing the data. The number of shards you set for an index should correspond to the size of an index, e.g. looking at the two indices store.sizes (one which has replicas set to 0 and the other has replicas set to 1) we can observe that each replica is a full copy of a primary shard.

store.size is the store size taken by primary and replica shards.
pri.store.size is the store size taken only by primary shards.

Total number of shards

First, try to estimate the total size of the data you plan to store in the index and the retention period, and then you can calculate the total number of shards using the formula:

⚠️ Ideally, shard sizes should be between 10GB and 50GB per shard, 10–30 GB for workloads that prioritize low latency (e.g., search workloads), or 30–50 GB (e.g. logging workloads).

Number of shards per index

To prevent hot nodes, OpenSearch distributes shards to instances based on count, where each instance receives as nearly as possible the same number of shards.

Use shard counts that are multiples of the data node count to ensure that each index is distributed evenly across data nodes. To ensure an even distribution of shards across the data nodes follow the formula:

i.e. If your cluster has 4 nodes and you want to distribute the shards across all nodes evenly, your index should have 8 shards. In other words, you should have at least one shard per data node.

Tunning cluster performance: Search (read) or Ingest (write)?

Search Intensive

Try to reduce the number of shards as much as possible.
Replicas improve search performance, so you might want more if you have a read-heavy workload.

Ingest Intensive

Try to have as many shards, as possible.
Each replica duplicates the indexing process (new documents are first indexed on the primary and then on any replicas) if you anticipate heavy indexing you can temporarily set the replica count value index.number_of_replicas to 0.
Increase Index refresh frequency: Indexing documents initially place them into a memory buffer. At this stage, the documents are not yet searchable. To make these documents searchable, a refresh operation is required. OpenSearch refreshes indexes that have received at least one search request in the last 30 seconds, every 1 second.

# GET /_cluster/settings?include_defaults=true
default": {
        "index": {
          "refresh_interval": "1s"
        }
      },

This means that documents written to an active index should typically become searchable within 1 second of being written to OpenSearch. This setting can be adjusted on a per-index basis.

Keep in mind a shorter refresh interval allows documents to become searchable more rapidly post-indexing, but it does so at the expense of increased resource utilisation.

Last but not least, some key takeaways: Seven rules for OpenSearch sizing

DEV Community: dejanualex

kubectl context like a pro

From Zero to EKS in Minutes

k8s debug pod

Kubernetes operator AWS ECR ecr-creds-refresher

ECR-creds-refresher

ECR-creds-refresher

Pre-requisites

Demo

Not All OSS Is Created Equal

Permissive licenses:

Restrictive licenses:

S̶u̶c̶c̶e̶s̶s̶f̶u̶l̶ stories: It's all about distribution

AWS Spot Instances: Business Case Essentials

The art of guesstimating

TL;DR YAML anchors

[Boost]

The Scoop On OpenSearch sizing

dejanualex for AWS Community Builders ・ Sep 2 '24

AWS Academy by PartyRock

The Scoop On OpenSearch sizing

Index size? Number of shards per index?

Avoid Oversharding and Node hot spotting

Shard Count

Total number of shards

Number of shards per index

Tunning cluster performance: Search (read) or Ingest (write)?

Links: