DEV Community: Luqman Bello

Designing an Internal Developer Platform (IDP) on AWS Using Backstage, EKS & Terraform

Luqman Bello — Mon, 24 Nov 2025 10:33:25 +0000

Introduction — Why IDPs Are Becoming Mandatory

If you've worked in engineering long enough, you already know the truth nobody wants to say out loud:

Most developers don’t want to touch YAML, Terraform, Kubernetes manifests, or IaC pipelines. They just want their service to run.

And honestly… can we blame them?
Between debugging a failing pod, hunting for IAM permissions, or figuring out why Terraform refuses to plan, developer energy gets drained on infrastructure instead of innovation.

That’s where Internal Developer Platforms (IDPs) come in.

IDPs are the “self-service layer” that removes friction.
The best IDPs allow developers to:

scaffold services
spin up infrastructure
deploy to multiple environments
observe their workloads
and do all this without knowing how the underlying platform works

Today, you’ll learn how to build such a platform using:

✔ Backstage (developer portal)
✔ Terraform (self-service IaC)
✔ ArgoCD (GitOps)
✔ Amazon EKS (runtime platform)
✔ AWS native services like IAM, KMS, Secrets Manager

This is the same pattern used at modern engineering orgs and unicorns.

So let’s build a real one, the right way.

The Four Layers of a Modern IDP

Every successful platform has the same 4 layers:

1. Developer Experience Layer  → Backstage
2. Platform Orchestration      → Terraform + GitOps (ArgoCD)
3. Runtime Infrastructure      → Amazon EKS + networking
4. Shared Services             → Observability, security, secrets, CI

Let’s break them down.

1. Developer Experience Layer — Backstage

Backstage gives you:

💠 Software Catalog

A single source of truth for all services:

Ownership
Deployments
Documentation
Tech stack
Status

💠 Golden Paths (Software Templates)

This is the real magic.

Instead of telling developers:

“Hey, here’s the Confluence page with our recommended architecture. Follow it.”

You give them a self-service template that does everything:

Create a Git repo
Generate CI/CD pipelines
Create Terraform infrastructure
Register service in Backstage
Deploy to EKS
Wire up monitoring & alerts

With one click.

2. Platform Orchestration — Terraform + GitOps

This is where your platform’s power comes from.

🟦 Terraform modules

These represent reusable “building blocks”:

microservice module
VPC module
RDS module
DynamoDB table module
S3 bucket module
EKS namespace module

Developers never write Terraform.
They fill in a simple form in Backstage → Backstage generates Terraform config → GitOps takes over → ArgoCD deploys it.

🟥 GitOps (ArgoCD)

This ensures:

everything is deployed from Git
automatic rollbacks
drift detection
clear audit trails
separation between build (CI) and deploy (CD)

Git is the source of truth.
ArgoCD is the engine.

3. Runtime Infrastructure — Amazon EKS

This is where workloads actually run.

Your EKS architecture includes:

Multi-tenant namespaces
IRSA (IAM Roles for Service Accounts)
Karpenter for fast autoscaling
Network Policies
Pod Security Standards
Ingress controller (ALB/Nginx/Ambassador)
Internal service mesh (optional)

EKS is your “runtime engine”, Backstage is the steering wheel, and GitOps is the autopilot.

4. Shared Services Layer

This includes all the “invisible” but necessary pieces:

✔ Secrets

AWS Secrets Manager + IRSA.

✔ Observability

Prometheus
Grafana
Tempo / Loki / OTel Collector

✔ Security

KMS encryption
Audit logs
IAM boundaries
Registry scanning

✔ CI Pipelines

GitHub Actions or GitLab CI.
CI builds → GitOps deploys.

Full Architecture flow diagram

🧬 Golden Path Template Example (Backstage)

Here’s a simplified template:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: aws-eks-service
  title: "Create a Service on EKS"
  description: "Scaffold a microservice with Terraform + GitOps"
spec:
  owner: platform-team
  parameters:
    - title: "Service Info"
      properties:
        name:
          type: string
          description: Name of the service
        owner:
          type: string
          description: Team name

  steps:
    - id: template
      name: Create Repo from Template
      action: fetch:template

    - id: terraform
      name: Terraform Module
      action: publish:github
      input:
        repoUrl: "{{ parameters.name }}"
        files:
          - terraform/main.tf

    - id: register
      name: Register in Backstage Catalog
      action: catalog:register

📦 Terraform Module Example (EKS Namespace)

module "namespace" {
  source = "git::https://github.com/org/platform-modules.git//eks-namespace"

  name      = var.service_name
  team      = var.team_name
  iam_roles = var.iam_roles
}

🔁 GitOps Flow (ArgoCD App)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myservice
spec:
  source:
    repoURL: https://github.com/org/myservice
    path: deploy/
  destination:
    namespace: myservice
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Real-World Challenges & Lessons Learned

No one tells you these things until you experience them:

Templates need constant updating
Teams will bypass your platform unless you make it genuinely delightful
Backstage plugin versioning can be chaotic
Keep the IDP simple: don’t over-engineer
EKS upgrades require strict compatibility planning
Terraform module contracts must be stable and versioned

But once it works, developer onboarding drops from weeks to minutes.

Final Thoughts — The Future of IDPs

The next evolution of IDPs will include:

AI-powered troubleshooting
Auto-generated infrastructure from natural language
Predictive autoscaling
Auto-remediation
Learning-based deployment risk scoring
AI-assisted documentation & code generation

And you’ll be ahead of the curve because you’re already building the foundation today.

Mastering Amazon EKS Auto Mode: Let Your Cluster Drive Itself (So You Can Work on the Fun Stuff)

Luqman Bello — Tue, 21 Oct 2025 11:10:59 +0000

Cloud‑native adoption has turned us into part‑time cluster mechanics. We spend evenings wrestling with YAML files, debugging tangled Helm charts and praying that our rollout doesn’t collide with a surprise Kubernetes upgrade. Wouldn’t it be nice to hand over the keys and let someone else handle the oil changes and tyre rotations?

That’s exactly what Amazon EKS Auto Mode promises. Announced at re:Invent 2024, it lifts much of the day‑to‑day operational burden from platform teams. In this article, I’ll unpack what it is, how it works, and why it might just give you back your evenings.

Why We Needed Auto Mode

If you’ve been building on Kubernetes for more than a few months, you know the drill: patching clusters, managing controllers, scaling nodes, and performing version upgrades. During an interview at re:Invent, Barry Cooks (AWS VP for Kubernetes) noted that customers increasingly ask AWS to “do more for us” because they simply don’t have time to manage clusters themselves. The platform engineering to‑do list is never empty, and the more mission‑critical your applications become, the less appealing it is to babysit infrastructure.

That’s why managed services like EKS exist: they abstract away control plane operations. Auto Mode takes this a step further by extending that abstraction into the data plane and cluster lifecycle tasks. Instead of worrying about which controller to install or when to apply security patches, you let AWS handle it for you.

What Is EKS Auto Mode?

EKS Auto Mode is a managed configuration of EKS designed to make Kubernetes almost “serverless.” When you opt in, AWS will:

Install and manage common controllers (such as autoscalers and networking add‑ons) for you.
Handle version lifecycle management and deliver one‑click upgrades.
Provision and patch EC2 nodes on your behalf, taking care of CVEs while still letting you benefit from on‑demand, reserved, or Spot pricing.

In essence, Auto Mode expands the “managed” surface area of EKS. You still interact with Kubernetes through the standard APIs, but the underlying mechanics of keeping the cluster healthy, secure, and up‑to‑date are handed over to AWS.

Under the Hood: Karpenter & the CNCF

If you’re familiar with Karpenter, AWS’s open‑source cluster autoscaler, Auto Mode will feel like a natural evolution. Karpenter dynamically provisions right‑sized nodes based on your workloads. In 2024, AWS donated Karpenter to the CNCF, signalling a commitment to open governance. Auto Mode runs Karpenter behind the scenes: when your workloads scale up or down, Karpenter decides what EC2 instances to launch and when to terminate them. You get the benefit of efficient autoscaling without having to maintain the tool yourself.

The diagram above illustrates the idea: your pods connect to a central service that provisions nodes and controllers on the fly. You focus on building and deploying containers while Auto Mode orchestrates the underlying compute and networking resources.

Getting Started with Auto Mode

If you are ready to take your hands off the wheel, here’s a high‑level overview of how to enable Auto Mode on a new cluster:

Create an EKS cluster with Auto Mode enabled. You can use the AWS CLI or eksctl. For example:

eksctl create cluster \
  --name my-auto-cluster \
  --region us-east-1 \
  --managed \
  --auto-mount-service-account-token=true \
  --enable-karpenter \
  --enable-auto-mode

The --enable-auto-mode flag tells EKS to handle controller installation and node lifecycle management for you.

Grant necessary IAM permissions. Auto Mode needs a service role that allows it to provision and patch EC2 instances. Make sure your cluster role includes AmazonEKSClusterPolicy, AmazonEKSServicePolicy, and AmazonEC2ContainerRegistryReadOnly.
Deploy your workloads as usual. Use kubectl or your GitOps pipeline to apply manifests. Karpenter and the EKS control plane will watch your pods and ensure the right amount of compute is available.
Enjoy simplified upgrades. When new Kubernetes versions are released, Auto Mode provides a single‑click upgrade path in the console or a one‑line command via the AWS CLI.

Use Cases & Predictions

It's not every use case that this is applicable for. It is particularly appealing for:

Small teams or startups without dedicated SREs.
Edge or hybrid deployments, where you want a consistent cluster experience across environments.
AI/ML workloads that require rapid scaling up and down.

According to an article from fariwinds.com, Industry analysts expect Karpenter adoption to explode in the coming years, and predict that organizations will consolidate clusters, experiment more with multi‑cloud and hybrid strategies, and even migrate away from legacy virtualization platforms. Auto Mode sits at the intersection of these trends: it abstracts away low‑level operations while still giving you the flexibility to run in multiple environments.

Potential Caveats

Auto Mode isn’t a one‑size‑fits‑all solution. You may still choose standard EKS if you need:

Fine‑grained control over add‑ons and node groups. Auto Mode manages common controllers automatically; if your architecture depends on niche or highly customized components, you might prefer manual control.
On‑premises custom networking. Hybrid Nodes and Auto Mode aim to bring on‑prem to the cloud, but networking requirements can be complex.
Regulatory or compliance controls that require explicit sign‑off on upgrades and patches.

To round this up, EKS Auto Mode isn’t a silver bullet—but it is a powerful tool in the ever‑expanding AWS toolbox. By delegating routine cluster maintenance to AWS, you free up valuable time to focus on what truly matters: delivering features, improving reliability, and delighting your users.

As with any new service, the best way to evaluate Auto Mode is to try it. Spin up a test cluster, deploy a sample workload, and see how it feels to let your infrastructure drive itself. Then come back and share your experiences because the community learns faster when we learn together. Arigato 🙇🏽

Building Production-Grade ECS Anywhere Infrastructure with Custom Capacity Providers

Luqman Bello — Fri, 01 Nov 2024 14:40:09 +0000

Hey there, fellow infrastructure engineers! If you're reading this, you're probably looking to level up your container game with ECS Anywhere. Maybe you've got a hybrid infrastructure setup, or perhaps you're dealing with edge locations that need the same love as your cloud workloads. Whatever brought you here, I've got you covered.

In this guide, we'll dive deep into building a production-ready ECS Anywhere infrastructure with custom capacity providers. But don't worry – we'll keep it real and focus on practical, battle-tested approaches that you can actually use.

What We'll Cover

Why ECS Anywhere (And Why Should You Care?)
The Architecture That Actually Works
Setting Things Up (The Right Way)
Custom Capacity Providers (The Secret Sauce)
Making It Production-Ready
When Things Go Wrong (And They Will)

Why ECS Anywhere?

Let's be honest – not everything belongs in the cloud. Whether you're dealing with regulatory requirements, existing infrastructure investments, or edge computing needs, sometimes you need to run containers outside AWS. That's where ECS Anywhere comes in.

Here's what makes it interesting:

☁️ Cloud Benefits + On-Prem Control = ECS Anywhere

But here's what they don't tell you in the basic tutorials: the default capacity provider might not cut it for production workloads. That's why we're building a custom one.

The Architecture That Actually Works

Before we dive into the code, let's talk architecture. Here's what we're building:

Why this setup? Because it:

Keeps your ops team sane with unified management
Handles real-world scaling scenarios
Doesn't fall apart under pressure

Setting Things Up

First things first. Here's what you'll need:

# Don't just copy-paste this - make sure you understand each part
aws --version  # Needs v2.13.0+

# Create your cluster with external capacity provider
aws ecs create-cluster \
    --cluster-name prod-hybrid \
    --capacity-providers EXTERNAL

🚨 Pro Tip: Always use a test environment first. I learned this the hard way when I accidentally scaled down production instances. Not fun.

The Secret Sauce: Custom Capacity Provider

This is where things get interesting. Here's a custom capacity provider that actually works in production:

class CustomCapacityProvider {
  constructor(private config: CapacityProviderConfig) {
    // Trust me, you want these logs
    this.setupLogging();
  }

  async evaluateCapacity(): Promise<void> {
    try {
      const metrics = await this.getMetrics();

      // Don't just check CPU - that's a rookie mistake
      if (this.needsScaling(metrics)) {
        await this.scaleCluster(metrics);
      }
    } catch (error) {
      // You'll thank me for this error handling later
      this.handleScalingError(error);
    }
  }

  private needsScaling(metrics: ClusterMetrics): boolean {
    // Real-world scaling logic that won't wake you up at 3 AM
    return metrics.cpuUtilization > 70 || 
           metrics.memoryUtilization > 80 ||
           metrics.pendingTasks > 0;
  }
}

Here's what makes this implementation special:

It handles edge cases (literally, if you're running at the edge)
It won't flap like a fish out of water during traffic spikes
It logs what you actually need to debug issues

Making It Production-Ready

Now, let's talk about what it takes to make this production-ready. Here are some battle-tested patterns:

Monitoring That Actually Helps

class MetricsPublisher {
  async publishMetrics(): Promise<void> {
    await cloudwatch.putMetricData({
      Namespace: 'ECS/CustomCapacityProvider',
      MetricData: [
        {
          // These are the metrics you'll actually look at
          MetricName: 'FailedTaskAllocation',
          Value: this.getFailedTaskCount(),
          Unit: 'Count'
        },
        // Add more metrics that matter
      ]
    }).promise();
  }
}

💡 Real Talk: Don't just monitor everything. Monitor what matters. My team once spent hours chasing a "problem" that turned out to be a noisy metric.

Security That Makes Sense

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:RegisterExternalInstance",
        "ecs:DeregisterExternalInstance"
      ],
      "Resource": "*",
      "Condition": {
        // This condition saved us during a security audit
        "StringEquals": {
          "aws:ResourceTag/Environment": "Production"
        }
      }
    }
  ]
}

When Things Go Wrong

Because they will. Here's your survival guide:

Common Issues I've Hit (So You Don't Have To)

The "Missing Instance" Problem

# First, check if SSM Agent is actually running
sudo systemctl status amazon-ssm-agent

# If it's not, here's the fix
sudo systemctl restart amazon-ssm-agent

The "Scaling Won't Stop" Issue

class ScalingManager {
  private async applyBackoff(): Promise<void> {
    // This backoff strategy saved our bacon during a traffic spike
    const backoffMinutes = Math.min(
      this.failureCount * 2,
      30
    );
    await this.wait(backoffMinutes);
  }
}

Real-World Debugging

Here's a debugging flow that's saved me countless hours:

Check the ECS agent logs
Verify Systems Manager connectivity
Look for capacity provider events
Check your custom metrics

# The holy grail of debugging commands
aws ecs describe-container-instances \
    --cluster prod-hybrid \
    --container-instances $INSTANCE_ID

Lessons Learned

After running this in production for a while, here are some key takeaways:

Start Small: Don't try to boil the ocean. Get a basic setup working and iterate.
Monitor Wisely: Focus on actionable metrics. Nobody wants another noisy dashboard.
Automate Recovery: Because nobody wants to SSH into servers at 3 AM.
Document Everything: Your future self will thank you.

Wrapping Up

Building a production-grade ECS Anywhere infrastructure isn't just about following AWS documentation. It's about understanding your workloads, planning for failure, and building systems that can be maintained by humans.

Remember:

Test thoroughly (seriously)
Start with a simple capacity provider
Add complexity only when needed
Keep those logs meaningful

What's Next?

If you're looking to take this further, consider:

Implementing cross-region failover
Adding custom metrics for your specific use case
Building automated testing for your capacity provider

Got questions? Hit me up in the comments. I'd love to hear about your ECS Anywhere adventures!

P.S. If you found this helpful, I'd love to hear about your implementation stories. Drop a comment below!

The Ultimate Comprehensive Guide to Setting Up Self-Hosted Airbyte on EKS Using Karpenter

Luqman Bello — Tue, 17 Sep 2024 12:13:56 +0000

Table of Contents

Introduction
Prerequisites
Setting Up EKS with Karpenter
- Creating an EKS Cluster
- Installing Karpenter
Deploying Airbyte on EKS
- Configuring Airbyte Resources
- Deploying Airbyte
Upgrading Airbyte OSS Version
Monitoring and Maintenance
Troubleshooting Common Errors

Introduction

Airbyte is an open-source data integration platform that enables you to consolidate your data from various sources to data warehouses, lakes, and databases. This 2024 guide walks you through setting up a production-ready, self-hosted Airbyte instance on Amazon EKS (Elastic Kubernetes Service) using Karpenter for dynamic node provisioning. The setup includes best practices for security, scalability, and maintenance.

Prerequisites

AWS Account: An active AWS account with appropriate permissions
AWS CLI: Version 2.13.0 or later
kubectl: Version 1.28 or later
Helm: Version 3.12 or later
eksctl: Version 0.163.0 or later
Basic knowledge: Kubernetes, AWS, and command-line operations

Setting Up EKS with Karpenter

Creating an EKS Cluster

Create a modern EKS cluster using eksctl with the latest supported version:

eksctl create cluster \
  --name airbyte-cluster \
  --region us-west-2 \
  --version 1.28 \
  --with-oidc \
  --zones us-west-2a,us-west-2b \
  --nodegroup-name standard-workers \
  --node-type t3.large \
  --nodes 2 \
  --nodes-min 1 \
  --nodes-max 4 \
  --managed \
  --addons aws-ebs-csi-driver,vpc-cni,coredns,kube-proxy \
  --node-volume-size 100 \
  --external-dns-access

Key changes from previous versions:

Updated to EKS 1.28 (latest stable)
Added essential EKS add-ons
Increased default node size to t3.large for better performance
Added larger root volume for containers
Enabled external DNS access for better integration

Installing Karpenter

Create AWS IAM Resources for Karpenter

Using the latest Karpenter IAM setup:

   export CLUSTER_NAME="airbyte-cluster"
   export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
   export AWS_REGION="us-west-2"

   curl -fsSL https://karpenter.sh/v0.32.0/getting-started/getting-started-with-eksctl/cloudformation.yaml  > cloudformation.yaml

   aws cloudformation deploy \
     --stack-name "karpenter-${CLUSTER_NAME}" \
     --template-file cloudformation.yaml \
     --capabilities CAPABILITY_NAMED_IAM \
     --parameter-overrides "ClusterName=${CLUSTER_NAME}"

Install Karpenter Controller

   helm repo add karpenter https://charts.karpenter.sh
   helm repo update

   helm install karpenter karpenter/karpenter \
     --namespace karpenter --create-namespace \
     --version v0.32.0 \
     --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::${AWS_ACCOUNT_ID}:role/karpenter-controller-${CLUSTER_NAME} \
     --set settings.aws.clusterName=${CLUSTER_NAME} \
     --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
     --set settings.aws.interruptionQueueName=karpenter-${CLUSTER_NAME} \
     --set controller.resources.requests.cpu=1 \
     --set controller.resources.requests.memory=1Gi \
     --set controller.resources.limits.cpu=1 \
     --set controller.resources.limits.memory=1Gi

Create a Modern Provisioner

Create provisioner.yaml:

   apiVersion: karpenter.sh/v1beta1
   kind: NodePool
   metadata:
     name: default
   spec:
     template:
       spec:
         requirements:
           - key: "kubernetes.io/arch"
             operator: In
             values: ["amd64"]
           - key: "kubernetes.io/os"
             operator: In
             values: ["linux"]
           - key: "karpenter.sh/capacity-type"
             operator: In
             values: ["on-demand"]
           - key: "node.kubernetes.io/instance-type"
             operator: In
             values: ["t3.large", "t3.xlarge", "m5.large", "m5.xlarge"]
         nodeClassRef:
           name: default
     limits:
       resources:
         cpu: 100
         memory: 100Gi
     disruption:
       consolidationPolicy: WhenUnderutilized
       consolidateAfter: 30s
     weight: 10
---
apiVersion: karpenter.sh/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  roleClassName: default
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
  tags:
    karpenter.sh/discovery: ${CLUSTER_NAME}

Apply the configuration:

   kubectl apply -f provisioner.yaml

Deploying Airbyte on EKS

Configuring Airbyte Resources

Create Namespace and Storage Class

   kubectl create namespace airbyte

   cat <<EOF | kubectl apply -f -
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: airbyte-storage
   provisioner: ebs.csi.aws.com
   parameters:
     type: gp3
     encrypted: "true"
   reclaimPolicy: Retain
   allowVolumeExpansion: true
   volumeBindingMode: WaitForFirstConsumer
   EOF

Set Up Persistent Storage

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: airbyte-pvc
     namespace: airbyte
   spec:
     storageClassName: airbyte-storage
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 50Gi

Deploying Airbyte

Add Airbyte Helm Repository

   helm repo add airbyte https://airbytehq.github.io/helm-charts
   helm repo update

Create Custom Values File

Create airbyte-values.yaml:

   global:
     serviceAccountName: airbyte-admin
     database:
       secretName: airbyte-db-secrets
     logs:
      storage:
        enabled: true

   webapp:
     resources:
       limits:
         cpu: "1"
         memory: 2Gi
       requests:
         cpu: "0.5"
         memory: 1Gi

   server:
     resources:
       limits:
         cpu: "2"
         memory: 4Gi
       requests:
         cpu: "1"
         memory: 2Gi

   worker:
     resources:
       limits:
         cpu: "2"
         memory: 4Gi
       requests:
         cpu: "1"
         memory: 2Gi

   persistence:
     enabled: true
     storageClass: airbyte-storage
     size: 50Gi

   metrics:
     enabled: true
     serviceMonitor:
       enabled: true

Install Airbyte

   helm install airbyte airbyte/airbyte \
     --namespace airbyte \
     --values airbyte-values.yaml \
     --version 0.50.0

Verify Deployment

   kubectl get pods -n airbyte
   kubectl get svc -n airbyte

Upgrading Airbyte OSS Version

Backup Before Upgrade

   # Backup using Velero or similar tool
   velero backup create airbyte-backup \
     --include-namespaces airbyte

Update Helm Repository

   helm repo update
   helm search repo airbyte/airbyte --versions

Upgrade Airbyte

   helm upgrade airbyte airbyte/airbyte \
     --namespace airbyte \
     --values airbyte-values.yaml \
     --version <NEW_VERSION> \
     --timeout 15m

Monitoring and Maintenance

Set Up Monitoring

   # Install Prometheus and Grafana
   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update

   helm install prometheus prometheus-community/kube-prometheus-stack \
     --namespace monitoring \
     --create-namespace

Configure Log Aggregation

Use AWS CloudWatch Logs:

   kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/cloudwatch-logs/cloudwatch-logs-configmap.yaml

Troubleshooting Common Errors

Pod Scheduling Issues

Symptom: Pods stuck in Pending state
Solution: Check Karpenter logs and node pool configuration:

  kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -f

Database Connection Issues

Symptom: Database connection errors in Airbyte logs
Solution: Verify secrets and connection strings:

  kubectl get secrets -n airbyte
  kubectl logs -n airbyte deployment/airbyte-server

Resource Constraints

Symptom: OOMKilled or CPU throttling
Solution: Adjust resource limits in values.yaml:

  resources:
    limits:
      memory: "4Gi"
      cpu: "2"
    requests:
      memory: "2Gi"
      cpu: "1"

Karpenter Node Provisioning Issues

Symptom: Nodes not being provisioned
Solution: Check NodePool and EC2NodeClass configurations:

  kubectl describe nodepool default
  kubectl describe ec2nodeclass default

This guide provides a robust foundation for running Airbyte on EKS with Karpenter. Remember to regularly check for updates and security patches for all components. For production environments, consider implementing additional security measures and backup strategies.

For questions, issues, or suggestions, please leave a comment below!

Managing AWS EKS Clusters Locally Using Lens

Luqman Bello — Tue, 10 Oct 2023 14:48:41 +0000

Kubernetes has undoubtedly become the go-to container orchestration platform, but managing its clusters can often be complex and tedious. This is where Lens comes in. Often dubbed as the "Kubernetes IDE,".

Lens is more than just a Kubernetes IDE; it's your one-stop-shop for effortless cluster management, offering a unique blend of usability and functionality. The tool shines exceptionally when working with Amazon Elastic Kubernetes Service (EKS), providing an intuitive and user-friendly interface to manage and monitor your clusters. In this article, we'll walk through the steps required to manage an AWS EKS cluster using Lens from a local machine.

Prerequisites

An active AWS account with the requisite permissions.
AWS CLI installed and configured
kubectl installed
EKS cluster up and running
Lens IDE installed on your local machine

Step-by-step guide:

Setting up AWS CLI and Authenticating:

Before we connect Lens to EKS, ensure your AWS CLI is authenticated and has access to the EKS cluster.

aws configure

Fill in your AWS Access Key ID, Secret Access Key, default region name, and default output format when prompted.
For a more detailed guide on how to setup AWS CLI, you can use Zacks Blog

Setting Up Your EKS Cluster

If you haven't got an EKS cluster running, you can set one up using the eksctl tool, as shown below:

eksctl create cluster --name my-cluster --region us-west-2 --nodegroup-name my-nodes --nodes 2 --nodes-min 1 --nodes-max 3 --managed

Here, my-cluster is your cluster name, and my-nodes is your node group. The cluster will be created in the us-west-2 AWS region with two EC2 instances. Feel free to adjust these as per your needs.

Installing Lens

You can download Lens from its official website and follow the installation guide for your respective operating system.

Connecting the Dots: kubectl and EKS

Before you can manage your EKS cluster with Lens, make sure your kubectl is configured to communicate with it. Run the following command to accomplish this:

aws eks update-kubeconfig --name my-cluster

This will update your Kubernetes configuration file (~/.kube/config), paving the way for Lens to connect to your EKS cluster.

Let's Dive into Lens

Connecting Lens to EKS

Start Lens: Open the Lens application and navigate to the cluster management view.
Add Cluster: Click on the "+" icon to initiate the addition of a new cluster.
Select kubeconfig: Choose the updated ~/.kube/config file.
Connect: After the cluster loads, hit the "Connect" button.

Navigating the Interface

Once connected, you'll be greeted with a wealth of options. From nodes, pods, and services to deployments and config maps, Lens makes it easier to manage them all.

Resource Management: CRUD Operations

Create: Use the "Create" button to upload a YAML configuration file or manually input the YAML data to create a new Kubernetes resource.
Update: Simply click on the resource you wish to update, make the required changes, and save.
Delete: To remove a resource, select it and click on the trash bin icon.

The Lens Advantage: Monitoring & Observability

Lens comes equipped with built-in monitoring features, offering real-time insights into your EKS cluster. Metrics like CPU utilization, memory usage, and disk I/O can be accessed straight from the dashboard.

Advance Features

Lens isn't just about monitoring; it's a full-fledged Kubernetes management IDE. You get features like:

1. Real-time cluster metrics
2. Direct terminal access to nodes and pods
3. An extensive catalog of extensions for more functionalities

Locking It Down: Security Aspects

Lens provides critical security features like Role-Based Access Control (RBAC) and Secrets management to ensure that your EKS cluster is well-secured.

To wrap up, Lens offers a compelling and intuitive way to manage your AWS EKS clusters, making it a must-have tool for anyone in the DevOps, Cloud Computing, or Kubernetes ecosystem. Its ease of setup and rich feature set make it an excellent choice for cluster management.

For further queries or discussion, feel free to drop a comment below. If you found this guide helpful, consider sharing it with your network!

Harnessing the Power of AWS Workflow Studio for DevOps Automation

Luqman Bello — Fri, 29 Sep 2023 16:27:40 +0000

Introduction:

In the dynamic realm of DevOps, the ability to streamline operational processes is invaluable. AWS Workflow Studio emerges as a compelling tool, offering a visual workflow designer to automate tasks and orchestrate AWS services seamlessly. This article delves into the key features of AWS Workflow Studio, its integrative capabilities with other AWS services, and the remarkable ease it brings to the DevOps arena.

Getting Started with AWS Workflow Studio:

AWS Workflow Studio provides a user-friendly interface to design, test, and deploy workflows. Even without extensive AWS Step Functions knowledge, DevOps professionals can navigate through the platform, designing workflows that automate complex operational tasks.

Visual Interface:

The drag-and-drop interface abstracts the complexity of underlying configurations, making it easier to visualize and design workflows.

AWS Services Integration:

AWS Workflow Studio facilitates seamless integration with other AWS services like AWS Lambda, SNS, and SQS. This integrative ability enhances the creation of robust automation solutions, like setting up notification systems or processing pipelines.

Deployment:

Once a workflow is designed, deploying it is straightforward. AWS Workflow Studio automatically generates the necessary AWS Step Functions code, which can be deployed directly or with minor modifications as per requirements.
The DevOps Impact:
The introduction of AWS Workflow Studio augments the DevOps toolkit in multiple ways:

Troubleshooting and Optimization:

The visual representation of workflows simplifies the troubleshooting process, helping to identify bottlenecks or errors swiftly.

Documentation:

Visual workflows act as self-documenting processes, promoting a better understanding among the team and ensuring everyone is on the same page regarding operational workflows.

Collaboration:

By providing a clear visualization of workflows, AWS Workflow Studio fosters better collaboration among development, operations, and other cross-functional teams.

Operational Efficiency:

Streamlining the design and deployment of workflows significantly enhances operational efficiency, freeing up time and resources for other critical tasks.

Anticipated Challenges and Mitigation Strategies:

While AWS Workflow Studio is a robust tool, integrating it into existing DevOps workflows might present challenges. Ensuring a smooth transition requires a well-thought-out strategy. One potential challenge could be the learning curve for professionals new to AWS Workflow Studio. Providing training sessions and creating comprehensive documentation can mitigate this challenge. Additionally, while the automatic generation of AWS Step Functions code accelerates workflow deployment, it may not always align with specific organizational standards or complex workflow requirements. A thorough review and possibly manual tweaking of the generated code might be necessary to ensure compliance and optimal performance.

Conclusion:

AWS Workflow Studio is more than just a workflow designer; it's an enabler for DevOps professionals striving for operational excellence. The ease of designing, testing, and deploying workflows, coupled with the seamless integration with other AWS services, makes it a noteworthy addition to the DevOps toolkit.

Using Amazon’s Kubernetes Distribution Everywhere with Amazon EKS Distro

Luqman Bello — Sat, 06 Aug 2022 19:25:00 +0000

Using Kubernetes in the public cloud is easy. Especially when using managed services like Amazon Elastic Kubernetes Service (Amazon EKS). EKS is one of the most popular Kubernetes distros. When using EKS, there is no need to administer examination level nodes, etcd nodes or other examination level components. This simplicity allows you to focus on your application. But in real life scenarios, you sometimes need to run Kubernetes clusters in on-premises environments. Maybe because of the regulation restrictions, compliance requirements or you may need the lowest latency when accessing your clusters or applications.

There are so many Kubernetes Distributions out there. Most of them are CNCF certified. But that means you need to choose from many options, try and implement one that suits your needs. Check the security part of your distro or find a suitable tool for your deployment. In other words, we all need standardization.

In December 2020, AWS announced the distribution of EKS. EKS Distro is a Kubernetes distribution built and powered by Amazon EKS managed, allowing you to deploy secure and reliable Kubernetes clusters in any environment. EKS Distro allows you to use the same tools and the Kubernetes version and dependencies with EKS. You also don't have to worry about security patches for the distribution because with each release of the EKS distribution you also get the latest patches and the EKS distribution follows the same EKS process to check Kubernetes versions. This means that you always use reliable and tested Kubernetes distributions in your environment.

EKS Distro is an open source project on GitHub. You can check out the repository at this link. https://github.com/aws/eks-distro/

You can install EKS Distro on metal servers or virtual machines in your data center or even in the environment of other public cloud providers. Unlike EKS, when using EKS Distro, you have to manage all control level nodes, nodes, etc. and control level components yourself. This comes with some additional operational burden, but is a huge benefit if you don't have to think about the security or reliability of your Kubernetes deployment.

As you can see from the screenshot above each EKS Deployment Option has its own features. On the right column there are options and features of EKS Distro. As I’ve mentioned before when using EKS Distro you need to have your own infrastructure and you need to manage the Control Plane. Also you can use different 3rd party CNI Plugins according to your needs. Biggest difference is that unlike EKS Anywhere, there are no Enterprise Support offerings from AWS with EKS Distro.

The project is on GitHub and supported by the Community. When you have any problems or when you want to contribute to the projects you can file an issue or find solutions from the previous issues on the repository.

Let’s see EKS Distro in action!

When installing EKS Distro, you can choose a launch partner’s installation options or you can use familiar community options like kubeadm or or kops.

I will demonstrate the installation of EKS Distro with kubeadm in this blog post.

First of all, for the installation with kubeadm, you need an RPM-based Linux system. I am using a CentOS system for this demonstration.

I have installed Docker 19.03 version, disabled swap and disabled SELinux on the machine.

I will install kubelet, kubectl and kubeadm with the commands below on the machine. I will install the 1.19 version of the Kubernetes in this demonstration.

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

cd /usr/bin

sudo wget https://distro.eks.amazonaws.com/kubernetes-1-19/releases/4/artifacts/kubernetes/v1.19.8/bin/linux/amd64/kubelet; \
sudo wget https://distro.eks.amazonaws.com/kubernetes-1-19/releases/4/artifacts/kubernetes/v1.19.8/bin/linux/amd64/kubeadm; \
sudo wget https://distro.eks.amazonaws.com/kubernetes-1-19/releases/4/artifacts/kubernetes/v1.19.8/bin/linux/amd64/kubectl

sudo chmod +x kubeadm kubectl kubelet

sudo systemctl enable kubelet

After enabling kubelet service, I am adding some arguments for kubeadm.

sudo mkdir /var/lib/kubelet

sudo vi /var/lib/kubelet/kubeadm-flags.env

KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd —network-plugin=cni —pod-infra-container-image=public.ecr.aws/eks-distro/kubernetes/pause:3.2"

I will pull the necessary EKS Distro container images and tag them accordingly.

sudo docker pull public.ecr.aws/eks-distro/kubernetes/pause:v1.19.8-eks-1-19-4;\
sudo docker pull public.ecr.aws/eks-distro/coredns/coredns:v1.8.0-eks-1-19-4;\
sudo docker pull public.ecr.aws/eks-distro/etcd-io/etcd:v3.4.14-eks-1-19-4;\
sudo docker tag public.ecr.aws/eks-distro/kubernetes/pause:v1.19.8-eks-1-19-4 public.ecr.aws/eks-distro/kubernetes/pause:3.2;\
sudo docker tag public.ecr.aws/eks-distro/coredns/coredns:v1.8.0-eks-1-19-4 public.ecr.aws/eks-distro/kubernetes/coredns:1.7.0;\
sudo docker tag public.ecr.aws/eks-distro/etcd-io/etcd:v3.4.14-eks-1-19-4 public.ecr.aws/eks-distro/kubernetes/etcd:3.4.13-0

I will add some other configurations as well.

sudo vi /etc/modules-load.d/k8s.conf

br_netfilter

sudo vi /etc/sysctl.d/99-k8s.conf

net.bridge.bridge-nf-call-iptables = 1

Now let’s initialize the cluster!

sudo kubeadm init --image-repository public.ecr.aws/eks-distro/kubernetes --kubernetes-version v1.19.8-eks-1-19-4

This output is mostly the same as the usual kubeadm init command output. As you can see from the screenshot, the output has the kubeadm join command for the worker nodes or the configuration for accessing the cluster with the kubeconfig file. By the way, let me do that and access my Kubernetes cluster installed with EKS Distro.

sudo mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Let’s run kubectl get nodes command and see the output.

As you can see, I am able to connect the cluster and see the kubectl get nodes command output but node is in NotReady status. The reason is I need to add a Pod Network addon and install a CNI Plugin to the cluster. I will use Calico CNI for this demonstration.

sudo curl https://docs.projectcalico.org/manifests/calico.yaml -O
kubectl apply -f calico.yaml

After installing the Calico CNI, my master node is now in Ready state.

I have configured the worker nodes with the same prerequisites like installing Docker and disabling swap. I will pull and tag the necessary container image for the Kubernetes cluster as well with these commands.

sudo docker pull public.ecr.aws/eks-distro/kubernetes/pause:v1.19.8-eks-1-19-4;\
sudo docker tag public.ecr.aws/eks-distro/kubernetes/pause:v1.19.8-eks-1-19-4

I can now move on with adding a worker node to the cluster. I will use the kubeadm join command from the kubeadm init command output.

When I run the kubectl get nodes command, I can see the other node in Ready state.

As you can see my worker node has now joined the cluster and I can see the pods in the kube-system namespace.

My Kubernetes cluster is installed with EKS Distro and ready for deploying application workloads!

Conclusion

Having a tested, verified and reliable Kubernetes Distribution for production workloads is extremely crucial. This is why EKS is one of the most used and most popular Kubernetes distribution. Being able to run the same Distribution Amazon uses with managed EKS service on any infrastructure and platform is a huge advantage.

If you have some compliance requirements or regulation restrictions and can not use public cloud platforms you can absolutely give EKS Distro a try.

Amazon Elastic Container Registry (Amazon ECR)

Luqman Bello — Sun, 29 Aug 2021 22:18:57 +0000

Amazon Elastic Container Registry (Amazon ECR) is an Amazon Web Service (AWS) product that stores, manages and deploys Docker images, which are managed clusters of Amazon EC2 instances. Amazon ECR allows all AWS developers to save configurations and quickly move them into a production environment, thus reducing overall workloads.

Amazon ECR provides a command-line interface (CLI) and APIs to manage repositories and integrated services, such as Amazon Elastic Container Service (Amazon ECS), which installs and manages the infrastructure for these containers. The primary difference between Amazon ECR and ECS is that while ECR provides the repository that stores all code that has been written and packaged as a Docker image, the ECS takes these files and actively uses them in the deployment of applications.

A developer can use the Docker command line interface to push or pull container images to or from an AWS region. Amazon ECR can be used wherever a Docker container service is running, including on-premises environments. AWS Elastic Beanstalk also supports Amazon ECR for multi-container environments.

How Amazon ECR works

First, Amazon Elastic Container Registry writes and packages code in the form of a Docker image. Next, it compresses, encrypts and manages access to the images -- including all tags and versions -- and controls image lifecycles. Finally, the Amazon ECS pulls the necessary Docker images from the ECR to be used in the deployment of apps and continues to manage containers everywhere -- including Amazon Elastic Kubernetes Service (Amazon EKS), AWS cloud and on premise networks.

Furthermore, Amazon ECR automatically encrypts container images at rest with Amazon Simple Storage Service (Amazon S3) server-side encryption and allows administrators to use AWS Identity and Access Management (AWS IAM) to create restrictions that limit access to each repository. The container registry stores container images in S3 for high availability.

Components of Amazon ECR

Amazon ECR includes:

Docker images - This is the file that is used to execute code within a Docker container.
Repository - The Docker images are stored in the Amazon ECR repository. Developers can push and pull images to the repository.
Repository policy - Developers can use these policies to manage access to the repositories and the images within them.
Registry - All AWS accounts receive access to Amazon ECR that allows them to create repositories and store images in them.
Authorization token - Before it can push and pull images, the Docker client must be recognized as an AWS account holder.

Amazon ECR security and other benefits

One of the greatest benefits provided by Amazon ECR is increased security. All images in Amazon ECR are transferred over HTTPS. Images at rest are automatically encrypted to ensure enhanced security. As mentioned before, developers can use AWS IAM to create policies that control permissions and manage access to images. This can be done without altering credentials directly on the EC2 instances. Policies can also be designed to control cross-account image sharing.

AWS security groups can be selected for the interface that control whether each host is allowed to interact with the interface. AWS security groups are virtual firewalls at the instance level that are easily created, attached and deleted. For example, there may be a security group assigned to all the EC2 instances in a cluster using an AWS Auto Scaling group. Developers can create the rule that allows the Amazon Virtual Private Cloud (Amazon VPC) endpoint to be accessed by all instances in this assigned security group.

Other benefits of Amazon ECR include:

High availability - The Amazon ECR architecture is highly scalable, durable and redundant. As a result, the Docker images are easily available and accessible and users can feasibly and dependably deploy new containers for their applications.
Streamlines workflow - Integration with Amazon ECS and the Docker CLI allows users to simplify their development and production work processes by facilitating continuous integration (CI) and continuous deployment (CD) in Amazon ECS. Furthermore, container images can be easily pushed to Amazon ECR with the Docker CLI. From there, Amazon ECS can easily pull the images directly and use them for production deployments.
Completely managed - Amazon ECR does not include any software that will need to be installed and managed or an infrastructure that has to be scaled. Users simply push images to ECR and pull them with any container management tool when they're needed.

Configuration and implementation of Amazon ECR

An AWS account is the first necessity when setting up Amazon ECR. When a user registers for an AWS account, they automatically get signed up for all of the services; they will only pay for the services they use.

Once the user has an AWS account, they can download the AWS Command Line Interface (AWS CLI) and Docker software.

All services in AWS require users to provide credentials in order to determine whether or not the user has permission to access the protected resources. The AWS console requires a password, however, use of AWS credentials is not recommended when accessing AWS. Instead, AWS IAM is recommended for a more secure authentication process. An AWS IAM user can access AWS using a special URL and their unique user credentials.

Amazon WorkDocs

Luqman Bello — Sun, 29 Aug 2021 21:49:17 +0000

Amazon WorkDocs is a fully managed, secure content creation, storage, and collaboration service. With Amazon WorkDocs, you can easily create, edit, and share content, and because it’s stored centrally on AWS, access it from anywhere on any device. Amazon WorkDocs makes it easy to collaborate with others, and lets you easily share content, provide rich feedback, and collaboratively edit documents. You can use Amazon WorkDocs to retire legacy file share infrastructure by moving file shares to the cloud. Amazon WorkDocs lets you integrate with your existing systems, and offers a rich API so that you can develop your own content-rich applications. Amazon WorkDocs is built on AWS, where your content is secured on the world's largest cloud infrastructure.

With Amazon WorkDocs, there are no upfront fees or commitments. You pay only for active user accounts, and the storage you use.

Benefits

Migrate your on premise file servers and reduce costs significantly
Securely share with internal teams and external users in real-time
Secure your content in the cloud
Bring content into your applications and processes
Route your documents using approval workflow
Extend your desktop to the cloud

Some of the use case of the Amazon WorkDocs is in helping to replace expensive legacy file sharing services. This system are mostly referred to as ECM solutions. With Amazon WorkDocs, you can easily migrate existing content from legacy network file shares to the cloud and your users can continue to access all their individual and team’s shared content from their native desktop file systems through WorkDocs Drive, or through the web user interface or mobile application.