DEV Community: kelvin manavar

Crafting the Perfect Golden AMI for Auto Scaling Groups in AWS

kelvin manavar — Mon, 22 Sep 2025 17:54:43 +0000

If you’ve ever worked with Auto Scaling Groups (ASGs) in AWS, you know the magic they bring, instances that launch, heal, and scale automatically based on demand. But here’s the catch: the speed and reliability of your scaling process often depends on the quality of the AMI (Amazon Machine Image) behind it.

This is where golden AMIs prove their value: reliable, pre-configured images built for production. Instead of launching a raw instance and installing everything on the fly, you spin up servers that are already dressed, polished, and ready to serve traffic.

So, how do you create one that truly deserves to be called golden? Let’s break it down.

Start with the OS Base Image 🖥️

The foundation matters. Choose a lightweight, secure, and well-supported base image - Amazon Linux 2, Ubuntu LTS, or RHEL are the usually chosen. Keep it patched to the latest updates before you bake your own version. No one wants to scale vulnerable servers.

Package & Dependency Installation 📦

What goes into the image depends on your philosophy:

Pre-install required software and configure (e.g., Nginx or Apache, PHP, Node.js, python, Java, etc.).
Install monitoring/logging agents (e.g., CloudWatch agent, SSM agent, AWS CodeDeploy agent).
Install AWS CLI

Application Code / Artifacts 👨‍💻

Decide whether to bake the application inside the AMI or fetch it dynamically:

Immutable pattern: Bake application code + dependencies in AMI.
Mutable pattern: Use UserData or CodeDeploy to pull the latest version during launch.

Security Hardening and Clean Up 🔐

Security isn’t optional. Before you turn an instance into an AMI:

Disable root SSH login.
Remove unnecessary users and packages.
Configure proper firewall rules (if required).
Apply CIS/benchmark hardening (if compliance required).
Clear out temporary files and old logs.
Remove existing SSH host keys (/etc/ssh/ssh_host_*) to avoid duplicate keys. This ensures that each instance feels unique, not a clone with security baggage.
Use cloud-init or systemd to generate new keys at boot.
Ensure the EC2 instance connect / key-pair login works as expected.

# Cleanup before AMI bake
sudo rm -f /etc/ssh/ssh_host_*
# Clean apt cache
sudo apt clean
sudo rm -rf /var/lib/apt/lists/*
# Clear bash history
history -c

Don’t Forget Cloud-Init 🧠

There are two ways to use below script to make sure that unique SSH host keys generated for every new instance.

When launching the EC2 via AWS Console put cloud-init script in to User data.
While working with auto scaling group setup we can specify script at user data when we create Launch Template (LT) or Launch Configuration (LC) for the ASG.

#cloud-config
ssh_deletekeys: true
ssh_genkeytypes:
  - rsa
  - ecdsa
  - ed25519
runcmd:
  - systemctl restart ssh

Note: If we remove existing SSH host keys then we must use cloud-init script otherwise we can not ssh in to new EC2 instances .

Performance Tuning 🛠️

Set tuned profiles or sysctl optimizations (if required).
Configure web/app server settings (PHP-FPM, Nginx, JVM heap, etc.).

Validation 🕵️

Never trust an untested image. Launch an instance from your new AMI and run through a checklist:

Ensure SSH/SSM works.
Ensure all services start automatically.
Ensure app/agents start correctly.
Ensure monitoring agent reporting data.

Only after passing these checks should you bless it as “golden.”

Final Thoughts 🎯

Golden AMIs might take a little extra effort to set up, but the payoff is huge. With a well-crafted image, your Auto Scaling Groups can launch faster, stay consistent, and reduce the risk of last-minute surprises during traffic spikes. Instead of wasting time configuring servers on the fly, you’re giving your infrastructure a head start with an image that’s already secure, patched, and production-ready.

In short, a solid golden AMI isn’t just a convenience, it’s the foundation of reliable, scalable infrastructure on AWS. Build it once, test it well, and let your Auto Scaling Groups do the rest.

Use Packer or AWS Image Builder to automate AMI creation. It makes your images reproducible and reduces human error.

Migrating from Kubernetes Ingress to Gateway API: A Step-by-Step Guide

kelvin manavar — Sun, 07 Sep 2025 04:15:53 +0000

In Kubernetes, one of the most critical tasks is managing how services inside the cluster are exposed to the outside world. For a long time, the Ingress API has been the standard way to handle HTTP(S) traffic, acting as the gateway that routes requests from external clients to the right services within the cluster. While Ingress has served its purpose well, it comes with limitations—particularly around extensibility, consistency across implementations, and support for advanced traffic management scenarios.

To address these gaps, Kubernetes introduced the Gateway API, a next-generation alternative to Ingress. Unlike Ingress, which primarily focuses on simple routing, the Gateway API provides a richer, role-oriented model. It is designed to be more expressive, flexible, and vendor-neutral, making it easier to define complex networking behaviors while maintaining a consistent user experience across different environments.

If your organization is currently relying on Ingress and considering a migration to the Gateway API, this guide will walk you through the process. We’ll explore why the Gateway API is worth adopting, what changes you need to be aware of, and the practical steps to migrate from your existing Ingress setup to the modern Gateway API within a running Kubernetes cluster.
To explore the reasons for migration, follow this link:
Link

Challenges in Migrating to Gateway API 🔥

Although the Gateway API brings powerful improvements over Ingress, transitioning to it is not always straightforward. Organizations should be prepared for the following challenges:

Controller Support – Not all Ingress controllers have full Gateway API compatibility yet. This can limit features or require choosing a new controller that natively supports Gateway.
Learning Curve – Gateway introduces new resources (Gateways, Routes, Policies) and a role-based design. Teams need time and training to adapt their workflows.
Feature Gaps – While Gateway API is more expressive, certain advanced use cases may still depend on vendor-specific extensions, reducing portability.
Operational Overhead – During migration, you may need to run both Ingress and Gateway in parallel, which increases configuration complexity and resource consumption.

Best Practices for Migrating to Gateway API 🌟

Start Small – Begin with non-critical services to test the migration workflow. This allows your team to build confidence, refine processes, and uncover challenges before moving mission-critical applications.
Leverage Role Separation – One of Gateway API’s strengths is its role-based design. Clearly define ownership: infrastructure teams can manage Gateways, while application developers focus on Routes. This separation improves security, scalability, and team efficiency.
Adopt GitOps Practices – Store Gateway and Route configurations in version control systems like Git. This enables collaboration, auditability, and quick rollbacks if issues arise during migration.
Monitor and Observe – Use logging, tracing, and monitoring tools to validate routing behavior, ensure performance, and detect anomalies early.
Stay updated – The Gateway API continues to evolve, with features like mTLS, gRPC routing, and traffic splitting being standardized. Keeping up-to-date helps your organization take advantage of new capabilities as they become stable.

Migration Strategy: Moving from Ingress to Gateway API 🎯

Migrating from Ingress to Gateway API requires planning, testing, and phased adoption. Below are the recommended steps.

Research & Planning:

Check your current setup: Collect details about existing Ingress objects, controllers, hosts, paths, TLS configurations, annotations, and backend services. kubectl get ingress -A -o yaml > all-ingresses.yaml
Select a compatible Gateway controller: The Gateway API is only a specification. you’ll need an implementation. Examples include:

NGINX ➡︎ NGINX Gateway Fabric
Traefik ➡︎ Traefik Proxy with Gateway API
HAProxy ➡︎ HAProxy kubernetes Ingress Controller with Gateway API
Istio ➡︎ Istio Gateway API

Map Ingress to Gateway API

IngressClass ➡︎ GatewayClass
Ingress ➡︎ Gateway + HTTPRoute
Review RBAC changes, as Gateway API introduces more granular access control.

Proof of Concept

Deploy Gateway API resources in a non-production namespace.
Validate routing, TLS, and service connectivity before rolling out further.

Run Both resources in Parallel

Keep Ingress for production traffic.
Deploy equivalent Gateway + HTTPRoute resources in parallel.
Use staging domains (e.g.,myapp.example.com) to validate behavior.

Migrate TLS

Unlike Ingress (which often uses annotations), Gateway API treats TLS as a first-class citizen.
Create TLS secrets and reference them in Gateway listeners.

Handle Vendor-Specific Annotations

Ingress often relies on annotations like nginx.ingress.kubernetes.io/rewrite-target. In Gateway API, these are replaced with native fields:

Header matching ➡︎ matches.headers
Path rewriting ➡︎ filters.requestRedirect, filters.urlRewrite
Traffic splitting ➡︎ backendRefs.weight Example:

rules:
- backendRefs:
  - name: v1-service
    port: 80
    weight: 80
  - name: v2-service
    port: 80
    weight: 20

Shift Production Traffic

Update DNS to point to the Gateway load balancer.
Monitor traffic using logs and metrics.
Roll back to Ingress if needed (since it still runs in parallel).

Decommission Ingress

Remove old Ingress objects.
Uninstall the Ingress controller.
Retain Gateway API resources as the single source of truth for service networking.

Migrating an existing Kubernetes Ingress resource to the Gateway API

Prerequisites ⚙️

A running Kubernetes cluster(1.24+) having deployment, services, pods running with ingress resource.

Step 1: Install Gateway API Resources

# Install Gateway API resources
kubectl kustomize "https://github.com/nginx/nginx-gateway-fabric/config/crd/gateway-api/standard?ref=v1.5.1" | kubectl apply -f -

# Verify installation
kubectl get crd | grep gateway

Step 2: Configure NGINX Gateway Fabric

We'll install NGINX Gateway Fabric as our Gateway API controller:

# Deploy NGINX Gateway Fabric CRDs
kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.1/deploy/crds.yaml

# Deploys NGINX Gateway Fabric with NGINX OSS using a Service type of NodePort
kubectl apply -f https://raw.githubusercontent.com/nginx/nginx-gateway-fabric/v1.6.1/deploy/nodeport/deploy.yaml

# Verify the deployment
kubectl get pods -n nginx-gateway

Let's configure specific ports for the gateway service:

# View the nginx-gateway service
kubectl get svc -n nginx-gateway nginx-gateway -o yaml

# Update the service to expose specific nodePort values
kubectl patch svc nginx-gateway -n nginx-gateway --type='json' -p='[
  {"op": "replace", "path": "/spec/ports/0/nodePort", "value": 30080},
  {"op": "replace", "path": "/spec/ports/1/nodePort", "value": 30081}
]'

# Verify the service has been updated
kubectl get svc -n nginx-gateway nginx-gateway

Step 3: Create GatewayClass and Gateway Resources

let's create the GatewayClass and Gateway resources. Save the following YAML to a file named gateway-resources.yaml:

---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: nginx
spec:
  controllerName: gateway.nginx.org/nginx-gateway-controller

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: nginx-gateway
  namespace: nginx-gateway
spec:
  gatewayClassName: nginx # Use the gateway class that matches your controller
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    hostname: myweb.k8s.local
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: web-tls-secret
        namespace: web-app
    allowedRoutes:
      namespaces:
        from: All
  - name: http
    port: 80
    protocol: HTTP
    hostname: myweb.k8s.local
    allowedRoutes:
      namespaces:
        from: All

Note: here, in above code we have used “web-tls-secret”. Which is already being used by Ingress resource.

# Apply the YAML: 
kubectl apply -f gateway-resources.yaml
# Verify the Gateway resource
kubectl get gateway -n nginx-gateway

Step 4: Create the HTTPRoute Resource

Create YAML file for HTTPRoute for http with name http.yaml.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web-route
  namespace: web-app
spec:
  parentRefs:
  - name: nginx-gateway
    kind: Gateway
    namespace: nginx-gateway
    sectionName: http
  hostnames:
  - myweb.k8s.local
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: web-service
      port: 80

Create YAML file for HTTPRoute for https with name https.yaml.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: web-route-https
  namespace: web-app
spec:
  parentRefs:
  - name: nginx-gateway
    kind: Gateway
    namespace: nginx-gateway
    sectionName: https
  hostnames:
  - myweb.k8s.local
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    backendRefs:
    - name: web-service
      port: 80

Note: Here, we have choose namespace=web-app at metadata of resource which is same as Deployment resource namespace.

# Apply the resource YAML:
kubectl apply -f http.yaml
kubectl apply -f https.yaml

# Verify the HTTPRoute resources
kubectl get httproute -n web-app

Step 5: Verify the Gateway API Configuration

# Check the Gateway status
kubectl describe gateway nginx-gateway -n web-app

# Check the HTTPRoute status
kubectl describe httproute web-route -n web-app

# Check if the HTTPRoute is properly bound to the Gateway
kubectl get httproute web-route -n web-app -o jsonpath='{.status.parents[0].conditions[?(@.type=="Accepted")].status}'
kubectl get httproute web-route-https -n web-app -o jsonpath='{.status.parents[0].conditions[?(@.type=="Accepted")].status}'

The output for the last command should be "True" if the HTTPRoute is properly accepted by the Gateway.
Note: If you are facing error with the listener, make sure that secret was created in the same namespace as gateway, if not, create the below reference grant.

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-gateway-to-web-app-secrets
  namespace: web-app 
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: nginx-gateway 
  to:
  - group: "" 
    kind: Secret
    name: web-tls-secret
EOF

Step 6: Test the Gateway API Configuration

Now, we are testing to see that application is accessible through the new Gateway API.

# Test the / endpoint
curl -v -H "Host: myweb.k8s.local" http://$NODE_IP:30080/

# Add the entry in /etc/hosts
# ${NODE_IP}   myweb.k8s.local

# Test the / endpoint
curl -v -k https://myweb.k8s.local:30081/

Step 7: After successful Verification Remove the Old Ingress Resource

kubectl get ingress -A
kubectl delete ingress <ingress-name> -n web-app

Final Thoughts 📝

Migrating from Ingress to Gateway API isn’t just about swapping tools. it’s about future-proofing your Kubernetes platform. With clearer roles, richer features, and fewer vendor lock-ins, Gateway API helps teams collaborate better while handling complex networking with ease. Start small, experiment, and scale gradually. Each step moves you closer to a more reliable, flexible, and modern foundation for your applications and the teams that build them.

Gateway API: The Future of Kubernetes Networking & Successor to the Ingress API

kelvin manavar — Tue, 26 Aug 2025 11:49:21 +0000

Introduction

When we talk about Kubernetes networking, one of the first things that comes up is Ingress. It’s been the go-to way of exposing applications to the outside world for years. But if you’ve worked with it, you already know the pain points: Ingress mostly focuses on HTTP/S traffic, behaves differently depending on which controller you use, and quickly becomes limiting when you need advanced routing, multi-tenancy, or protocols beyond HTTP.

This is exactly where the Gateway API steps in. Think of it as the next generation of Kubernetes networking—built to be more flexible, more consistent across implementations, and friendlier for different teams to collaborate on. It supports not just HTTP but also TCP, TLS, and gRPC, while giving you cleaner ways to manage policies and traffic routing at scale.

Gateway API aims to solve the issues we face with ingress

What is the Gateway API?

The Gateway API is a set of Kubernetes resources that bring more power and flexibility to how we manage networking. Instead of relying on rigid configurations, it allows you to dynamically provision infrastructure and set up advanced traffic routing in a clean, consistent way.

What makes it stand out is its design: it’s extensible, so it can grow with your needs; role-oriented, so different teams (like platform engineers, infra teams, and app developers) can manage their parts without conflict; and protocol-aware, meaning it works seamlessly across HTTP, TCP, TLS, gRPC, and beyond.
In short, Gateway API makes exposing and managing network services in Kubernetes smarter, more flexible, and ready for modern workloads.

Gateway API design concepts

The Gateway API was built with a few guiding principles that shape how it works and why it’s more powerful than Ingress:

Role-oriented Design: Gateway API kinds are modeled after organizational roles that are responsible for managing Kubernetes service networking:

Infrastructure Provider: Manages infrastructure that allows multiple isolated clusters to serve multiple tenants, e.g. a cloud provider.
Cluster Operator: Manages clusters and is typically concerned with policies, network access, application permissions, etc.
Application Developer: Manages an application running in a cluster and is typically concerned with application-level configuration and service composition. Portable: Gateway API resources are Kubernetes custom resources (CRDs), and they work consistently across different implementations, so you’re not locked into a single vendor.

Expressiveness: In addition to HTTP host/path matching and TLS, Gateway API can express capabilities like HTTP header manipulation, traffic weighting & mirroring, TCP/UDP routing, and other capabilities that were only possible in Ingress through custom annotations.

Extensible: The API is flexible enough to let you plug in custom resources at different layers, giving you fine-grained control and customization possible at the appropriate places within the API structure.

Key Resources in the Gateway API

Gateway API has three stable API kinds:

GatewayClass: These are cluster-wide resources that work like templates for networking. They define the behavior and configuration for Gateways created from them. If you’ve used StorageClasses in Kubernetes, it’s a similar concept—only here it’s applied to the networking data plane instead of storage.
A sample GatewayClass CRD:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: my-gateway-class
spec:
  controllerName: example.com/gateway-controller

Gateway: These are the actual instances created from a GatewayClass. You can think of them as the “real” networking components that handle traffic. A Gateway could be an in-cluster proxy, a hardware load balancer, or even a cloud provider’s load balancer. It takes the traffic from outside the cluster. Gateways are attached to respective GatewayClass.
A sample Gateway CRD:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: default
spec:
  gatewayClassName: my-gateway-class
  listeners:
    - name: http
      protocol: HTTP
      port: 80

Routes: Which are not a single resource, but represent many different protocol-specific Route resources. The HTTPRoute has matching, filtering, and routing rules that get applied to Gateways that can process HTTP and HTTPS traffic. Similarly, there are TCPRoutes, UDPRoutes, and TLSRoutes which also have protocol-specific semantics. This model also allows the Gateway API to incrementally expand its protocol support in the future.

    apiVersion: gateway.networking.k8s.io/v1
    kind: HTTPRoute
    metadata:
      name: my-httproute
      namespace: default
    spec:
      parentRefs:
        - name: my-gateway
      hostnames:
        - "example.com"
      rules:
        - matches:
            - path:
                type: PathPrefix
                value: /app
          backendRefs:
            - name: my-service
              port: 80

How the Request Flows

Let’s walk through what happens when a request goes through a Gateway acting as a reverse proxy:

A client tries to reach http://www.example.com.
The client’s DNS resolver looks up www.example.com and gets back the IP address of the Gateway.
The client sends the HTTP request to that Gateway IP. The reverse proxy receives it and checks the Host header to find the right configuration defined by the Gateway and its attached HTTPRoute.
If needed, the proxy can do extra checks like matching headers or paths based on the rules in the HTTPRoute.
It can also tweak the request, for example by adding or removing headers, using filters defined in the route.
Finally, the reverse proxy forwards the request to the right backend services.

Note: Gateway API is the successor to the Ingress API. However, it does not include the Ingress kind. As a result, a one-time conversion from your existing Ingress resources to Gateway API resources is necessary.

If you are already using ingress then want to use Gateway API, we require migrating Ingress resources to Gateway API resources.

Common Use Cases for Gateway API:

Multi-tenant clusters → Perfect for environments where multiple teams share the same cluster but still need isolation and clear boundaries.
Progressive delivery → Enables deployment strategies like canary releases or blue/green rollouts, making it easier to test changes safely before full rollout.
Beyond HTTP → Unlike traditional Ingress, Gateway API supports other protocols too—like TCP, gRPC, and WebSockets—making it a versatile choice for modern applications.

Final Thoughts:

Kubernetes has made deploying and scaling applications much easier, but networking has always been one of its tougher challenges. Ingress did the job for simple scenarios, but as apps, teams, and protocols grew more complex, its limits quickly showed.

That’s where the Gateway API comes in. It’s the next generation of Kubernetes networking-flexible, extensible, and designed for today’s workloads. With role-based design, multi-protocol support, and consistency across implementations, it brings infrastructure providers, operators, and developers onto the same page.

Whether you’re just starting with Kubernetes or planning for the future, Gateway API is more than just an upgrade to Ingress-it’s the new foundation for Kubernetes networking.

In the next article, I will write about how to migrate existing Ingress resources to Gateway API resources.