DEV Community: Karthi Mahadevan

Kubernetes Gateway API and my experience

Karthi Mahadevan — Wed, 05 Nov 2025 17:05:01 +0000

https://gateway-api.sigs.k8s.io/
https://kubernetes.io/docs/concepts/services-networking/gateway/

The traditional Kubernetes Ingress resource has served as the primary mechanism for exposing HTTP(S) applications to the outside world. It defines host- and path-based routing rules that an Ingress Controller (like NGINX or AWS ALB Ingress Controller) translates into load balancer configurations.

While effective for simple setups, Ingress has several limitations:

Limited expressiveness — routing rules and filters are basic.

Controller-specific annotations — leading to vendor lock-in.

Difficult separation of concerns — operations and developers share the same configuration surface.

The Gateway API was designed by the Kubernetes networking SIG to address these challenges. It introduces a richer and more modular model:

GatewayClass: Defines the type of underlying infrastructure (e.g., AWS ALB).

Gateway: Represents the actual entry point into the cluster.

Route (e.g., HTTPRoute, TCPRoute, GRPCRoute): Defines routing rules, attached to one or more Gateways.

Service: Still used as the backend target for traffic, just like before.

n an AWS EKS cluster, the AWS Load Balancer Controller can interpret Gateway and Route resources to automatically provision and configure AWS Application Load Balancers (ALBs) or Network Load Balancers (NLBs).

A typical traffic flow looks like this:
Client → (AWS ALB via Gateway) → Gateway (reverse proxy) → HTTPRoute → Service → Pod

This is conceptually similar to the classic:
Client → (AWS ALB) → Ingress Controller → Service → Pod

Main take away for me is
Reduced Vendor Lock-In
Gateway API defines a consistent, cloud-agnostic schema.
The same Gateway and Route manifests can work across AWS, GCP, Azure, or on-prem — you only change the GatewayClass to fit the environment. No more provider-specific annotations cluttering your YAML.

New Python Package manager (UV) and Using Ray

Karthi Mahadevan — Fri, 14 Mar 2025 11:00:56 +0000

While I was in my earlier platform engineer role, one of my teammate said about UV, I didn't pay attention to it. But when I started using it, I realised how fast it is.

While I was reading and learning about Ray , I got to know they use UV internally. Nice.

How we created EKS cluster

Karthi Mahadevan — Fri, 28 Feb 2025 10:26:58 +0000

we used a cluster template.
This is a YAML configuration file for eksctl (a CLI tool for creating and managing Amazon EKS clusters). It’s designed to provision a Kubernetes cluster on AWS with specific networking, IAM, and node group settings.

Templating:
The configuration uses environment variables (e.g., ${EKS_CLUSTER_NAME}, ${VPC_ID}, etc.) to inject values dynamically. This makes it flexible and reusable across different environments (e.g., dev, staging, prod).

Resources That Will Be Created

EKS Cluster:
    A new EKS cluster is created with a specified name, version, and region.
    Cluster metadata includes tags for identification and categorization (such as Environment, Application, and Role).

VPC & Networking:
    VPC Configuration: Uses a pre-existing VPC (via ${VPC_ID}) with defined CIDRs.
    Subnets:
        Private Subnets: Three subnets (private-one, private-two, private-three) for worker nodes.
        Public Subnets: Three public subnets for resources that might need internet connectivity.
    Endpoint Access: Cluster endpoints are configured for private access only (public access is disabled).

IAM Configuration & Service Accounts:
    OIDC: The cluster is configured to work with OIDC, which is essential for associating IAM roles with Kubernetes service accounts.
    Service Accounts: Multiple service accounts are defined for specific roles:
        Cluster Autoscaler: For automatically adjusting the number of nodes.
        EBS CSI Controller: For managing persistent storage volumes.
        AWS Load Balancer Controller: To manage AWS load balancers.
        External DNS, CloudWatch Exporter, Cert Manager, Secrets Manager, Parameter Store, and others: Each with tailored IAM policies (either pre-defined “wellKnownPolicies” or custom policies via inline JSON).

Addons:
    The vpc-cni addon is included with an attached policy for managing networking on EKS worker nodes.

Managed Node Groups:
    Core Node Group:
        Designed to run primary workloads.
        Uses Bottlerocket AMI for enhanced security and performance.
        Configured with multiple instance types, private networking, EBS volumes (with encryption), scaling parameters (min, desired, max sizes), and specific labels/tags (including those for cluster autoscaler integration).


CloudWatch Logging:
    Enables cluster logging for all available log types, which is crucial for monitoring and troubleshooting.

Patterns and Structure

Modular Design:
The configuration cleanly separates different concerns:
    Cluster and VPC settings (networking, endpoints)
    IAM and Service Accounts (roles, policies)
    Node Groups (resource definitions, scaling, labels)
    Addons and Logging

Environment-Driven Templating:
Using placeholders for values allows this file to be easily adapted to different environments or clusters by simply setting environment variables during deployment.

Best Practices:
    Security: Private endpoints and encrypted volumes.
    Scalability: Defined node groups with autoscaling tags.
    Observability: CloudWatch logging is enabled for full cluster visibility.
    Separation of Responsibilities: Distinct IAM roles for different services minimize permissions and adhere to least-privilege principles.

While configuration sets up the core EKS cluster, IAM roles, managed node groups, and logging, the following additions can help to achieve a more production-ready environment:

Secrets Encryption: Encrypt Kubernetes secrets using a customer-managed KMS key.
Network Policies: Enforce pod-to-pod and pod-to-external communication restrictions.
Enhanced RBAC and Audit Logging: Refine RBAC rules and enable detailed audit logs.
Refined Security Groups: Use restrictive security groups, including at the pod level.
AWS Integrations: Leverage CloudTrail, GuardDuty, and managed add-ons for improved security and observability.
Backup/DR Planning: Implement backup strategies to safeguard your cluster and workloads.

How I use Cloudwatch and fluentbit

Karthi Mahadevan — Fri, 28 Feb 2025 10:13:47 +0000

Fluent Bit is deployed as a DaemonSet (a POD that runs on every node of the cluster). When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich log with some more information.

This will enable container (open policy agent) logs available in aws cloudwatch. The log group name where the logs will be /aws/containerinsights/${CLUSTER_NAME}/application ; here CLUSTER_NAME will be "tooling" for prod.

fluentbit.yaml will have

Here’s how the ClusterRole, ClusterRoleBinding, and ConfigMap are linked and their roles in this configuration:

ClusterRole

The ClusterRole named fluent-bit-role defines the permissions that Fluent Bit requires to interact with Kubernetes resources. It specifies:

Non-resource URL access: Allows access to /metrics with the get verb.
Resource access: Grants permissions to namespaces, pods, pods/logs, nodes, and nodes/proxy with the get, list, and watch verbs.

ClusterRoleBinding

The ClusterRoleBinding named fluent-bit-role-binding links the ClusterRole to a subject, enabling Fluent Bit to use the permissions.

Subject: The ServiceAccount named fluent-bit in the logging namespace.
RoleRef: Specifies that the binding refers to the fluent-bit-role ClusterRole.

This linkage ensures that the fluent-bit ServiceAccount has the necessary permissions to collect logs and interact with Kubernetes objects.

ConfigMap

The ConfigMap named fluent-bit-config provides configuration data for Fluent Bit. It contains:

Fluent Bit configurations: Specifies input sources (e.g., application logs), filtering (e.g., Kubernetes metadata), and output destinations (e.g., CloudWatch Logs).
Parser definitions: Defines parsers for structured log formats, such as docker and syslog.

How They Are Linked

Permissions for Log Access:
- The fluent-bit DaemonSet runs pods using the fluent-bit ServiceAccount.
- The fluent-bit-role-binding binds the fluent-bit-role ClusterRole to the fluent-bit ServiceAccount.
- This setup allows Fluent Bit to access logs, Kubernetes metadata, and node information.
Configuration Data:
- The DaemonSet mounts the fluent-bit-config ConfigMap to /fluent-bit/etc/ within its pods.
- Fluent Bit reads configurations from this directory to process logs according to the defined rules.

This structure ensures Fluent Bit operates with the correct permissions and configurations in a Kubernetes environment. Let me know if you need further clarification or adjustments!

Gitlab CI pipelines output

Karthi Mahadevan — Mon, 10 Feb 2025 12:24:39 +0000

To be considered valid YAML, you must wrap the entire command in single quotes. If the command already uses single quotes, you should change them to double quotes (") if possible:

job:
  script:
    - 'curl --request POST --header "Content-Type: application/json" "https://gitlab/api/v4/projects"'`

This Stackoverflow got bit more info
https://stackoverflow.com/questions/3790454/how-do-i-break-a-string-in-yaml-over-multiple-lines/21699210#21699210

Starting in GitLab 16.7 and GitLab Runner 16.7, you can now enable a feature flag titled FF_SCRIPT_SECTIONS, which will add a collapsible output section to the CI job log for multi-line command script blocks. This feature flag changes the log output for CI jobs that execute within the Bash shell.

A pipeline with a multi-line command in the script block for the build-job

variables:
FF_PRINT_POD_EVENTS: "true"
FF_USE_POWERSHELL_PATH_RESOLVER: "true"
FF_SCRIPT_SECTIONS: "true"

this will be helpful when you have multi line script and want to see the complete output.

AWS Lambda RIC - Runtime interface Client

Karthi Mahadevan — Thu, 06 Feb 2025 17:35:13 +0000

Why did we choose lambda ric ?

Docker images can handle larger deployments (up to 10GB)
Perfect for bundling extensive resources like opa policies
More efficient than ZIP files for large codebases
Better layer management and caching

Standardization Benefits

Consistent environments across development and production
Same container runs locally and in Lambda
Simplified CI/CD pipelines
Uniform testing environment

More..

Custom runtime configurations
Specific system libraries
Large framework requirements

Local testing details are in https://github.com/aws/aws-lambda-python-runtime-interface-client?tab=readme-ov-file#local-testing

We heavily use chainguard images,so build Docker image

FROM cgr.dev/chainguard/python
ARG LAMBDARIC_VERSION=3.0.0
RUN python -m pip install --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host=files.pythonhosted.org --no-cache-dir awslambdaric=="${LAMBDARIC_VERSION}"
COPY --chown=root:root --chmod=755 src/ ./src # your function handlers 
ENTRYPOINT [ "python", "-m", "awslambdaric" ]
CMD [   "src/handler.receiver"  ]

 docker build -t $docker_build_name -f your.Dockerfile .

Remember to push the image to ECR :) Or just test via

Run your Lambda image function using the docker run command.

docker run -d -v ~/.aws-lambda-rie:/aws-lambda -p 9000:8080 \
    --entrypoint /aws-lambda/aws-lambda-rie \
    myfunction:latest \
        /usr/local/bin/python -m awslambdaric app.handler

This runs the image as a container and starts up an endpoint locally at http://localhost:9000/2015-03-31/functions/function/invocations.

Post an event to the following endpoint using a curl command:

curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'

Remember to check the logs and clean later


docker logs "$(docker ps -q)"
echo" Stop and prune Docker containers"
docker stop "$(docker ps -a -q)"
docker system prune

What I use daily - python development

Karthi Mahadevan — Fri, 31 Jan 2025 22:33:30 +0000

Best Practices I follow during python development in current project.

Use Virtual Environments to keep dependencies isolated.
vscode to develop
Pytest, mockito for tests
Document Code using MkDocs and publish it as site in gitlab pages. ( Sooner we are moving to Backstage)
Taskfile , designed to streamline development processes related to testing package building, and publishing for a Python project. task is responsible for creating the source distribution (sdist) and wheel packages of the application using poetry. task also publishes the built package to a specified repository (GitLab in our case).
Entire org using Renovate to automate the management of software dependencies, ensuring projects remain up-to-date with the latest versions and security patches.
We use pre-commit on every commit to enforce our linting and style, use pre-commit install --install-hooks to enable it.
Mypy a static type checker that helps ensure type correctness in Python code. so we can catch type-related errors before runtime.
Ruff , a linter and formatter that can check for style issues, potential bugs across Python code.
some 10+ Pre-commit hooks.
Strict Commitlint format
- Gitleaks for detecting sensitive information (like API keys and passwords)
few more for Python project focusing on linting, testing, mutation testing, type checking, and ensuring code quality.

a sample function formatted according to Ruff's style conventions in our project would look like

def calculate_area(radius: float, pi: float = 3.14159) -> float:
    """Calculate the area of a circle.

    Args:
        radius (float): The radius of the circle.
        pi (float, optional): The value of pi. Defaults to 3.14159.

    Returns:
        float: The area of the circle.

    Raises:
        ValueError: If the radius is negative.
    """
    if radius < 0:
        raise ValueError("Radius must be non-negative.")

    return pi * (radius ** 2)

Local OPA using docker

Karthi Mahadevan — Fri, 31 Jan 2025 14:47:19 +0000

Docker Setup for OPA

Below is a Dockerfile to build and run OPA with your policies:

FROM openpolicyagent/opa:0.70.0 AS opa USER 1005 COPY opa/policies /global

Build and Run the Docker Image

Build the Docker image:

docker build -t opa -f opa/Dockerfile .

Test the policies:

docker run -it --rm opa test -f pretty -v -b /global

Run the OPA server:

docker run -it --rm --name opa -p 8181:8181 opa run --server --ignore "*_test.rego" --addr :8181 -b /global

GitLab DevSecOps developer

Karthi Mahadevan — Mon, 25 Nov 2024 15:03:56 +0000

As a GitLab DevSecOps developer utilizes the NIST framework to enhance security throughout the software development lifecycle. By integrating security practices into every phase of development, this developer ensures that vulnerabilities are identified and mitigated early, aligning with NIST's Secure Software Development Framework (SSDF) principles. This approach not only fosters collaboration among development, security, and operations teams but also automates security checks through tools like Static Application Security Testing (SAST) and Dynamic Application Security Testing (DAST). By leveraging GitLab's capabilities, such as centralized vulnerability management and compliance tracking, the developer effectively maintains a secure and efficient workflow that meets both organizational and regulatory standards.

thanks to bytebytego Alex

Lean development

Karthi Mahadevan — Tue, 28 Mar 2023 14:58:02 +0000

I decided to pick this topic because in work, we had Retro meeting and everyone suggested some improvments, then I realised all the suggestion can be put into one of these 8. Here are those 8 buckets.

In today's fast-paced software development world, efficiency is key. Developers are always looking for ways to optimize their processes and deliver high-quality products to their customers quickly. One methodology that has gained popularity in recent years is Lean. Originally developed for manufacturing, the Lean methodology has since been adapted to many different industries, including software development.

At its core, Lean is all about minimizing waste and maximizing value. In software development, this means identifying and eliminating the eight types of waste that can occur in the development process. These wastes include defects, overproduction, waiting, non-utilized talent, transportation, inventory, motion, and extra processing.

Defects

Defects are any work that has errors, bugs, or requires rework. Defects are one of the most common types of waste in software development and can be caused by a variety of factors, such as poor requirements, inadequate testing, or miscommunication. To minimize defects, teams should focus on developing and implementing robust testing procedures, including automated testing and code reviews. Continuous integration and continuous delivery (CI/CD) pipelines can also help to catch defects early in the development process, reducing the need for rework.

Overproduction

Overproduction is creating more code or features than what is required by the customer. This can happen when developers make assumptions about what the customer wants, or when requirements are not clearly defined. Overproduction can lead to wasted effort and resources, as well as a longer time to market. To minimize overproduction, teams should focus on delivering small, incremental features that provide value to the customer. This approach allows for frequent feedback and ensures that the team is building what the customer actually needs.

Waiting

Waiting is time spent waiting for feedback, approvals, or resources. Waiting can be a major source of waste in software development, as it can lead to delays and decreased productivity. To minimize waiting, teams should aim to have a fast and efficient feedback loop. This can be achieved through regular check-ins, agile methodologies, and using tools that enable collaboration and communication.

Non-utilized talent

Non-utilized talent is underutilizing the skills, abilities, or ideas of team members. This can happen when team members are not given the opportunity to contribute or when their contributions are not valued. To minimize non-utilized talent, teams should focus on creating a culture of collaboration and empowerment. This can be achieved through regular team building activities, cross-functional training, and ensuring that team members have a voice in the development process.

Transportation

Transportation is the movement of information or code that does not add value. This can include moving code between development environments, or between teams. Transportation can be a major source of waste in software development, as it can lead to delays and errors. To minimize transportation, teams should focus on creating a streamlined development environment, with clear processes and tools that enable efficient collaboration.

Inventory

Inventory is creating or storing unfinished work, code, or features. This can happen when requirements change, or when work is not completed on time. Inventory can lead to wasted effort and resources, as well as a longer time to market. To minimize inventory, teams should focus on delivering small, incremental features that provide value to the customer. This approach allows for frequent feedback and ensures that work is completed in a timely manner.

Motion

Motion is extra movement or effort that does not add value. This can include excessive clicking or scrolling, or moving between different tools or systems. Motion can be a major source of waste in software development, as it can lead to decreased productivity and increased frustration. To minimize motion, teams should focus on creating a streamlined development environment

Stop saying "you forgot to …" in code review

Karthi Mahadevan — Wed, 22 Mar 2023 12:12:08 +0000

On startup, Danger reads a Dangerfile from the project root. Danger code in GitLab is decomposed into a set of helpers and plugins, all within the danger/ subdirectory, so ours just tells Danger to load it all. Danger then runs each plugin against the merge request, collecting the output from each. A plugin may output notifications, warnings, or errors, all of which are copied to the CI job’s log. If an error happens, the CI job (and so the entire pipeline) fails.

On merge requests, Danger also copies the output to a comment on the MR itself, increasing visibility.

Danger is a gem that runs in the CI environment, like any other analysis tool. What sets it apart from (for example, RuboCop) is that it’s designed to allow you to easily write arbitrary code to test properties of your code or changes. To this end, it provides a set of common helpers and access to information about what has actually changed in your environment, then runs your code!

If Danger is asking you to change something about your merge request, it’s best just to make the change.

check this site for more info : https://danger.systems

We are using danger and seeing lots of benefits !

Deleting NameSpace and Helm

Karthi Mahadevan — Mon, 06 Mar 2023 14:13:23 +0000

For Helm Release stuck in Uninstalling state you have to do

k delete secrets sh.helm.release.v1.name.VERSION-N

and for Namespace deletion, ensure there is no finalizers hook!