DEV Community: Senthil Kumaran

The Prow Upgrade Story

Senthil Kumaran — Mon, 16 Mar 2026 05:40:35 +0000

One of my first experiences with Kubernetes was with using Prow. Prow is a Kubernetes CI/CD system. In the Mesosphere/D2IQ days, I had to transition the CI jobs that we were running in Jenkins on Mesos/Marathon to Prow on Kubernetes. In order to learn Kubernetes, I suggested we set up Prow to run the CI jobs for our Mesosphere/D2IQ frontend. That was the first task that I undertook that would help me understand Kubernetes . It was simple but a critical CI/CD system that handles job execution and GitHub automation through policy enforcement. Anyone who has either contributed to Kubernetes or worked with the testing-sig will have come across the weird-looking Prow robots working with your pull requests. And in the GitHub-based ChatOps, there are multiple /slash-commands that are utilized to interact with the state of the pull request.

The component listeners listen to these webhooks and act on the state and the command. Just like regular CI/CD systems, they trigger a job on the event, and the job usually clones the repository at the pull request HEAD and runs a script that exercises the tests and finally gives a success or failure signal.
The intelligence of running the tests is baked into the script that runs the tests. Prow does not do anything fancy that is expected from any continuous integration system. It simply runs the job and returns the results.

The first task before setting up the jobs was setting up the Kubernetes cluster and then deploying Prow.

The various nouns in this diagram are the components of the Prow that we will see.

GitHub sends webhooks to Prow. Prow runs your tests as Kubernetes pods. Prow reports the results back to GitHub. That's it. The core pipeline flows down the center: Hook receives GitHub webhooks, creates a ProwJob custom resource, the Controller Manager picks it up and schedules a pod in the Cluster.

So the setup chain was simple.

I used Terraform to manage the infrastructure state. Terraform was used to create the S3 buckets to store the Terraform state. We used to have a DynamoDB table for locking. It surprises me in hindsight, and perhaps even surprised me back then, but we did use a DynamoDB table just for locking. This is to avoid multiple terraform apply commands acting on the same cluster. The cluster creation was controlled with users with IAM permissions for various operations.

This is how we built the infrastructure for Kubernetes. Any Kubernetes distribution can be used; at that time, I remember we were using a home-built one, which the company, D2IQ was selling. For the worker nodes, I had provisioned very large instances — ones with 16 vCPUs and 64 GB of RAM — plenty of headroom for Prow's microservices and the test pods they'd spawn. Since Prow runs the CI job as a Kubernetes job, it needs those resources to spawn the job and run.

I had the cluster infrastructure stored in Terraform state files; Terraform created the S3 buckets, DynamoDB lock table, and the IAM users.
For our Kubernetes cluster, I had a Native K8s cluster on EC2 using a home-grown technology called Konvoy while at Mesosphere/D2IQ.
I set up the Prow application on the cluster using the Custom Resource Definitions, and since CI systems are often internal, I had to set up an Internal domain configured to point to Prow's Hook and Deck endpoints, and set up cert-manager for TLS on that internal domain.

Once the Kubernetes cluster was provisioned, the cluster came up cleanly. The nodes were healthy, the API server responding, CoreDNS resolving.

I had to apply the ProwJob Custom Resource Definitions to set up the application. We created two secrets that Prow needs to talk to GitHub: an HMAC (Hash-Based Message Authentication Code, a shared secret) token for validating incoming webhooks, and a GitHub App's private key for authenticating API calls.
I deployed this full set of Prow components using the S3-backed starter manifest.

Once the Prow system was set up, the Prow services would be running. The Hook component would be waiting for webhooks, Deck serving the UI, the Prow Controller Manager ready to schedule jobs. Components like Tide watch for mergeable PRs, a component called Horologium handles periodic jobs. Systems like Crier report back to GitHub, and Sinker quietly cleans up after everyone else.

Wiring up the domain

When you architect a CI system for developers, as I did, the primary responsibility is to make the system reliable. Prow needs to be reachable from the public internet — specifically, GitHub needs to send HTTPS webhook requests to Hook's endpoint whenever something happens in the subscribed repositories.

That means in the Kubernetes world, you need a domain, an Ingress, and a valid TLS (Transport Layer Security) certificate.

The internal domain had to be provisioned using Terraform, and with the domain I set up the Ingress that routed /hook traffic to the Hook Service and everything else to Deck. Even these were provisioned in the Terraform state files.

We used cert-manager to handle the TLS. I pointed an internal domain at the cluster, set up an Ingress that routed /hook traffic to the Hook service and everything else to Deck, and used cert-manager to handle TLS. cert-manager would automatically provision and renew certificates, wiring them into the Ingress as a Kubernetes Secret. For cert-manager, we configured a ClusterIssuer, created a Certificate resource for the Prow domain, and let cert-manager do its thing.

All of these were done in Terraform state files, so that when we did a terraform apply, the domain was created, the certificate manager was set up, certificates were gathered, and the system was operational serving traffic over HTTPS.
Once the certificate was issued and the Ingress had valid TLS termination, we went to the GitHub App at the org-level, set the webhook URL to point at the Prow domain's /hook path, and plugged in the HMAC secret.

It works. Open a test PR, and within seconds the Hook receives the webhook, the trigger plugin creates the ProwJob, the Controller Manager scheduled a pod, the Crier reported results back to GitHub. The green checkmark appearing on the PR status means all is good, and life is good.

Kubernetes Upgrades

Kubernetes upgrades are talked about as a frequent source of pain points. Even when you use a Managed Kubernetes Provider, when you have made complex changes to your cluster and operate business-critical workloads, teams are seldom confident to upgrade their Kubernetes clusters without a good plan in place. So, in my Kubernetes cluster, as we updated our Kubernetes version and Prow version, we had to upgrade the components too. As I explained above, the internal domain over HTTPS was set up with a separate Terraform plan.
And when we upgraded cert-manager, it started to fail. That's when the Hook endpoint stopped receiving webhooks from GitHub.

The Failure Chain

Here's what happened:

The cert-manager upgrade caused the webhook pod to restart. During the upgrade, cert-manager's own internal webhook — which validates cert-manager CRDs — went through a transition period where it was unavailable. This was due to a bug in the version of cert-manager we were upgrading to.
When cert-manager's cainjector or controller had issues during the upgrade, the Certificate resource couldn't renew and the Secret wasn't properly updated, as the TLS certificate for Prow's Ingress was tied to cert-manager. During the upgrade, the Ingress lost its valid TLS certificate, and without a valid cert, the HTTPS endpoint that GitHub was hitting for webhooks started returning TLS errors.

GitHub webhook deliveries started failing — this can be seen in the GitHub interface showing that webhook deliveries had failed — and all CI/CD stopped. No webhooks meant no presubmit jobs, no postsubmit jobs, no /retest commands, nothing. Prow was running perfectly fine internally, but it was invisible to the outside world.

Everything inside the cluster looked healthy:

# All pods running?

kubectl get pods -n prow
kubectl get pods -n cert-manager

# ProwJobs being created?  No new ones
kubectl get prowjobs -n prow --sort-by=.metadata.creationTimestamp

# Hook logs showing any incoming webhooks? Nothing
kubectl logs -l app=hook -n prow -f

# Certificate status?
kubectl describe certificate prow-tls -n prow
# This is where I finally saw the problem

The kubectl describe certificate output showed the certificate was in a failed state — cert-manager couldn't issue or renew it because the upgrade had left its internal state inconsistent.

# Check cert-manager's own health
kubectl get pods -n cert-manager
kubectl logs -l app.kubernetes.io/name=webhook -n cert-manager

The webhooks were unhealthy, so I had to delete and reinstall cert-manager cleanly.

I was doing everything using Terraform. Manually mutating state when everything else is maintained in Terraform was going to be tricky. Terraform does not like that. Also, certain versions and dependencies of cert-manager were a problem, so I had to find the right combination of versions and encode it.

# If the webhook is unhealthy, you may need to
# delete and reinstall cert-manager cleanly

kubectl delete -f https://github.com/cert-manager/cert-manager/releases/download/v<OLD>/cert-manager.yaml
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v<NEW>/cert-manager.yaml

Once everything was stable, I had to wait for the cert-manager rollout to stabilize and force the certificate renewal.

# Wait for everything to stabilize
kubectl -n cert-manager rollout status deployment/cert-manager-webhook

# Force certificate renewal
kubectl delete secret prow-tls-secret -n prow
# cert-manager will recreate it from the Certificate resource

# Verify
kubectl describe certificate prow-tls -n prow
# Should show "Certificate is up to date and has not expired"

Once cert-manager was stable, the GitHub webhook deliveries should be back and hooks succeeding again.

kubectl logs -l app=hook -n prow -f
# Should see incoming webhook events

This downtime affected the productivity of my group for multiple hours. And fixing something when things were down was a challenging one.

Some of the Important Lessons I have learned from this experience are

Cert-manager upgrades are not zero-downtime for your services.

When you upgrade cert-manager, its internal admission webhook restarts. During that window, if any Certificate resources need to be reconciled, things can break. If your Ingress TLS depends on cert-manager and the timing is bad, your endpoints go dark.
The best way would be to decouple the TLS from cert-manager for critical webhook endpoints during the upgrade.

The Prow Hook endpoint is the single point of entry.

Everything in Prow depends on Hook receiving webhooks. If Hook is unreachable, whether due to TLS issues, DNS problems, or network configuration, the CI/CD pipeline goes silent. This is an important design consideration for anyone designing an application architecture on Prow.

Having a "break glass" procedure for webhook endpoints.

As in, you can manually create a TLS secret and attach it to your Ingress without cert-manager. When cert-manager is the thing that's broken, you need a way to get your endpoints back up while you fix it:

# Create a self-signed cert to restore connectivity
openssl req -x509 -nodes -days 30 -newkey rsa:2048 \
  -keyout /tmp/tls.key -out /tmp/tls.crt \
  -subj "/CN=prow.your-internal-domain.com"

kubectl create secret tls prow-tls-secret \
  --cert=/tmp/tls.crt --key=/tmp/tls.key \
  -n prow --dry-run=client -o yaml | kubectl apply -f -

This was a firefighting story that helped me realize how infrastructure and complex applications stand on some very simple dependencies.

Hitori

Senthil Kumaran — Sun, 18 Jan 2026 19:22:56 +0000

I wrote an implementation of the game Hitori using Claude Code in agent mode.

It is a puzzle that I developed after looking at a game called Hitori in Linux. I was studying the architecture of it and noted down how that game was written.

I thought that I should make an implementation of it.

I tried to make a Django implementation where Django serves as the backend and the frontend is in JavaScript. I set up the foundation for it and then pointed Claude Code to the original implementation of Hitori, which is written in C in Linux. I asked it to develop a similar one for a client-server architecture that I could play in a web browser using a Python backend and JavaScript frontend.

I enabled it in agent mode and gave it to dangerously skip permissions. I asked it to implement the game and gave some Yes's for the prompts it asked, then went about with it. I had asked it to make a game and then went to sleep.

When I woke up the next morning, I saw that the complete game was implemented. To my surprise, I didn't have to do anything. The whole game was there and I just played it. I was like, "Oh my goodness." I was amazed for a full day at what had just happened.

But then, to take it further, I thought there should be a login system and game board improvements. That took some time, and then I deployed it on a Kubernetes cluster. Even that was easy - I didn't have to do anything.

This is the first game I developed where I didn't have to open an editor or IDE at all. I did everything completely from the command prompt. This was something new. I thought I should capture this moment, so I'm writing about it in my blog. That's pretty much it.

You can enjoy playing this game at https://hitori.learntosolveit.com

Visionary - Photo Of the Day Browser Extension

Senthil Kumaran — Sat, 10 Jan 2026 16:23:38 +0000

I built a browser plugin called "Visionary" that overlays meaningful descriptions and context directly onto stunning pictures of the day.

I noticed that existing picture-of-the-day plugins were built over two decades ago and never evolved to harness the capabilities of modern artificial intelligence. AI can transform the picture viewing experience by distilling complex descriptions into accessible insights and providing references to explore the core concepts in the photo more deeply.

You can get a sense of how this works by visiting
https://picture.learntosolveit.com

Get it for Chrome or other Stores

https://chromewebstore.google.com/detail/picture-of-the-day/conkiboijidpjkmflbmagmcebddpphli

Games: Hitori

Senthil Kumaran — Sat, 22 Nov 2025 15:07:58 +0000

Hitori is a Japanese logic puzzle played on a grid of numbers. The objective is to eliminate numbers so that no row or column contains duplicate numbers, all shaded cells are isolated (not touching horizontally or vertically), and unshaded cells create a single connected group, that is you can you move from one unshaded cell to another unshaded cell following the path vertically or horizontally.

Here’s how a typical Hitori puzzle appears:

How to Play:

Shade cells to eliminate duplicate numbers from each row and column.
Shaded (black) cells cannot be adjacent to each other vertically or horizontally.
All non-shaded (white) cells must be connected.

The source code of this project is here https://gitlab.gnome.org/GNOME/hitori and the architecture is explained here.

Games: Biniax 2

Senthil Kumaran — Sun, 16 Nov 2025 15:54:18 +0000

Biniax is a very simple, but interesting, vertical scrolling , matching game. You have to move you piece to collapse against the piece
of same color. The game ends when you are no longer able to collapse the pieces.

This is how we start.

The game is written in ANSI C and uses SDL. I had to fix the source code for compilation, it is maintained here
https://github.com/orsenthil/biniax2/tree/main, and the architecture gives the architectural concepts for developement.

Games: Chromono

Senthil Kumaran — Sun, 16 Nov 2025 07:11:46 +0000

Chromono is a visually appealing and aesthetic circular puzzle game. The game engine and physics is enticing. The rules are simple, you only have to make the circle of the same color overlap/touch with circle with the same patch. The game introduce challenges with bounce, physics and tricky circular structures.

The source code lives here, https://github.com/orsenthil/chromono/tree/main, and I had to fix for the build and runtime issues on Linux for OpenGL.

The game is written in C++11, and uses OpenGL. The architecture and technologies to learn are exciting and modern too.

Games: Atomix

Senthil Kumaran — Sat, 15 Nov 2025 15:52:20 +0000

Atomix is a very simple sliding block tiling puzzle game, in which, the user has to slide the atoms to form the molecule given as a challenge.

The source code is maintained here https://gitlab.gnome.org/GNOME/atomix/-/tree/main

The game is written in C, with the object-oriented capabilities provided by a system called GObject. The windowing toolkit used is GTK.
The architecture and learning concepts make it a highly approachable game.

Games: achilles-1

Senthil Kumaran — Sat, 15 Nov 2025 14:32:21 +0000

Achilles is a simulation, not really a game. In Achilles, the world is given an X coordinate, Z coordinate, number of food items and number of organisms.
The organisms are not given any objective. But they have functions like they have vision, and learning systems powered by neural networks.
Food spawns randomly across the world. Food decays if it is left alone.
When the organisms see food in proximity, they eat the food. When the organisms see each other, if they are similar, they mate. If there is enough strength and difference, they can attack each other.

And the world evolves!

Here is a screenshot of the evolution

Here is the attack in progress.

The program is written in C++ and uses OpenGL to display the world. The package that is present in many Linux operating systems does not behave the way we want. I had to check out the source code, and fix the bugs in the program.

The source of the maintained version lives in GitHub here: https://github.com/orsenthil/achilles-1; there is a lifetime worth of learning concepts used by this simple program.

Games 2048-QT

Senthil Kumaran — Tue, 11 Nov 2025 14:28:05 +0000

This is a port of 2048 game written in QT framework

The source code here https://github.com/xiaoyong/2048-Qt shows a very simple, and modular approach.
The QT game load is written in C++, but the game logic itself is written in qml and javascript. The interface components are in C++, the game logic is in Javascript, which has a native integration with QML, Qt Markup Language; and this whole program is orchestrated via QT framework.

Games- Amoebax

Senthil Kumaran — Sun, 09 Nov 2025 18:37:16 +0000

Amoebax is a falling puzzle game, tetris like game. The player has to arrange a group of four.

You have to arrange the falling tiles into groups of four. It has a two player game mechanics too, where in, while you try to ensure that you dont crash to the top, when you remove the paired blocks, you send it to your opponent. You win when your opponent's game is full.

The opponent is usually an AI. The game is written in C++, and uses libraries like SDL, zlib, ogg and vorbis.

The source code here is https://gitlab.com/perita/amoebax/, there is an emscripten port of the game https://peritasoft.com/amoebax/index.html which can be played online by the author, Jordi Frita.

When examining the source code, it is written in modular C++ and uses multiple game design concepts like state design pattern to manage different game states, singleton for creating an instance of the game, factory to create ai players, strategy to implement player behaviors, template for customizable steps, observer for communication, and RAII for dealing with sound and image resources.

Games - AisleRiot Zebra

Senthil Kumaran — Sun, 09 Nov 2025 15:02:51 +0000

It has become very easy to learn new things. In these series of blog poss, I will be sharing #100games2025, I will be sharing about 100 different new games I have learned in 2025.

The Zebra variant of AisleRiot game is played with two decks. The top row has the two decks to arrange. The player has to arrange the cards with opposite colors. Like Black and Red go alternate on top of each. The bottom is same as solitaire, but in the zebra variant, the top row is arranged alternately. When a slot in the bottom row, called tableau has an empty slot, it is automatically filled in with the latest in the waste pile or from the stock pile.

Coin Flip Simulator

Senthil Kumaran — Wed, 25 Jun 2025 22:48:53 +0000

This is a coin flip simulator. It compare theoretical binomial distribution with experimental results


import random
import math
import matplotlib.pyplot as plt
import numpy as np
from collections import Counter

def Nchoosek(N, k):
    return math.factorial(N) / (math.factorial(k) * math.factorial(N - k))

def binomial(n,k, p):
    return Nchoosek(n, k) * math.pow(p, k) * math.pow(1-p, n-k)


def run_experiment(n_flips, n_experiments, p_heads=0.5):
    """
    Simulate coin flips and count number of heads.

    :param n_flips: Number of coin flips per experiment.
    :param n_experiments: Number of experiments to run.
    :param p_heads: Probability of getting heads (default 0.5 for a fair coin)

    :return: List of head counts for each experiment.
    """
    results = []

    for experiment in range(n_experiments):
        flips = [random.random() < p_heads for _ in range(n_flips)]
        heads_count = sum(flips)
        results.append(heads_count)

    return results


def compare_theoretical_vs_experimental():
    """Compare theoretical binomial distribution with experimental results."""
    # Parameters
    n_flips = 10          # Number of coin flips per experiment
    n_experiments = 1000  # Number of experiments to run
    p_heads = 0.5         # Probability of heads (fair coin)

    # Run the experiment
    print(f"Running {n_experiments} experiments with {n_flips} coin flips each...")
    results = run_experiment(n_flips, n_experiments, p_heads)

    # Count occurrences of each number of heads
    result_counts = Counter(results)

    # Convert to probabilities
    experimental_probs = {k: v/n_experiments for k, v in result_counts.items()}

    # Calculate theoretical probabilities using your binomial function
    theoretical_probs = {k: binomial(n_flips, k, p_heads) for k in range(n_flips + 1)}

    # Prepare data for plotting
    k_values = list(range(n_flips + 1))
    theoretical_values = [theoretical_probs[k] for k in k_values]
    experimental_values = [experimental_probs.get(k, 0) for k in k_values]

    # Plot the results
    plt.figure(figsize=(12, 6))

    # Bar width and positions
    width = 0.35
    x = np.array(k_values)

    # Create bars
    plt.bar(x - width/2, theoretical_values, width, label='Theoretical Probability')
    plt.bar(x + width/2, experimental_values, width, label='Experimental Probability')

    # Add labels and title
    plt.xlabel('Number of Heads')
    plt.ylabel('Probability')
    plt.title(f'Binomial Distribution: {n_flips} coin flips, p(heads)={p_heads}')
    plt.xticks(k_values)
    plt.legend()
    plt.grid(axis='y', linestyle='--', alpha=0.7)

    # Add a text box with statistics
    mean_theoretical = n_flips * p_heads
    var_theoretical = n_flips * p_heads * (1 - p_heads)
    mean_experimental = sum(k * count for k, count in result_counts.items()) / n_experiments

    stats_text = f"Theoretical Mean: {mean_theoretical}\n"
    stats_text += f"Theoretical Variance: {var_theoretical}\n"
    stats_text += f"Experimental Mean: {mean_experimental:.2f}"

    plt.figtext(0.15, 0.8, stats_text, bbox=dict(facecolor='white', alpha=0.8))

    # Show plot
    plt.tight_layout()
    #plt.show()
    plt.savefig('coinflip.png')

    # Print summary
    print("\nSummary:")
    print(f"Theoretical Mean: {mean_theoretical}")
    print(f"Theoretical Variance: {var_theoretical}")
    print(f"Experimental Mean: {mean_experimental:.2f}")

    # Print probabilities table
    print("\nProbabilities Table:")
    print("Heads | Theoretical | Experimental | Difference")
    print("-" * 50)
    for k in k_values:
        theo = theoretical_probs[k]
        exp = experimental_probs.get(k, 0)
        diff = exp - theo
        print(f"{k:5d} | {theo:.6f} | {exp:.6f} | {diff:.6f}")

compare_theoretical_vs_experimental()