DEV Community: POTHURAJU JAYAKRISHNA YADAV

"It Works on My Machine" — How a Shared Redis Database Broke Authentication Across Environments

POTHURAJU JAYAKRISHNA YADAV — Sat, 03 Oct 2026 03:02:23 +0000

We've all heard the classic developer line: "But it works on my local machine!" Usually it's a punchline. This time, it was the starting point of a debugging journey that taught me a lesson I won't forget about environment isolation, Redis, and the subtle ways shared infrastructure can betray you.

In this post I'll walk through a real production-adjacent incident I ran into recently: the application ran perfectly on local, worked fine when I ran it myself, but kept failing in the Dev and QA environments with authentication errors coming from a third-party vendor. The root cause turned out to be something that looked completely innocent until the very end.

If you run multiple environments that share backing services, this one is worth reading to the end.

The Setup

Our application is a fairly standard web service. Without going into proprietary details, the moving parts relevant to this story are:

The application — runs in multiple environments: local (developer machines), Dev, and QA.
MySQL — our primary relational data store.
Redis — used for caching and, importantly, for storing session/authentication tokens.
A third-party vendor server — we authenticate against it, and it issues tokens we store and reuse.

The flow that mattered: a user logs in, we obtain an authentication token from the third-party vendor, and we cache that token in Redis so we don't have to re-authenticate on every request.

Simple enough. Until it wasn't.

The Symptom

The report came in like this:

"The dev team runs the application locally and everything works. But in Dev and QA, it's not working."

That phrasing is a trap. "Not working" can mean a hundred things. The actual symptom, once I dug in, was an authentication error returned by the third-party vendor server — but only in the shared Dev and QA environments. Local was perfectly happy.

The confusing part: the error was intermittent and seemed to have nothing to do with what any individual developer was doing. One minute a user was authenticated, the next minute they were getting kicked out, and the vendor was rejecting tokens that should have been valid.

Classic "works on my machine" territory. Which is exactly why I started where most of us start.

The Investigation (and Every Dead End Along the Way)

I want to be honest about the order I actually went through this, because the dead ends are the most instructive part. Debugging is rarely a straight line, and each thing I ruled out narrowed the search.

Step 1: "Did someone change the code?"

The first instinct when something works in one place and breaks in another: what changed?

I checked the recent commits and diffs between what was running locally and what was deployed to Dev and QA. I was looking for any code change — a new feature, a refactor, a tweak to the auth flow, anything.

Result: No code changes. The code running locally was effectively the same code running in Dev and QA. If the code is identical, the code probably isn't the problem.

That's an important deduction: when the same code behaves differently in different places, the difference is almost always in the environment, not the code.

Step 2: "Is it a dependency version mismatch?"

The next usual suspect: package versions. Maybe local had a different version of a library than the deployed environments — a different Redis client, a different HTTP client, a different auth SDK. Version drift between environments is a very common cause of "works here, not there."

So I compared the dependency versions across environments. Lockfiles, installed package versions, the whole list.

Result: Same versions everywhere. Another dead end, but another data point: it's not a dependency problem.

Step 3: "Let me reproduce it myself"

At this point I did the thing you should always do — I stopped theorizing and ran the application myself, locally, trying to reproduce the exact failing flow.

Result: It worked perfectly for me. No authentication errors. Everything flowed as expected.

This is the most frustrating stage of any bug hunt. The thing refuses to break when you're watching it. But it's also a huge clue: if it works in isolation on local but fails in the shared environments, the problem lives in something the shared environments have in common that local doesn't.

Step 4: "Maybe the backing services are misconfigured"

Now I turned my attention to the infrastructure. The application depends on Redis and MySQL. My hypothesis: maybe Redis or MySQL in Dev/QA was misconfigured — wrong settings, wrong memory limits, wrong eviction policy, something.

I went through the server-side configuration for these services across environments.

Result: The server-side configurations were all the same. The Redis and MySQL servers themselves looked correctly and identically configured. Yet another dead end for the "something is broken" theory.

By now a pattern was forming in my head. Nothing was broken. Everything was configured the same. And yet behavior differed. That usually means something is being shared when it shouldn't be.

Step 5: "Let me look at the application environment config"

The final place to look was the application's own environment configuration — the .env-style config that tells each environment which Redis, which MySQL, which vendor endpoint to talk to.

And here's where it clicked.

I looked at where the application was pointed for Redis, and I saw that Dev and QA were pointed at the same Redis instance and the same Redis database.

That was the smoking gun.

The Root Cause

Here's what was actually happening.

Redis supports multiple logical databases within a single instance (numbered 0, 1, 2, and so on by default). Our Dev and QA environments were both pointed at:

the same Redis host, and
the same Redis logical database.

Because Redis stores data as keys, and both environments used the same key naming for authentication tokens, the two environments were writing to and reading from the same slots.

Now layer the authentication flow on top of that:

A developer logs into Dev. The app authenticates against the third-party vendor, gets a token, and stores it in Redis under some key — say, a key derived from the user's identity.
The same user (or the same key) logs into QA. The app authenticates again, gets a new token from the vendor, and overwrites the same Redis key.
Back in Dev, the next request pulls the token from Redis — but now it's holding the QA session's token, or a token that the vendor has effectively superseded.
The third-party vendor sees a token that doesn't match the session it expects and rejects it with an authentication error.

In other words: logging into one environment was effectively logging the user out of the other. The two environments were stomping all over each other's tokens because they shared the same Redis database.

That perfectly explained every observation:

Why no code change mattered — the code was correct; the data store was shared.
Why versions didn't matter — it was never a dependency issue.
Why it worked on local — local pointed at its own isolated Redis, so nothing else was overwriting its tokens.
Why it worked when I ran it myself — I was the only one touching those keys at that moment.
Why it was intermittent — it only broke when activity in Dev and QA overlapped on the same user/keys.
Why the vendor threw the auth error — the token it received had been replaced by the other environment's token.

The services weren't misconfigured in the sense of being broken. They were misconfigured in the sense of being shared across environments that should have been isolated.

The Fix

The fix was refreshingly small once the cause was understood: give each environment its own Redis database.

Redis makes this easy because a single instance exposes multiple numbered logical databases. By changing the Redis database index (and ideally the host) per environment in the application's environment config, each environment got its own isolated keyspace.

Conceptually, the configuration went from this:

# Dev
REDIS_HOST=shared-redis
REDIS_DB=0

# QA
REDIS_HOST=shared-redis
REDIS_DB=0      # <-- same instance AND same DB as Dev

to this:

# Dev
REDIS_HOST=shared-redis
REDIS_DB=0

# QA
REDIS_HOST=shared-redis
REDIS_DB=1      # <-- separate logical database

After separating the Redis databases per environment, the token collisions stopped. Logging into Dev no longer touched QA's tokens, the vendor stopped rejecting tokens, and the authentication errors disappeared entirely.

A note on logical databases vs. separate instances: Using separate Redis logical databases (REDIS_DB) solved our immediate problem and is a perfectly valid quick fix. For stronger isolation, the better long-term approach is a completely separate Redis instance per environment, since logical databases still share the same memory, CPU, eviction policy, and FLUSHALL blast radius. Another robust option is key prefixing / namespacing (for example, prefixing every key with the environment name) so collisions are impossible even on a shared database. Pick the level of isolation that matches your risk tolerance.

Why This Was So Easy to Miss

This is the kind of bug that hides in plain sight, and I think it's worth reflecting on why.

Nothing was broken. Every service was up. Every config "looked" correct in isolation. There was no stack trace pointing at Redis — the error surfaced at the third-party vendor, which is as far downstream from the real cause as you can get.
The error was misleading. An authentication failure from an external vendor makes you suspect credentials, tokens, network, or the vendor itself — not your cache layer.
It only manifested under concurrency. When one environment was quiet, the other worked fine. That's why it never reproduced on demand.
Shared infrastructure is often invisible in day-to-day work. You rarely stop and ask, "Wait, are these two environments literally the same Redis database?" You assume isolation that was never actually there.

The real lesson is that the error message pointed at the symptom (vendor auth rejection), while the cause was several layers upstream (shared cache state). Good debugging means following the chain backward past the first plausible-looking culprit.

Lessons Learned

Here are the takeaways I'm carrying forward, and that I'd offer to anyone running multi-environment systems.

1. Environments must be isolated — all the way down

It's not enough to have separate application deployments. Every backing service — Redis, MySQL, message queues, object storage, search indexes — needs to be isolated per environment, or at minimum namespaced so there's no chance of cross-environment collision. Shared state between environments is a latent bug waiting for enough concurrent traffic to surface.

2. "Same code, different behavior" means look at the environment

When identical code behaves differently across environments, the bug is almost never in the code. It's in configuration, data, infrastructure, or state. Save yourself time by ruling out code quickly and moving to the environment.

3. Follow the error upstream, past the obvious suspect

The vendor threw the authentication error, but the vendor wasn't the problem. The first component that reports an error is frequently just the victim of something further up the chain. Keep asking "but why did that happen?" until you reach a cause that explains all your observations, not just some of them.

4. The config file deserves the same scrutiny as the code

I checked code, versions, and server-side service configuration before I looked carefully at the application's environment config — which is exactly where the problem was. Environment/config files are code. Diff them across environments as a first-class debugging step.

5. Reproduce in isolation to localize the problem

The fact that it worked on local and worked when I ran it alone was itself the key insight: the problem only existed where state was shared. "Can I reproduce it in isolation?" is a powerful question — the answer tells you whether the bug is in the component or in its surroundings.

6. Design keyspaces defensively

Even if you believe your environments are isolated today, prefix your cache keys with the environment name. It costs almost nothing and makes an entire category of cross-environment collisions structurally impossible. Defense in depth applies to cache keys too.

A Simple Mental Checklist for "Works Here, Not There" Bugs

Next time you hit a bug that works in one place but fails in another, run through this:

Did the code change? Diff it. If it's identical, move on.
Did the dependencies change? Compare versions and lockfiles. If identical, move on.
Can you reproduce it in isolation? If it only breaks in a shared environment, suspect shared state.
Are the backing services misconfigured? Check server-side config.
Where is each environment actually pointed? Check the application's env/config — host, port, database index, credentials, endpoints.
Is anything being shared that shouldn't be? Same database, same keyspace, same bucket, same queue.
Follow the error upstream until you find a cause that explains every symptom.

In my case, the answer was hiding at step 5 and 6 the whole time.

Closing Thoughts

The bug felt enormous while I was chasing it — authentication failing in Dev and QA, a third-party vendor rejecting tokens, nothing reproducing locally. The fix was one line of configuration: point each environment at its own Redis database.

That contrast is the whole story of debugging distributed systems. The symptoms are loud and scattered; the cause is quiet and specific. The work is in patiently ruling things out until the quiet cause has nowhere left to hide.

If you take one thing away: isolate your environments completely, and never assume that two environments pointed at "Redis" are pointed at different Redis. Verify it. Because the day your traffic overlaps, that shared database will find you.

Have you run into a similar shared-infrastructure gotcha? I'd love to hear how it surfaced for you and how you tracked it down. Drop it in the comments.

How I Eliminated Lambda Deployment Downtime with Terraform and Blue-Green Deployments

POTHURAJU JAYAKRISHNA YADAV — Sat, 06 Jun 2026 11:31:09 +0000

I Got Tired of Lambda Deployment Headaches, So I Built My Own Blue-Green Deployment System

We've all had that moment.

You push a new Lambda version to production, hit deploy, and then spend the next few minutes refreshing dashboards and logs hoping nothing breaks.

I've been there more times than I'd like to admit.

Sometimes deployments worked perfectly. Other times I would see unexpected 503 errors, traffic routing issues, or situations where rolling back wasn't as fast as I wanted. The Lambda deployment itself wasn't usually the problem. The challenge was everything around it—traffic routing, validation, and rollback.

After hitting the same deployment issues multiple times, I decided to build a deployment model that matched how I actually wanted releases to work.

The result was a Terraform-driven blue-green deployment framework for AWS Lambda that gives me:

Zero-downtime deployments
IP-based canary testing
Weighted traffic shifting
Instant rollback
No CodeDeploy
No Route53 dependency
No hardcoded VPC IDs

Everything is controlled through Terraform.

The Problem I Wanted to Solve

AWS already provides deployment options for Lambda through aliases and CodeDeploy.

For many teams, that's enough.

But in my case, there were a few limitations that kept bothering me.

1. I Wanted to Test Production Before Users Did

When deploying a new version, I wanted a way to access the new release from my own machine while everyone else continued using the stable version.

With standard Lambda deployments, that wasn't straightforward.

2. Rollbacks Needed To Be Instant

If something goes wrong after deployment, I don't want to wait for another deployment process.

I want traffic to switch back immediately.

3. I Wanted Full Traffic Control

Sometimes I want:

10% rollout
25% rollout
50% rollout
75% rollout

Not just predefined deployment strategies.

4. I Wanted Less Moving Parts

CodeDeploy works, but it introduces additional services, configurations, hooks, and deployment definitions.

For this project, I wanted something simpler.

The Idea

Instead of thinking about Lambda deployments as version updates, I started thinking about them the same way ECS blue-green deployments work.

What if two Lambda environments always existed?

One would be stable.

One would contain the new release.

Traffic could simply move between them.

That led to this architecture:

Blue Lambda (current production)
Green Lambda (new version)
Blue Target Group
Green Target Group
ALB Listener Rules controlling traffic

The important detail is that both environments always exist.

Nothing is destroyed.

Nothing is recreated.

Traffic is simply redirected.

That's what eliminates deployment interruptions.

Everything Is Controlled Through Terraform

The deployment workflow is driven through a simple Terraform configuration.

nginx-lambda = {
  family          = "nginx-svc"

  blue_image      = ".../nginx-lambda:blue"
  green_image     = ".../nginx-lambda:green"

  active_color    = "blue"

  activate_canary = false
  green_weight    = 0
}

Whenever I want to change deployment behavior, I update the values and run:

terraform apply

That's it.

No deployment pipelines.

No manual traffic switching.

No complicated release process.

Phase 1: Canary Testing

The first thing I do after deploying a new version is test it myself.

I enable canary mode:

activate_canary = true
green_weight    = 0

This creates a higher-priority ALB rule.

If requests come from my whitelisted IP address:

My IP → Green Version

Everyone else still reaches:

Users → Blue Version

This allows me to validate:

Application functionality
API responses
Headers
Authentication flows
Database interactions

All in production.

Without exposing the new version to actual users.

Phase 2: Gradual Traffic Shifting

Once canary testing looks good, I begin moving real traffic.

For example:

green_weight = 50

Now:

50% → Blue
50% → Green

The Application Load Balancer handles traffic distribution automatically.

Because both target groups already exist and already have Lambda functions attached, traffic shifting happens smoothly.

No downtime.

No target registration delays.

No deployment gaps.

Phase 3: Full Promotion

When confidence is high enough, I move all traffic to Green.

green_weight = 100

At this point:

100% Users → Green

But here's the part I like most.

Blue still exists.

It's sitting there untouched.

Ready for rollback.

Phase 4: Instant Rollback

Every deployment system claims to support rollback.

The real question is:

How fast?

With this setup, rollback is literally:

green_weight = 0

Apply Terraform.

Traffic immediately returns to Blue.

No redeployment.

No waiting for Lambda versions.

No waiting for target group registration.

No waiting for containers to start.

The previous stable environment never disappeared.

What About The Next Release?

This was actually one of the hardest design problems.

Let's say:

Blue = v1
Green = v2

After promoting v2, what happens when v3 arrives?

The answer is surprisingly simple.

Rotate the images.

blue_image  = ".../nginx-lambda:green"
green_image = ".../nginx-lambda:yellow"

Now:

Blue  = v2 (stable)
Green = v3 (candidate)

The deployment cycle repeats.

Canary.

Weighted rollout.

Promotion.

Rollback if needed.

Forever.

Why I Stopped Seeing 503 Errors

Before building this, I experimented with several approaches.

Lambda Alias Switching

This occasionally introduced timing issues during updates.

Dynamic Target Groups

Using conditional resources sometimes caused Terraform to destroy resources before listener rules updated.

Permission Propagation Delays

Lambda permissions occasionally weren't fully propagated before attachments occurred.

The common problem was simple:

Resources were being created and destroyed during deployment.

Eventually I realized the best deployment is often the one that changes the least.

So I stopped changing infrastructure during releases.

Instead:

Blue always exists
Green always exists
Target groups always exist
Attachments always exist

The only thing that changes is traffic routing.

That's why the process is so reliable.

GitHub Repository

https://github.com/jayakrishnayadav24/lambda-blue-green-deployment

Final Thoughts

The biggest lesson from building this wasn't about Lambda or Terraform.

It was about reducing moving parts.

Instead of making deployments smarter, I made them simpler.

Both environments stay alive.

Traffic moves gradually.

Rollback is immediate.

And I can personally validate a release before customers ever touch it.

After several production deployments using this pattern, it's become one of my favorite deployment strategies for Lambda workloads.

If you've been fighting deployment-related downtime or complicated release workflows, you might find this approach useful too.

What If You Need Two ArgoCD Instances in One EKS Cluster?

POTHURAJU JAYAKRISHNA YADAV — Sat, 30 May 2026 08:26:58 +0000

Most ArgoCD tutorials start the same way:

Deploy ArgoCD.

Connect a Git repository.

Create an Application.

Done.

But what happens when multiple teams start sharing the same Kubernetes cluster?

Recently I was exploring a scenario where a single Amazon EKS cluster needed to support two different ArgoCD environments:

A Custom ArgoCD instance for the Platform Team
A Managed ArgoCD instance for Application Teams

The Custom ArgoCD would manage infrastructure components such as:

cert-manager
external-dns
monitoring
ingress controllers

While the Managed ArgoCD would be used by application teams to deploy:

APIs
frontends
microservices
business applications

The challenge was figuring out how to keep both environments isolated while still sharing the same EKS cluster, Cognito User Pool, and AWS Application Load Balancer.

At first, I thought:

Why not just use a single ArgoCD instance?

The more I thought about it, the more questions came up.

How do we separate responsibilities?
How do we avoid accidental changes?
How do we provide SSO?
How do we keep costs under control?

That led me down an interesting path:

Can we run two ArgoCD instances inside the same EKS cluster, authenticate both using Cognito SSO, and still share the same ALB?

Turns out we can.

This article walks through the architecture, challenges, lessons learned, and a few surprises along the way.

The Scenario

Imagine you have a Kubernetes cluster used by multiple teams.

The Platform Team owns:

cert-manager
external-dns
monitoring
ingress controllers
cluster add-ons

The Application Teams own:

APIs
frontends
microservices
business applications

A single ArgoCD instance technically works.

But should both groups operate from the same GitOps control plane?

For smaller environments, probably yes.

For larger environments, things start getting messy.

Different permissions.

Different ownership boundaries.

Different operational responsibilities.

That was the motivation behind splitting ArgoCD into two instances.

The Goal

The architecture I wanted looked like this:

EKS Cluster

├── ArgoCD (Platform Team) - Custom argocd
│     ├── cert-manager
│     ├── external-dns
│     └── monitoring
│
└── ArgoCD (Application Teams) - Managed argocd
      ├── frontend
      ├── backend
      └── business applications

And on top of that:

Single Sign-On
Fully automated provisioning
Infrastructure as Code
Minimal AWS cost

The Final Architecture

The final design consisted of:

Developer
    |
    | HTTPS
    v

Shared AWS ALB
(group.name = argocd)

    |
    +---------------------+
    |                     |
    v                     v

argocd              argocd2
Namespace:          Namespace:
argocd              argocd-managed

Platform Team       Application Teams

    |                     |
    +----------+----------+
               |
               v

AWS Cognito User Pool

Both ArgoCD instances run inside the same EKS cluster.

The difference is that each lives in its own namespace.

argocd
argocd-managed

This gives logical separation without requiring another cluster.

Why Cognito?

Before choosing Cognito, I looked at AWS IAM Identity Center.

Identity Center is generally the preferred enterprise solution.

However, for this learning exercise I wanted everything automated using Terraform.

My ideal deployment process was:

terraform apply

and automatically create:

User Pool
App Clients
Groups
Users
Passwords

without opening the AWS Console.

Cognito made that surprisingly easy.

Building the Authentication Layer

The first step was creating a Cognito User Pool.

Each ArgoCD instance gets its own App Client.

Visually, it looked like this:

Cognito User Pool

├── App Client
│     └── ArgoCD Platform
│
└── App Client
      └── ArgoCD Managed

Here's what it looked like in AWS.

One thing I particularly liked was eliminating manual user creation.

Terraform creates:

User Pool
Domain
App Clients
Groups
Admin User

Everything becomes Infrastructure as Code.

No temporary passwords.

No manual console work.

No forgotten setup steps.

Connecting ArgoCD to Cognito

ArgoCD supports OIDC natively.

That means Cognito can act as the identity provider.

The authentication flow becomes:

User
  |
  v
ArgoCD
  |
  v
Cognito Hosted UI
  |
  v
Authentication
  |
  v
OIDC Callback
  |
  v
ArgoCD Dashboard

Once logged in, ArgoCD reads the Cognito groups from the token and maps them to ArgoCD roles.

That means administrators are controlled centrally through Cognito instead of managing users inside ArgoCD itself.

The Shared ALB Experiment

This was probably the most interesting part.

Initially I considered creating two separate Application Load Balancers.

One for each ArgoCD instance.

That would work.

But it also means:

More AWS resources
More management
More cost

Then I remembered the AWS Load Balancer Controller supports Ingress Groups.

That changed everything.

Instead of:

ALB #1 → ArgoCD
ALB #2 → ArgoCD Managed

I could use:

One ALB
Two Hostnames
Two ArgoCD Instances

The ALB creates listener rules based on the hostname.

And from Kubernetes:

Both ingress resources point to the same ALB.

The routing happens automatically based on the Host header.

A nice cost optimization with very little additional complexity.

The Problem Nobody Warned Me About

The first ArgoCD deployment worked perfectly.

The second one failed immediately.

The error looked something like this:

customresourcedefinitions.apiextensions.k8s.io already exists

At first it wasn't obvious.

Then it clicked.

ArgoCD installs CRDs.

CRDs are cluster-scoped resources.

The first installation already created them.

The second installation attempted to create them again.

Kubernetes refused.

The fix was simple:

crds:
  install: false

Once I disabled CRD installation for the second ArgoCD instance, everything worked.

It was one of those issues that takes a while to understand but only seconds to fix.

The Login Experience

After deployment, the user experience was exactly what I wanted.

First, users see the ArgoCD login screen.

Selecting the Cognito login option redirects users to the Hosted UI.

After successful authentication, users land directly inside ArgoCD.

No local accounts.

No shared admin credentials.

No Kubernetes secret lookups.

Just SSO.

Another Small Gotcha

While testing Applications, I ran into this error:

app path is not a directory

I had accidentally pointed ArgoCD to a YAML file instead of a directory.

ArgoCD expects:

path: manifests/

not:

path: manifests/app.yaml

A small detail, but one that can waste a surprising amount of time when you're learning.

What I Learned

The most valuable lesson wasn't about Cognito.

It wasn't about Terraform.

It wasn't even about ArgoCD.

It was about ownership.

As environments grow, separating responsibilities becomes more important than simply making things work.

Running a Custom ArgoCD for platform operations and a Managed ArgoCD for application teams gave:

Better operational boundaries
Cleaner RBAC
Clear ownership between teams
Reduced risk of accidental changes
Independent GitOps workflows

The platform team can safely manage infrastructure applications while application teams operate in their own ArgoCD environment without interfering with cluster-level services.

And because both instances share a single ALB, the cost impact remained minimal.

Would I Do This Again?

For a small team?

Probably not.

A single ArgoCD instance is easier to manage.

For organizations with dedicated platform teams and application teams?

Absolutely.

The pattern scales nicely, provides cleaner ownership boundaries, and integrates well with SSO.

Most importantly, the entire environment can be created with a single:

terraform apply

which is exactly what I wanted from the beginning.

If you've experimented with multiple ArgoCD instances or different GitOps multi-tenancy approaches, I'd love to hear how you've solved it.

Happy GitOps 🚀

GitHub Repository

https://github.com/jayakrishnayadav24/eks-argocd-cognito-sso

Zero-Downtime Blue-Green and IP-Based Canary Deployments on ECS Fargate

POTHURAJU JAYAKRISHNA YADAV — Sat, 23 May 2026 08:36:49 +0000

Most ECS blue-green deployment tutorials eventually lead to the same stack:

AWS CodeDeploy
Deployment groups
AppSpec files
Lifecycle hooks
Weighted traffic shifting
Complex rollback orchestration

And while CodeDeploy works, I kept running into one practical limitation during real deployments:

I couldn’t let my internal team validate a new release on the actual production URL before exposing it to customers.

That became the entire motivation behind this setup.

I didn’t want:

separate staging domains
duplicate ALBs
temporary preview environments
“almost production” testing

I wanted something much simpler:

Internal users should see the new version first
Customers should continue seeing the stable version
Both should use the same production domain
Rollback should be immediate
Deployments should remain fully zero downtime

So I built a Terraform-driven deployment workflow using:

ECS Fargate
Application Load Balancer (ALB)
ALB listener priorities
Source IP routing
Terraform

without using CodeDeploy.

After running this setup in practice, I ended up preferring it for many ECS workloads.

The Core Idea

Both BLUE and GREEN environments run behind the same ALB.

Internal office/VPN IPs get routed to GREEN first.

Everyone else continues hitting BLUE.

That means QA and internal teams can validate the new release directly on the real production infrastructure before public rollout begins.

Same:

domain
SSL certificate
ALB
authentication flow
redirects
networking path

No “staging surprises” later.

A lot of deployment issues only appear on the real production routing path.

Real Example

Internal users open:

https://nginx.jayakrishnayadav.cloud

…and immediately see the GREEN version.

Meanwhile, public users continue seeing BLUE.

No DNS switching.

No duplicate infrastructure.

Just ALB listener routing.

Architecture Overview

The deployment flow looks like this:

                ┌────────────────────┐
                │   Application LB   │
                └─────────┬──────────┘
                          │
         ┌────────────────┴────────────────┐
         │                                 │
 Internal Office/VPN IPs             Public Users
         │                                 │
         ▼                                 ▼
   GREEN Target Group               BLUE Target Group
         │                                 │
    ECS GREEN Tasks                  ECS BLUE Tasks

The canary routing rule gets evaluated first.

If the request source IP matches internal CIDRs, traffic goes to GREEN.

Everything else falls back to BLUE.

Terraform Structure

I kept the Terraform layout modular so it could be reused across multiple services.

.
├── main.tf
├── variables.tf
├── outputs.tf
├── env/
│   ├── backend.hcl
│   └── terraform.tfvars
├── modules/
│   ├── vpc/
│   ├── iam/
│   ├── alb/
│   ├── ecs-cluster/
│   └── ecs-blue-green-service/
└── scripts/
    └── zero-downtime-test.sh

Each ECS service gets:

BLUE ECS service
GREEN ECS service
BLUE target group
GREEN target group
production listener rule
optional canary listener rule

ALB Listener Rule Logic

The entire deployment behavior depends on ALB listener priorities.

The canary listener rule gets evaluated first.

If the request source IP matches internal CIDRs, traffic gets forwarded to GREEN.

resource "aws_lb_listener_rule" "canary" {
  count    = var.activate_canary ? 1 : 0
  priority = 99

  condition {
    source_ip {
      values = var.canary_source_ips
    }
  }

  condition {
    host_header {
      values = ["nginx.jayakrishnayadav.cloud"]
    }
  }

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.green.arn
  }
}

The production rule remains below it:

resource "aws_lb_listener_rule" "production" {
  priority = 100

  condition {
    host_header {
      values = ["nginx.jayakrishnayadav.cloud"]
    }
  }

  action {
    type             = "forward"
    target_group_arn = local.active_target_group
  }
}

That’s it.

No weighted routing.

No lifecycle hooks.

Just listener priorities.

Real Deployment Workflow

This wasn’t built as a theoretical architecture exercise.

I tested the rollout flow directly from Terraform while continuously validating traffic behavior against live ECS Fargate services.

Terraform initialization:

terraform init -backend-config=env/backend.hcl

Deployment apply:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

During canary validation, I continuously verified my public IP:

curl ifconfig.me

That mattered because the ALB source-IP rule decides whether traffic reaches:

BLUE
GREEN

Once my IP matched the configured canary CIDRs, traffic immediately started routing to GREEN.

Deployment Flow

The nice part about this setup is that everything becomes variable-driven.

Step 1 — Normal Production State

BLUE handles all production traffic.

GREEN remains scaled down.

enable_canary   = false
activate_canary = false
promote_to_all  = false

Apply:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

Result:

BLUE active
GREEN inactive
minimal Fargate cost

Step 2 — Start GREEN Tasks

Now we start the GREEN environment.

enable_canary   = true
activate_canary = false
promote_to_all  = false

Apply again:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

At this stage:

GREEN tasks start
ECS health checks complete
ALB target registration completes
no production traffic reaches GREEN yet

Users never hit partially starting containers.

Step 3 — Internal Canary Validation

Now we enable canary routing.

enable_canary   = true
activate_canary = true
promote_to_all  = false

Apply again:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

Now:

internal office/VPN users hit GREEN
public users continue hitting BLUE

This became the most valuable phase of the deployment workflow.

Because now:

QA validates production behavior
developers inspect logs
authentication flows get tested
sessions and redirects get verified

while customers remain completely unaffected.

Internal Canary Routing

This is the ALB listener rules view while canary routing is enabled.

The priority 99 rule matches internal source IPs and forwards them to GREEN, while everyone else continues hitting BLUE.

Step 4 — Promote GREEN to Production

Once validation looks good:

enable_canary   = true
activate_canary = false
promote_to_all  = true

Apply again:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

Now:

production listener switches to GREEN
BLUE scales down
all users see the new version

No downtime occurs.

Traffic simply moves from one target group to another.

Verifying Zero Downtime

I didn’t want to assume the deployment was safe.

I wanted to verify it continuously during rollout.

So I used a simple curl-based validation script that continuously hit both applications while traffic shifted between BLUE and GREEN.

for i in {1..100}
do
  for url in \
    "https://nginx.jayakrishnayadav.cloud/" \
    "https://apache.jayakrishnayadav.cloud/"
  do
    response=$(curl -k -s -w " HTTPSTATUS:%{http_code}" "$url")

    body=${response% HTTPSTATUS:*}
    status=${response##*HTTPSTATUS:}

    if [[ $body == *"BLUE - v"* ]]; then
      color="BLUE"
    elif [[ $body == *"GREEN - v"* ]]; then
      color="GREEN"
    else
      color="UNKNOWN"
    fi

    echo "Run: $i | URL: $url | Status: $status | Version: $color"
  done
done

Output during deployment:

You can clearly see:

HTTP 200 responses throughout deployment
no failed requests
no 503s
clean traffic movement from BLUE to GREEN

That confirmed the deployment was genuinely zero downtime.

Production Promotion View

After promotion:

the canary rule disappears
the production listener points directly to GREEN
all traffic reaches the new version
BLUE scales down to zero

Clean and simple.

Rollback

Rollback became extremely simple.

I just reverted the Terraform variables:

enable_canary   = false
activate_canary = false
promote_to_all  = false

Apply Terraform again:

terraform apply \
  -var-file=env/terraform.tfvars \
  -lock=false \
  -auto-approve

ALB immediately routes traffic back to BLUE.

The rollback process stays predictable because traffic switching is entirely controlled through ALB listener rules.

HTTPS Configuration

The ALB uses ACM certificates for HTTPS.

Listeners:

Port 80 → redirect to HTTPS
Port 443 → production traffic
optional internal listener → restricted to internal CIDRs

Example:

test_listener_allowed_cidrs = [
  "160.30.39.198/32"
]

That keeps internal preview traffic private while still using the same production infrastructure.

Cost Optimization

One thing I specifically wanted to avoid was permanently doubling infrastructure cost.

Normal state:

only BLUE tasks run

Deployment window:

BLUE + GREEN both run temporarily

After promotion:

BLUE scales down again

So infrastructure cost only increases briefly during deployments.

Final Thoughts

This project started because I wanted a very practical deployment workflow:

Internal users should validate the new version on the actual production URL before customers ever see it.

Once I implemented that using ALB listener priorities and source IP routing, I realized I no longer really needed CodeDeploy for this workflow.

The end result became:

simpler
easier to operate
easier to rollback
easier to debug
easier to reason about
fully zero downtime

And because everything is Terraform-driven, the deployment process stays reproducible and predictable.

GitHub Repository

Full Terraform implementation:

https://github.com/jayakrishnayadav24/ecs-blue-green-deployment/tree/canary

I Replaced 47 DevOps Scripts With One AI Agent — Here’s What Happened

POTHURAJU JAYAKRISHNA YADAV — Fri, 17 Apr 2026 10:15:34 +0000

The Hook: I Was Wrong About Automation

I thought I automated DevOps.

I had 47 deployment scripts.

Then I started replacing them with an AI agent —
and most scripts became unnecessary.

Not by following instructions.
By making decisions.

And the 2 AM debugging stopped.

Note: The code shown here is simplified for clarity.
The GitHub repo contains a more modular implementation.

That's when I realized:

I wasn't automating.
I was hardcoding decisions.

I gave Claude a list of AWS tools and one instruction: "Deploy this app."

No hardcoded logic.
No decision trees.

Just: describe the goal, Claude figures out how.

Deployment time: 3 hours → in minutes for most cases.

(Built this while deploying real workloads on AWS: Docker, ECS, EC2, IAM.)

🎯 Is This For You?

This post is for:

DevOps engineers with 10+ deployment scripts
Platform engineers building internal developer platforms
Anyone exploring AI agents beyond chatbots
Teams on AWS (Docker, EC2, ECS, IAM)

If manual deployment takes > 30 minutes, read this.

✨ What's Possible

Old way (manual):

aws ecs create-cluster --cluster-name prod
aws ecs register-task-definition --family myapp ...
# [50+ commands, 3 hours, manual debugging]

New way (agent):

agent.run("Deploy FastAPI with PostgreSQL, auto-scale to 20, "
          "minimal IAM perms, CloudWatch monitoring")

# ✅ Done in minutes for most cases, with significantly reduced debugging

No scripts.

Just Claude thinking out loud about your infrastructure.

🚀 How It Actually Works

You describe the goal (natural language):

"Deploy my app on 5 ECS tasks, auto-scale to 20 on high CPU"

Claude gets these tools:

dispatcher = {
    "ecs__create_cluster": ecs.create_cluster,
    "ecs__register_task": ecs.register_task_definition,
    "ecs__create_service": ecs.create_service,
    "iam__create_role": iam.create_role,  # For permissions
}

Claude reasons autonomously:

"User wants ECS deployment.
 I need to:
 1. Check if cluster exists
 2. Register task definition
 3. Create service with 5 tasks
 4. Setup auto-scaling
 5. Verify it's running"

It executes:

Call ecs__create_cluster() → Get cluster ARN
Call ecs__register_task() → Get task definition
Call ecs__create_service() → Get running tasks
Call ecs__setup_autoscaling() → Confirmed
Return: "All 5 tasks running, auto-scaling 5-20"

** Minimal manual intervention in most cases. Just reasoning + execution + feedback loops.**

⚠️ The Problem I Started With

I had:

deploy_docker.py — 150 lines of Docker logic
deploy_ec2.py — 200 lines of EC2 logic
deploy_ecs.py — 300 lines of ECS logic
3 routers trying to chain them together
0 way to handle "deploy to both EC2 AND ECS"

Every new service = new script. Every new workflow = rewrite everything.

Solution: Reduce rigid scripts — let the agent handle orchestration logic dynamically.

Instead of routing to specific tools, give Claude all available tools and let it decide which ones to use, in which order, adapting as it goes.

# All tools in one place
tools = {
    "docker__run": run_container,
    "ec2__create": create_instance,
    "ecs__deploy": create_service,
    "iam__create_role": create_role,
    # ... more tools
}

# Let Claude orchestrate
agent.run("Deploy with Docker locally, then scale to ECS production")
# Claude figures out: Docker first, then ECS, then IAM for permissions

🏗️ The Architecture (3 Layers)

┌──────────────────────────────┐
│ User Goal (natural language) │  
│ "Deploy scalable production  │
│  stack with auto-scaling"    │
└──────────┬───────────────────┘
           │
           ▼
    ┌─────────────────────────────────────────────┐
    │  Claude (Bedrock)                           │
    │  ← Reads goal + available tools             │
    │  ← Decides sequence of actions              │
    │  ← Adapts when things fail                  │
    └──────────────┬────────────────────────────┘
                   │
    ┌──────────────┴──────────────┐
    │                             │
    ▼                             ▼
┌─────────────┐  ┌──────────────────┐
│ AWS APIs    │  │ Conversation     │
│ (via boto3) │  │ Memory (DynamoDB)│
│ ← Executes  │  │ ← Recalls setup  │
│   decisions │  │   from last week │
└─────────────┘  └──────────────────┘
    │                    │
    └────────┬───────────┘
             │
    ┌────────▼─────────┐
    │ Actual Resources │
    │ EC2, ECS, Docker │
    │      IAM, etc    │
    └──────────────────┘

3 moving parts:

Claude — Reasons about the task
Tools — Execute AWS API calls
Memory — Remember past deployments for coherence

🔧 The Setup (Code Foundation)

Here's the entire base system in ~100 lines. Full code on GitHub.

import boto3
from agents.memory import load_history, save_message

MODEL = "apac.amazon.nova-lite-v1:0"  # Fast, cheap 

class BaseAgent:
    """Foundation for all agents (Docker, EC2, ECS, IAM)"""
    AGENT_KEY = None  # "docker", "ec2", etc.
    SYSTEM_PROMPT = ""  # Agent's personality
    CAPABILITIES = []  # What this agent can do

    def __init__(self, name: str, region: str = "us-east-1"):
        self.name = name
        self.session = boto3.Session(region_name=region)
        self._bedrock = None

    @property
    def bedrock(self):
        """Lazy load Bedrock connection (only when needed)"""
        if not self._bedrock:
            self._bedrock = self.session.client("bedrock-runtime")
        return self._bedrock

    def run(self, task: str, session_id: str = None) -> dict:
        """The agentic loop: think → decide → act → repeat"""
        tools = self.get_tools()
        dispatcher = self.get_dispatcher()
        messages = load_history(session_id) + [{"role": "user", "content": [{"text": task}]}]

        for _ in range(10):  # Max 10 iterations to prevent infinite loops
            # Ask Claude what to do
            response = self.bedrock.converse(
                modelId=MODEL,
                system=self.SYSTEM_PROMPT,
                messages=messages,
                tools=tools,
                toolUseDepth=2,
            )

            # Is Claude done?
            if response["stopReason"] == "endTurn":
                return {"status": "SUCCESS", "message": response["content"]}

            # Execute tools Claude wants to call
            tool_results = []
            for tool_use in response["content"]:
                tool_name = tool_use["toolUse"]["name"]
                try:
                    result = dispatcher<a href="tool_use["toolUse"]["input"]">tool_name</a>
                except Exception as e:
                    result = {"error": str(e), "status": "failed"}

                tool_results.append({"toolUseId": tool_use["toolUse"]["toolUseId"],
                                    "content": [{"json": result}]})

            # Add Claude's decision + results to conversation
            messages.append({"role": "assistant", "content": response["content"]})
            messages.append({"role": "user", "content": [{"toolResult": tr} for tr in tool_results]})

            # Save for future reference
            if session_id:
                save_message(session_id, messages[-1])

        return {"status": "FAILED", "reason": "Max iterations reached"}

What's happening:

Lazy loading (@property bedrock): Don't connect to Claude until needed
Tool loop: Call tools, capture results, show Claude the output
Error handling: Don't crash—tell Claude what failed, let it adapt
Memory: Save each turn so Claude remembers next week

🚀 Agents in Action: 3 Real Examples

DockerAgent: Deploy Locally

class DockerAgent(BaseAgent):
    AGENT_KEY = "docker"
    SYSTEM_PROMPT = """You manage Docker containers.
Rules: Pull image first, check if container exists, use sensible defaults."""

    CAPABILITIES = [
        {"name": "list_containers", "description": "List running containers"},
        {"name": "run_container", "description": "Pull and run image"},
        {"name": "stop_container", "description": "Stop a container"},
    ]

Usage:

docker_agent = DockerAgent()
docker_agent.run("Deploy FastAPI on port 8000 with health check")
# Claude calls: list_containers → run_container → confirms it's running

EC2Agent: Scale to Cloud

class EC2Agent(BaseAgent):
    AGENT_KEY = "ec2"
    SYSTEM_PROMPT = """You manage EC2 instances.
Rules: Tag for organization, verify security groups, create new→test→retire old."""

    CAPABILITIES = [
        {"name": "describe_instances", "description": "List instances"},
        {"name": "create_instance", "description": "Launch new instance"},
        {"name": "stop_instance", "description": "Stop instance"},
    ]

Usage:

ec2_agent = EC2Agent()
ec2_agent.run("Create 2 t2.micro instances, tag as app-server")
# Claude: checks if instances exist → creates 2 → returns IPs → confirms running

ECSAgent: Production Auto-Scaling

class ECSAgent(BaseAgent):
    AGENT_KEY = "ecs"
    SYSTEM_PROMPT = """You manage ECS services at production scale.
Rules: Task definition first, use Fargate, always configure auto-scaling."""

    CAPABILITIES = [
        {"name": "create_cluster", "description": "Create ECS cluster"},
        {"name": "register_task_definition", "description": "Register blueprint"},
        {"name": "create_service", "description": "Deploy service"},
        {"name": "setup_autoscaling", "description": "Configure scaling rules"},
    ]

Usage:

ecs_agent = ECSAgent()
ecs_agent.run("Deploy with 5 tasks, auto-scale to 20 on high CPU, health monitoring")
# Claude orchestrates: cluster → task definition → service → autoscaling → verification

Real Usage: From Local to Production

Scenario: Deploy FastAPI from laptop to production in minutes for most cases, with significantly reduced debugging.

Stage 1: Test locally

docker_agent.run("Run myapp:latest on port 8000", session_id="prod_001")
# ✅ Docker container running

Stage 2: Scale to cloud

ec2_agent.run("Create 2 instances for load balancing", session_id="prod_001")
# ✅ 2 EC2 instances up (same session = Claude remembers port 8000)

Stage 3: Auto-scaling production

ecs_agent.run("Deploy with 5-20 auto-scaling, CloudWatch monitoring", session_id="prod_001")
# ✅ 5 ECS tasks running, auto-scaling 5-20 based on CPU

All with the same session_id: Claude remembers the image name, port, and configuration from Stage 1. Everything connects seamlessly.

📚 Memory: Agents Remember

# Day 1
agent.run("Deploy on Docker with port 8000", session_id="user_123")

# Week later
agent.run("Move this to ECS for production", session_id="user_123")
# Claude reads history from DynamoDB:
# "I remember this app used port 8000. I'll keep that for ECS too."

DynamoDB stores every conversation turn. Claude recalls past decisions and uses them for coherence.

🛡️ Production-Ready Code Patterns

This is what the user feedback emphasized. Real code needs safety.

Pattern 1: Error Handling (Don't Crash, Recover)

# ❌ Wrong: crashes on error
result = dispatcher[tool_name](tool_input)

# ✅ Right: tell Claude about the error
try:
    result = dispatcher[tool_name](tool_input)
except Exception as e:
    result = {
        "error": str(e),
        "status": "failed",
        "request_id": request_id,  # For debugging
    }
# Claude sees the error and tries a different approach

Why this matters: If Docker pull fails, don't give up. Tell Claude: "Pull failed, but let me check if the image is locally cached." Claude adapts.

Pattern 2: Timeouts (Prevent Hanging)

from functools import wraps
import signal

def with_timeout(seconds=30):
    """Prevent tools from running forever"""
    def decorator(func):
        def wrapper(*args, **kwargs):
            def handler(signum, frame):
                raise TimeoutError(f"Tool execution exceeded {seconds}s")

            signal.signal(signal.SIGALRM, handler)
            signal.alarm(seconds)
            try:
                result = func(*args, **kwargs)
                signal.alarm(0)  # Cancel timeout
                return result
            except TimeoutError as e:
                return {"error": str(e), "status": "timeout"}
        return wrapper
    return decorator

@with_timeout(seconds=60)
def create_instance(params):
    """If EC2 creation hangs, timeout after 60 seconds"""
    ec2 = boto3.client("ec2")
    return ec2.create_instances(**params)

Pattern 3: Input Validation (Prevent Bad Requests)

def create_instance(params):
    """Validate before executing"""
    # Check required fields
    required = ["ImageId", "InstanceType"]
    for field in required:
        if field not in params:
            return {"error": f"Missing required field: {field}", "status": "failed"}

    # Validate instance type
    valid_types = ["t2.micro", "t2.small", "t3.medium"]
    if params["InstanceType"] not in valid_types:
        return {
            "error": f"InstanceType must be one of {valid_types}",
            "status": "failed"
        }

    # Now safe to execute
    ec2 = boto3.client("ec2")
    return ec2.create_instances(**params)

Pattern 4: Audit Logging (Proof for Compliance)

import json
from datetime import datetime

def log_action(request_id: str, user: str, action: str, params: dict, result: dict):
    """Log everything for audits"""
    log_entry = {
        "request_id": request_id,
        "timestamp": datetime.utcnow().isoformat(),
        "user": user,
        "action": action,
        "params": json.dumps(params),  # What was requested
        "result_status": result.get("status"),
        "result_code": result.get("error"),
    }

    # Save to CloudWatch Logs or DynamoDB
    dynamodb.put_item(TableName="agent_audit", Item=log_entry)

# In the tool dispatcher:
result = dispatcher[tool_name](tool_input)
log_action(request_id, user_id, tool_name, tool_input, result)

Pattern 5: Cost Guards (Don't Deploy Expensive Mistakes)

MONTHLY_BUDGET = 1000  # $1000/month limit

def estimate_cost(action: str, params: dict) -> float:
    """Estimate AWS cost before executing"""
    if action == "create_instance":
        instance_type = params.get("InstanceType", "t2.micro")
        hourly_rates = {
            "t2.micro": 0.012,
            "t2.small": 0.023,
            "t3.medium": 0.042,
        }
        months_running = 1
        return hourly_rates.get(instance_type, 0) * 730 * months_running

    elif action == "create_rds":
        # RDS: ~$400/month for 20GB
        return 400

    return 0

# In the runnable:
estimated = estimate_cost(tool_name, tool_input)
if estimated > MONTHLY_BUDGET:
    return {
        "error": f"Cost ${estimated:.2f} exceeds budget ${MONTHLY_BUDGET}",
        "status": "rejected",
    }
# Only execute if under budget
result = dispatcher[tool_name](tool_input)

Pattern 6: Role-Based Access (Security Boundaries)

# Define who can do what
USER_PERMISSIONS = {
    "admin": ["create_instance", "delete_instance", "create_role", "delete_role"],
    "developer": ["create_instance", "stop_instance", "deploy_to_ecs"],
    "readonly": ["describe_instances", "list_containers", "get_logs"],
}

def check_permission(user_role: str, action: str) -> bool:
    """Prevent unauthorized actions"""
    allowed = USER_PERMISSIONS.get(user_role, [])
    if action not in allowed:
        return False
    return True

# Before executing:
if not check_permission(user_role, tool_name):
    return {
        "error": f"User role '{user_role}' cannot perform '{tool_name}'",
        "status": "permission_denied",
    }
result = dispatcher[tool_name](tool_input)

✅ Why Agents > Scripts

	Scripts	Agents
New workflow	Rewrite code	Claude adapts
Error recovery	Crashes	Tries alternatives
Reasoning	None (hardcoded)	Full decision log
Maintenance	Grows linearly	One framework
Learning	Manual proof	Automatic audit trail

🎯 Getting Started

Just 3 commands:

git clone https://github.com/jayakrishnayadav24/ai-agents
cd ai-agents
pip install -r requirements.txt

Then try:

from agents.docker_agent import DockerAgent

agent = DockerAgent()
response = agent.run(
    "Deploy FastAPI app on port 8000 with health check"
)
print(response)

That's it. Claude handles the rest.

📚 What's Next (Part 2)

This was the concept. Github:

✅ Full working code (all agents)
✅ DynamoDB setup for memory
✅ Deploying agents to Lambda
✅ Real production patterns (request tracing, cost estimation)
✅ Demo with actual deployments

⚠️ Where This Breaks

Complex compliance environments (manual approvals still needed)
Cost estimation is approximate
Requires well-defined tools (garbage in → garbage out)

The Bottom Line

Before: 47 scripts, 3 hours/deployment, constant debugging

After: 1 agent framework, in minutes for most cases, with significantly reduced debugging/deployment.

Why? Because Claude doesn't follow scripts. Claude plans and executes the steps based on the goal.

It adapts. It learns. It remembers.

That's not automation anymore. That's the future of infrastructure.

How many deployment scripts are you still maintaining?

10+? 20+? 50+?

I want to see how bad this problem is. Drop a comment 👇

Your feedback shapes Part 2.

The User Interface & Ground-Truth Testing

POTHURAJU JAYAKRISHNA YADAV — Tue, 14 Apr 2026 10:49:18 +0000

Part 5 of 5 | ← Part 4 | Complete Series

Streamlit Overview

Streamlit lets you build data apps in pure Python—no HTML, CSS, or JavaScript needed.

import streamlit as st
from datetime import datetime
import pandas as pd
import requests

# Page config
st.set_page_config(
    page_title="IPL AI Assistant",
    page_icon="🏏",
    layout="wide",
)

# Title
st.title("🏏 IPL AI Assistant")
st.subtitle("Predictions + Q&A Powered by ML")

# Tabs
tab1, tab2, tab3 = st.tabs(["💬 Chat", "🎯 Predict", "📊 Metrics"])

Tab 1: Chat Interface

with tab1:
    st.header("Ask Anything")

    # Initialize session state
    if "messages" not in st.session_state:
        st.session_state.messages = []

    # Display chat history
    for message in st.session_state.messages:
        with st.chat_message(message["role"]):
            st.markdown(message["content"])

    # User input
    if user_input := st.chat_input("Ask about IPL..."):
        st.session_state.messages.append({"role": "user", "content": user_input})

        with st.chat_message("user"):
            st.markdown(user_input)

        # Call backend
        try:
            response = requests.post(
                "http://localhost:8000/chat",
                json={"message": user_input},
                timeout=5,
            )
            response.raise_for_status()

            result = response.json()
            assistant_message = result.get("message", "I couldn't understand that.")

            st.session_state.messages.append({
                "role": "assistant",
                "content": assistant_message,
            })

            with st.chat_message("assistant"):
                st.markdown(assistant_message)

        except Exception as e:
            st.error(f"❌ Backend error: {str(e)}")

Key Concepts:

Session State: st.session_state.messages persists across reruns
- When user submits a message, Streamlit reruns the entire script
- Session state preserves conversation history
- Without it: chat history disappears on each input
Chat Message: st.chat_message() renders messages with role-based styling
- "user" = right-aligned, blue background
- "assistant" = left-aligned, gray background
Chat Input: st.chat_input() provides a textbox with submission handling
- Returns None until user submits
- Automatically clears after submission

Tab 2: Prediction Interface

with tab2:
    st.header("Match Prediction Simulator")

    col1, col2 = st.columns(2)

    with col1:
        st.subheader("Teams")
        batting_team = st.selectbox(
            "Batting Team:",
            [
                "Mumbai Indians",
                "Chennai Super Kings",
                "Royal Challengers Bangalore",
                "Kolkata Knight Riders",
                # ... all 10 teams
            ],
        )
        bowling_team = st.selectbox(
            "Bowling Team:",
            options=["All teams except batting team"],
            index=0,
        )

    with col2:
        st.subheader("Venue")
        venue = st.text_input("Ground name:", "Wankhede")

    st.subheader("Pre-Match Form")
    col1, col2, col3, col4 = st.columns(4)

    with col1:
        h2h_rate = st.slider(
            "H2H Win Rate (Batting Team)",
            0.0, 1.0, 0.5, 0.05,
        )

    with col2:
        overall_rate = st.slider(
            "Overall Win Rate",
            0.0, 1.0, 0.5, 0.05,
        )

    with col3:
        venue_rate = st.slider(
            "Venue Win Rate",
            0.0, 1.0, 0.5, 0.05,
        )

    with col4:
        rolling_rate = st.slider(
            "Last 5 Matches Win Rate",
            0.0, 1.0, 0.5, 0.05,
        )

    st.subheader("Toss")
    col1, col2 = st.columns(2)

    with col1:
        toss_win = st.radio("Who won toss?", ["Batting Team", "Bowling Team"])
        toss_win = 1 if toss_win == "Batting Team" else 0

    with col2:
        toss_choice = st.radio("Toss choice?", ["Bat", "Field"])
        toss_choice = toss_choice.lower()

    # Predict button
    if st.button("🎯 Predict Match Outcome", use_container_width=True):
        try:
            response = requests.post(
                "http://localhost:8000/predict",
                json={
                    "batting_team": batting_team,
                    "bowling_team": bowling_team,
                    "venue": venue,
                    "h2h_rate": h2h_rate,
                    "overall_rate": overall_rate,
                    "venue_rate": venue_rate,
                    "rolling_rate": rolling_rate,
                    "toss_win": toss_win,
                    "toss_choice": toss_choice,
                },
                timeout=5,
            )
            response.raise_for_status()

            result = response.json()
            winner = result["winner"]
            confidence = result["confidence"]

            st.success(
                f"### 🏆 {winner} wins!\n"
                f"**Confidence:** {confidence:.1%}"
            )

            # Show prediction breakdown
            st.info(
                f"**Model:** {result['model']}\n\n"
                f"**Reasoning:**\n"
                f"- H2H: {h2h_rate:.0%}\n"
                f"- Form: {rolling_rate:.0%}\n"
                f"- Venue: {venue_rate:.0%}\n"
            )

        except Exception as e:
            st.error(f"❌ Prediction failed: {str(e)}")

Key UI Patterns:

st.selectbox() — Dropdown selector
st.slider() — Range input (0.0-1.0)
st.radio() — Single-choice radio buttons
st.columns() — Grid layout (col1, col2, etc.)
st.button() — Form submission
st.success/error/info() — Colored alerts

Tab 3: Metrics & Transparency

with tab3:
    st.header("Model Performance")

    # Load metrics
    import json
    with open("models/metrics.json") as f:
        metrics = json.load(f)

    col1, col2, col3 = st.columns(3)
    col1.metric("Test Accuracy", f"{metrics['accuracy']:.1%}")
    col2.metric("Precision", f"{metrics['precision']:.1%}")
    col3.metric("Recall", f"{metrics['recall']:.1%}")

    st.subheader("Confusion Matrix")
    st.image("models/confusion_matrix.png", use_column_width=True)

    st.subheader("Feature Importance")
    importance_df = pd.DataFrame({
        "Feature": ["h2h_rate", "rolling_rate", "venue_rate", ...],
        "Importance": [0.32, 0.28, 0.18, ...],
    }).sort_values("Importance", ascending=False)

    st.bar_chart(importance_df.set_index("Feature"))

    st.subheader("Q&A Engine")
    st.info(
        f"**Total Q&A Pairs:** 42,523\n\n"
        f"**Vocabulary Size:** 18,394\n\n"
        f"**Match Strategy:** TF-IDF + Cosine Similarity (threshold: 0.15)\n\n"
        f"**Coverage:** {(42523 / 50000 * 100):.1f}% of expected cricket topics"
    )

Testing: The Foundation of Trust

Good tests = confidence in deployment.

Test Structure

# tests/test_qa.py

import pytest
import pandas as pd
from src.build_qa_model import answer_question
from joblib import load

# Load test data
test_df = pd.read_csv("datasets/ipl_2008_2024_complete.csv")
qa_model = load("models/qa_model.joblib")

# Extract Q&A components
tfidf = qa_model["tfidf"]
Q_matrix = qa_model["Q_matrix"]
answers = qa_model["answers"]

Test 1: Specific Match Facts

def test_match_lookup():
    """Can we answer specific match questions?"""
    questions = [
        "Who won the match on 2024-04-01 between MI and RR?",
        "How many runs were scored by MI in 2024-04-01?",
        "What was the result of MI vs RR on 2024-04-01?",
    ]

    for question in questions:
        answer, score = answer_question(
            question, tfidf, Q_matrix, answers, threshold=0.15
        )

        assert answer is not None, f"Failed on: {question}"
        assert len(answer) > 10, f"Answer too short: {answer}"
        assert score > 0.15, f"Confidence too low: {score}"

Why data-driven?

Not hardcoded: "answer == 'Mumbai Indians wins'"
CSV-based: Pulls real facts from dataset
Robust: Works even if answer phrasing changes

Test 2: Aggregate Statistics

def test_most_wins():
    """Can we retrieve aggregate stats?"""
    questions = [
        "Which team has won most matches?",
        "Who has most IPL titles?",
        "Team with highest win percentage?",
    ]

    for question in questions:
        answer, score = answer_question(
            question, tfidf, Q_matrix, answers, threshold=0.15
        )

        # Verify answer is a valid team name
        assert answer is not None
        valid_teams = ["Mumbai Indians", "CSK", "RCB", ...]
        assert any(team in answer for team in valid_teams)

Test 3: Head-to-Head

def test_head_to_head():
    """Can we answer H2H questions?"""
    questions = [
        "Head-to-head record between MI and CSK?",
        "Does MI have winning record vs KKR?",
        "Who dominates MI vs RR?",
    ]

    for question in questions:
        answer, score = answer_question(
            question, tfidf, Q_matrix, answers, threshold=0.15
        )

        assert answer is not None
        # H2H answers contain numbers (win counts)
        assert any(char.isdigit() for char in answer)

Test 4: Threshold Behavior

def test_threshold_protects_low_confidence():
    """Low-confidence matches are rejected."""
    nonsense = "xyzabc qwerty asdfgh"  # Gibberish

    answer, score = answer_question(
        nonsense, tfidf, Q_matrix, answers, threshold=0.15
    )

    # Model shouldn't hallucinate
    assert answer is None, f"Got answer for nonsense: {answer}"
    assert score < 0.15

Test 5: ML Model Predictions

def test_model_predictions():
    """Can we predict match winners?"""
    from src.train import normalize_teams, engineer_features
    from joblib import load

    ml_model = load("models/model.joblib")
    pipeline = ml_model["pipeline"]

    # Create a test case
    test_row = test_df.iloc[0].copy()  # Use real match
    test_row["date"] = "2023-05-01"  # Test on recent data

    # Engineer features
    features_df = engineer_features(
        test_df[test_df["date"] < "2023-01-01"],  # Historical
        test_row,
    )

    # Predict
    prediction = pipeline.predict(features_df)
    prob = pipeline.predict_proba(features_df)

    assert prediction[0] in [0, 1]  # Binary classification
    assert 0 <= max(prob[0]) <= 1  # Valid probability
    assert abs(sum(prob[0]) - 1.0) < 0.01  # Probabilities sum to 1

Test 6: Feature Sanity

def test_feature_ranges():
    """Are features in valid ranges?"""
    from src.features import engineer_features

    # Get one match
    test_match = test_df.iloc[0]

    # Engineer
    features = engineer_features(test_df, test_match)

    # Check rates (should be 0-1)
    assert 0 <= features["h2h_rate"].values[0] <= 1
    assert 0 <= features["overall_rate"].values[0] <= 1
    assert 0 <= features["venue_rate"].values[0] <= 1
    assert 0 <= features["rolling_rate"].values[0] <= 1

    # Check binary fields
    assert features["toss_win"].values[0] in [0, 1]

Test 7: No Data Leakage

def test_future_data_not_used():
    """Ensure no future data affects past predictions."""
    from src.features import engineer_features

    # Engineer a 2023 match using only 2022 history
    before_date = "2023-01-01"
    hist_data = test_df[test_df["date"] < before_date]

    test_match = test_df[
        (test_df["date"] >= before_date) & 
        (test_df["date"] < "2023-02-01")
    ].iloc[0]

    features = engineer_features(hist_data, test_match)

    # Features should only use hist_data, not test_match
    # (This is enforced in engine_features with before_date guards)
    assert features is not None

    # Verify: no 2023 data leaked into 2022 rates
    assert all(hist_data["date"] < before_date)

Running Tests

# Install test dependencies
pip install pytest

# Run all tests
pytest tests/

# Run with verbose output
pytest tests/ -v

# Run specific test
pytest tests/test_qa.py::test_match_lookup -v

# Coverage report
pytest tests/ --cov=src --cov-report=html

Deployment

Option 1: Streamlit Cloud

# Push to GitHub
git push origin main

# Link in Streamlit Cloud (streamlit.io/cloud)
# - Select repository
# - Select app.py
# - Auto-deploys on push

Live in 2 minutes, updates automatically.

Option 2: Docker

FROM python:3.11

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

EXPOSE 8501

CMD ["streamlit", "run", "src/streamlit_app.py", "--server.port=8501"]

docker build -t ipl-app .
docker run -p 8501:8501 ipl-app

Visit http://localhost:8501

Option 3: Heroku / Railway

# Deploy with one command
heroku create ipl-ai-assistant
git push heroku main

App runs at ipl-ai-assistant.herokuapp.com

Performance Monitoring

# Log predictions to dataframe
import logging
from datetime import datetime

logging.basicConfig(filename="predictions.log", level=logging.INFO)

@st.cache_data
def get_prediction_logs():
    return pd.read_csv("predictions.log")

# Dashboard
st.line_chart(
    get_prediction_logs()
    .groupby("hour")["confidence"]
    .mean()
)

Track:

Average confidence over time
Common queries
Backend latency

Common Issues & Solutions

Issue	Solution
"Connection refused"	Backend not running on localhost:8000
Chat history disappears	Use st.session_state, not regular variables
Predictions slow	Enable model caching, use lazy loading
Tests fail on new data	Read from CSV, not hardcoded values
Threshold too strict	Lower from 0.15 to 0.10 for more results

Summary: The Complete System

Component	Purpose	Technology
Feature Engineering	Calculate pre-match metrics	pandas, numpy
ML Model	Predict match winners	scikit-learn
Q&A Engine	Answer cricket questions	TF-IDF, cosine similarity
FastAPI Backend	Intelligent routing, lazy loading	FastAPI, uvicorn
Streamlit Frontend	Chat, prediction, metrics	Streamlit
Testing	Verify correctness	pytest, CSV-based assertions

You now have:

✅ Production-ready predictions (61.8% accuracy)

✅ Intelligent Q&A with 42K learning pairs

✅ Low-latency API (<20ms per request)

✅ Smooth UI with session persistence

✅ 22 tests ensuring reliability

✅ Multiple deployment options

What's Next?

Deploy this and message me your results! 🚀

Repository: https://github.com/jayakrishnayadav24/ipl-ai-assistant

Series Complete 🎉

← Part 4: Backend Routing | ← Return to Series

Built with 💚 for cricket fans. Questions? DM me!

Smart Backend Routing — Predictions vs Q&A Intelligently

POTHURAJU JAYAKRISHNA YADAV — Tue, 14 Apr 2026 10:47:09 +0000

Part 4 of 5 | ← Part 3 | Part 5 → | View Series

The Hook: Two Systems, One Decision

I had two systems:

One predicts match winners.

One answers 42,000 questions.

Now the real problem:

How do you decide which one to use?

This is where most systems fail, Not because of bad models. Because of bad decisions.

Users call /predict for Q&A. They call /qa for predictions. Chaos.

⚠️ Mistake I Almost Made: Separate APIs

At first, I built two separate endpoints:

/predict  → Only for predictions
/qa       → Only for Q&A

Users had to decide which one to call.

It was confusing. Unintuitive. Wrong.

That's when I realized:

The system should decide — not the user.

The Architecture

from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware

app = FastAPI(
    title="IPL AI Assistant",
    description="Predictions + Q&A for cricket",
    version="1.0"
)

# Allow frontend to call backend
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_methods=["*"],
    allow_headers=["*"],
)

Lazy Loading: The Secret Sauce

Stop here for a second.

This optimization seems small.

But it changes everything about how your system scales.

Instead of loading models at startup:

# ❌ Wrong way (startup)
model = joblib.load("models/model.joblib")  # 500ms
qa_index = joblib.load("models/qa_model.joblib")  # 400ms
# Server ready: 900ms

# ✅ Right way (lazy)
BUNDLE = None
QA_BUNDLE = None

def get_bundle():
    global BUNDLE
    if BUNDLE is None:
        BUNDLE = joblib.load("models/model.joblib")  # On first use
        print("[INFO] ML model loaded")
    return BUNDLE

def get_qa():
    global QA_BUNDLE
    if QA_BUNDLE is None:
        QA_BUNDLE = joblib.load("models/qa_model.joblib")  # On first use
        print("[INFO] Q&A index loaded")
    return QA_BUNDLE

Benefits:

Faster startup (150ms vs 900ms)
Serverless-friendly (AWS Lambda, Google Cloud)
Optional loading (if users never ask, models never load)

Cost savings: If you deploy 100 servers, save 100 × 750ms = 75 seconds total startup.

Endpoint 1: Health Check

@app.get("/health")
def health_check():
    """Called by load balancers every 10 seconds."""
    return {
        "status": "ok",
        "version": "1.0",
        "timestamp": datetime.utcnow().isoformat()
    }

If this returns 200, traffic flows. If timeout, traffic reroutes. Fast failure detection.

Endpoint 2: Metadata

@app.get("/model-info")
def model_info():
    """Tell clients what models are running."""
    bundle = get_bundle()  # First call triggers load

    return {
        "ml_model": {
            "type": "GradientBoostingClassifier",
            "test_accuracy": 0.618,
            "features": 13,
        },
        "qa_engine": {
            "qa_pairs": 42523,
            "vocabulary": 18394,
            "threshold": 0.15,
        },
    }

Clients call this before predictions: "Are you ready? What accuracy?"

Endpoint 3: Raw Prediction

from pydantic import BaseModel

class PredictionRequest(BaseModel):
    batting_team: str
    bowling_team: str
    venue: str
    h2h_rate: float        # 0.0-1.0
    overall_rate: float    # 0.0-1.0
    venue_rate: float      # 0.0-1.0
    rolling_rate: float    # 0.0-1.0
    toss_win: int          # 0 or 1
    toss_choice: str       # "bat" or "field"

@app.post("/predict")
def predict_winner(request: PredictionRequest):
    """Raw ML model prediction."""
    bundle = get_bundle()

    # Create feature vector
    df = pd.DataFrame([{
        "batting_team": request.batting_team,
        "bowling_team": request.bowling_team,
        "venue": request.venue,
        "h2h_rate": request.h2h_rate,
        # ... other rates
        "toss_win": request.toss_win,
        "toss_choice": request.toss_choice,
    }])

    # Predict
    prediction = bundle["pipeline"].predict(df)[0]
    confidence = bundle["pipeline"].predict_proba(df)[0]
    winner = bundle["label_encoder"].inverse_transform([prediction])[0]

    return {
        "winner": winner,
        "confidence": float(max(confidence)),
        "model": "GradientBoostingClassifier",
    }

Example:

POST /predict
{
  "batting_team": "Mumbai Indians",
  "bowling_team": "Chennai Super Kings",
  "venue": "Wankhede",
  "h2h_rate": 0.54,
  "overall_rate": 0.55,
  "venue_rate": 0.60,
  "rolling_rate": 0.52,
  "toss_win": 1,
  "toss_choice": "bat"
}

Response:
{
  "winner": "Mumbai Indians",
  "confidence": 0.62
}

Endpoint 4: Intelligent Chat Router

Stop for a moment.

This endpoint is doing something powerful.

It's not just forwarding requests. It's making decisions for the user.

This is the magic endpoint:

class ChatRequest(BaseModel):
    message: str

@app.post("/chat")
def handle_chat(request: ChatRequest):
    """
    Routes to prediction or Q&A automatically.
    """
    message = request.message.lower()

    # Step 1: Detect intent
    is_prediction = detect_prediction_intent(message)

    # Step 2: Extract teams
    teams = extract_teams(message)

    # Step 3: Route
    if is_prediction and len(teams) == 2:
        return handle_prediction(teams[0], teams[1])
    else:
        return handle_qa(message)

Intent Detection

def detect_prediction_intent(message: str) -> bool:
    """Is user asking for a prediction?"""
    keywords = [
        "will", "would", "who will", "predict",
        "who wins", "vs", "against", "beat",
    ]
    return any(kw in message for kw in keywords)

Examples:

"Will MI beat KKR?" → True (has "will")
"MI vs CSK?" → True (has "vs")
"How many sixes?" → False
"Does toss matter?" → False

Team Extraction

from difflib import get_close_matches

def extract_teams(message: str) -> list:
    """Find team names, even with typos."""
    bundle = get_bundle()
    team_names = list(bundle["team_index"].keys())

    found_teams = []
    words = message.split()

    # Exact matching
    for team in team_names:
        if team.lower() in message.lower():
            found_teams.append(team)

    # Fuzzy matching (for abbreviations like "mi" → "Mumbai Indians")
    for word in words:
        matches = get_close_matches(word, team_names, n=1, cutoff=0.8)
        if matches and matches[0] not in found_teams:
            found_teams.append(matches[0])

    return found_teams[:2]  # At most 2 teams

Examples:

"Will mi beat kkr?" 
→ Fuzzy match "mi" → "Mumbai Indians"
→ Fuzzy match "kkr" → "Kolkata Knight Riders"
→ Return ["Mumbai Indians", "Kolkata Knight Riders"]

"who wins, royal challengers bangalore vs sunrisers?"
→ Exact match "royal challengers bangalore"
→ Fuzzy match "sunrisers" → "Sunrisers Hyderabad"
→ Return ["RCB", "SRH"]

Prediction Handling

def handle_prediction(team1: str, team2: str) -> dict:
    """Compute features, run model, format response."""
    bundle = get_bundle()
    history_df = bundle["history_df"]

    # Compute live rates from historical data
    h2h = compute_h2h_rate(team1, team2, history_df)
    overall1 = compute_overall_rate(team1, history_df)
    overall2 = compute_overall_rate(team2, history_df)

    # Create prediction request
    pred_req = PredictionRequest(
        batting_team=team1,
        bowling_team=team2,
        venue="TBD",
        h2h_rate=h2h,
        overall_rate=overall1,
        venue_rate=0.5,
        rolling_rate=compute_rolling_rate(team1, history_df),
        toss_win=1,
        toss_choice="bat",
    )

    # Call /predict
    prediction = predict_winner(pred_req)

    # Format as chat response
    return {
        "type": "prediction",
        "message": f"🏆 **{prediction['winner']}** likely wins ({prediction['confidence']:.0%})",
        "confidence": prediction['confidence'],
    }

Q&A Handling

def handle_qa(message: str) -> dict:
    """Call Q&A retrieval system."""
    qa = get_qa()

    answer, score = answer_question(
        message,
        qa["tfidf"],
        qa["Q_matrix"],
        qa["answers"],
        threshold=0.15,
    )

    if answer is None:
        return {
            "type": "qa",
            "message": "🤔 I'm not confident about that.",
            "confidence": score,
        }

    return {
        "type": "qa",
        "message": answer,
        "confidence": score,
    }

Error Handling

@app.post("/chat")
def handle_chat(request: ChatRequest):
    try:
        message = request.message.lower()

        # ... routing logic ...

        return result

    except ValueError as e:
        return {"error": "Invalid input"}

    except Exception as e:
        # Log to monitoring (Sentry, DataDog)
        logger.error(str(e))
        return {"error": "Internal server error"}

Never return stack traces. Log errors, return friendly messages.

Performance Characteristics

Endpoint	Latency	Queries/sec
/health	<1ms	10,000+
/model-info	50ms	200
/predict	10ms	1,000
/chat (ML)	20ms	500
/chat (Q&A)	5ms	2,000

One $5 server handles 10,000 queries/day easily.

Deployment Options

Option 1: Traditional Server

pip install fastapi uvicorn scikit-learn pandas

uvicorn app:app --host 0.0.0.0 --port 8000

API available at http://localhost:8000

Option 2: Docker Container

FROM python:3.11

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY . .

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

docker build -t ipl-api .
docker run -p 8000:8000 ipl-api

Option 3: AWS Lambda (Serverless)

# With Lambda, lazy loading is critical
# Models load on first invocation, persist across requests

from mangum import Mangum

handler = Mangum(app)

Deploy with SAM or Serverless Framework.

Testing the Backend

# Health check
curl http://localhost:8000/health
# {"status": "ok"}

# Model info
curl http://localhost:8000/model-info
# {"ml_model": {...}, "qa_engine": {...}}

# Prediction
curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{
    "batting_team": "Mumbai Indians",
    "bowling_team": "Chennai Super Kings",
    ...
  }'

# Chat
curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Will MI beat CSK?"}'

Scaling Tips

Use gunicorn for multiple workers:

   gunicorn -w 4 -k uvicorn.workers.UvicornWorker app:app

Enable caching for Q&A (results don't change):

   from functools import lru_cache

   @lru_cache(maxsize=10000)
   def answer_question_cached(question):
       return answer_question(question, ...)

Load balance across multiple servers:
- Send requests to different servers
- Each loads models independently
- Scales to 1000s of requests/sec

💡 Key Insight: Routing is Intelligence

AI isn't just about models.

It's about deciding which model to use, when to use it, and when NOT to use it.

Your ML algorithm is 61.8% accurate. That's good.

But your routing logic? That's 100% accurate.

Because it knows the user's intent before they do.

🎤 The Mic Drop Lesson

The smartest part of this system isn't the model.

It's the decision of when NOT to use it.

When a user asks "How many sixes?", the system says: "This is Q&A, not ML."

When they ask "Will MI beat CSK?", it says: "This is prediction, not lookup."

That's what makes it feel intelligent.

What's in Part 5 (Frontend + Testing)

Final post: Making it all visible and reliable:

✅ Streamlit frontend (3 tabs)

✅ Session state (conversation persistence)

✅ 22 ground-truth tests

✅ Deployment (Streamlit Cloud, Docker)

✅ Common pitfalls

Sneak preview: Your tests should pull data from CSV, not be hardcoded. Here's why it matters...

This is Part 4 of 5. Subscribe for the finale! 🏏

← Part 3: Q&A Engine | Part 5: Frontend + Testing →

42,000 Instant Answers Without APIs — TF-IDF Q&A System

POTHURAJU JAYAKRISHNA YADAV — Tue, 14 Apr 2026 10:45:56 +0000

Part 3 of 5 | ← Part 2 | Part 4 → | View Series

The Hook: I Almost Used GPT

I built the prediction engine.

Now I needed a Q&A system to answer 42,000 cricket facts.

First instinct: Use GPT.

Open the API, feed it data, done.

Then I calculated the cost.

⚡ WARNING: If you're using LLMs for factual retrieval, you're probably overpaying and getting hallucinations. Here's what I built instead: a system that answers 42,000 questions, zero APIs, zero hallucinations, <5ms per query.

⚠️ Mistake I Almost Made: Going with LLMs

The math looked easy:

GPT-4: $0.01 per request

10,000 users per day = $100/day

That's $3,000/month.

For what?

Reciting facts that are already in my database.

Stop here for a second.

There's something wrong with using AI when search is enough.

And there's something worse: hallucinations.

An LLM might confidently say "Suryakumar hit 12 sixes in that match" when he actually hit 8.

For cricket facts, that's not acceptable.

That's when I stopped.

The Real Solution: TF-IDF + Cosine Similarity

This one line changed everything:

That question changed everything.

What if I didn't need AI at all?

Instead:

Generate 42,000 Q&A pairs from raw data
Vectorize all questions using TF-IDF
At query time: vectorize user's question, find closest match, return answer

Cost: $0 after initial computation

Latency: <5ms per query

Accuracy: 100% (only returns facts from data)

What is TF-IDF?

Stop here for a moment.

This algorithm is simple, but it's doing heavy lifting.

TF-IDF = Term Frequency × Inverse Document Frequency

Term Frequency (TF):
  How often does word X appear in THIS question?
  (More = higher weight)

Inverse Document Frequency (IDF):
  How many documents contain word X?
  (Rare words = higher weight)

TF-IDF = TF × IDF
  Common word ("the"): low TF-IDF (appears everywhere)
  Specific word ("Suryakumar"): high TF-IDF (appears rarely)

Result: Important words get high weight, noise gets filtered out.

Generating 42,000 Q&A Pairs

Type 1: Per-Match Facts (35 variations per match)

For every match, generate multiple question phrasings:

def generate_match_qa(match_id, date, batting_team, winner, pom, runs):
    answer = (
        f"Match {match_id} | {date} | {batting_team} scored {runs} | "
        f"Winner: {winner} | Player of Match: {pom}"
    )

    questions = [
        f"who won match {match_id}",
        f"player of the match {match_id}",
        f"how many runs did {batting_team} score match {match_id}",
        f"pom match {match_id}",
        f"who was man of match {match_id}",
        # ... 30 more variations covering:
        # - Different phrasings
        # - Natural language variants  
        # - Abbreviations (POM, 6s, 4s, RR, PP)
        # - Casual phrasing ("tell me about match X")
    ]

    return [(q, answer) for q in questions]

Result: ~36,000 pairs from ~1,000 matches.

Each answer identical (same match, 35 questions). This trains the vectorizer that "player of match", "pom", "man of match" all mean the same thing.

Type 2: Aggregate Stats

# Most Wins
top_team = df["winner"].value_counts().index[0]  # Mumbai Indians
top_wins = df["winner"].value_counts().iloc[0]   # 69 wins

questions = [
    "which team has most wins",
    "most successful team in ipl",
    "best team ipl history",
]
answer = f"Mumbai Indians has most wins: {top_wins}"

# Toss Impact
toss_win_rate = df["toss_winner"].eq(df["winner"]).mean() * 100
questions = [
    "does toss matter",
    "toss impact on match",
    "toss win match win",
]
answer = f"Toss wins lead to match wins {toss_win_rate:.1f}% of time"

Result: ~3,000 aggregate pairs.

Type 3: Head-to-Head Records

for team1 in teams:
    for team2 in teams[teams.index(team1)+1:]:
        h2h_matches = df[
            ((df["team1"] == team1) & (df["team2"] == team2)) |
            ((df["team1"] == team2) & (df["team2"] == team1))
        ]

        wins = h2h_matches["winner"].value_counts()
        answer = f"H2H: {team1} {wins[team1]} wins, {team2} {wins[team2]} wins"

        questions = [
            f"head to head {team1} vs {team2}",
            f"{team1} vs {team2} record",
            f"h2h {team1} {team2}",
        ]

Result: ~3,000 H2H pairs.

Total: 42,000+ pairs

Building the TF-IDF Index

from sklearn.feature_extraction.text import TfidfVectorizer

# Initialize vectorizer
tfidf = TfidfVectorizer(
    ngram_range=(1, 3),  # Unigrams, bigrams, trigrams
    min_df=1,            # Keep all terms
    analyzer="word"      # Split by spaces
)

# Extract questions and fit
questions = [pair[0] for pair in qa_pairs]
Q_matrix = tfidf.fit_transform(questions)

print(f"Vectorized {len(questions)} questions")
print(f"Vocabulary: {Q_matrix.shape[1]}")
# Output:
# "Vectorized 42523 questions"
# "Vocabulary: 18394"

Matrix shape: (42,523 × 18,394)

42,523 rows: One per question
18,394 columns: Unique vocabulary terms
Sparse: Most cells are 0 (most terms don't appear in most questions)

Why ngram_range=(1, 3)?

Unigrams (1 word):
  "player", "of", "the", "match"

Bigrams (2 words):
  "player of", "of the", "the match"

Trigrams (3 words):
  "player of the"

Without bigrams/trigrams:

Questions vectorized separately:
  "player" gets weight 0.05 (appears in tons of questions)
  "of" gets weight 0.02 (extremely common)
  "the" gets weight 0.01 (super common)

Result: Query "player of the match" matches EVERY question with "player"
→ False positives everywhere

With bigrams/trigrams:

"player of the" as a UNIT gets weight 0.80 (specific to match contexts)
"of" alone still gets weight 0.01

Result: Query "player of the match" matches ONLY match-specific questions
→ No false positives

Query-Time Retrieval

from sklearn.metrics.pairwise import cosine_similarity

def answer_question(user_question, tfidf, Q_matrix, answers, threshold=0.15):
    """
    Algorithm:
    1. Vectorize user's question (same vectorizer)
    2. Compute cosine similarity against all 42K questions
    3. Find best match
    4. Return if confident enough
    """
    # Step 1: Vectorize user input
    q_vec = tfidf.transform([user_question.lower()])
    # Shape: (1, 18394) vector

    # Step 2: Compute similarity to all 42K questions
    similarities = cosine_similarity(q_vec, Q_matrix)[0]
    # Shape: (42000,) array of similarity scores

    # Step 3: Find best match
    best_idx = np.argmax(similarities)
    best_score = similarities[best_idx]

    # Step 4: Return only if confident
    if best_score < threshold:
        return None, best_score

    return answers[best_idx], best_score

Real Examples

Example 1: Good Match ✅

User: "Who was player of the match in match 335982?"

Vectorization: TF-IDF creates 1×18394 vector
Similarity:    Computed against all 42K
Best match:    Index 5203 (similarity: 0.92)
Question 5203: "player of the match in match 335982"
Answer:        "...Match 335982 | ...Player of Match: Suryakumar Yadav"

Confidence gate: 0.92 > 0.15 → ✅ Return answer

Example 2: Fuzzy Match ✅

User: "who was pom in match 335982?"

Vectorization: "pom" is abbreviation, matches "player of match" trigram
Similarity:    0.85 (strong match due to trigram)
Best match:    Question 5204 "pom in match 335982"
Answer:        Same as above

Confidence gate: 0.85 > 0.15 → ✅ Return answer

Example 3: Off-Topic ❌

User: "Can I learn TF-IDF for machine learning?"

Vectorization: Cricket vocabulary doesn't have "learn", "machine", "for"
Similarity:    Best match is 0.08 (coincidental overlap)
Best match:    Some random match fact

Confidence gate: 0.08 < 0.15 → ❌ Reject
Response:       "🤔 I'm not confident about that"

The threshold (0.15) acts as a graceful uncertainty gate.

Memory Efficiency

Without sparse matrices (dense):
  42,523 × 18,394 × 8 bytes = 6.2GB

With sparse matrix (TF-IDF default):
  Only non-zero entries stored (~20MB compressed)

Result: 310x smaller! Fits in memory easily.

Performance Metrics

Metric	Value
Q&A pairs	42,523
Vocabulary size	18,394
Index file size	20MB
Query time	<5ms
Memory (at rest)	20MB
Memory (loaded)	150MB
Queries per second (1 server)	10,000+

Cost: Single $5 server handles 100,000 queries/day with zero API calls.

💡 Key Insight: You Don't Always Need AI

This system answers 42,000 questions.

Without a single LLM.

Without neural networks.

Without anything fancy.

Just math.

And clean data.

Why This Beats Alternatives

Approach	Cost	Speed	Accuracy	Hallucination Risk
TF-IDF (Ours)	$0	<5ms	100%	0%
LLM API (GPT)	$100/day	500ms	95%	5-10%
Hybrid (LLM + retrieval)	$50/day	200ms	99%	1%
Database + SQL	$50/month	50ms	100%	0%

We chose TF-IDF because:

✅ Zero hallucinations (only returns data facts)
✅ Instant (<5ms)
✅ Free (no APIs)
✅ Transparent (we control everything)

For this use case, TF-IDF is optimal.

Intent Detection (Going Beyond Direct Match)

Raw Q&A answer is often verbose. If user asks "How many sixes?", returning full match details is wrong.

def detect_intent(question):
    if any(w in question for w in ["sixes", "6", "boundary"]):
        return "sixes"
    if any(w in question for w in ["player", "man", "pom"]):
        return "player_of_match"
    if any(w in question for w in ["winner", "win", "result"]):
        return "winner"
    return "summary"

def format_answer(raw_answer, intent):
    if intent == "sixes":
        import re
        m = re.search(r"6s (\d+)", raw_answer)
        return f"🔥 Sixes: {m.group(1)}" if m else raw_answer

    if intent == "player_of_match":
        m = re.search(r"Player of Match: ([^\n]+)", raw_answer)
        return f"🌟 {m.group(1)}" if m else raw_answer

    return raw_answer

Result: Clean, focused answers instead of walls of text.

Model Serialization

qa_bundle = {
    "tfidf": tfidf,              # Fitted vectorizer
    "Q_matrix": Q_matrix,        # Pre-computed TF-IDF vectors  
    "questions": questions,      # For debugging
    "answers": answers,          # Parallel to questions
}

joblib.dump(qa_bundle, "models/qa_model.joblib")

Everything in one file. Load once at startup, reuse forever.

🎤 The Mic Drop Lesson

TF-IDF won't impress anyone at a conference.

But it's fast.

It's accurate.

It never lies.

Most importantly: it costs nothing.

Sometimes the boring solution is the best one.

What's in Part 4 (FastAPI Backend)

Next post: How to route questions intelligently:

✅ Lazy loading (optional until first use)

✅ Intent detection (prediction vs Q&A)

✅ Team extraction (fuzzy matching)

✅ Smart routing (/chat does both)

✅ Error handling (timeout, API errors)

Sneak preview: One endpoint handles both systems. The backend decides which AI to use.

This is Part 3 of 5. Subscribe to follow! 🏏

← Part 2: Prediction Engine | Part 4: Smart Backend →

Building the Perfect Prediction Engine — Feature Engineering Without Cheating

POTHURAJU JAYAKRISHNA YADAV — Tue, 14 Apr 2026 10:11:37 +0000

Part 2 of 5 | ← Part 1 | Part 3 → | View Series

The Hook: 95% Accuracy That Was Completely Wrong

I built a prediction model.

It gave me 95% accuracy.

I was ecstatic. I thought I'd cracked it.

Then I deployed it.

Predictions were garbage.

⚡ WARNING: Most IPL prediction models you see online are wrong. Not because of the algorithm. Because they're leaking future data. You're about to learn how to build one that doesn't.

⚠️ Mistake I Made: The Data Leakage Trap

The problem?

I was using runs and wickets as features.

But here's the thing — I calculated them from the same match I was predicting.

The model wasn't predicting.

It was cheating.

It was like asking "Will Team A win?" and then telling the model "Here are Team A's final stats from that match."

Of course it knew the answer.

Stop here for a second.

That's the entire problem with most ML models.

They don't learn to predict. They learn to memorize the future.

That 95% accuracy? Total illusion. The moment I tested on real, unseen data: 30% accuracy. Worse than flipping a coin.

The Real Challenge

So I started over.

This time: Only historical data. No future data leakage.

I had 13 pre-match features to compute for 1,076 matches, using only historical data available before the match.

The naive approach: Loop through each match, compute features. Takes 45 minutes.

The real question: How do you compute features correctly AND fast?

The answer: Vectorized pandas operations + smart grouping + the date filter from hell.

This is the critical part:

Every single feature must be computed from data that exists BEFORE the match.

Not after.

Before.

One line: matches["date"] < before_date

That line is doing ALL the work.

Four Historical Rate Functions

1️⃣ Head-to-Head Win Rate

How many times has Team A beaten Team B historically?

💡 Key Line: matches["date"] < before_date

This is the data leakage guard. Without it, you're looking at the future. With it, you're looking at the past only.

def _h2h_win_rate(matches, team1, team2, before_date):
    """Win rate of team1 vs team2 in all past matches."""
    past = matches[
        (matches["date"] < before_date) &  # ← THE CRITICAL LINE
        (
            ((matches["batting_team"] == team1) & (matches["bowling_team"] == team2)) |
            ((matches["batting_team"] == team2) & (matches["bowling_team"] == team1))
        )
    ]
    if past.empty:
        return 0.5  # Neutral if no history

    wins_by_team1 = (past["winner"] == team1).sum()
    return wins_by_team1 / len(past)

Real example: MI vs CSK

38 past matches (filtered by before_date)
MI won: 18
CSK won: 20
→ h2h_rate_MI = 18/38 = 0.474 (47.4%)

Notice what's missing?

Stats from the match we're predicting.

That's the whole point.

2️⃣ Overall Win Rate

What percentage of matches does this team win (all-time)?

This is the team's DNA.

Some teams are just better. Structurally. Over 16 years.

def _overall_win_rate(matches, team, before_date):
    """Win percentage across all matches."""
    past = matches[
        (matches["date"] < before_date) &  # ← Still have the guard
        ((matches["batting_team"] == team) | (matches["bowling_team"] == team))
    ]
    return (past["winner"] == team).sum() / len(past)

Real example:

Mumbai Indians
Matches before 2024-01-01: 200
Wins: 90
→ 45% win rate (structurally strong)
They win nearly half their matches. Consistently.

Kings XI Punjab
Matches: 200
Wins: 70
→ 35% win rate (historically weaker)
They win about a third.

The signal is HUGE:

MI at 45% vs KXIP at 35% = 10 percentage points of pure team quality.

The model learns this. And it matters.

3️⃣ Venue Win Rate

Home field advantage.

Is it real?

Yes.

By how much?

A lot.

def _venue_win_rate(matches, team, venue, before_date):
    """Win rate at a specific venue."""
    past = matches[
        (matches["date"] < before_date) &
        (matches["venue"] == venue) &  # ← Filter to this ground only
        ((matches["batting_team"] == team) | (matches["bowling_team"] == team))
    ]
    return (past["winner"] == team).sum() / len(past) if not past.empty else 0.5

Real example — Mumbai Indians:

Ground	Record	Win Rate
Wankhede (Home)	18-10	64%
Eden Gardens (Away)	4-11	27%
Narendra Modi Stadium	8-7	53%

Notice that 64% at home?

Then 27% at KKR's fortress (Eden Gardens).

That's the magic of venue. Same team. 37 percentage points difference. Same players. Different ground.

👉 The model learns this. It matters.

4️⃣ Rolling Win Rate (Momentum)

Is a team hot or cold?

Last 5 matches.

Won how many?

That's momentum.

def _rolling_win_rate(matches, team, before_date, n=5):
    """Recent form: last n matches."""
    past = matches[
        (matches["date"] < before_date) &
        ((matches["batting_team"] == team) | (matches["bowling_team"] == team))
    ].tail(n)  # ← LAST n matches only

    return (past["winner"] == team).sum() / len(past) if not past.empty else 0.5

Psychological impact:

Winning streak (5-0): Confidence high, execution sharp → morale boost
Losing streak (0-5): Doubt creeping in, mistakes multiplying → morale crash

Real example — the moment that shows momentum:

Match Day: 2024-01-20 (MI vs CSK)

MI's last 5 matches:
Loss → Loss → Win → Win → Win
→ rolling_rate = 60% (recovering, momentum UP)

CSK's last 5 matches:  
Loss → Loss → Loss → Loss → Loss
→ rolling_rate = 0% (in free fall)

What would the model predict?
MI: 60% form advantage
CSK: 0% form advantage
→ MI becomes heavy favorite

Plot twist: You'd think momentum would be the #1 predictor.

It's not. (Spoiler: we'll see why later.)

Assembling Features: The engine_features() Function

Now let's put all four features together:

def engineer_features(df):
    """
    Main feature engineering function.
    Input: Raw DataFrame (2,217 rows)
    Output: ML-ready DataFrame (932 rows, 13 features)
    """
    # Step 1: One row per match
    matches = df[df["innings"] == 1].copy().reset_index(drop=True)

    # Step 2: Chronological order (CRITICAL)
    matches = matches.sort_values("date").reset_index(drop=True)

    # Step 3: Compute features for each match
    rows = []
    for _, row in matches.iterrows():
        t1, t2 = row["batting_team"], row["bowling_team"]
        date = row["date"]  # ← THIS date is the cutoff
        venue = row["venue"]
        winner = row["winner"]

        if pd.isna(winner):
            continue  # Skip abandoned matches

        rows.append({
            "team1": t1,
            "team2": t2,
            "venue": venue,
            "toss_winner_is_team1": 1 if row["toss_winner"] == t1 else 0,
            "toss_decision": row.get("toss_decision", "unknown"),

            # Historical rates (ALL filtered by date < before_date)
            "h2h_win_rate_t1": _h2h_win_rate(matches, t1, t2, date),
            "overall_win_rate_t1": _overall_win_rate(matches, t1, date),
            "overall_win_rate_t2": _overall_win_rate(matches, t2, date),
            "venue_win_rate_t1": _venue_win_rate(matches, t1, venue, date),
            "venue_win_rate_t2": _venue_win_rate(matches, t2, venue, date),
            "rolling_win_rate_t1": _rolling_win_rate(matches, t1, date),
            "rolling_win_rate_t2": _rolling_win_rate(matches, t2, date),

            "winner": winner,  # ← What actually happened
            "date": date,
        })

    return pd.DataFrame(rows)

What you get:

Shape: (932 rows, 15 columns)

✓ 932 matches (2008-2024, all cleaned)
✓ 15 columns:
  - 4 metadata (team1, team2, venue, date)
  - 2 toss info
  - 7 historical rates
  - 1 target (winner)
  - 1 date

The magic: Every single feature is computed BEFORE the match. No cheating.

⚠️ The Second Mistake: Wrong Train-Test Split

I almost made this mistake again.

❌ The Trap (Random Split)

X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)

What happens:

Training set: [2024, 2008, 2021, 2010, 2023, ...]
Test set:     [2015, 2022, 2009, 2019, 2024, ...]

Model sees 2024 during training.
Model tests on 2008 data.

→ Completely backwards.
→ 95% fake accuracy. 30% real accuracy in production.

Why I almost fell for it:

Sklearn's train_test_split is beautiful. Clean API. Works great.

For time-series data, it's a disaster.

✅ The Right Way (Temporal Split)

split_date = pd.Timestamp("2023-01-01")
train_df = engineered_df[engineered_df["date"] < split_date]
test_df = engineered_df[engineered_df["date"] >= split_date]

Timeline (the way reality works):

2008 ──────────────────────── 2023 ────── 2024
├─ TRAIN: 15 years history ─┤├─ TEST: 2 years unseen ─┤
         ↓                              ↓
    Model learns               Model evaluates
    (only past)                (only future)

💡 Key Rule: Train sees only the past. Test sees only the future.

Anything else is cheating.

Result: 61.8% test accuracy (real, honest, deployable).

Model Tournament: 4 Algorithms Battle

Now which algorithm should I use?

candidates = {
    "Logistic Regression": LogisticRegression(...),
    "Random Forest": RandomForestClassifier(n_estimators=200, max_depth=8),
    "Gradient Boosting": GradientBoostingClassifier(n_estimators=200, max_depth=4),
    "XGBoost": XGBClassifier(n_estimators=200, max_depth=4),
}

Let them fight on the same data.

Drumroll...

Model	Test Accuracy	Cross-Val Mean	Notes
Logistic Regression	46.5%	48.9%	Too linear
Random Forest	50.7%	53.4%	Better, but basic
🏆 Gradient Boosting	61.8%	48.3%	WINNER
XGBoost	58.3%	47.1%	Close 2nd, overfits

The accuracy gap visualized:

Random Guessing: ████░░░░░░░░░░░░░  12.5%
Logistic Reg:    ███████░░░░░░░░░░░  46.5%
Random Forest:   ██████████░░░░░░░░  50.7%
XGBoost:         ████████████░░░░    58.3%
Gradient Boost:  ████████████████░   61.8% ← Better patterns, not fancier math

💡 Scale it up:

Random baseline (8 teams, pick randomly): 12.5% accuracy

Gradient Boosting: 61.8%

→ 5x better than just guessing.

Why Gradient Boosting crushed it:

Model	Why It Failed
Logistic Regression	Linear model. Can't capture "Team A beats Team B when at home AND have won 3 straight" logic.
Random Forest	Decent, but treats features independently. Misses interactions.
Gradient Boosting	Iterative learning. Each tree corrects the previous tree's mistakes. Learns nuances.
XGBoost	Same idea as GB, but overfits slightly on this small dataset.

Lesson: Better features > better algorithms.

But better algorithms + better features? Now you're cooking.

🔍 Feature Importance: The Shocking Truth

# From the trained Gradient Boosting model
feature_importance = {
    "overall_win_rate_t1": 0.152,     # 15.2% — DOMINATES
    "h2h_win_rate_t1": 0.128,         # 12.8%
    "toss_winner_is_team1": 0.053,    # 5.3% — Wait, that's LOW
    "rolling_win_rate_t1": 0.049,     # 4.9% — Surprising!
    "venue_win_rate_t1": 0.042,       # 4.2%
}

The results shocked me:

1. Overall Team Quality Dominates (15.2%)

A fundamentally strong team beats fundamentally weak teams.

Consistently.

This is the biggest signal.

MI at 45% win rate (all-time) vs KXIP at 35% (all-time)?

The model puts 15.2% weight on that difference alone.

2. H2H History Matters (12.8%)

Historical head-to-head is powerful.

If Team A has beaten Team B 70% of the time historically?

That's signal.

But notice: It's less than overall quality.

3. ⚠️ Toss is Overrated (5.3%)

Cricket myth: "Winning the toss wins the match."

Reality: Toss matters ~5% of the time.

Bigger factors:

Better team
Better history vs opponent
Better recent form

Toss is just noise compared to these.

4. 😲 Momentum is Weak (4.9%)

This is the shocking one.

You'd think "last 5 matches" would be huge.

It's the 4th most important feature.

Why?

Recent form is volatile. Wins vary. Losses happen.

Team quality is stable. It's baked in over 16 years.

The model trusts stability over noise.

5. Venue Advantage is Real but Limited (4.2%)

Home field helps.

But it's less predictive than everything else.

Why?

Because team quality already accounts for most of the variance.

The Production Pipeline

Now let's make this production-ready.

How do you package this for production?

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder

# Numeric features: standardize to mean 0, std 1
numeric_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

# Categorical features: convert to numbers
categorical_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='unknown')),
    ('onehot', OneHotEncoder(sparse=False, handle_unknown='ignore'))
])

# Combine everything
preprocessor = ColumnTransformer([
    ('num', numeric_transformer, numeric_cols),
    ('cat', categorical_transformer, categorical_cols)
])

# Full pipeline: preprocess → train → predict
pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('model', GradientBoostingClassifier(n_estimators=200, max_depth=4))
])

# Train on 2008-2022, never look at 2023-2024 during training
pipeline.fit(X_train, y_train)
accuracy = pipeline.score(X_test, y_test)  # 0.618 ✓

print(f"Model learned: {accuracy * 100:.1f}% accuracy on unseen data")

What this does:

Raw data → Normalize → One-hot encode → Gradient Boosting → Prediction

Every input gets processed the same way. Every time.

Packaging for the Real World

Now save everything:

bundle = {
    "pipeline": pipeline,           # ← Everything needed to predict
    "features": feature_names,      # ← What features it expects
    "label_encoder": label_encoder, # ← Team name mappings
    "metrics": {
        "test_accuracy": 0.618,     # ← How good
        "cv_mean": 0.483,           # ← Cross-val mean
        "cv_std": 0.028,            # ← Cross-val std
    },
    "feature_importance": importance_dict,  # ← What matters
}

joblib.dump(bundle, "models/model.joblib")
# Result: ~30MB file
# Contains: preprocessor, model, metadata, everything

Later, load it:

bundle = joblib.load("models/model.joblib")
pipeline = bundle["pipeline"]

# Use it
result = pipeline.predict_proba(new_data)[0]
# → [0.38, 0.62]  (38% one team, 62% the other)

Prediction Time: Deploy It

User asks: "Will MI beat CSK tomorrow at Wankhede?"

# Step 1: Compute features from historical data ONLY
# (no future data, no match results yet)
features = {
    "h2h_rate": 0.54,        # MI vs CSK history → from database
    "overall_rate": 0.55,    # MI all-time → from database
    "venue_rate": 0.60,      # MI at Wankhede → from database
    "rolling_rate": 0.52,    # MI last 5 matches → from database
    "toss_win": 1,           # MI won toss → from live toss result
}

# Step 2: Send to pipeline
X = pd.DataFrame([features])
prediction = pipeline.predict_proba(X)[0]
# Returns: [0.38, 0.62]
#          (38% CSK wins, 62% MI wins)

# Step 3: Return to user
response = {
    "winner": "Mumbai Indians",
    "confidence": 0.62,  # 62% confident
    "explanation": "MI's strong overall record (55%) + venue advantage (60%) + H2H lead (54%)"
}

No magic. No hallucinations. No APIs.

Just math.

🚨 Other Mistakes I Almost Made

Mistake: Hardcoding Features

# ❌ WRONG
def predict(team1, team2):
    h2h = 0.54  # Hardcoded!!!!
    rolling = 0.52  # Hardcoded!!!!
    return model.predict([[h2h, rolling]])

# Problem: Data changes, your code doesn't.
# Forever stuck at 0.54.

Mistake: Inconsistent Team Name Formats

# ❌ WRONG
matches_where_mi_played = matches[
    (matches["team1"] == "Mumbai Indians") |  # What if CSV has "MI"?
    (matches["team2"] == "Mumbai Indians")
]

# Query returns nothing. Silent failure.
# You have no idea why.

✅ Right Way: Normalize Everything

# At load time: standardize all team names
team_mapping = {
    "MI": "Mumbai Indians",
    "CSK": "Chennai Super Kings",
    "DC": "Delhi Capitals",
}
df["team1"] = df["team1"].map(team_mapping).fillna(df["team1"])
df["team2"] = df["team2"].map(team_mapping).fillna(df["team2"])

# Now all queries work
h2h = _h2h_win_rate(df, "Mumbai Indians", "Chennai Super Kings", date)

💡 The Single Biggest Lesson

The gap between 46% accuracy and 61.8% didn't come from fancier algorithms.

It came from not cheating.

Most ML models fail because they leak information. They train on the future. They use data that wouldn't exist at prediction time.

This model works because every single feature is computed from the past only.

The four historical rate functions (H2H, Overall, Venue, Rolling) do all the heavy lifting.

Gradient Boosting is nice. But it's the features that matter.

🎯 The Myth vs Reality

Myth	Reality
"Better algorithms = better predictions"	Better features >> better algorithms
"Toss determines the match"	Toss is ~5% of the variance
"Recent form is everything"	Recent form is 4.9% of the variance
"Home field is huge"	Home field is 4.2% (team quality matters more)
"You need deep learning"	Simple feature engineering beats 95% of models

🎤 The Mic Drop Lesson

And I almost shipped that model.

No data leakage guards, no temporal split, just raw 95% accuracy shipped to production.

Here's what happened:

The model didn't get better because of Gradient Boosting.

It got better because it stopped cheating.

Most ML projects fail for the same reason.

They learn from the future.

This one doesn't.

That's it. That's why it works.

What's Next (Part 3: Q&A Engine)

So we built a prediction engine that doesn't hallucinate.

Now the harder part: How do you answer 42,000 questions without APIs or LLMs?

✅ Generate Q&A pairs from raw data

✅ TF-IDF (simple, fast, accurate)

✅ Search in <5ms

✅ Zero hallucinations

✅ Why expensive APIs are overkill

Sneak preview: TF-IDF won't impress at parties. But it's 100% accurate and costs $0.00 per query.

Sometimes boring wins.

🏏 This is Part 2 of 5. Subscribe to follow the series!

← Part 1: The Problem | Part 3: Q&A Engine →

Part 1: The Problem No One Solves — Building an IPL AI That Predicts AND Answers

POTHURAJU JAYAKRISHNA YADAV — Tue, 14 Apr 2026 09:55:21 +0000

This is Part 1 of a 5-part series on building a complete ML system. Part 2 → | Part 3 → | Part 4 → | Part 5 →

The Cricket App Nobody Asked For

Last year, I was watching IPL matches and thinking about the perfect fantasy cricket assistant.

Not just another prediction model.

Not just a stats dashboard.

But something that could:

Answer any factual question instantly ("Who was player of the match on March 27, 2024?")
Predict match winners with real confidence scores
Route questions intelligently (knowing when to use each engine)

I'd see people on Twitter asking:

"Has CSK beaten MI more times than the other way around?"
"Does winning the toss actually help?"
"Who will win if MI bat against CSK tomorrow?"

The problem: The first two need retrieval. The third needs ML. Nobody was solving both in one system.

At least, not in a simple, production-ready way.

So I built it.

What This System Actually Does

Example 1: Match Prediction

User: "Who will win if Mumbai Indians bat against Chennai Super Kings?"

System: Runs Gradient Boosting model with pre-match features
- H2H record: 54%
- Overall win rate: 55%
- Venue history: 60%
- Recent form: 52%

Response: 🏆 Chennai Super Kings (64% confidence)

Example 2: Factual Lookup

User: "Who was player of the match in match 335982?"

System: TF-IDF cosine similarity against 42,000 Q&A pairs

Response: 🌟 Suryakumar Yadav

Example 3: Historical Analysis

User: "Head to head KKR vs RCB?"

System: Retrieves team record from Q&A index

Response:
- KKR: 9 wins
- RCB: 8 wins
- (Total: 17 innings)

The magic? It automatically decides which engine to use based on the question.

🎯 How It Looks in Action

Here's what a real conversation looks like:

User: "Who will win if MI bat against CSK?"
System: Runs the ML model → 🏆 CSK (64% confidence)

User: "How many runs did MI score in match 335982?"
System: Queries Q&A index → 🏏 Mumbai Indians scored 222 runs

User: "Does toss matter?"
System: Retrieves aggregate stats → Only 51.3% of toss winners win the match

Two different engines, one seamless experience.

🎨 Quick Visual Overview

User Input (Natural Language)
    ↓
Intent Detection
    ├─ Prediction keywords? → ML Engine
    ├─ Match ID? → Q&A Lookup
    └─ Otherwise → Q&A Retrieval
    ↓
Intelligent Response
    └─ Routed through FastAPI backend

The Architecture (Simple Version)

CSV Data (2,217 rows)
    ↓
Normalization (fix team names, dates)
    ↓
Feature Engineering (pre-match only)
    ├─→ ML Model (Gradient Boosting, 61.8%)
    └─→ Q&A Index (42,000 pairs)
         ↓
    FastAPI Backend (/chat, /predict, /health)
         ↓
    Streamlit Frontend (3 tabs: Chat, Predict, Metrics)

Critical design: Only .joblib files are used at runtime. No CSV dependency. Makes it:

✅ Fast (no CSV parsing)

✅ Portable (single deploy)

✅ Docker-friendly

The Dataset Problem Nobody Talks About

The IPL dataset spans 2008–2024 (2,217 rows). Looks clean, right?

It's not.

Problem 1: Team Name Chaos

Delhi Daredevils became Delhi Capitals in 2019. Kings XI Punjab became Punjab Kings in 2021.

If you don't normalize:

# ❌ WRONG
Delhi Daredevils: 5 wins (2015-2018)
Delhi Capitals:   2 wins (2020-2024)
Total: 7 wins (WRONG! It's one team)

The fix:

TEAM_NAME_MAP = {
    "Royal Challengers Bengaluru": "Royal Challengers Bangalore",
    "Delhi Daredevils": "Delhi Capitals",
    "Kings XI Punjab": "Punjab Kings",
}

df[team_cols] = df[team_cols].replace(TEAM_NAME_MAP)

One function. Called once. All team names normalized.

Problem 2: Date Type Matters

# ❌ WRONG (string comparison)
df[df["date"] < "2023-01-01"]
# "2024" < "2023" alphabetically → TRUE (backwards!)

# ✅ RIGHT (datetime comparison)
df["date"] = pd.to_datetime(df["date"])
df[df["date"] < pd.Timestamp("2023-01-01")]

The Real Problem: Data Leakage

Here's where most sports prediction projects fail silently.

Imagine building a predictor with:

features = ["total_runs", "wickets", "run_rate", "boundaries"]

Your model trains on 500 matches and gets 95% accuracy.

In production? 45% accuracy.

Why? Because total_runs is only known after the match. Your model is using information that doesn't exist at prediction time.

This is data leakage, and it's invisible.

The Guard: Time-Based Filtering

For Match X (played Jan 15, 2024), I only use matches before Jan 15:

def _h2h_win_rate(matches, team1, team2, before_date):
    past = matches[
        (matches["date"] < before_date) &  # ← THE GUARD
        (
            ((matches["batting_team"] == team1) & 
             (matches["bowling_team"] == team2)) |
            ((matches["batting_team"] == team2) & 
             (matches["bowling_team"] == team1))
        )
    ]
    if past.empty:
        return 0.5
    return (past["winner"] == team1).sum() / len(past)

That one line ensures the model only learns from past data.

Four historical metrics:

H2H win rate — Past match record
Overall win rate — All-time record
Venue win rate — Ground-specific history
Rolling win rate — Recent form (last 5 matches)

13 features total. All pre-match. Zero leakage.

Why This Matters

The difference between 46% (baseline) and 61.8% (Gradient Boosting) came from:

✅ Proper feature engineering (H2H, venue, momentum)
✅ No data leakage (pre-match features only)
✅ Time-based validation (train on past, test on future)

That last one is critical. If I'd used random 80/20 split:

❌ WRONG:
- Training: 2008-2024 mixed
- Testing: 2008-2024 mixed
- Model trains on 2024 data, tests on 2021 data (backwards!)

✅ RIGHT:
- Training: 932 matches (2008-2022)
- Testing: 144 matches (2023-2024)
- Simulates real deployment

The Normalized Dataset

After all fixes:

Metric	Value
Total rows	2,217
Unique matches	1,076
Seasons	17 (2008-2024)
Teams	8 (normalized)
Pre-match features	13 — all derived strictly from past data
Training set	932 matches
Test set	144 matches

Ready for feature engineering.

What's in Part 2 (Prediction Engine)

Next post: I'll show you:

✅ How to assemble 13 features cleanly

✅ Why Gradient Boosting won (surprise: size matters)

✅ Feature importances — what actually matters?

✅ Model comparison (61.8% explained)

✅ Confidence scoring

Sneak preview: Most people think toss is important. The model learned toss is almost irrelevant (5.3% importance). Head-to-head records? That's the signal (12.8%).

The Moment It Clicked

At one point, my model showed ~95% accuracy on the training set — and I was genuinely excited.

Then I realized it was using future data. Total_runs, wickets, boundaries — all known only after the match ended.

That was the moment I understood how easy it is to fool yourself in machine learning. The model wasn't learning cricket patterns. It was memorizing match outcomes.

I rebuilt everything with strictly pre-match features. Accuracy dropped to 61.8%. But suddenly, it actually worked in the real world.

🏁 Final Thought

Most ML systems don't fail because of models.

They fail because of:

Bad data (unclean, inconsistent teams/dates)
Leakage (using information that shouldn't exist at prediction time)
Wrong validation (random splits on time-series data)

Fix those three things — and even simple models become powerful.

Part 2 dives into how I built the prediction engine with these principles in mind.

This is Part 1 of 5. Ready for the deep dive? Part 2: Prediction Engine →

← Back to Series

Terraform Modular EKS + Istio — Part 5

POTHURAJU JAYAKRISHNA YADAV — Fri, 27 Mar 2026 02:49:35 +0000

CSI Drivers (How Storage Actually Works in EKS)

So far:

VPC → network
IAM → permissions
EKS → control plane
Nodes → compute

Now comes the part many people ignore:

👉 Storage

Without this:

Pods can’t persist data
Databases won’t work
Logs disappear on restart

This module installs CSI drivers, which allow Kubernetes to use AWS storage.

📂 Module Files

modules/csi-driver/
├── main.tf
├── variables.tf
└── outputs.tf

📄 variables.tf

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "ebs_csi_role_arn" {
  description = "IAM role ARN for EBS CSI driver"
  type        = string
}

variable "s3_csi_role_arn" {
  description = "IAM role ARN for S3 CSI driver"
  type        = string
}

variable "node_group_dependency" {
  description = "Node group dependency to ensure nodes exist first"
  type        = any
}

🧠 What these inputs mean

cluster_name → where to install addons
ebs_csi_role_arn → IAM role for EBS driver
s3_csi_role_arn → IAM role for S3 driver
node_group_dependency → ensures nodes exist first

⚠️ Important insight

This module depends on:

👉 IAM (for IRSA roles)
👉 Node groups (for scheduling pods)

📄 main.tf

1. EBS CSI Driver

resource "aws_eks_addon" "ebs_csi" {
  cluster_name             = var.cluster_name
  addon_name               = "aws-ebs-csi-driver"
  service_account_role_arn = var.ebs_csi_role_arn

  depends_on = [var.node_group_dependency]
}

🧠 What this does

Installs:

👉 AWS EBS CSI Driver inside cluster

What is CSI?

CSI = Container Storage Interface

👉 It allows Kubernetes to talk to AWS storage.

What EBS CSI enables

Create EBS volumes
Attach volumes to pods
Persist data

Important line

service_account_role_arn = var.ebs_csi_role_arn

👉 This is IRSA

This means:

Pod gets IAM role
Pod can call AWS APIs

Without this

❌ Pod cannot create volumes
❌ PVC fails

2. S3 CSI Driver

resource "aws_eks_addon" "s3_csi" {
  cluster_name             = var.cluster_name
  addon_name               = "aws-mountpoint-s3-csi-driver"
  service_account_role_arn = var.s3_csi_role_arn

  depends_on = [var.node_group_dependency]
}

🧠 What this does

Installs:

👉 S3 CSI driver

What it enables

Mount S3 bucket as:

Pod → S3 bucket (like filesystem)

Use cases

logs
shared storage
backups

3. Dependency Handling

depends_on = [var.node_group_dependency]

Why this is critical

CSI driver runs as pods.

👉 Pods need nodes

So order must be:

Nodes → CSI Driver

Without this

Addon installs
But pods fail to schedule

📄 outputs.tf

output "ebs_csi_addon_id" {
  description = "EBS CSI addon ID"
  value       = aws_eks_addon.ebs_csi.id
}

output "s3_csi_addon_id" {
  description = "S3 CSI addon ID"
  value       = aws_eks_addon.s3_csi.id
}

🧠 Why outputs matter

Used for:

tracking addon deployment
debugging
dependencies in future modules

🔥 What You Actually Built

Kubernetes Pod
      │
      │
CSI Driver
      │
      │
AWS Storage (EBS / S3)

⚠️ Real Issues People Face

Missing IRSA → access denied
No node dependency → pods fail
Wrong role → volume attach fails
Forgetting CSI → PVC stuck in pending

🧠 Key Takeaways

Kubernetes doesn’t manage storage directly
CSI drivers connect Kubernetes to AWS
IRSA is required for secure access
EBS = block storage
S3 = object storage

🚀 Next

In Part 6:

👉 AWS Load Balancer Controller
👉 How ALB actually integrates with Kubernetes
👉 Why target-type: ip matters

At this point, your cluster can:

run workloads
persist data

Now we move to traffic layer.

Terraform Modular EKS + Istio — Part 4

POTHURAJU JAYAKRISHNA YADAV — Fri, 27 Mar 2026 02:47:10 +0000

EKS Node Groups (Where Your Cluster Actually Gets Compute)

In the previous part, we created the EKS control plane.

At that point:

Kubernetes API exists
Cluster is reachable

But:

👉 There are no machines to run workloads

That’s where Node Groups come in.

This module creates the actual EC2 instances that:

join the cluster
run pods
execute your applications

📂 Module Files

modules/eks-nodes/
├── main.tf
├── variables.tf
└── outputs.tf

📄 variables.tf

variable "cluster_name" {
  description = "Name of the EKS cluster"
  type        = string
}

variable "node_group_name" {
  description = "Name of the EKS node group"
  type        = string
}

variable "node_role_arn" {
  description = "ARN of the EKS node group IAM role"
  type        = string
}

variable "subnet_ids" {
  description = "List of subnet IDs"
  type        = list(string)
}

variable "instance_types" {
  description = "List of instance types"
  type        = list(string)
  default     = ["t3.large"]
}

variable "desired_size" {
  description = "Desired number of nodes"
  type        = number
  default     = 1
}

variable "max_size" {
  description = "Maximum number of nodes"
  type        = number
  default     = 2
}

variable "min_size" {
  description = "Minimum number of nodes"
  type        = number
  default     = 1
}

variable "max_unavailable" {
  description = "Maximum number of nodes unavailable during update"
  type        = number
  default     = 1
}

variable "labels" {
  description = "Key-value map of Kubernetes labels"
  type        = map(string)
  default     = {}
}

🧠 What these variables control

This module is completely configurable:

cluster → which EKS cluster to join
instance_types → what machines to use
scaling → how many nodes
labels → Kubernetes scheduling

Example thinking

dev → small nodes (t3.medium)
prod → bigger nodes (m5.large)

👉 Same code, different behavior.

📄 main.tf

1. Node Group Resource

resource "aws_eks_node_group" "nodes" {
  cluster_name    = var.cluster_name
  node_group_name = var.node_group_name
  node_role_arn   = var.node_role_arn
  subnet_ids      = var.subnet_ids

What this does

Creates:

👉 EC2 instances managed by EKS

These instances:

automatically join the cluster
register as Kubernetes nodes

Important inputs

cluster_name → which cluster to join
node_role_arn → permissions for nodes
subnet_ids → where nodes are created

Subnet choice

You are passing:

👉 private subnets

This means:

nodes do NOT have public IP
more secure
traffic goes through NAT

2. Instance Types

instance_types = var.instance_types

What this controls

Defines:

CPU
Memory
cost

Example:

t3.large → 2 vCPU, 8GB RAM

3. Scaling Configuration

scaling_config {
  desired_size = var.desired_size
  max_size     = var.max_size
  min_size     = var.min_size
}

Meaning

desired_size → current running nodes
min_size → minimum nodes
max_size → maximum nodes

Example

min = 1
desired = 2
max = 5

👉 Cluster can scale between 1–5 nodes

4. Update Configuration

update_config {
  max_unavailable = var.max_unavailable
}

Why this matters

Controls rolling updates.

Example:

max_unavailable = 1

👉 Only 1 node can be down during update

Why important

prevents downtime
controls deployment safety

5. Labels

labels = var.labels

What labels do

Labels are used by Kubernetes for:

scheduling
targeting workloads

Example:

env = dev
type = backend

👉 Later you can do:

nodeSelector:
  type: backend

6. Dependency

depends_on = [var.node_role_arn]

Why this is needed

Even though role is passed:

Terraform might not always guarantee order.

So this ensures:

👉 IAM role exists before node creation

📄 outputs.tf

output "node_group_arn" {
  description = "Amazon Resource Name (ARN) of the EKS Node Group"
  value       = aws_eks_node_group.nodes.arn
}

output "node_group_status" {
  description = "Status of the EKS Node Group"
  value       = aws_eks_node_group.nodes.status
}

🧠 Why outputs matter

These outputs help in:

debugging
monitoring
integration with other modules

🔥 What You Actually Built

EKS Control Plane
        │
        │
Managed Node Group (EC2 Instances)
        │
        │
Kubernetes Pods run here

⚠️ Real Issues People Face

Wrong subnet → nodes can’t join
Missing IAM role → node creation fails
No NAT → nodes can’t pull images
Too small instance → pods crash

🧠 Key Takeaways

Node group = actual compute layer
Control plane alone is useless without nodes
Scaling config controls capacity
Labels help in workload placement

🚀 Next

In Part 5:

👉 Addons + CSI Driver
👉 How storage works in EKS
👉 Why IRSA becomes critical

At this point, your cluster is alive — now we make it usable.