DEV Community: Debasish Mohanty

How I Debugged a KV-Cache Offloading Bug in vLLM

Debasish Mohanty — Sun, 20 Sep 2026 16:56:16 +0000

How I Debugged a KV-Cache Offloading Bug in vLLM

LLM inference performance is often limited by GPU memory rather than raw compute.

One of the problems I worked on in vLLM involved KV-cache offloading for models using mixed KV-cache groups.

The failure was subtle: the existing logic assumed a single KV-cache group layout, but models with mixed groups could require different block calculations.

The problem

KV-cache is one of the largest consumers of GPU memory during autoregressive generation.

When the GPU cannot keep the required KV-cache resident, vLLM can offload cache blocks to another memory tier.

The existing implementation relied on block_size for its calculations.

That assumption was not sufficient for models with mixed KV-cache groups.

The result was incorrect chunking during KV-cache offloading.

This affected models with architectures such as:

DeepSeek-V4-Flash
Gemma-4

What was wrong

The important distinction was between the configured block size and the number of blocks that should actually be processed together.

Using the existing value directly worked for the common case, but broke when different KV-cache groups had different requirements.

The bug was therefore not simply a memory-capacity problem.

It was an assumption in the API and its downstream calculation.

The fix

I introduced:

blocks_per_chunk

while keeping the existing:

block_size

behavior backward compatible.

The important part was avoiding a breaking change for existing users of the KV-cache implementation.

The new value allows the offloading logic to operate correctly when KV-cache groups require different chunking behavior.

Why this matters

LLM infrastructure bugs are often not obvious application failures.

The model can load.

The request can start.

The GPU can be healthy.

And the system can still produce incorrect behavior because an internal assumption doesn't hold for a particular model architecture.

That is why I find infrastructure debugging interesting.

The problem is usually several layers below the API surface:

Model
↓
Attention / KV Cache
↓
Memory Manager
↓
GPU Memory
↓
Runtime
↓
Kubernetes / Cloud Infrastructure

A production inference system needs every layer to agree about the same assumptions.

The upstream contribution

The fix was submitted upstream to vLLM:

PR #48878

https://github.com/vllm-project/vllm/pull/48878

The change was designed to preserve existing behavior while handling mixed KV-cache groups correctly.

What I learned

The biggest lesson was simple:

The bug was in the assumption, not the runtime.

When debugging inference infrastructure, I now try to identify the invariant first:

What does the API promise?
What does the scheduler assume?
What does the memory manager calculate?
Does that assumption still hold for newer model architectures?

That approach is often more useful than starting from the final symptom.

I work on infrastructure underneath LLM inference:

AWS → Kubernetes → vLLM → GPU

I'm particularly interested in GPU memory, KV-cache management, scheduling, autoscaling, observability, and making inference infrastructure faster and more cost-efficient.

GitHub: https://github.com/Debasish-87

Website: https://www.debasishmohanty.in/

Building an Enterprise Release Governance Platform for Kubernetes (DevSecOps + CI/CD)

Debasish Mohanty — Fri, 13 Mar 2026 08:35:18 +0000

Introduction

Modern CI/CD pipelines often rely heavily on test results to decide whether a release should proceed.

However, in real production environments, tests passing alone does not guarantee a safe release.

For example:

A deployment may pass all tests
But the Kubernetes cluster might already be under pressure
Other services may be crashlooping
Security vulnerabilities might exist in dependencies

To explore this problem, I built a platform called Enterprise Release Governance System (ERGS).

The goal of ERGS is to transform a traditional CI/CD pipeline into a release intelligence system that evaluates multiple signals before allowing a release.

The Problem with Traditional CI/CD

Typical pipelines usually follow a pattern like this:

Run tests
Build artifacts
Deploy

This approach ignores important signals such as:

security vulnerabilities
dependency risks
cluster health
runtime platform stability

In large systems, releasing without considering these signals can introduce serious operational risks.

What is ERGS?

Enterprise Release Governance System (ERGS) is a governance layer on top of CI/CD pipelines.

It integrates multiple validation layers and generates a final release decision:

GO → Safe to release
HOLD → Manual review required
NO-GO → Release blocked

The entire system runs inside GitHub Actions pipelines and produces consolidated reports.

Project repository:

👉 https://github.com/Debasish-87/ReleaseGuard

ERGS High-Level Architecture

                     ┌─────────────────────────┐
                     │     Developer Commit     │
                     │     GitHub Repository    │
                     └─────────────┬───────────┘
                                   │
                                   ▼
                       ┌────────────────────┐
                       │   GitHub Actions    │
                       │  CI/CD Pipeline     │
                       └─────────┬───────────┘
                                 │
                                 ▼

        ┌───────────────────────────────────────────────────┐
        │              RELEASE GOVERNANCE PIPELINE           │
        └───────────────────────────────────────────────────┘

      ┌───────────────┐
      │ Layer 1       │
      │ Application   │
      │ Testing       │
      │ (Allure)      │
      └──────┬────────┘
             │
             ▼

      ┌───────────────┐
      │ Layer 2       │
      │ DevSecOps     │
      │ Security Scan │
      │               │
      │ Semgrep       │
      │ Trivy         │
      │ Gitleaks      │
      └──────┬────────┘
             │
             ▼

      ┌───────────────┐
      │ Layer 3       │
      │ SBOM &        │
      │ Dependency    │
      │ Security      │
      │               │
      │ Syft          │
      │ Grype         │
      └──────┬────────┘
             │
             ▼

      ┌───────────────┐
      │ Layer 4       │
      │ Kubernetes    │
      │ Platform      │
      │ Validation    │
      │ (KPQE)        │
      │               │
      │ Node checks   │
      │ Pod health    │
      │ Crashloops    │
      └──────┬────────┘
             │
             ▼

      ┌───────────────┐
      │ Layer 5       │
      │ Release       │
      │ Dashboard     │
      │               │
      │ Consolidated  │
      │ Reports       │
      └──────┬────────┘
             │
             ▼

      ┌───────────────┐
      │ Layer 6       │
      │ Final         │
      │ Decision      │
      │ Engine        │
      │               │
      │ GO / HOLD     │
      │ / NO-GO       │
      └──────┬────────┘
             │
             ▼

      ┌───────────────────────────┐
      │ GitHub Pages Reports      │
      │                           │
      │ /allure                   │
      │ /security                 │
      │ /sbom                     │
      │ /kpqe                     │
      │ /dashboard                │
      │ /decision                 │
      └───────────────────────────┘

System Architecture

The pipeline evaluates releases using multiple layers.

Layer 1 — Automated Testing

Application tests are executed and results are published using Allure reports.

Outputs include:

Allure HTML report
test execution summary
testing intelligence signals

Layer 2 — DevSecOps Security Scans

Security analysis is performed using several tools:

Semgrep → static code analysis
Trivy → vulnerability scanning
Gitleaks → secret detection

These tools identify potential security risks before deployment.

Layer 3 — SBOM Generation

Software Bill of Materials (SBOM) is generated using Syft.

Dependencies are then scanned for vulnerabilities using Grype.

Outputs include:

CycloneDX SBOM
vulnerability reports

This ensures visibility into software supply chain risks.

Layer 4 — Kubernetes Platform Validation

Before approving a release, the pipeline validates the cluster health.

The platform checks:

node readiness
pod crashloops
restart risk signals
overall cluster health

This stage is implemented using Kubernetes Platform Quality Engineering (KPQE).

Layer 5 — Release Intelligence Dashboard

All signals are merged into a consolidated dashboard.

The dashboard provides:

release summary
testing insights
security scan results
SBOM vulnerability status
Kubernetes readiness signals

Layer 6 — Final Decision Engine

Finally, a decision engine evaluates governance rules.

Example logic:

GO

tests passed
no critical vulnerabilities
cluster health acceptable

HOLD

tests pass but security warnings exist

NO-GO

tests fail
critical vulnerabilities detected
cluster validation fails

The system generates a final output:

final-decision.jsonjson

Example Release Governance Decision

A typical decision might look like this:

json { "releaseDecision": "GO", "tests": "passed", "security": "clean", "clusterStatus": "healthy" }

This provides a clear automated decision for CI/CD pipelines.

Why Release Governance Matters

Modern production environments are extremely complex.

A safe release decision should consider:

code quality
security posture
supply chain risks
infrastructure health

By combining these signals, we can significantly improve release reliability.

Demo

Demo video:

https://youtu.be/rC9K4sqsgE0

Project repository:

https://github.com/Debasish-87/ReleaseGuard

Conclusion

CI/CD pipelines are excellent for automation, but they often lack governance intelligence.

A governance layer like ERGS allows teams to make smarter release decisions by combining:

testing signals
security scans
dependency analysis
Kubernetes cluster health

Instead of relying only on tests passing, releases can be evaluated with a holistic risk perspective.

💬 I'm curious how other teams handle release governance in Kubernetes environments.

Do you validate cluster health before deployments, or rely only on CI/CD pipeline checks?

DEV Community: Debasish Mohanty

How I Debugged a KV-Cache Offloading Bug in vLLM

How I Debugged a KV-Cache Offloading Bug in vLLM

The problem

What was wrong

The fix

Why this matters

The upstream contribution

What I learned

More

Building an Enterprise Release Governance Platform for Kubernetes (DevSecOps + CI/CD)

Introduction

The Problem with Traditional CI/CD

What is ERGS?

ERGS High-Level Architecture

System Architecture

Layer 1 — Automated Testing

Layer 2 — DevSecOps Security Scans

Layer 3 — SBOM Generation

Layer 4 — Kubernetes Platform Validation

Layer 5 — Release Intelligence Dashboard

Layer 6 — Final Decision Engine

GO

HOLD

NO-GO

Example Release Governance Decision

Why Release Governance Matters

Demo

Conclusion