DEV Community: Nishant Modak

The 6 Questions to Ask Before Adding a High-Cardinality Label

Nishant Modak — Mon, 19 Jan 2026 20:38:17 +0000

Last month, a team I was talking to added a pod_id label to debug a networking issue. Seemed harmless - only 200 pods.

But with 50 metrics per pod and 2-minute pod churn during deployments, they created 150,000 new series per hour. Prometheus memory climbed from 8GB to 32GB in a week. They didn't notice until it OOMKilled during a production incident.

The fix took 10 minutes. The outage took 3 hours. The postmortem took a week.

The Checklist

Before adding any label that could explode, ask:

1. Which system stores this?

Prometheus pays cardinality costs at write time (memory). ClickHouse pays at query time (aggregation). Know your failure mode.

2. Is this for alerting or investigation?

Alerting must be bounded. Investigation can be unbounded - but maybe shouldn't live in Prometheus.

3. What's the expected cardinality?

distinct_values × other_label_combinations = series count
200 pods × 50 metrics × 10 endpoints = 100,000 series. Per deployment.

4. What's the growth rate?

Will this 10x in a year? Containers, request IDs, user IDs - these grow with traffic.

5. Is there a fallback?

Can you drop this label via metric_relabel_configs if it explodes? Test this before you need it.

6. Who owns this label?

When it causes problems at 3am, who gets paged?

Metrics to Watch

Before cardinality bites:

  prometheus_tsdb_head_series              # Active series count
  prometheus_tsdb_head_chunks_created_total # Rate of new chunks
  prometheus_tsdb_symbol_table_size_bytes  # Memory for interned strings
  process_resident_memory_bytes            # Actual memory usage

If head_series grows faster than expected, you have a problem brewing.

Going Deeper

I wrote a full breakdown of how Prometheus and ClickHouse handle cardinality differently at the storage engine level - head blocks, posting lists, Gorilla encoding, columnar storage, GROUP BY explosions.

https://last9.io/blog/high-cardinality-metrics-prometheus-clickhouse/ - covers why they fail in completely different ways and how to design pipelines knowing that.

Debugging Microservices Like a Pro: How Trace IDs Saved My Production Incident

Nishant Modak — Mon, 19 Jan 2026 06:39:26 +0000

Last week, our checkout service started timing out randomly. 2% of requests, no pattern, no obvious cause. Here's how I tracked it down using trace IDs.

## The Problem

Our architecture:

API Gateway → Auth Service → Cart Service → Payment Service → Notification Service

Users reported "payment failed" but logs showed success everywhere. Classic distributed systems nightmare.

## The Fix: Following the Trace ID

Every request gets a unique trace ID at the gateway. I grep'd for a failing request's trace ID across all services:

  grep "trace_id=abc123" /var/log/*.log

Found it: Cart Service was returning 200, but with an empty response body. Payment Service treated empty as "no items" and silently skipped.

Without trace IDs, I'd still be searching.

Trace ID vs Correlation ID

One thing that confused me early on: what's the difference between a trace ID and correlation ID?

https://last9.io/blog/correlation-id-vs-trace-id/ - they follow a request across service boundaries with parent-child span relationships. Correlation IDs are simpler - just a shared identifier without the hierarchy.

For debugging microservices, trace IDs give you the full picture.

Quick Implementation (Node.js)

`const { trace } = require('@opentelemetry/api');

app.use((req, res, next) => {
const span = trace.getActiveSpan();
const traceId = span?.spanContext().traceId;
req.traceId = traceId;
console.log([${traceId}] ${req.method} ${req.path});
next();
});`

Key Takeaways

Always propagate trace IDs across services
Log the trace ID with every log line
Use distributed tracing tools (Jaeger, Tempo, Last9) to visualize
When debugging, start with the trace ID and follow it

What's your go-to debugging strategy for microservices? Drop a comment!

Is Bun Production-Ready in 2026? A Practical Assessment

Nishant Modak — Fri, 16 Jan 2026 22:00:00 +0000

Is Bun Production-Ready in 2026? A Practical Assessment

Bun has come a long way since its initial release. With the Anthropic acquisition in December 2025, the project now has significant backing and a clear path forward. But is it ready for your production workloads?

What's Changed Recently

Bun 1.3+ Features

The recent releases have focused on developer experience:

Zero-config frontend dev: Run bun index.html directly — it handles hot reloading, ES modules, React transpilation. No Vite or Webpack config needed.
Built-in database clients: Bun.SQL supports PostgreSQL, MySQL, and SQLite natively.
Package security: bun pm check integrates with Socket.dev for vulnerability scanning.

The Anthropic Factor

The acquisition signals long-term investment. Bun powers Claude Code (which hit $1B ARR), so Anthropic has strong incentive to keep it stable and performant. The team remains the same, and it stays MIT-licensed.

When Bun Makes Sense

Based on production experience, here's where Bun shines:

Good fits:

APIs and microservices (fast cold starts matter)
CLI tools and scripts (native TypeScript, fast startup)
Internal tooling (speed up dev cycles)
SSR apps with React/Vue (built-in bundling)

Proceed with caution:

Apps heavily dependent on native Node modules
Workloads requiring every Node.js API to match exactly
Mission-critical systems without thorough dependency testing

Practical Considerations

Pin your versions — Bun's patch releases sometimes include new features
Test your dependencies — Most npm packages work, but edge cases exist
Check Node.js API coverage — Some APIs have gaps or behave slightly differently

Getting Started

If you're evaluating Bun for a new project, check out this comprehensive getting started guide that covers installation, configuration, and use cases in detail.

Bottom Line

Bun is production-viable for many workloads in 2026. The Anthropic backing reduces abandonment risk, and the tooling has matured. Start with lower-stakes projects, validate your specific dependencies, and scale from there.

What's your experience with Bun in production? Drop a comment below.

Why High-Cardinality Metrics Break Everything

Nishant Modak — Thu, 01 Jan 2026 20:56:38 +0000

High-cardinality metrics are one of those ideas that sound obviously right—until you try to use them in production.

In theory, they promise precision. Instead of averages and rollups, you get specificity: per-request, per-user ID, per-container, per-feature insights. The kind of detail engineers instinctively want when something is on fire.

And then things start breaking.

Not immediately. Not loudly. But quietly—often in ways that feel like mysterious bugs until you realize the system itself was never designed for this shape of data.

What makes high-cardinality failures especially painful is that nothing crashes. Dashboards still load. Alerts still fire. Deploys continue as usual. The only early signal is often an unexplainable cost spike or queries that suddenly feel sluggish during incidents.

Under the hood, the reason is mechanical. Every unique label combination creates a new time series. Each series needs storage, index entries, memory during ingestion, and ongoing compaction work. As cardinality grows, cost and query complexity don’t scale linearly—they multiply.

At query time, the problem gets worse. Filters that once narrowed the search space stop being selective. Queries fan out across hundreds of thousands—or millions—of sparse, short-lived series. The query engine isn’t broken; it’s doing exactly what it was asked to do, across far more data than anyone realized they’d created.

The most dangerous failure mode isn’t cost or performance—it’s trust. Charts flicker. Series appear and disappear. Queries return inconsistent shapes. Engineers stop believing what they see and quietly fall back to logs, not because logs are better, but because they’re predictable.

The takeaway isn’t that high cardinality is bad. It’s that unbounded, accidental cardinality shows up later as cost surprises, slow queries, and trust erosion unless systems are explicitly designed for it.

If this sounds familiar, the full post walks through:

why these failures are hard to detect early
the systems-level mechanics behind them and
the patterns teams use to make high-cardinality metrics survivable in practice

Complete article here
https://last9.io/blog/why-high-cardinality-metrics-break/

Log Anything vs Log Everything

Nishant Modak — Wed, 16 Oct 2024 02:37:07 +0000

Log Everything vs. Log Anything ⚡️

https://last9.io/blog/log-anything-vs-log-everything/

Log Everything:

Structured, consistent logging across services
High-cardinality data that adds context
Events that tell a story about system behavior

Fine wine, complex yet clear. Helps future you debug at 3 AM.

Log Anything:

Random console.log("here") sprinkled like confetti
Unstructured text that's a pain to parse
Non-thoughtful severity levels.

Mystery juice, might be tasty, might be toxic. Future you curses past you at 3 AM.

Log Models

Nishant Modak — Sat, 28 Sep 2024 04:29:41 +0000

If LLM Models are all the rage, can we use them for understanding our telemetry data and in particular, logs?

What models are available so we can use that?

https://nishantmodak.github.io/engineering/2024/09/27/log-models.html

Logging in Python: Best Practices

Nishant Modak — Fri, 20 Sep 2024 17:47:59 +0000

Today we're going to talk about something that's about as exciting as watching paint dry, but twice as important: Python logging.

Now, I know what you're thinking. "Nishant, I'd rather stick a fork in my eye than deal with logging." Well, tough noodles, because if you want to be a real programmer - you know, the kind that doesn't spend their weekends hunting for that one bug that's been driving the entire dev team insane - you need to master the art of logging.

You see, logging is like the black box in an airplane. When everything's going smoothly, nobody gives a hoot about it. But the moment your code decides to take a nosedive into the Pacific Ocean of runtime errors, you'll be thanking your lucky stars for every log message you wrote. It's the difference between spending five minutes identifying the problem and spending five days growing a beard while you comb through your entire codebase.

So buckle up. We're about to embark on a magical journey through the land of Python logging. By the end of this article, you'll be logging like a lumberjack on steroids. And who knows? You might even enjoy it. (Okay, that's a stretch, but at least you won't hate it as much.)

What is Python logging?
Python logging module
Printing vs logging
Python logging examples

Snippet 1: Creating a logger with a handler and a formatter
Snippet 2: Logging to a file
Snippet 3: Using logging in a class

Types of Python logging methods
How to get started with Python logging
Advantages and disadvantages of Python logging
Python logging platforms
Basic Python logging concepts
Python logging configuration
Configuring the library
Customizing via factory functions
Configuring using Configparse-Format Files
Python logging performance
Configuration-based considerations
Code-based considerations
Advanced logging techniques
Contextual logging with extra parameter
Using LoggerAdapter for adding context
Logging exceptions
Best practices for Python logging

Now that we've got our roadmap, let's dive into the nitty-gritty of Python logging. Trust me, it's going to be a wild ride. Well, as wild as logging can be, which is about as wild as a sloth on sedatives. But hey, you're here to learn, not to party, right? https://last9.io/blog/python-logging-best-practices/

Logging in Python: Best Practices

Nishant Modak — Fri, 20 Sep 2024 17:47:59 +0000

Today we're going to talk about something that's about as exciting as watching paint dry, but twice as important: Python logging.

What is Python logging?
Python logging module
Printing vs logging
Python logging examples

Snippet 1: Creating a logger with a handler and a formatter
Snippet 2: Logging to a file
Snippet 3: Using logging in a class

Types of Python logging methods
How to get started with Python logging
Advantages and disadvantages of Python logging
Python logging platforms
Basic Python logging concepts
Python logging configuration
Configuring the library
Customizing via factory functions
Configuring using Configparse-Format Files
Python logging performance
Configuration-based considerations
Code-based considerations
Advanced logging techniques
Contextual logging with extra parameter
Using LoggerAdapter for adding context
Logging exceptions
Best practices for Python logging

Prometheus Remote Write

Nishant Modak — Mon, 16 Sep 2024 23:33:57 +0000

Ever felt like your Prometheus setup is about to burst at the seams? You're not alone. We've all been there, watching our monitoring system groan under the weight of a million time series.

But fear not! I've just penned an epic saga on taming the beast that is Prometheus remote write. We're talking queue wizardry, cardinality kung-fu, and relabeling magic that'll make your metrics flow like butter.
Oh, and there's a juicy example where we turned a metric firehose into a well-behaved garden sprinkler. Spoiler: It involves a 60% CPU diet and a 70% latency liposuction.

Curious? Hop over to the blogpost for the full scoop.

Optimizing remote write performance

Happy optimizing, and may your dashboards be ever green! 🚀📉

Logging in Golang

Nishant Modak — Sat, 14 Sep 2024 05:44:52 +0000

Practical insights into Golang logging, including how to use the log package, popular third-party libraries, and tips for structured logging.

Table of Contents

Introduction to Golang Logging
The Standard Library: log Package How I Learned to Stop Worrying and Love fmt.Println()
Popular Third-Party Logging Libraries Because Reinventing the wheel is so 2000s
Structured Logging in Go JSON: Its whats for dinner
Configuring Log Levels and Output Formats Choosing your adventure
Integrating with Observability Platforms Because logs are lonely without metrics and traces
Best Practices and Performance Considerations How to not shoot yourself in the foot
Real-World Examples I really have used this stuff myself
Conclusion Log everything; but log it right with a schema (Otel)

Golang logging guide for developers

Watermelon Metrics

Nishant Modak — Tue, 13 Jul 2021 04:38:42 +0000

Watermelon Metrics; A situation where individual dashboards look green, but the overall performance is broken and red inside.

https://blog.last9.io/need-for-systems-observability/

Nobody likes to wait in a Queue

Nishant Modak — Fri, 24 Jul 2020 07:25:21 +0000

Nobody likes to wait in a queue. Almost everyone in a queue at immigration or in a road traffic has considered

What if I move to the other queue? Thats going faster
Why haven't they added more servers / units / lanes to service?
This is just the bad time of the day. Should have started early.

Its a science! These things can be answered by formulas and you can study these systems formally.

The mathematical field of studying queues is called Queueing Theory.

Queueing Theory 101

Definitions

System Structures

Key Concepts

Arrivals
How things arrive? Stream, Batch
Is there a normal arrival distribution?
Are the arrival times finite / infinite?
Service
Kind of resources needed to service
How long does it take ?
Number of servers available
How many can be serviced?
Parallel / Serial?
Single / Multiple Queues
Preemption (Priority)
Queue Characteristics
Balking (people/things don't join, if queue is long)
Reneging (queue too long, leave mid way)
Jockeying (switch queue)
Finite / Infinite (queue lengths)

Questions we are interested in

How long does a customer (*request) have to wait
What is the probability of customer having to wait longer
Average length of queue
Expected utilisation of the server at any given point

To Decide

Is it worthwhile to improve on service times
‘Auto Scale!’
Should we introduce priorities
Is waiting area adequate

Little's Law

L = ƛW

ƛ = Arrival Rate
W = Waiting times
L = Total being serviced

Presentation

SRE Platform