DEV Community: tejaswipandava

Kafka Interview Prep (Part 1) — Know Your Fundamentals Cold

tejaswipandava — Wed, 07 Oct 2026 04:41:00 +0000

Kafka is mandatory for any Backend / Java / Microservices interview. This is Part 1 of a
two-part prep sheet — it covers the core architecture, partitioning, brokers, and delivery
semantics questions that come up in almost every interview. Part 2 moves into real-world
design and production-failure reasoning, aimed at senior/lead-level depth.

📎 Continue to Part 2 → Kafka Interview Prep Part 2 — Production Scenarios & Real-World Design
once you're comfortable with everything below.

1. Kafka Architecture

Producer → Kafka Cluster [Broker 1, 2, 3] → Consumer Group. Zookeeper (legacy) or KRaft
(modern, Zookeeper-less mode) manages cluster metadata (broker list, partition leaders, ACLs).

Topic — a named stream of messages, split into partitions for parallelism.
Partition — an ordered, append-only log. Each partition has one leader broker and N replicas (followers) for fault tolerance.
Broker — a Kafka server that stores partitions and serves reads/writes.
Consumer Group — a set of consumers that jointly consume a topic; each partition is consumed by exactly one consumer within a group at a time.
Zookeeper vs. KRaft — Zookeeper was the original external metadata store; KRaft (Kafka Raft) is the newer built-in consensus protocol that removes the Zookeeper dependency entirely (default since Kafka 3.x / mandatory from Kafka 4.0).

Producer(s) → [Broker 1 | Broker 2 | Broker 3] → Consumer Group (C1, C2, C3...)
                     ↑
              KRaft / Zookeeper (metadata, leader election)

2. acks=0, acks=1, acks=all?

Controls how many broker acknowledgments a producer waits for before considering a write
successful — the core producer durability vs. latency trade-off.

acks=0 — Producer doesn't wait for any acknowledgment. Fastest, but data can be lost if the broker fails before the write is even attempted.
acks=1 — Only the partition leader acknowledges the write. Faster than all, but if the leader crashes before followers replicate the message, it's lost.
acks=all (or -1) — All in-sync replicas (ISR) must acknowledge. Safest option — no acknowledged write is lost as long as at least one ISR survives — but slowest, since it waits for replication round-trips.

3. How to prevent duplicate payment processing?

Standard go-to answer:

Enable the idempotent producer: enable.idempotence=true — Kafka assigns each producer a unique ID + sequence number per partition, so the broker can detect and drop duplicate retries automatically. (Note: this only dedupes broker-side retries of the same send — it does not protect against an application-level duplicate, like the user double-clicking "Pay".)
Store processed message IDs in a DB/cache as an idempotency key — before processing, check if the ID was already handled; if so, skip (or return the cached result).
Use Kafka Transactions (transactional.id) combined with exactly-once semantics (EOS) for read-process-write workflows spanning multiple topics/partitions.

4. What if 6 partitions and 8 consumers in one group?

Only 6 consumers will be active (one per partition); the remaining 2 sit idle with no
partitions assigned. Max active consumers in a group = number of partitions — you cannot
parallelize consumption beyond the partition count within a single group. (Scaling further
requires more partitions.)

5. How does Kafka decide which partition a message goes to?

Every message is routed to a partition using one of three rules, checked in this order:

Explicit partition number provided — if the producer explicitly sets a partition number on the record, Kafka sends it there directly, no hashing involved (see Q6).
Key is provided, no explicit partition — Kafka runs the key through a partitioner (by default, a hash of the key) and maps it to a partition number. The same key always maps to the same partition, which is exactly how ordering-per-key is achieved.
No key, no explicit partition — Kafka spreads messages across partitions in a round-robin-ish / sticky-batching fashion, just to balance load evenly, with no ordering guarantee between any two messages.

6. Can a producer decide exactly which partition its message lands in?

Yes. The producer API lets you pass a partition number directly when sending a record, which
bypasses hashing entirely and forces that message into that exact partition. This is used
sparingly — e.g., for manual load balancing or testing — because it bypasses the key-based
routing that normally keeps ordering and distribution automatic and consistent.

7. Do all messages from one producer end up in the same partition?

No — a producer is not pinned to one partition. Each individual message is routed
independently based on Q5's rules (its own key, or round-robin if keyless). A single producer
instance can and typically does write to every partition of a topic; what stays consistent is
only that messages sharing the same key always land in the same partition, regardless of which
producer sent them.

8. What exactly is a broker, and where does it fit in?

A broker is a single Kafka server process — the thing that actually stores partition data on
disk and serves producer writes / consumer reads over the network. A Kafka cluster is simply
a group of brokers working together. Producers and consumers don't talk to "Kafka" as an
abstract thing — they open connections to specific brokers, which is why the client needs
cluster metadata (which broker currently leads which partition) before it can send or read data.

9. Can one broker host multiple partitions?

Yes, routinely. A broker typically hosts partitions (as leader and/or replica) from many
different topics simultaneously — there's no one-partition-per-broker restriction. With 3
brokers and a topic of 6 partitions, for example, each broker would naturally end up leading
roughly 2 of those partitions, plus hosting replicas of others.

10. What's the relationship between a topic and a broker? Can one broker host multiple topics?

A topic is a logical name for a stream of data; a broker is a physical server. They're
independent concepts — a topic's partitions are spread across the brokers in the cluster
(for parallelism and fault tolerance), and conversely, a single broker commonly stores partitions
belonging to many different topics at once. So yes — one broker hosting multiple topics (and
multiple partitions per topic) is the normal, expected setup, not an edge case.

11. If a producer targets a specific partition and that partition's leader is down, what happens?

The producer's metadata (refreshed periodically, and on error) tells it which broker currently
leads each partition. If the leader is down, the producer's send fails against the stale leader,
triggers a metadata refresh, and the cluster's controller will have already promoted an
in-sync replica (see Part 2 for the full failover story) to be the new leader. The producer then
retries against the new leader transparently (assuming retries are enabled, which is the
default) — from the caller's point of view this looks like a brief delay/retry, not a hard
failure, as long as an in-sync replica was available to take over.

12. Does Kafka put a hard cap on partitions, brokers, consumers, or producers?

No single hard-coded number — these are practical/operational limits, not protocol limits:

Partitions per topic / per cluster — technically unbounded, but every partition has a real memory/file-handle/metadata cost per broker, and very high partition counts (tens of thousands) slow down leader elections and controller operations. Teams size this deliberately, not infinitely.
Brokers per cluster — no fixed cap; production clusters range from a handful to hundreds of brokers.
Consumers in a group — not capped by Kafka itself, but only as many are ever active as there are partitions (see Q4) — any extra consumers in the same group just sit idle.
Producers — effectively unlimited; any number of producers can write to the same topic concurrently.

The practical takeaway for an interview: the real constraint is almost always partition count
(it caps parallelism and has a real operational cost), not some hidden Kafka maximum.

13. Why did Kafka move away from Zookeeper to KRaft?

Zookeeper was an external system Kafka depended on purely to store cluster metadata (broker
list, partition leaders, ACLs) and run leader elections — meaning every Kafka deployment actually
had to run and operate two distributed systems. This added real costs:

Operational overhead — a second system to deploy, monitor, patch, and scale separately.
Scalability ceiling — Zookeeper itself became a bottleneck for clusters with very large numbers of partitions, since every metadata change was a round-trip to Zookeeper.
Slower recovery — controller failover (re-electing which broker manages cluster metadata) was slower when routed through an external system.

KRaft (Kafka Raft) replaces Zookeeper with a built-in consensus protocol (Raft) run by the
Kafka brokers themselves — metadata now lives inside Kafka as just another replicated log. This
means one system instead of two, faster controller failover, and the ability to scale to far
more partitions. KRaft has been production-ready since Kafka 3.x and is mandatory (Zookeeper
support removed) from Kafka 4.0 onward.

14. How to maintain ordering for same Order ID?

Use the Order ID as the message key. Kafka's default partitioner hashes the key to
deterministically route all messages with the same key to the same partition. Since a
partition is a strictly ordered log, all events for that Order ID are processed in order.

⚠️ Ordering is guaranteed WITHIN a partition only — there's no ordering guarantee across
partitions/topics.

15. What is Dead Letter Topic (DLT)?

When a message fails processing after exhausting all configured retries, instead of dropping it
or blocking the partition indefinitely, it's routed to a separate Dead Letter Topic for
manual inspection, alerting, or reprocessing later. This ensures no message is silently lost
and a single poison-pill message doesn't stall the whole consumer.

16. At-Most-Once vs At-Least-Once vs Exactly-Once?

Semantic	Config	Behavior
At-Most-Once	`acks=0` + commit offset before processing completes	May lose messages, never duplicates
At-Least-Once	`acks=all` + commit offset after processing completes	Never loses messages, may duplicate
Exactly-Once	Idempotent producer + Kafka Transactions + manual offset commit	Never loses, never duplicates

At-Least-Once is the most common default in practice, paired with idempotent consumer logic
(dedup by message ID) to approximate exactly-once behavior without the full transactional
overhead.

⚠️ Nuance worth mentioning if pressed: with Kafka's default auto-commit (enable.auto.commit=true),
offsets are actually committed on a background timer (auto.commit.interval.ms), not strictly
"right before" or "right after" a given message — so depending on exactly when a crash happens
relative to that timer, auto-commit can land you in either at-most-once or at-least-once
territory. For a predictable guarantee, disable auto-commit and commit manually at the point you
intend (before vs. after processing).

17. Consumer Lag?

The difference between the latest offset (last message produced to a partition) and the
consumer's committed offset (last message it has processed). High lag means the consumer is
falling behind the producer's rate — a key signal for scaling consumers or investigating slow
processing.

Monitor via: Kafka UI / AKHQ, Grafana + Prometheus (via JMX exporter), or the CLI:

kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group <group-id>

18. Kafka vs RabbitMQ?

	Kafka	RabbitMQ
Model	Log-based (durable, replayable log)	Queue-based (messages removed once consumed)
Throughput	Very high (millions of msgs/sec)	Moderate
Use case	Event streaming, event sourcing, analytics pipelines	Task queues, RPC-style messaging, work distribution
Ordering	Per-partition ordering	Per-queue ordering
Replay	Yes — consumers can re-read from any offset	No — once acked, message is gone
Routing	Simple (topic/partition/key)	Rich (exchanges: direct, topic, fanout, headers)

Rule of thumb: Kafka for high-throughput event streaming and replayable logs; RabbitMQ for
flexible task/work queues with complex routing needs.

19. Real Design: Order → Payment → Inventory

Order Service  → order-topic   → Payment Service
                                        |
                                        ▼
                                 payment-topic
                                        |
                                        ▼
                               Inventory Service

Each service consumes events relevant to it and produces new events downstream — this is
the classic event-driven microservices / choreography pattern:

Order Service places an order → publishes to order-topic.
Payment Service consumes order-topic, processes payment → publishes result to payment-topic.
Inventory Service consumes payment-topic → reserves/deducts stock.

Benefits: asynchronous (services don't block on each other), scalable (each service
scales independently by adding consumers/partitions), fault-tolerant (a downstream service
being briefly down doesn't lose events — they sit in the topic until it recovers).

Trade-off to mention in an interview: this requires careful handling of failure/compensation
(e.g., what happens if Inventory can't fulfill after Payment succeeded? → Saga pattern with
compensating transactions) and idempotency at each consumption step (see Q3 and Q16).

20. What is the role of the `__consumer_offsets` topic?

Kafka stores committed consumer offsets in an internal, compacted topic called
__consumer_offsets (instead of an external store like Zookeeper, as in older Kafka versions).
This lets consumers resume from their last committed position after a restart or rebalance.

21. What is a rebalance, and why can it be disruptive?

A rebalance occurs when consumers join/leave a group (e.g., scaling up, a consumer crash, or
a deployment), triggering redistribution of partitions among the remaining/new consumers. During
a rebalance, consumption pauses ("stop-the-world"), which can hurt throughput/latency if
rebalances happen frequently. Mitigations: Incremental Cooperative Rebalancing (Cooperative Sticky assignor) minimizes disruption by only reassigning partitions that need to move, instead
of revoking all partitions from all consumers.

22. What is ISR (In-Sync Replica)?

The set of replicas (leader + followers) that are fully caught up with the leader's log within an
allowed lag (replica.lag.time.max.ms). acks=all requires acknowledgment specifically from all
current ISR members — if a replica falls too far behind, it's dropped from the ISR and no longer
blocks writes.

23. What happens if a partition leader dies?

Kafka's controller (elected via KRaft/Zookeeper) detects the failure and promotes one of the
in-sync replicas (ISR) to become the new leader — never an out-of-sync replica, since that
could silently lose committed data (this is the unclean.leader.election.enable setting, which
defaults to false for exactly this reason). Producers/consumers refresh their metadata and
redirect requests to the new leader — this is largely transparent to clients, though it can cause
a brief availability blip during the election.

24. How does Kafka achieve high throughput?

Sequential disk I/O — append-only log writes are much faster than random I/O.
Zero-copy transfer — sendfile() lets Kafka send data from disk to network socket without copying through user-space memory.
Batching & compression — producers batch multiple messages per request and can compress batches (gzip, snappy, lz4, zstd) to reduce network/disk overhead.
Partitioning — parallelism across brokers and consumers.

25. What is log compaction?

An alternative retention policy (vs. time/size-based deletion) where Kafka retains only the
latest value for each key in a topic, removing older duplicate-keyed records in the
background. Useful for topics representing "current state" (e.g., a changelog topic backing a
KTable) rather than an event history.

26. What is the difference between a Kafka Producer's `linger.ms` and `batch.size`?

Both control batching behavior on the producer:

batch.size — max bytes to batch before sending (per partition).
linger.ms — max time to wait for more messages to accumulate before sending, even if the batch isn't full. Tuning these trades off latency (lower values) against throughput/efficiency (higher values, larger batches).

27. What is Kafka Streams / KSQL used for?

Kafka Streams is a Java library for building stream-processing applications directly on top
of Kafka (stateful transformations, joins, windowed aggregations) without needing a separate
processing cluster (like Spark/Flink). ksqlDB provides a SQL-like interface over the same
capabilities for simpler use cases.

28. How do you handle schema evolution in Kafka messages?

Use a Schema Registry (e.g., Confluent Schema Registry) with Avro/Protobuf/JSON Schema
to enforce and version message schemas. Configure compatibility modes (backward, forward, full)
so producers/consumers can evolve independently without breaking each other — critical in a
microservices setup where teams deploy on different timelines.

29. What is the difference between `poll()` timeout and `max.poll.interval.ms`?

poll() timeout — how long a single call to poll() blocks waiting for new records if none are immediately available.
max.poll.interval.ms — the max allowed time between successive poll() calls before the consumer is considered dead and triggers a rebalance. Important when message processing itself is slow — if processing takes longer than this interval without calling poll() again, the consumer gets kicked out of the group.

📎 Up next: Kafka Interview Prep Part 2 — Production Scenarios & Real-World Design, covering
the Zomato live-location case study and senior/lead-level production failure scenarios
(duplicate processing, consumer lag diagnosis, multi-event-type topics).

Idempotency - "The customer clicked 'Pay' three times. How do you ensure they're charged only once?"

tejaswipandava — Tue, 06 Oct 2026 12:45:00 +0000

Most know what idempotency is; very few can clearly explain why it's needed and how its actually implemented.

🤔 The Problem

Imagine a payment system. The user clicks "Pay." The request reaches the server and is
processed successfully — but the response never reaches the user because of a network timeout,
a dropped connection, or a slow proxy.

What does the user do? They click "Pay" again.

Without idempotency, this retry can cause the API to:

Create multiple orders
Charge the customer duplicate payments
Deduct inventory more than once
Produce a very angry customer

The core issue: the client cannot tell the difference between "my request never arrived" and "my
request succeeded but the response got lost." Both look identical from the client's side — a
timeout — but calling the API again is only safe in the first case.

💡 The Solution: Idempotency Keys

Generate a unique Idempotency Key for each logical request (not each HTTP call — the same
logical intent, e.g., "charge this cart," keeps the same key across retries).

First request with a given key → process it normally, then store the response alongside the key.
Retry with the same key → the server recognizes it's already been handled, and simply returns the previously stored response instead of processing the request again.
No duplicate processing, no matter how many times the client retries.

The core guarantee: Same request + same key = same result, every time.

Client                         Server
  │  POST /pay (Idempotency-Key: abc123)
  ├──────────────────────────────►
  │                               │  key "abc123" not seen before
  │                               │  → process payment
  │                               │  → store {key: abc123, response: {...}}
  │  ◄── 200 OK {charge_id: X}────┤
  │  (response lost in transit)   │
  │
  │  Retry: POST /pay (Idempotency-Key: abc123)
  ├──────────────────────────────►
  │                               │  key "abc123" already exists
  │                               │  → skip processing, return stored response
  │  ◄── 200 OK {charge_id: X}────┤   (same charge_id, no new charge)

🌍 Where Idempotency Is Used

Anywhere duplicate side effects are unacceptable:

Payment APIs
Order creation
Money transfers
Ticket booking
Inventory updates

🎯 Common Questions

1. What is idempotency?

An operation is idempotent if performing it multiple times has the same effect as performing it
once. In HTTP/API terms: retrying an idempotent request should never change the outcome beyond
what the first successful call already did.

2. Why is POST not idempotent by default?

By HTTP semantics, POST typically means "create a new resource" — calling it twice naturally
creates two resources (two orders, two charges), because the server has no built-in way to know
the second call is a retry of the first rather than a genuinely new request. GET, PUT, and
DELETE are idempotent by convention/spec (see Q6), but POST is not — which is exactly why
payment/order-creation APIs (which are inherently POST-based, since they create something) need
an explicit idempotency mechanism layered on top; it doesn't come for free from the HTTP verb.

3. How do idempotency keys work?

The client generates a unique key (typically a UUID) per logical operation — generated once and reused across all retries of that same operation, never regenerated on retry.
The client sends it with the request, usually as a header (e.g., Idempotency-Key: <uuid>).
The server checks a fast lookup store for that key:
- Not seen before → process the request, store {key → response}, return the response.
- Already seen, completed → skip reprocessing, return the stored response directly.
- Already seen, still in-flight → the request is currently being processed by a concurrent call; the server should make the retry wait (or return a 409 Conflict telling the client to retry after a short delay) rather than let both proceed and race each other.

4. Where should idempotency keys be stored?

A fast, shared, low-latency key-value store — Redis is the typical choice, since idempotency
checks sit directly on the hot path of every write request and need sub-millisecond lookups. Some
systems back this with a database unique constraint on the idempotency key column instead of
(or in addition to) Redis, trading a bit of latency for stronger durability guarantees (see the
Redis-crash question below for why this matters).

5. How long should they be retained?

Long enough to cover the realistic retry/timeout window of the client, but not indefinitely:

Too short → a legitimate late retry (e.g., a mobile client that was offline for a few minutes) is treated as a brand-new request, causing a duplicate.
Too long → the store grows unbounded, wasting memory/storage on keys nobody will ever reuse.

A common choice is 24 hours for payment-style APIs (covers virtually all realistic client
retry/backoff windows), implemented via a TTL on the stored key.

6. Which HTTP methods are idempotent?

Method	Idempotent?	Why
`GET`	✅ Yes	Read-only, no side effects regardless of call count
`PUT`	✅ Yes	Replaces a resource with a given representation — calling it N times leaves the same end state as calling it once
`DELETE`	✅ Yes	Deleting an already-deleted resource still results in "resource doesn't exist"
`POST`	❌ No (by default)	Semantically "create" — repeated calls create repeated resources unless an idempotency key is explicitly layered on top
`PATCH`	❌ Not guaranteed	Depends on the semantics of the patch — e.g., "increment balance by 10" is not idempotent, "set status to X" is

💡Tip

Don't say:

❌ "Idempotency prevents duplicate requests."

This is imprecise — idempotency doesn't prevent the client from sending duplicate requests at
all (they will, especially on timeout/retry); it prevents duplicate requests from causing duplicate
side effects.

Say instead:

✅ "Idempotency ensures the same request produces the same outcome. A common implementation uses
an Idempotency Key to identify retries and return the original response instead of processing
the request again."

This shows you understand both the problem (retries are inevitable and indistinguishable from
first attempts) and the solution mechanism (key-based deduplication with a stored response),
not just the buzzword.

💬 Follow-up: What if Redis (storing the idempotency keys) crashes?

This is the natural next question which tests your understanding of the trade-offs behind your own solution, not just the happy path.

Key considerations to raise:

In-memory Redis alone is a single point of failure for a correctness-critical mechanism.
If Redis crashes and loses data (no persistence configured), a request that was already
processed but not yet acknowledged to the client can be reprocessed on retry, since the
server no longer remembers seeing that key — defeating the entire point of idempotency at
exactly the moment it matters most.
Mitigations, roughly from cheapest to strongest:
- Redis persistence (AOF/RDB) + replication — reduces data loss on a crash, but doesn't eliminate the window between a write and it being durably persisted/replicated.
- Backing store with a durable, unique constraint — write the idempotency key to a relational DB (or DynamoDB/Cassandra with a unique key constraint) as the source of truth, using Redis only as a fast-path cache in front of it. If Redis is down or cold, fall back to the DB — slower, but still correct. A DB INSERT ... ON CONFLICT DO NOTHING (or equivalent unique constraint violation) gives you atomic "claim this key or fail" semantics even without Redis.
- Idempotency at the payment provider itself, as a second line of defense — many payment processors (Stripe, etc.) support their own idempotency keys passed through to them. Even if our own store fails and we accidentally call the provider twice with the same key, the provider's own deduplication prevents an actual double charge. This is the "defense in depth" answer: don't rely on a single layer being perfectly durable.
Fail-closed vs. fail-open trade-off: if the idempotency store is completely unavailable and
there's no fallback, should the request be rejected (fail-closed — safe, but hurts availability)
or allowed through with the retry risk accepted (fail-open — available, but risks a duplicate)?
For payments specifically, I'd lean fail-closed (reject with a clear retriable error) rather
than risk a duplicate charge — the cost of a duplicate payment is much higher than the cost of a
brief availability blip on the write path.

Strong answer framing: "I wouldn't rely on Redis alone for something this correctness-critical.
I'd use Redis as a fast-path cache, back it with a durable store with a unique constraint as the
real source of truth, and pass the same idempotency key through to the downstream payment provider
as a second independent layer of protection — so no single component failing causes an actual
double charge."

Latency Percentiles (p50/p90/p95/p99)

tejaswipandava — Mon, 05 Oct 2026 10:11:31 +0000

Latency is the time an API takes to respond to a request — the delay between when the request is sent and when the response is received.

1. Problem

A single "average latency" number hides the experience of your worst-served users and makes capacity/overload decisions impossible. A service can report a perfectly healthy average while a meaningful slice of real users — often the ones on slow networks, hitting a cold cache, or landing on a degraded instance — have a broken experience. You need a metric that tells you how bad the tail of your traffic looks, not just the middle of it.

2. Constraint

Latency is never normally distributed. It has a long right tail caused by GC pauses, cold starts, lock contention, cache misses, slow downstream calls, and retries. Averages are dominated by the bulk of fast requests and are mathematically insensitive to a worsening tail.
Consumers of a service (other teams, external clients) need a concrete, published promise of how fast the service responds — not a vague "it's usually fast." This requires an agreed-upon number, not a feeling.
In a real architecture, a single service rarely acts alone — it fans out to multiple upstream APIs, databases, and third-party services:

Upstream API 1 ──┐
Upstream API 2 ──┼──►  our service  ──┬──► DB 1
Upstream API 3 ──┘                    ├──► DB 2
                                       └──► DB 3

When a request depends on several downstream calls, the overall response is only as fast as the slowest one. This means tail latency compounds across a call graph — a problem a single-hop percentile number doesn't reveal on its own (see 3.5).

Computing an exact percentile requires sorting every recorded latency value, which doesn't scale once you're measuring millions of requests per minute across many endpoints.

3. Decision

3.1 Define and publish an SLA, backed by an SLO

An SLA (Service Level Agreement) is the formal promise between a service provider and its consumers — it spells out what will be offered and the standard the provider commits to meeting. Underneath that promise sits an SLO (Service Level Objective) — the measurable target (e.g., "p99 < 120ms") that the team actually tracks day to day. The SLA is the external contract; the SLO is the internal number engineering is held to in order to honor it.

3.2 Track percentiles, not the average

A percentile simply answers: "what's the response time for X% of requests?" A pXX value means XX% of requests completed at or below that time — the remaining (100 − XX)% took longer. Reading left to right below, each percentile looks at a progressively smaller, slower slice of traffic:

Percentile	On 100 requests...	What it tells you	What it represents	When to use it
p50	50 requests were faster than this value	middle response time	the typical user's experience	baseline performance
p90	90 requests were faster; 10 were slower	early warning sign	the leading edge of your slow traffic	first signal that something is starting to degrade
p95	95 requests were faster; 5 were slower	overall quality	the bulk of your users, including the slower ones	the performance budget most teams commit to
p99	99 requests were faster; only 1 was slower	reliability	your worst-case users	debugging the tail and latency-critical paths

Practical guidance: start by optimizing for p95 — it's achievable, actionable, and catches real degradation without you chasing every last outlier. Only once p95 is consistently met should you graduate to tightening p99; chasing the tail before the bulk of traffic is stable is usually wasted engineering effort. (And don't stop at p50 alone — it's a useful baseline, but by definition it's blind to the slower half of your traffic, which is exactly what you need to catch before it spreads.)

3.3 Compute percentiles with approximate streaming structures, not sorted arrays

At scale, monitoring tools like Grafana and Prometheus don't sort every raw latency value — they use streaming histogram-based approximations (e.g., Prometheus histograms, HDR Histogram, t-digest) that give a percentile estimate with a small, bounded error at a fraction of the cost.

3.4 Monitor trends, not single data points

Collect response times per endpoint.
Alert on p95/p99 trend lines rather than a single instantaneous reading — this catches genuine, sustained degradation and filters out one-off noise and spikes.
Compare p95/p99 across regions to catch localized/regional performance issues that a global average would mask.

3.5 Account for tail latency amplification in fan-out calls

If your service calls out to multiple downstream dependencies to answer a single request (as in the diagram above), the chance that at least one of them is slow grows quickly with the number of calls. For example, if each downstream call independently has a 1% chance of being "slow" (its own p99), and your service makes 50 such calls to answer one request, the probability that the overall request is slowed by at least one of them is roughly 1 - (0.99)^50 ≈ 39% — far worse than the 1% tail of any individual dependency. This is why a service's end-to-end p99 is often dominated by its worst dependency, not its average one, and why latency budgets are usually allocated per hop rather than assumed to simply "add up."

4. Trade-off

Percentile-based SLOs require more sophisticated monitoring infrastructure (histogram-aware backends like Prometheus/Grafana) than a single average — accepted because the alternative actively hides the problem you're trying to catch.
Approximate percentile structures (t-digest, HDR Histogram) introduce a small, bounded estimation error — accepted as negligible compared to the cost of exact computation at production volume.
Pushing hard for a tight p99 across every dependency in a fan-out architecture is costly (redundant calls, hedged requests, over-provisioned capacity) — accepted selectively, only on the paths where the business impact of a slow tail actually justifies it. Chasing p99.99 everywhere, immediately is the extreme version of this mistake: expensive for most endpoints, with diminishing returns that rarely justify the cost outside of genuinely latency-critical paths.

5. Failure Mode

Metrics blind spot: dashboards show a stable p50 (and even a stable average) while p95/p99 silently climbs for days. A narrow but real degradation — e.g., one bad database replica serving 5% of traffic — can go undetected until it's wide enough to affect the median too. Mitigation: always alert explicitly on p95/p99, never just p50.
Noisy single-sample alerting: alerting on one bad percentile reading from a single window causes alert fatigue and desensitizes the team to real issues. Mitigation: alert on trend over a rolling window, not an instantaneous spike.
Tail latency amplification goes unnoticed: teams monitor the service's own handler time but not each downstream call individually, so the end-to-end p99 looks "mysteriously" worse than any single dependency's reported p99, with no obvious owner to investigate. Mitigation: track and budget latency per hop (see 3.5), not just end-to-end.

Where to look first

When p95/p99 degrades, these are the usual suspects:

DNS and TLS handshake overhead.
Cold starts of services/instances.
Missing or cold cache.
Slow database queries (missing indexes, lock contention).
Connection pool exhaustion.
Slow downstream dependencies.
Garbage collector pauses.