Your gRPC stream is "healthy" and serving nothing: synthetic monitoring for server-side streams

Daniil R — Mon, 27 Jul 2026 13:26:37 +0000

By Daniil Romashov — SRE/DevOps engineer. The tool described here is open source:
github.com/youngpabl0/grpc-streams-checker (Apache-2.0).

Uptime checks are a solved problem for request/response APIs: hit the endpoint, read the status code, done.
Server-side streaming RPCs break that model completely. A gRPC stream isn't a request and a response — it's a channel the server holds open and pushes messages into over time. The
connection can be up, the handshake can succeed, the health endpoint can be green — and yet the
only thing that matters, frames arriving with valid data, can be silently broken.
Your probe says "healthy" while consumers get nothing.

I hit this running real-time market-data streams behind a gRPC-web/Envoy stack. In a realistic
production topology the stream crosses several hops before anyone consumes it:

client ──▶ LB ──▶ Envoy (grpc-web / HTTP/2) ──▶ backend ──▶ upstream feed

Every hop is a place where the stream dies while everything still looks healthy:

an idle timeout on the LB or proxy quietly kills long-lived connections;
Envoy / grpc-web translation buffers or drops frames while the TCP session stays up;
the backend keeps the stream open but its upstream feed went quiet — nothing to push;
frames arrive, but they're garbage — empty payloads, missing fields — after a bad deploy.

None of this is visible to an HTTP 200 probe or a standard gRPC health check.
"The stream stopped
producing" was simultaneously our worst failure mode and the one nothing caught — customers reported it before monitoring did. So I built a synthetic checker for it, ran it in production for
a year across six streaming methods, and have now open-sourced a clean rewrite.

The core idea: consume the stream like a real client

The only probe that answers "is this stream actually serving valid data?" is a real client.
For each configured stream, on a infinite cycle, grpc-streams-checker doing:

loads the .proto at runtime (via @grpc/proto-loader — no codegen, no stale stubs; adding a stream to monitoring = pointing at a file and naming a method);
opens the server-side streaming RPC — TLS or plaintext, auth metadata if needed;
collects N frames (not just the first!) within a time budget, measuring time-to-first-frame and the largest gap between consecutive frames;
validates every frame's payload — required fields present, values matching exact strings or regexes — because "frames are flowing" and "frames are correct" are different failures;
cancels the stream and exports the outcome as Prometheus metrics.

Point 3 deserves emphasis. A stream that emits one frame and hangs is a different (and sneakier) failure than one that never starts. Demanding, say, minFrames: 3 per check verifies sustained flow.
And the largest inter-frame gap turns out to be a great early-warning signal: a stuttering
producer degrades there long before it goes fully dark.

Point 4 catches the failure nobody instruments for: the deploy that keeps the stream up but breaks the payload.
An empty symbol field in a price tick is an outage for the consumer even though every transport-level signal is green.

Metrics designed for alerting, not just dashboards

Every failure mode gets its own result label, and — the key design decision — every per-stream metric carries a method label, so per-method dashboards and ownership-routed alerts fall out of the label set for couple of minutes:

grpc_stream_up{stream,method,addr,dc}                      # 1 = frames arrived AND payloads valid
grpc_stream_check_total{...,result}                        # ok|timeout|insufficient_frames|validation_failed|error
grpc_stream_error_total{...,code}                          # UNAVAILABLE, DEADLINE_EXCEEDED, UNAUTHENTICATED…
grpc_stream_first_frame_ms / _duration_seconds (histogram) # degrades before it breaks
grpc_stream_max_inter_frame_ms                             # sustained-flow health
grpc_stream_frames_last_check / _total                     # did we get the 3-5 we demanded?
grpc_stream_validation_fail_total{...,rule}                # missing_field | mismatch | empty_frame
grpc_stream_last_success_timestamp_seconds                 # staleness alerting
grpc_stream_last_run_timestamp_seconds                     # a dead checker must never look like health

Success rate by method — one query:

sum by (method) (rate(grpc_stream_check_total{result="ok"}[5m]))
/
sum by (method) (rate(grpc_stream_check_total[5m]))

The repo ships ready-made Prometheus alert rules
— stream down, slow first frame, stuttering producer, invalid payloads, repeated gRPC errors, and
"the checker itself died" (silence must never masquerade as health) — plus
Kubernetes manifests
with probes, resource limits, and a ServiceMonitor.

Try it in 32 seconds

The repo includes a demo gRPC server with intentional failure modes:

git clone https://github.com/youngpabl0/grpc-streams-checker.git
cd grpc-streams-checker && npm ci

node examples/demo-server.js &        # a PriceStream on :50051
cp streams.example.json streams.json
node src/index.js --once
# → {"stream":"price_stream","result":"ok","frames":3,"firstFrameMs":223,...}

node examples/demo-server.js --silent   # opens, never sends → result: timeout
node examples/demo-server.js --broken   # empty fields       → result: validation_failed
node examples/demo-server.js --slow=2000 # stuttering        → max_inter_frame_ms grows

--once exits non-zero on any failure, so the same binary doubles as a CI gate or cron probe.

What a use in production taught me

Per-method labels pay for themselves. With team-routed alerting on top, each squad gets paged only for its own streams — one checker serves the whole org.
Time-to-first-frame is the canary. Every full outage we had was preceded by visible first-frame degradation. Alert on the trend, not just the corpse.
Validate payloads. Two of our worst incidents were "stream up, data wrong." Transport metrics can't see that class of failure at all.
Monitor the monitor. last_run_timestamp with a "checker is dead" alert is non-negotiable; a crashed checker otherwise reads as a fleet of healthy streams.

The pattern generalizes to any long-lived push transport — WebSocket, SSE, queue consumers. The
value is turning "is data actually flowing, and is it valid?" into labelled metrics you page
yourself about, instead of something a customer reports.

Daniil Romashov is a Site Reliability / DevOps engineer specializing in reliability and
observability for high-load systems.
Code and manifests: github.com/youngpabl0/grpc-streams-checker.

DEV Community: Daniil R

Your gRPC stream is "healthy" and serving nothing: synthetic monitoring for server-side streams

The core idea: consume the stream like a real client

Metrics designed for alerting, not just dashboards

Try it in 32 seconds

What a use in production taught me