Observability - A Counter in RAM, an ID in a Header, and a Batch Export

Tarek E — Fri, 14 Aug 2026 21:49:37 +0000

For a long time, my mental model of observability was this: you import an SDK, sprinkle some calls through your code, each call fires off data to a server somewhere, and a dashboard reads it back. A logging system with extra steps.

That model is wrong in a specific, interesting way. And I couldn't see how it was wrong until I stopped looking at the dashboards and started looking at what actually gets emitted, and how.

The seductive wrong model

The wrong model is seductive because the plumbing really does look identical. Logging: emit, store, search. Observability: emit, store, query. Same loop, right?

So my working theory became: observability is logging plus some fancy logic to analyze the logs.

Close. But no. The difference isn't in the analysis. It's in the emission — and it splits into three mechanisms that have almost nothing in common with each other.

Descent one: metrics aren't events at all

A metric is not a record you write. It's a number sitting in your app's memory.

requests_total.increment()     // 1, 2, 3...
request_duration.record(0.23)  // adds to a histogram

Nothing is sent when this line runs. The number just changes in RAM. Periodically — every 15 seconds, say — either a backend scrapes an endpoint your app exposes, or a collector ships the current values out.

That's why metrics are absurdly cheap: a million requests is one counter reading "1,000,000", not a million records. You could never reconstruct a clean p99 latency graph by parsing log text. The histogram was built for it at write time.

And the stateless-container objection answers itself: the in-memory counter is disposable. Each instance flushes to the backend on a schedule (on serverless, a sidecar collector even does a final flush at shutdown), and the backend sums across instances. The durable truth never lived in your app.

Descent two: logs are the familiar part

Logs work exactly the way I always assumed everything worked: an event, written out, shipped, searched.

The only upgrade observability adds is one field, injected automatically:

log.info("payment failed", { trace_id: "abc123" })

That trace_id looks trivial. It's the glue for everything else.

Descent three: the trace, or the part that felt like magic

Distributed tracing was the piece I couldn't demystify. A request hops across five services and somehow the platform shows me the whole path as one tree. Surely something sophisticated is happening.

It isn't.

When service A calls service B, the SDK stuffs the trace ID into the outgoing request. For HTTP, it's literally a header — traceparent, a W3C standard. For gRPC, the same ID rides in the metadata (which is HTTP/2 headers underneath). For a message queue, it goes in the message headers.

Service B reads the header and says: I'm part of trace abc, my parent is span 1. Each service exports its spans independently. The backend reassembles the tree later, matching parent and child IDs like a linked list.

That's the entire trick. No agent watching the network. No distributed coordination. An ID, copied into a header, over and over.

The click

Here's where it snapped into place.

Observability is not one system. It's three mechanically different emitters that happen to share a pipeline:

a number in RAM, sampled on a timer
an event, enriched with an ID and shipped
an ID in a header, propagated hop by hop and reassembled later

And the reason no amount of after-the-fact log analysis can replace it: if the trace ID was never propagated at request time, no analysis can invent it. The structure has to exist when the data is born. The dashboard is downstream of a decision made in a header, milliseconds earlier, inside your request path.

Oh. It's just a counter, a log line with an ID, and a header. That's it.

The fancy part was never the analysis. It was agreeing — across languages, frameworks, and vendors — on what to emit and what to call it. That's what OpenTelemetry actually is: not a tool, a treaty.

Observability is the ability to understand a system's internal state purely from what it emits — so you can answer questions you never thought to ask in advance. It works because the emitted signals carry their relationships with them: metrics as live numbers, logs as events, traces as an ID passed hand-to-hand through every request. The linking happens at emission time; everything after is just reading.

OSI Model - The Internet Is Just Envelopes Thrown Away at Every Router

Tarek E — Fri, 14 Aug 2026 21:48:02 +0000

The wrong mental model

For a long time I pictured a packet as a thing. A little sealed capsule that leaves my laptop, hops from router to router, and arrives at a server on the other side of the planet — the same object the whole way, like a parcel with my address on it.

That picture is wrong. Not slightly wrong. Structurally wrong.

Start at the top

When my browser sends an HTTP request, that request is just data. The layers below it exist to move that data, and each one wraps it in its own header.

Transport (TCP) wraps it in a segment: source port, destination port, sequence numbers.
Network (IP) wraps that in a packet: source IP, destination IP, TTL.
Data link (Ethernet, Wi-Fi) wraps that in a frame: destination MAC first, then source MAC, then a checksum trailer.

Nesting dolls. Fine. That part I had right.

What I had wrong was what happens to those dolls in transit.

Descend: what actually changes per hop

Here's the thing I had to sit with. The destination IP on my packet is the server's IP from the very first instant. Not my router's. The packet is addressed to the far end before it leaves my machine.

But the frame around it? That's addressed to my router's MAC. The frame only knows about the next device on the wire. It's local. It has a range of exactly one hop.

So my router receives the frame, strips it off, and throws it away.

Then it reads the packet inside, decides where to send it next, decrements the TTL, recomputes the header checksum — yes, the IP header is modified at every single router, even when the addresses aren't — and builds a brand new frame addressed to the next hop's MAC.

That happens at every router. Dozens of times. Frame born, frame dead. Every wire gets its own envelope.

One detour the packet can't avoid

The IP addresses do stay constant end-to-end — with one exception, and it happens once, right at my front door.

My home router does NAT: it swaps my private source IP (192.168.x.x) for the public one my ISP assigned, usually rewrites the source port too, and recomputes the TCP checksum because that checksum covers the IPs. After that single rewrite, the addressing is fixed for the rest of the journey.

So the packet isn't immutable either. It's a letter that gets its return address corrected once, and its postmark stamped at every sorting facility.

The click

And then it landed: the packet never travels.

Nothing travels except signals on a wire. What crosses the internet is a pattern — a description of a packet, re-serialized into a fresh frame at every hop, carried one link at a time by envelopes that are shredded on arrival.

My HTTP request doesn't ride across the ocean. It gets rebuilt, hop by hop, thirty times, and the only reason the far end receives "the same" request is that every router faithfully copies the layers above the frame into the next envelope.

DEV Community: Tarek E

Observability - A Counter in RAM, an ID in a Header, and a Batch Export

The seductive wrong model

Descent one: metrics aren't events at all

Descent two: logs are the familiar part

Descent three: the trace, or the part that felt like magic

The click

OSI Model - The Internet Is Just Envelopes Thrown Away at Every Router

The wrong mental model

Start at the top

Descend: what actually changes per hop

One detour the packet can't avoid

The click