DEV Community: Xinyang Wu

A Working Map of Computer Networking

Xinyang Wu — Mon, 03 Aug 2026 08:39:29 +0000

Layering is the whole trick

Networking looks like a pile of acronyms until you notice the one organizing idea behind all of it: layering. Each layer solves exactly one problem and offers a clean interface to the layer above. That is why you can swap WiFi for a cable — a link-layer change — and your browser, living at the application layer, never notices.

The OSI seven-layer model is the theory; the five-layer TCP/IP model is what people actually reason with:

Layer	Job	Representative protocols	Addressing unit
Application	conversation rules between programs	HTTP, DNS, SSH, SMTP	—
Transport	end-to-end delivery, reliability, ports	TCP, UDP	port
Network	addressing and routing across networks	IP, ICMP	IP address
Link	delivery within one physical network	Ethernet, ARP	MAC address
Physical	bits to signals	copper, fiber, radio	—

The mechanism that stitches the layers together is encapsulation. As data moves down the stack, each layer wraps it in its own header: your HTTP request becomes the payload of a TCP segment, which becomes the payload of an IP packet, which becomes the payload of an Ethernet frame. On the receiving side, each layer peels off its header and hands the payload up. Once encapsulation clicks, most of networking stops being trivia and starts being a map.

TCP vs UDP

The transport layer offers two very different contracts:

	TCP	UDP
Connection	connection-oriented (handshake first)	connectionless (just send)
Reliability	lost segments are retransmitted	no guarantees; packets may vanish
Ordering	delivered in order	no ordering promise
Overhead	heavier, 20-byte header	light, 8-byte header
Typical use	web pages, file transfer, mail	video calls, games, DNS queries

My favorite mnemonic: TCP is a phone call — you establish the line first, confirm the other side is listening, and speak in order. UDP is a postcard — you drop it in the mailbox and hope.

The interesting question is why anyone would choose the postcard. Real-time video is the canonical answer: in a live stream, a dropped frame is better than a stalled one. If TCP paused the stream to retransmit a frame from 300 ms ago, you would get exactly the freezing behavior you hate. A brief visual glitch and moving on is the correct trade, so real-time media rides on UDP and handles loss at the application layer.

Opening and closing a connection

TCP's reliability starts before any data flows, with the three-way handshake:

client -> SYN         "I want to connect; my sequence number is x"
server -> SYN + ACK   "Ready on my side; my sequence is y, and I saw your x"
client -> ACK         "I saw your y — data can flow"

Why three messages and not two? Because both sides need evidence that both directions work — that each peer can send and receive. After two messages, the server still has no proof the client ever heard its reply. Two-message setup also opens the door to a stale SYN that wandered the network for a while: it could conjure a phantom connection the client never intended. The third message closes both gaps.

Teardown takes four messages:

client -> FIN   "No more data from me"
server -> ACK   "Understood"
server -> FIN   "I am finished too"
client -> ACK   "Goodbye"

Why do the ACK and FIN not always merge the way SYN and ACK do during setup? Because closing is one-directional. When the client says it has nothing left to send, the server may still have responses in flight. It acknowledges the client's FIN immediately, keeps sending until it is done, and only then sends its own FIN. Half-closed connections are a feature, not an accident. That said, when the server has nothing left to send — the typical short-lived HTTP exchange — it does combine the two into a single FIN+ACK segment, collapsing the close to three segments, which is exactly what most packet captures show.

From URL to pixels

Typing a URL and hitting enter exercises every layer in sequence, which makes it the best single walkthrough in networking:

DNS lookup — the domain name becomes an IP address (application layer, usually over UDP).
TCP handshake — three messages to the server's IP (transport layer).
TLS handshake — for HTTPS, the two sides negotiate a symmetric key so everything after is encrypted.
HTTP request — GET / plus headers travels through the established, encrypted channel.
Response — the server returns a status code and the HTML body.
Render — the browser parses HTML, CSS, and JavaScript, building the page and often firing more requests for images and scripts, each repeating some of the steps above.
Teardown — connections close with the four-message sequence when they are no longer needed.

Every hop names its layer: DNS at the application layer, TCP at transport, IP routing at the network layer, Ethernet or WiFi at the link layer underneath it all. If you can narrate this end to end, the layer model has done its job.

HTTP essentials

HTTP method semantics are worth internalizing, because retry logic and caching depend on them:

Method	Idempotent	Safe (read-only)	Meaning
GET	yes	yes	fetch
PUT	yes	no	set or overwrite
DELETE	yes	no	remove
POST	no	no	create

Idempotent means repeating the request leaves the system in the same state — which is why a client may safely retry a timed-out PUT but should think twice before retrying a POST.

Status codes come in classes: 1xx informational, 2xx success (200), 3xx redirection (301, 304), 4xx client errors (401, 403, 404), 5xx server errors (500, 502, 503). The class tells you who to blame before you read the digits.

Finally, HTTP is stateless: the protocol itself remembers nothing between requests. Login state is layered on top with cookies, server-side sessions, or signed tokens — three different answers to the same question of how a stateless protocol keeps recognizing you.

DNS resolution

Name resolution walks a chain of lookups — most of them caches — and the first hit wins:

browser cache -> hosts file (/etc/hosts) -> OS DNS cache -> local resolver (often the ISP's)
    -> root servers (.) -> TLD servers (.com) -> authoritative servers

One distinction worth keeping sharp: the hosts file is static configuration, not a cache. The OS DNS cache proper — systemd-resolved on Linux, mDNSResponder on macOS — stores previous answers with their TTLs and can simply be flushed when it goes stale, while a hosts-file entry sits there until someone edits it out. When you are chasing a wrong DNS answer, knowing which of the two you are looking at decides the fix.

The local resolver typically does the legwork for you — a recursive query — walking from the root to the top-level domain to the authoritative server, then caching the answer with its TTL so the next lookup short-circuits. Queries usually travel over UDP port 53 because they are tiny and speed matters; large responses and zone transfers fall back to TCP.

Flow control vs congestion control

These two TCP mechanisms are easy to conflate because both slow a sender down. They protect different victims:

	Flow control	Congestion control
Protects	the receiving host	the network as a whole
Mechanism	sliding window — the receiving side advertises how much it can accept	slow start, congestion avoidance, fast retransmit
Viewpoint	point-to-point, between two endpoints	global, across everyone sharing the path

Flow control is a courtesy between two machines: do not overrun the buffer at the far end. Congestion control is a social contract: probe the shared network gently, back off when packets start disappearing, because loss is the network's way of saying it is full.

Addresses and NAT

An IPv4 address is 32 bits, and three ranges are reserved for private networks: 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. CIDR notation encodes the split between network and host bits: /24 means the first 24 bits name the network, leaving 8 bits for hosts — 256 addresses, of which about 254 are usable.

NAT is why your whole household shares one public IP. The router maps many private addresses to a single public one, distinguishing flows by port number. It is also why inbound connections to a machine behind NAT are hard: nobody outside can address it directly, which is the root problem that port forwarding and tunneling solutions exist to solve.

Where to go deeper

Three threads I want to pull on next:

TLS in detail — the hybrid-encryption handshake, best studied side by side with SSH's version of the same idea. I walked through the SSH stages in a separate note.
HTTP/1.1 vs 2 vs 3 — head-of-line blocking, multiplexing, and why HTTP/3 abandons TCP for QUIC.
Load balancing and CDNs — where this map meets systems design.

None of this is exotic. But having the map means that when something breaks — a timeout, a refused connection, a stale DNS answer — you know which layer to interrogate first.

RAG Beyond the Demo: Pipeline, Citations, Evaluation, and When Not to Bother

Xinyang Wu — Mon, 03 Aug 2026 08:38:17 +0000

What RAG buys you

Retrieval-augmented generation has a one-sentence core: retrieve relevant documents first, put them in the prompt, and let the model answer from them. That single move addresses three structural weaknesses of an LLM on its own. Hallucination gets reduced, because the model is conditioned on real documents you supplied — reduced, not eliminated, since nothing forces the decoder to stay inside the retrieved context, which is why faithfulness evaluation shows up later in this post. Stale or private knowledge stops requiring retraining, because updating the system means swapping documents, not weights. And context cost stays under control, because you send the few relevant chunks instead of the whole corpus on every request.

The interesting decision is when not to use it. I think of it as a triangle. If the corpus is small enough to fit in the context window, long context is the degenerate case of RAG — just send everything. If the corpus is large, or you need cost control, or you need citations, retrieval earns its place. And if what you actually want to change is the model's capability or style rather than its knowledge, neither helps: that is fine-tuning territory.

Two pipelines, not one

Every RAG diagram I find useful separates two paths that run on different schedules:

offline: load -> chunk -> embed -> index
online:  query -> embed -> retrieve top-k -> (rerank) -> assemble context -> generate -> cite

The offline path runs when documents change. The online path runs per request. Keeping them mentally separate matters because they fail differently: the offline path fails silently (bad chunking quietly poisons every future answer), while the online path fails loudly and per-query. Evaluation, covered below, attaches to each path separately too.

Chunking itself is a trade-off with no free setting. Chunks around 400 tokens with 10–20% overlap are a sane default; the overlap exists so a sentence split across a boundary still survives intact in one chunk. Larger chunks carry more context but retrieve more coarsely; smaller chunks retrieve precisely but strand facts without their surroundings.

Embeddings without the magic

An embedding model maps text to a dense vector — a few hundred to a couple thousand dimensions — such that semantically similar texts land near each other, measured by cosine similarity or dot product. Whether the vectors come from a hosted API or a local sentence-transformers model is completely transparent to the rest of the pipeline, which is a genuinely nice property: you can start local and swap later.

A question worth answering precisely: why does this need a Transformer at all — would a simple MLP do? No, for two reasons. First, an MLP wants fixed-dimension input, and text is a variable-length token sequence; you need an encoder plus pooling just to produce something fixed-size. Second, and more fundamentally, the semantics live in large-scale contrastive pretraining, not in the layer shapes. The baseline spectrum makes this visible: BM25 sees only surface word overlap, averaged word vectors lose word order and context, and contrastively trained bi-encoders are what finally make "vector near" mean "meaning near".

One linear-algebra fact does most of the work in a minimal implementation: if you L2-normalize the vectors, the dot product is cosine similarity, so retrieval collapses to a single matrix-vector product. The whole retriever fits in a few lines:

import numpy as np
from sentence_transformers import SentenceTransformer

def chunk(text, size=80, overlap=20):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, len(words), step)] or [text]

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
chunks = [(source, c) for source, text in DOCS for c in chunk(text)]
index = model.encode([c for _, c in chunks], normalize_embeddings=True)  # (N, 384)

def retrieve(query, k=3):
    q = model.encode([query], normalize_embeddings=True)[0]
    scores = index @ q                    # dot product == cosine, thanks to normalization
    top = np.argsort(scores)[::-1][:k]
    return [(chunks[i][0], chunks[i][1], float(scores[i])) for i in top]

If you switch to a hosted embeddings API, only the encode calls change — but normalize the returned vectors yourself (APIs do not always guarantee it), and never mix models between indexing and querying: changing the embedding model means rebuilding the index.

When brute force is the right call

Exact nearest-neighbor search is O(N) per query, which sounds like a problem until you put numbers on N. Approximate indexes — HNSW graphs, IVF clustering — exist to trade a little recall for orders-of-magnitude faster queries at millions of vectors. At tens of pages of documentation, N is a few hundred chunks, and the brute-force matrix product above runs in microseconds.

Reaching for a vector database at that scale adds operational surface and an approximation error for zero benefit. Declining to deploy one is not a shortcut; it is a sizing decision, and being able to say precisely when HNSW starts paying for itself is the part worth knowing.

Two-stage retrieval and hybrid search

When retrieval quality does become the bottleneck, two upgrades come before anything exotic.

Two-stage retrieval exploits an asymmetry: bi-encoders are fast but coarse, cross-encoders are accurate but slow. So recall broadly with the bi-encoder — say top-50 — then let a cross-encoder reranker rescore just those candidates down to a top-5. You pay the expensive model only on a short list.

Hybrid search covers the failure mode dense vectors are worst at: exact jargon, abbreviations, and identifiers that the embedding model never learned to place well. Run BM25 alongside the dense retriever and fuse the two rankings with reciprocal rank fusion. A first version legitimately skips both upgrades; the point is knowing which symptom each one treats.

Citations: soft vs hard

Here is the observation that reframed citations for me: the query never connects to sources — the retrieved chunks do, and that connection is already fixed inside retrieve() before the model generates a single token.

That gives you two mechanisms. The soft one embeds source markers in the context and prompts the model to repeat them inline. It gives sentence-level granularity, but the model can mis-attribute, skip citations, or invent sources that were never retrieved. The hard one skips the model entirely: the application already knows which chunks it passed in, so it returns them as structured data alongside the generated text.

def answer(query: str) -> dict:
    hits = retrieve(query)
    context = "\n\n".join(f"[{source}] {text}" for source, text, _ in hits)
    text = call_llm(context, query)
    return {
        "answer": text,
        "sources": [{"source": s, "score": score} for s, _, score in hits],
    }

The structured list is deterministic — those chunks are, with certainty, what the answer was conditioned on — but coarse: it says which documents were used, not which sentence supports which claim. Production systems combine both: hard sources as the authoritative record, soft inline markers for granularity.

Evaluate the two halves separately

A RAG system breaks in two independent places, so evaluating it end-to-end mostly tells you that it is wrong, not where. Split the evaluation at the retrieval boundary.

Retrieval evaluation is deterministic and should come first. Build a golden set of queries mapped to their relevant chunks, then measure recall@k (of the relevant chunks, how many made the top-k — the metric that matters most, because what is never retrieved can never be cited), plus MRR for ranking quality and precision@k for noise.

Generation evaluation is fuzzier and usually uses an LLM as judge, scoring faithfulness (is every claim in the answer supported by the context?) and answer relevance — the RAGAS-style criteria. Judges carry known biases toward position, verbosity, and their own outputs, so use a strong judge model and calibrate with human spot-checks rather than trusting scores blindly.

Golden sets come from three places: hand labeling (best way to start), synthetic generation (ask a model to write the question a given chunk answers), and mining real queries from logs once you have traffic.

Prompt caching changes the calculus

Prompt caching is the mechanism that quietly moves the RAG-versus-long-context break-even. Under causal attention, a token's representation depends only on what precedes it, so providers can cache the KV state of a request's prefix and skip recomputing it when the next request starts with the byte-identical prefix.

The iron rule follows directly: change one early byte and everything after it is invalidated. Requests render in a fixed order — tools, then system, then messages — so the static corpus belongs as early as possible and the volatile user query at the very end. The classic silent cache-killers are timestamps, request IDs, unsorted JSON serialization, and per-user values injected into the system prompt; the symptom is a cached-token counter in the usage stats that stays at zero forever.

Providers differ in the details — some cache automatically past a minimum prefix length, others want explicit cache breakpoints — but cached reads generally cost on the order of a tenth of normal input tokens. That changes the economics: for a mid-sized corpus, you can park the whole thing in a cached prefix and pay full price only for each query. The break-even point where RAG beats long context moves noticeably further out than the un-cached math suggests.

Self-test

Why does L2-normalizing embeddings let a plain dot product stand in for cosine similarity, and what silently goes wrong if you skip the normalization?
Your system returns a fluent, confidently wrong answer. Which half of the pipeline do you check first, and with which metric?
What can prompt-inline citations do that deterministic retrieval-based sources cannot — and what is the reliability cost?
A teammate adds a timestamp to the system prompt of a long-context setup that relies on prompt caching. What happens to cost, and how would you detect it from usage stats?

Refactoring OpenTelemetry Metrics: from God Class to Per-Subsystem Modules

Xinyang Wu — Mon, 03 Aug 2026 08:37:41 +0000

The constraint that shapes everything

A refactor that recently landed in a codebase I work in replaced our metrics god class with per-subsystem modules. The design was a colleague's work, not mine. But studying the change, and migrating my own instruments over to the new pattern, taught me more about OpenTelemetry's design constraints than any tutorial had.

The constraint that shapes everything is easy to state: an OTel instrument — a counter, a histogram, an observable gauge — can be created once per process and never unregistered. Register the same instrument twice and, depending on the SDK, you get warnings or subtly duplicated data. So every metrics architecture in an OTel codebase is really an answer to one question: where does the process-level singleton live?

The old answer in our codebase was one global Metrics object, built at startup and threaded through the application by dependency injection. That does solve the singleton problem. It also creates a monster.

The anti-pattern: one god class of mixins

Because every subsystem needed its instruments on that one global object, the object accreted everything. The composition mechanism was multiple inheritance:

class GeneralMetricsMixin: ...      # each defines instrument fields + an _init_*() method
class AcquisitionMetricsMixin: ...
class CameraMetricsMixin: ...

class Metrics(GeneralMetricsMixin, AcquisitionMetricsMixin, ..., CameraMetricsMixin):
    def __post_init__(self):
        self._init_general_metrics()
        self._init_acquisition_metrics()
        ...

Beyond the usual god-object complaints — the class kept growing, and metric definitions lived far from the code that recorded them — mixins add a sharper failure mode. Mixins compose through Python's method resolution order, and same-named attributes silently override each other. If two mixins both define _duration_metric, the MRO keeps one and the other is shadowed without any error. A metric can simply stop being recorded, and nothing tells you.

The old code carried the fingerprint of this hazard:

def measurement_stopped(self):
    GeneralMetricsMixin.measurement_stopped(self)  # explicit class to dodge a name collision

When plain self.measurement_stopped() is unsafe and methods must be called through an explicit base class, the design is already fighting you. The general lesson is the classic one: composition over inheritance. Independent small classes cannot shadow each other's fields.

The replacement pattern

The refactor deleted the god class and gave each subsystem its own module with a uniform shape: a scope constant, a frozen dataclass holding the expensive OTel handles, a memoized factory, and a cheap wrapper class you can construct per request.

import dataclasses as dc
import functools

from opentelemetry import metrics

_SCOPE = "myapp.api.prediction"

@dc.dataclass(frozen=True, slots=True)
class _Instruments:               # holds only the OTel handles
    failure_counter: metrics.Counter
    duration: metrics.Histogram

@functools.cache                  # memoization as the process-level singleton
def _get_instruments() -> _Instruments:
    meter = metrics.get_meter(_SCOPE)
    return _Instruments(
        failure_counter=meter.create_counter(name=f"{meter.name}.failures"),
        duration=meter.create_histogram(name=f"{meter.name}.duration", unit="s"),
    )

class PredictionMetrics:          # cheap; construct one per request
    def __init__(self, settings):
        self._instruments = _get_instruments()
        self._settings = settings

    def record_failure(self, *, cause: str) -> None:
        self._instruments.failure_counter.add(
            1, {**_settings_tags(self._settings), "cause": cause}
        )

@functools.cache is doing the singleton's job: the factory body runs once per process, which is exactly the guarantee the instruments need, with no global object and no injection boilerplate.

There is a nice separation hiding in this shape. _Instruments is the expensive shared state — created once, immutable, identical for every caller. PredictionMetrics is the cheap per-request context — it binds the current settings in its constructor so call sites don't have to pass them into every method. That split of heavyweight shared state from lightweight context is the flyweight idea, and it shows up everywhere: HTTP clients, database pools, loaded model weights.

Just as important: each subsystem's metric definitions now live next to the code that records them. Changing one feature touches one module, not a central registry.

Metric names are a public API

The most instructive detail of the refactor is what did not change: the emitted metric names. The internals were rewritten, but myapp.api.prediction.failures kept flowing under exactly that name.

Dashboards, monitors, and saved queries reference metric names as plain strings in another system. No compiler sees them, no test fails when they break. That makes a metric name a contract: internals can change freely, but renaming a metric is a breaking change for everyone downstream. When our scope was eventually renamed, the same change also updated the monitoring docs and the affected dashboards — a rename is a migration, not an edit.

The same discipline applies to database column names, REST paths, and event schemas: anything other systems address by name is a public API, whatever your codebase thinks of it.

Cardinality is the bill

OTel calls them attributes, some backends say tags, Prometheus says labels — same thing: key-value pairs on each data point, so you can later filter and group by them. The backend stores one time series per unique combination of metric name and tag values, and that product is what you pay for.

Our failure counter carries tags like model_variant (5 values), aggregated (2), remove_outliers (2), normalization (3), and cause (4):

5 × 2 × 2 × 3 × 4 = 240 time series   # fine

Now add one tag holding a raw floating-point input, with effectively unlimited distinct values. Every distinct float multiplies those 240 series, and you get millions of series, a painful invoice, and slow queries — a classic cardinality explosion.

Two defenses survived the refactor:

# 1. Collapse unbounded values before tagging: strips float noise
#    so 2.0000001 becomes "2"; the parameter only takes a handful
#    of values in practice anyway.
def _scale_tag(scale_factor: float) -> str:
    return f"{scale_factor:g}"

The second is a rule of thumb: log it, don't tag it. Per-request detail — input names, exact intermediate values — goes into structured logs, which handle high cardinality natively and carry the trace id, so you can pivot from a metric spike to the traces and logs of the requests behind it. I wrote more about that pivot in OpenTelemetry from first principles.

Tags are for bounded dimensions: enums, booleans, small buckets. Everything else has a better home.

Push vs pull instruments

OTel has two ways to feed a metric, and picking wrong makes code awkward.

	Push (synchronous)	Pull (observable)
Model	report when an event happens	callback answers "what is it now?"
Instruments	`Counter`, `Histogram` via `.add` / `.record`	`ObservableGauge` with a callback
Fits	discrete events: failures, durations	levels: temperature, queue depth

Events push; levels get pulled. A failure is a moment in time, so you .add(1) when it happens. A queue depth has no event — it just is — so the SDK periodically invokes your callback at collection time.

Sometimes business code wants a push-style API for something that is really a level. The codebase bridges that with a small helper: callers set() the latest value whenever they like, and a registered callback hands the current snapshot to the SDK when collection runs. The concurrency detail is worth copying — hold the lock only to copy, never while yielding:

class ObservableMetric:
    """Push-style set() bridged onto a pull-style observable gauge."""

    def set(self, value: float, attributes: dict) -> None:
        key = tuple(sorted(attributes.items()))
        with self._lock:
            self._values[key] = value

    def _observe(self, options):
        with self._lock:
            snapshot = list(self._values.items())   # copy under the lock
        for key, value in snapshot:                 # yield outside it
            yield metrics.Observation(value, dict(key))

Keeping the critical section to a dict copy means a slow exporter can never block the hot path that calls set().

Self-test

Why does OpenTelemetry force some kind of process-level singleton on you, and how does functools.cache satisfy it?
What is the silent failure mode of composing metrics via mixins, and what code smell is its fingerprint?
A counter has five tags with 5, 2, 2, 3, and 4 possible values. How many time series is that — and what happens when someone adds a raw-float tag?
You need to track failed requests and current queue depth. Which instrument type fits each, and why?

OpenTelemetry from First Principles: Three Signals, One Identity

Xinyang Wu — Mon, 03 Aug 2026 08:37:05 +0000

Why OpenTelemetry exists

OpenTelemetry is a vendor-neutral telemetry standard: a unified API, an SDK, and a wire protocol (OTLP). The point of the split is decoupling how telemetry is produced from which backend consumes it. Your business code writes against the OTel API; if you move from Datadog to Prometheus and Grafana, you change the exporter configuration at startup and touch zero lines of application logic.

That is the headline benefit, but it undersells the design. The real advantage over running three separate tools — a metrics library, a tracing library, a logging pipeline — is that one SDK produces all three signals, and they share a Resource (a common identity) and a context (a common thread). Everything interesting about OTel follows from that sharing.

Three signals, three questions

Each signal answers a different question, and the differences in shape are not accidental:

Signal	Question	Character
Metrics	What broke, and how badly?	Pre-aggregated, cheap, low-cardinality, alertable
Traces	Where in the request did it break?	Causal chain plus timing, per request
Logs	Why, in full detail?	Arbitrary structure, high cardinality

Metrics are the alarm: myapp.api.prediction.failures climbing tells you something is wrong. A trace decomposes one request into a tree of timed spans and tells you which stage is slow or failing. Logs carry the arbitrary, high-cardinality detail — input names, intermediate values — that explains why.

There is a well-known critique, associated with Charity Majors, that the "three pillars" framing misses the point: having three data stores is not observability. Being able to slice by high-cardinality dimensions and pivot freely between signals is. I find that critique exactly right, and it reframes the rest of this post: the mechanisms below are what turn three disconnected monitoring systems into one investigation workflow — metric spike, jump to a representative trace, jump to that request's logs.

The vocabulary that makes docs readable

Five terms unlock most of the OTel documentation:

Resource — who is producing telemetry: service.name, service.version, host.name. The shared identity of all three signals.
Scope (a Meter, Tracer, or Logger) — which module is producing it: metrics.get_meter("myapp.api.prediction") is a namespace, not a global.
Instrument / Span — the concrete gauge: a Counter or Histogram for metrics, a named unit of work with start and end times for traces.
Attributes — key-value dimensions on a measurement or span. Same concept as Datadog tags and Prometheus labels.
OTLP — the wire protocol from SDK to collector, typically gRPC on port 4317.

The API/SDK split

The most consequential design decision in OTel is that the API and the SDK are separate packages with separate roles.

The API is lightweight and is the only thing libraries depend on: get_meter, create_counter, .add(), start_as_current_span. If no SDK is installed, every one of these calls is a no-op — no errors, near-zero overhead. That is why a third-party library can ship built-in instrumentation without forcing a backend, an exporter, or any cost on applications that do not care.

The SDK is assembled exactly once, at application startup: tracer, meter, and logger providers, batch processors, exporters, sampling policy. The application — and only the application — decides whether telemetry is exported, where it goes, and how much is kept.

In practice this looks like instrument definitions scattered across modules (API calls only) plus one setup_telemetry(app) function in a telemetry.py that wires up the three providers and points them at a collector.

Correlation mechanism 1: shared Resource

The coarsest correlation mechanism is also the easiest to overlook. Create one Resource and hang it on all three providers:

resource = Resource.create({
    SERVICE_NAME: service_name,
    SERVICE_VERSION: service_version,
})

TracerProvider(resource=resource)
MeterProvider(resource=resource)
LoggerProvider(resource=resource)

Now service.name is stamped on every metric point, every span, and every log record, and a single filter in the backend — service:myapp — scopes all three signals to the same service at once. It sounds trivial. It stops being trivial the first time you try to correlate signals from a codebase where metrics and logs were configured by different people with different naming.

Correlation mechanism 2: trace context

Trace context is the spine of the whole system, and it operates at two levels.

In-process, the Python SDK keeps an ambient current span in contextvars, which survives await. tracer.start_as_current_span("prediction.combine") makes the new span current; any child span created inside — by you or by a library — attaches to it automatically, no parent-passing required.

Cross-process, the W3C traceparent header carries the context over HTTP:

traceparent: 00-<trace_id: 32 hex>-<parent_span_id: 16 hex>-<flags: 2 hex>

Client instrumentation injects it on outbound requests; server instrumentation extracts it on inbound ones. One user request becomes one trace across services.

Ambient context is also where I learned my favorite failure mode. The global FastAPIInstrumentor().instrument() call works by monkey-patching FastAPI.__init__, so it only affects app instances created after it runs. In one codebase I work in, the apps were built at module import time, before telemetry setup — so they were silently uninstrumented. No error, no warning. No SERVER span meant no per-request root anchor, so manually created spans grafted themselves onto whatever stale ambient context was left on the event loop, and showed up inside unrelated traces from hours earlier. The fix was the instance-level FastAPIInstrumentor.instrument_app(app) (order-independent), plus a regression test pinning the initialization contract.

The lessons generalize: ambient context is implicit global state — it never raises, it just quietly gives you wrong parentage. The server span is the per-request root anchor; without it, every manual span is adrift. And monkey-patch-style auto-instrumentation always has an initialization-order contract worth locking down with a test.

Correlation mechanism 3: logs that know their trace

Connecting logs to traces requires no application changes at all. The OTLP LoggingHandler reads the current span context at emit time and stamps trace_id and span_id onto every log record. Write structured logs — fields as attributes, not interpolated prose — and each line is automatically linked to the request that produced it:

LOGGER.info("prediction diagnostics", extra={
    "prediction.model_variant": variant,
    "prediction.input_names": names,  # high cardinality: logs, not metric tags
})

One backend click takes you from a trace to that request's logs, and back.

A second war story lives here. After a dependency upgrade, every log line appeared twice in the backend — one copy complete, one missing attributes. The cause: a newer version of the logging instrumentation library began auto-installing its own root logging handler, alongside the handler the codebase had added manually. Two OTLP handlers on the root logger, two exports per record. The fix was making the manual handler the only export path and leaving a comment explaining why. The durable lesson: your telemetry export path must be unique and understood, and "duplicate log lines, one copy with fewer attributes" is the diagnostic fingerprint of a double handler.

Correlation mechanism 4: metrics that point at traces

Two routes connect a metric spike back to concrete requests.

The spec's route is exemplars: by default, a measurement recorded inside a sampled span carries that span's trace_id as an exemplar, so a histogram bucket can link directly to a representative trace.

The pragmatic route, which I trust more day to day, is shared dimensions: put the same bounded attributes, with the same names and values, on both metric tags and span attributes — prediction.model_variant, prediction.aggregated, and so on. Then a spike on some metric slice becomes a trace search filtered by identical dimensions, which becomes one request's spans, which link to that request's logs. Spike to trace to log in two pivots, built entirely out of naming discipline.

One gotcha: temporality

Metric temporality is the classic OTel-to-backend integration trap. Delta temporality reports only each export interval's increment; cumulative reports the running total since process start. Counters and Histograms are usually best exported as delta — that is what lets a backend show per-interval minima and maxima — while an UpDownCounter measures a level, so cumulative matches its semantics:

temporality = {
    Counter: AggregationTemporality.DELTA,
    Histogram: AggregationTemporality.DELTA,
    UpDownCounter: AggregationTemporality.CUMULATIVE,
}

The trap is that OTLP's default is cumulative while some backends prefer delta, and the mismatch produces charts that look plausible and are wrong. Configure it explicitly and write down why.

Self-test

Five questions worth being able to answer without looking up:

What breaks if two services share a Resource but not trace context — what can you still correlate, and what can you not?
Why can a third-party library safely call create_counter even if the application never installs an SDK?
A manually created span shows up inside a trace from hours ago. What is the likely mechanism, and what was missing?
Every log line appears twice in your backend, one copy with fewer attributes. What is your first hypothesis?
Why is delta temporality the usual choice for a Histogram but not for an UpDownCounter?

A Weekend of Kubernetes: FastAPI, Helm, and Breaking Things on Purpose

Xinyang Wu — Mon, 03 Aug 2026 08:36:30 +0000

The setup

Reading about Pods and Services never made Kubernetes stick for me, so I gave it a weekend with a concrete goal: take two FastAPI services — an API service and an auth service — and get them running on a local kind cluster behind a Helm chart shaped like something you would actually run in production.

Two constraints kept the exercise honest. First, everything had to be additive: no service code changes at all, just a Dockerfile, a kind config, and a chart. If the deployment layer needs the application to change, the deployment layer is wrong. Second, the chart had to go beyond a hello-world Deployment — config and secrets handled properly, liveness and readiness probes, autoscaling, and ingress with host-based routing, one hostname per service.

The image is a slim multi-stage build that runs as a non-root user. The kind config maps host ports 80 and 443 into the node, so ingress traffic works from the host with plain curl:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
      - containerPort: 443
        hostPort: 443

Cluster and namespace are both named demo. A small Makefile wraps the lifecycle: make up builds the image, creates the cluster, installs ingress-nginx, and runs helm upgrade --install; make down deletes everything. Being able to rebuild from zero in a few minutes mattered more than I expected — it is what made the breakage experiments later in this post cheap.

Chart anatomy

The chart contains the standard cast: Deployment, Service, ConfigMap, Secret, Ingress, HorizontalPodAutoscaler, ServiceAccount, plus dev and prod values overlays and a helm test pod that curls both services after every install.

The part I like most: one image, two Deployments, templated from a single services: map in the values file.

services:
  api:
    command: ["python", "-m", "myapp.api"]
    port: 8000
    host: api.demo.localhost
  auth:
    command: ["python", "-m", "myapp.auth"]
    port: 8001
    host: auth.demo.localhost

The templates range over this map. Both services run the same image; the command: field overrides the image's default CMD, so one build artifact becomes two different processes on two different ports, each behind its own ingress hostname. Adding a third service is one new values entry and zero template edits — I verified this by adding a dummy whoami service and watching a Deployment, a Service, and an ingress route appear for it without touching a single template. That is the test of a good chart: growth is data, not code.

How a secret becomes config

The single most clarifying exercise of the weekend was tracing one value — the auth signing key — from the values file all the way into the running process:

values.yaml  (secrets: section, overridable per environment)
      |  helm install renders the secret template
      v
Secret manifest    stringData: { app.toml: <the complete TOML> }
      |  applied to the cluster
      v
etcd               data: { app.toml: <base64> }    <- encoded, NOT encrypted
      |  kubelet starts the pod
      v
Pod                /config/app.toml  (tmpfs volume, never written to disk)
                   APP_CONFIG_FILE=/config/app.toml  (env var from the ConfigMap)
      |  application startup
      v
config = {**defaults, **toml.load(os.environ["APP_CONFIG_FILE"])}

Two details in that pipeline are worth pausing on. The first is that base64 is an encoding, not encryption — anyone with etcd access or broad RBAC can decode a Secret. A Kubernetes Secret is a distribution mechanism, not a vault. The second is the last line: the application merges the loaded TOML over its defaults with a shallow dict merge. That one implementation detail dictates chart design. If the Secret carried only a partial overlay — say, just the section containing the signing key — the shallow merge would replace that entire section and silently drop its sibling defaults. So the whole config file lives in the Secret. Slightly inelegant, but correct, and the kind of constraint you only find by reading the app's config-loading code rather than the Helm docs.

Small tricks that carry the chart

Roll pods when config changes. Kubernetes does not restart pods just because a Secret they mount was updated. The standard fix is a checksum annotation on the pod template:

annotations:
  checksum/config: '{{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}'

A config change produces a new hash, the pod template changes, and a rolling update happens for exactly the right reason.

Let the HPA own the replica count. If replicas: is set on the Deployment while an HPA is active, every helm upgrade fights the autoscaler and resets its scaling decision. The chart omits replicas entirely whenever autoscaling is enabled.

Probe a free endpoint. Liveness and readiness point at /openapi.json — FastAPI serves it automatically, it needs no auth, and it touches no downstream dependency. It answers exactly one question, "is this process up and serving HTTP", which is the right question for a liveness probe to ask.

Skip the registry. kind load docker-image copies the locally built image straight into the cluster node, and pullPolicy: IfNotPresent stops the kubelet from trying to pull it. Local iteration needs no registry at all.

Diff before every upgrade. The helm-diff plugin renders what an upgrade would change and shows it before anything is applied. Reading that diff is the habit that separates "it worked" from "I know what it did".

Breaking things on purpose

The most valuable hours of the weekend were spent breaking a working deployment in controlled ways, because each breakage pairs a symptom with the one command that diagnoses it.

Wrong image tag. Pods sit in ImagePullBackOff.

kubectl -n demo describe pod <pod>    # Events show the pull error and the exact image name

Corrupt the TOML inside the Secret. The container starts, Python throws during config load, the process exits, the kubelet restarts it: CrashLoopBackOff.

kubectl -n demo logs <pod> --previous    # the traceback from the crashed attempt

The --previous flag matters — the current container may be mid-restart with an empty log.

Selector that matches nothing. Edit the Service's selector so it no longer matches the pod labels — or reinstall the chart with a mismatched pod-template label — and the Service silently loses its endpoints. Everything looks green — pods Running, Service present — yet the ingress returns 503.

kubectl -n demo get endpoints    # an empty ENDPOINTS column means selector mismatch

The obvious version of this experiment — relabeling a live pod — does not stay broken, and why it heals is its own lesson. Each Deployment's ReplicaSet selects pods by the same per-service labels the Service uses, so the moment a pod's label changes, the ReplicaSet disowns it and spins up a replacement that matches again. Endpoints repopulate within seconds and the ingress recovers. That quarantine behavior is a classic debugging move in its own right: strip a label from a misbehaving pod and it drops out of the Service, replaced by a fresh pod, while the original stays alive for inspection.

Scale the ingress controller to zero. curl gets connection refused, while kubectl get ingress still shows a perfectly healthy-looking object. The lesson generalizes: Kubernetes objects describe intent, not the current state of the data path. The Ingress resource is inert configuration; the controller pod does the actual work.

The common thread: symptoms surface at different layers, and debugging speed is mostly about knowing which layer emits which symptom before you start typing.

What I'd add next

The chart is deliberately local-only, and each gap points at a well-known production tool:

TLS via cert-manager with a self-signed cluster issuer.
External Secrets Operator, so real secrets never live in a values file.
Observability: kube-prometheus-stack plus an OpenTelemetry Collector, and one Grafana dashboard for the two services — see OpenTelemetry from first principles for why I want metrics, traces, and logs wired through one pipeline.
CI that builds the image per commit and pushes the packaged chart to an OCI registry.
NetworkPolicies restricting which pods may talk to the auth service, and a PodDisruptionBudget.

Each of these is roughly a weekend of its own, which feels like the right unit for this kind of learning.

Self-test

Five questions I now expect myself to answer cold. If any of them feels fuzzy to you, that section of the docs is worth a re-read.

Why is there a ReplicaSet between a Deployment and its Pods? The Deployment owns the rollout strategy; each ReplicaSet just guarantees the replica count for one revision of the pod template.
A Service has no endpoints — what three things do you check? The selector matches the pod labels; the pods pass their readiness probes; the Service port maps to the container port.
What triggers a rolling update? Any change to the pod template — image tag, env vars, resources, or the config checksum annotation.
The HPA shows <unknown> for CPU — why? Either metrics-server is missing (on kind it also needs --kubelet-insecure-tls), or the container declares no CPU request to compute a percentage against.
Are Kubernetes Secrets encrypted? No — base64-encoded in etcd. Real deployments layer on etcd encryption at rest, tight RBAC, and an external secret store.

An Agent Is a Loop: a Working Mental Model for Agentic Systems

Xinyang Wu — Mon, 03 Aug 2026 08:35:54 +0000

The one-sentence definition

Strip away the vendor decks and an agent is exactly this: a language model placed inside a loop that can call tools, remember things, and hand control back to a human when it gets stuck. Everything else — orchestration frameworks, memory stores, multi-agent topologies — is engineering around that loop.

The loop has a name: ReAct (Reasoning + Acting). Reason about the current state, take one action, observe the result, reason again, until the goal is met. In pseudocode it is embarrassingly small:

def run_agent(task: str) -> str:
    messages = [SYSTEM_PROMPT, user_message(task)]
    while True:
        reply = llm(messages, tools=TOOL_SCHEMAS)
        if not reply.tool_calls:
            return reply.text                       # done: final answer
        messages.append(reply)                      # keep the model's own turn
        for call in reply.tool_calls:
            result = execute(call)                  # code runs the tool
            messages.append(tool_result(call.id, result))

Every concept below is an answer to an engineering problem this loop creates.

Five layers

When I sketch an agentic system, I draw five layers:

Governance — how much autonomy the agent gets, and when it must escalate to a human.
The orchestration loop — reason, act, observe, repeat until done.
Capabilities — tool calling, retrieval and grounding, integration with real workflows.
State — short-term context plus long-term memory.
The factory — how agents are created and configured in the first place.

The loop sits in the middle. Governance constrains it from above; capabilities and state feed it from below; the factory stamps out configured instances of it. The layers are worth keeping separate because they fail differently: a grounding bug produces confident nonsense, while a governance bug produces an agent that deletes something it should have asked about first.

Who owns the control flow

The most consequential design decision is who decides the next step.

Model-driven control flow lets the LLM pick the next tool call. It is flexible and handles situations you never anticipated — and it is also unpredictable, hard to reproduce, and capable of wandering off or looping forever.

Deterministic workflows hard-code the control flow and use the model only inside individual nodes. They are reproducible, testable, and cheap.

Between the extremes sit a few named patterns: prompt chaining (a fixed pipeline of LLM steps), routing (classify the input first, then dispatch), orchestrator-worker (one coordinator decomposes a task and farms pieces out), and evaluator-optimizer (generate, critique, revise).

The real question is never "should we build an agent?" It is which decisions go to the model and which stay in code. Default to a workflow; hand a decision to the model only when the next step genuinely cannot be known in advance.

Tool calling and the security boundary

Tool calling is how the loop touches the world, and its mechanics matter:

The model receives a list of tools, each with a name, a description, and a JSON schema for its parameters.
When the model wants a tool, it emits a structured call — a name plus arguments. It does not execute anything.
The harness — your program — validates and executes the call, then appends the result to the conversation.
The model reads the result and continues.

Step 3 is the security boundary. The model requests; code decides. Every permission check, rate limit, and audit log lives in the harness, which is precisely why the model must never execute anything directly.

MCP (Model Context Protocol) standardizes the plumbing — a common way to plug tools and data sources into any model, a USB port for tools instead of a custom connector per integration.

Two failure modes recur. Vague tool descriptions produce erratic tool choice: the description is the interface, so write it like documentation. And too many tools degrade selection quality — past a couple dozen, retrieve over the tool catalog and present only the relevant few.

Grounding

A model's knowledge is frozen at training time, and it fills gaps confidently and wrongly. Grounding means binding answers to verifiable sources, and the mainstream implementation is RAG: embed the question, find the most similar chunks in a vector store, put them in the prompt, and answer from them.

Each stage has its own failure point. Chunks too large drown the signal in noise; too small, they sever meaning. An embedding model mismatched to the domain retrieves plausible-looking irrelevance. Pure vector search misses exact keywords, which is why serious pipelines run hybrid search (BM25 plus vectors) and add a reranker to reorder candidates. Citations on the final answer keep it auditable.

The slogan worth remembering: retrieval is the means; grounding is the goal. I go deeper on the pipeline and its evaluation in RAG beyond the demo.

State

An agent has two memories with very different physics.

Short-term memory is the context window — finite and expensive. When it fills up, the options are: compaction (summarize older turns), a sliding window (drop the oldest), offloading (write intermediate results to an external store), and retrieval (pull back only the relevant fragments on demand).

Long-term memory is anything that survives the session: plain files and structured records (deterministic, human-inspectable) or vector stores (fetched back by semantic similarity). Two more terms earn their keep: checkpointing, so a crashed run restarts from the last saved state instead of from zero, and the episodic versus semantic split — a log of what happened versus distilled knowledge extracted from it.

One agent or many

Multi-agent architectures buy exactly three things:

Context isolation — the big one. A sub-agent starts with a fresh context window, so the coordinator is not bloated by every detail of every subtask.
Parallelism — independent subtasks run concurrently.
Specialization — each agent gets a narrow role prompt and a small tool set.

The costs are real: coordination overhead, errors that amplify as they propagate between agents, higher latency, multiplied token spend, and much harder debugging.

Multi-agent systems are not smarter; they are better at isolating context and running in parallel. If a single agent with well-chosen tools can do the job, use the single agent.

Autonomy and escalation

Autonomy is a spectrum, not a switch:

Level	Meaning
Suggest-only	The system proposes; a human performs every action
Human-in-the-loop	The agent proposes actions; each one needs approval first
Human-on-the-loop	The agent acts on its own; a human monitors and can stop it
Full autonomy	No human in the process

My rule of thumb: how strictly an action is gated should scale with its irreversibility times its blast radius. Reads and queries can run fully automatically. Writing a local file is fine to review after the fact. Sending email, moving money, deleting data, publishing anything outward — a human approves first, every time.

Escalation is the safety valve that makes autonomy tolerable. The agent stops and hands control back when it is uncertain, blocked, missing permissions, or facing a high-risk irreversible action. Mechanisms include confidence thresholds, an explicit ask-a-human tool, and review queues; in multi-agent setups, a stuck worker escalates to its orchestrator. The framing I hold onto: escalation is not failure — it is the responsible default. Better to ask one question too many than to act irreversibly while unsure.

Agent factories

Once one agent works, you want twenty, and the factory pattern applies directly. An agent is fully described by its configuration: a role prompt, a tool set, a model choice, and a permission set. Define those as data — declaratively — and a factory spawns instances from templates on demand.

The payoff is the same as everywhere the pattern shows up: consistency, versioned definitions, and cheap extension. Adding a new agent means adding configuration, not writing code.

The whole model in one paragraph

A factory turns declarative configuration into agents. Each agent runs a ReAct orchestration loop; inside the loop it acts through tool calls that the harness executes, learns what it never knew through retrieval, and remembers through short- and long-term state. Whether you run one loop or several depends on whether you need context isolation and parallelism. How much the loop may do without asking is set by its autonomy level, and when it exceeds that level — or simply gets stuck — an escalation path hands control back to a human. Workflow integration then wires the whole thing into real systems: triggers start it, hooks instrument it, and approval checkpoints keep the irreversible steps honest.

Your SSH Key Isn’t Always the Problem: A Layer-by-Layer Debugging Guide

Xinyang Wu — Sun, 19 Jul 2026 06:55:41 +0000

I once ran:

ssh demo-server

and immediately suspected my SSH key.

The actual error was:

ssh: Could not resolve hostname demo-server: Name or service not known

My key was innocent. SSH had not opened a TCP connection, exchanged a single protocol message, checked the server's identity, or attempted user authentication. It did not even know which IP address to contact.

That failure taught me a more useful way to debug SSH: treat every error as a timestamp. It tells you how far the connection progressed before it stopped.

Instead of changing keys, restarting services, and deleting known_hosts entries at random, ask one question:

Which layer did I fail to reach?

Here is the model I now use.

The connection has seven checkpoints

When you type ssh demo-server, the useful mental model is:

configuration
    → name resolution
    → TCP connection
    → SSH transport and key exchange
    → server verification
    → user authentication
    → channel creation

Each stage depends on the previous one. A failure in name resolution cannot be fixed by rotating an authentication key, because the authentication stage never happened.

These are diagnostic checkpoints, not seven separate wire-protocol layers. In the SSH protocol, server host-key authentication is part of transport key exchange; I separate verification here because it produces its own recognizable failures and fixes.

Stage 0: SSH resolves its configuration

Before contacting the network, the client combines command-line options with SSH configuration, usually from ~/.ssh/config and the system configuration.

Imagine this entry:

Host lab-*
    User dev

Running ssh lab-gpu matches the pattern and sets the remote user to dev. It does not give lab-gpu an IP address. Without a HostName, the destination is still literally lab-gpu, so the operating system must resolve that name later.

A complete alias might look like this:

Host demo-server
    HostName 192.0.2.10
    User dev
    Port 22
    IdentityFile ~/.ssh/id_ed25519

The fastest way to inspect what SSH actually derived is:

ssh -G demo-server

For a compact view:

ssh -G demo-server | grep -E '^(hostname|user|port|identityfile) '

This is safer than reasoning from one config block by eye. Multiple matching blocks may contribute settings, and for most options SSH uses the first value it obtains. “The most specific block wins” is therefore a misleading model.

If the resolved hostname, user, port, or identity file is wrong, stay at this stage. The network is not your problem yet.

Stage 1: The operating system resolves the hostname

SSH next needs to turn the resolved hostname into an address. Depending on the machine, that may involve:

DNS
/etc/hosts
a VPN or overlay network's DNS
other system name-service mechanisms

One subtle trap is confusing a deployment tool's inventory with the operating system's resolver. An inventory might contain:

lab-gpu ansible_host=192.0.2.10

Ansible understands that mapping. OpenSSH does not automatically read Ansible's inventory. Unless lab-gpu is also defined through SSH configuration or a system resolver, ssh lab-gpu can still fail with:

Could not resolve hostname

That message is unusually precise: configuration completed, but the client could not obtain an address. No packet reached an SSH server.

First use ssh -G to find the effective hostname; resolver tools do not read ~/.ssh/config aliases. If that effective value is a name such as server.example.com, check the Linux system name-service path with:

getent hosts server.example.com

On macOS, the equivalent system lookup can be inspected with:

dscacheutil -q host -a name server.example.com

To query DNS specifically rather than the full system resolver path, use:

dig +short server.example.com

Also check whether the name only exists while a company VPN, mesh VPN, or private DNS service is active.

Stage 2: The client opens a TCP connection

Once the client has an address, it tries to connect to the configured port—normally port 22.

At this point, similar-looking messages describe different conditions:

Connection timed out: packets may be dropped by a firewall, the route may be broken, the VPN may be missing, or the host may be unavailable.
No route to host: the local networking stack cannot find a usable path, or an intermediate device reports that the destination is unreachable.
Connection refused: the client received an active rejection, usually because the destination has nothing listening on that port. An intermediary can also reject it.

A small TCP probe against the effective address and port from ssh -G helps isolate this layer:

nc -vz 192.0.2.10 22

This does not prove that SSH authentication will work. It only answers a narrower and very useful question: can I establish direct TCP connectivity to this host and port? If SSH uses ProxyJump or ProxyCommand, a probe from your machine does not test the same path.

Stage 3: SSH negotiates a secure transport

After TCP succeeds, the client and server exchange SSH protocol versions and negotiate algorithms. They perform a key exchange to derive fresh session keys, and the server proves control of its host key by signing data from the exchange.

A successful exchange creates the encrypted, integrity-protected transport used by the rest of the session. Modern cipher suites often provide authenticated encryption directly, so it is better to think in terms of confidentiality and integrity than to assume every connection uses a separate MAC algorithm.

At the wire-protocol level, the host-key proof belongs to this exchange. The next checkpoint is the client's decision about whether to trust that key.

Failures here look different from network failures. Examples include protocol banners that never arrive, an early connection reset, or messages such as “no matching key exchange method found.” These often point to incompatible algorithm policies, a non-SSH service on the port, an intermediary, or a server-side SSH problem.

Stage 4: The client verifies the server

Encryption is not enough if you encrypt a session to the wrong machine. The client therefore checks the server's host key against ~/.ssh/known_hosts or another configured host-key database.

On a first connection, many setups use trust on first use: you verify and accept the fingerprint, and the client remembers it. On later connections, the key should match.

If you see:

REMOTE HOST IDENTIFICATION HAS CHANGED!

the connection has already passed name resolution, TCP, and enough of the SSH handshake to receive a host key. The warning could mean a legitimate rebuild or address reassignment—but it could also indicate misrouting or an active attack.

Do not make “delete the known_hosts entry” your automatic response. First verify the new fingerprint through a trusted, independent channel. Remove or replace the old entry only after you understand why it changed.

Stage 5: The server authenticates the user

Only now does your personal SSH key enter the story.

With public-key authentication:

the private key remains on the client;
the corresponding public key is normally listed in the remote account's authorized_keys;
the client signs session-bound authentication data to prove that it controls the private key;
the server verifies that signature with the public key.

The private key is not uploaded to the server.

Two similarly named files have completely different jobs:

File	Usually lives on	Purpose
`known_hosts`	client	Verifies the server's identity
`authorized_keys`	server	Lists public keys allowed to authenticate as a user

This distinction explains why the following error is actually evidence of partial success:

Permission denied (publickey,password)

The client resolved a destination, connected over TCP, negotiated SSH, and accepted the server identity. It then failed to authenticate the requested user.

Now—and only now—it makes sense to inspect the resolved username, offered identities, file permissions, account policy, SSH agent, and the remote authorized_keys entry.

Stage 6: SSH opens a channel

A successful authentication does not necessarily mean you will receive an interactive shell. SSH can open different channels for:

an interactive shell
a single remote command
SFTP
local, remote, or dynamic port forwarding

For example:

ssh -N -o ExitOnForwardFailure=yes \
  -L 127.0.0.1:8080:127.0.0.1:3000 demo-server

asks the client to listen only on the local loopback interface at port 8080, without running a remote command. A connection to that port travels through the encrypted SSH session, then the remote SSH server connects to 127.0.0.1:3000 from its own network context. ExitOnForwardFailure catches failure to establish the requested listener, although it cannot guarantee that the target service will accept a later forwarded connection.

If authentication succeeds but forwarding reports “administratively prohibited” or “open failed,” the problem is at the channel or destination-policy layer—not with DNS or your key.

Read the error as a stage marker

Here is the compact version I wish I had earlier:

Error or symptom	Last relevant stage	Check next
`Could not resolve hostname`	name resolution	effective `HostName`, DNS, hosts file, VPN
`No route to host`	network path	routes, VPN, interface, gateway
`Connection timed out`	TCP path	reachability, firewall, correct address and port
`Connection refused`	TCP destination	SSH service and listening port
no matching key-exchange algorithm	SSH transport	client/server algorithm policies
host identification changed	server verification	independently verify the new fingerprint
`Permission denied (publickey)`	user authentication	resolved user, offered key, `authorized_keys`, account policy
forwarding is prohibited or open fails	channel creation	SSH server policy and target reachability

The table is not a replacement for logs, but it prevents category errors. It tells you where to begin.

My three-step first response

When an SSH connection fails, I now start with these steps.

First, inspect the effective configuration:

ssh -G demo-server

Second, on a Unix-like server, get a stage-by-stage trace without password prompts and make a successful probe exit immediately:

ssh -vvv -T -o BatchMode=yes -o ConnectTimeout=5 demo-server true

Verbose output usually makes the boundary visible: configuration applied, address resolved, connection attempted, host key checked, identities offered, or channel rejected.

Be careful before sharing that output. Sanitize hostnames, usernames, addresses, paths, key fingerprints, and other infrastructure details.

Third, test the suspected boundary independently. Resolve the effective hostname if resolution failed; probe the effective address and port if they exist; inspect offered keys only if authentication was actually reached. Remember that resolver and TCP tools do not apply your SSH alias or proxy configuration for you.

Debug the earliest failure, not the loudest theory

SSH combines configuration, naming, networking, cryptography, identity, and multiplexed channels behind one short command. That makes it feel mysterious—but it also makes its errors surprisingly informative.

The next time ssh fails, read the error as a timestamp, identify the earliest incomplete checkpoint, and test that boundary directly. A DNS problem does not need a new key. A refused TCP connection does not need a cleared known_hosts. And an authentication error is proof that several lower layers already worked.

That one mental model turns SSH debugging from guesswork into a sequence of small, testable questions.

10 Cross-Topic Lessons from a 123/138 LeetCode Sprint

Xinyang Wu — Sat, 06 Jun 2026 10:02:16 +0000

I first published this retrospective 46 problems into a 12-week DSA sprint. My tracker is now at 123 / 138 slots: I have worked through Week 11 and am 2 / 12 into Week 12. The tracker deliberately repeats a few problems across topics, so 123 is a progress metric—not a claim of 123 unique accepted problems.

The expanded tracker did not produce dozens of unrelated tricks. It made the same failures show up in new disguises. A bad state definition in dynamic programming felt a lot like an under-specified sliding window. Marking a BFS node too late felt like inserting into a hash map too early. A wrong heap invariant looked suspiciously like a wrong binary-search invariant.

These are the ten lessons that survived that repetition. The bugs below are not hypothetical warnings. They are mistakes I wrote, plus the smallest counterexamples that finally made the mistake obvious.

1. If I cannot define the state in one sentence, I am not ready to update it

My first House Robber state was internally inconsistent:

dp[1] = nums[1]
dp[i] = max(nums[i] + dp[i - 2], dp[i - 1])

The initialization says dp[1] means “money from robbing house 1.” The recurrence needs it to mean “the best result using houses 0 through 1.” Those are different contracts.

The counterexample is [2, 1, 1, 2]. Initializing dp[1] = 1 permanently loses the valid choice of taking the first house. The recurrence returns 3 instead of 4. I added a separate vmax to patch the output, but the wrong state had already contaminated later states. A downstream maximum could not repair an upstream definition.

The correct base case has the same meaning as the recurrence:

prev2 = nums[0]
prev1 = max(nums[0], nums[1])

for x in nums[2:]:
    prev2, prev1 = prev1, max(prev1, prev2 + x)

The same lesson appeared outside DP:

In Fruit Into Baskets, I stored only the kinds of fruit in a deque. That could not answer “when has this kind completely left the window?” The input [1,1,1,2,3,3] exposed it: my window kept three kinds and reported 5 instead of 4. The state needed counts, not just names.
In Maximum Product Subarray, keeping only the largest product ending here loses a negative value that may become the next maximum. For [2,-5,-2,-4,3], the state must retain both the current maximum and minimum to recover the answer 24.
In Longest Palindromic Substring, one left endpoint per right endpoint is not enough. Palindromicity depends on both boundaries, so the state naturally becomes dp[left][right].

My current rule is simple: finish the sentence “this state contains exactly…” before writing an update. If I need a patch variable later, I first suspect that sentence.

2. The empty state is data, not a special case

Prefix sums taught me to put prefix[0] = 0 in front. Later topics made the idea more general: the empty state should be represented by the identity that cooperates with the operation.

Operation	Empty-state value	Why it works
Sum	`0`	`x + 0 = x`
Product	`1`	`x * 1 = x`
Minimum	`inf`	a real candidate always beats it
Maximum	`-inf`	a real candidate always beats it
Counting-DP empty choice	`1`	there is exactly one way to choose nothing
Unreachable count	`0`	no valid construction has reached this state
Prefix-frequency count	`{0: 1}`	one empty prefix exists
Earliest prefix index	`{0: -1}`	the empty prefix ends before index 0

One of my earliest versions used an inclusive prefix array, appended a zero at the end, and relied on Python's negative indexing:

return prefix[right] - prefix[left - 1]  # left == 0 reads prefix[-1]

It passed, but only because prefix[-1] wrapped around to the hidden zero. The same code is an out-of-bounds bug in languages without negative indexing. The honest form is:

prefix = [0]
for x in nums:
    prefix.append(prefix[-1] + x)

return prefix[right + 1] - prefix[left]

Coin Change exposed the other half of the rule. I used -1 for “unreachable” while minimizing. Then unreachable states won every min, and -1 + 1 created fake zero-cost solutions. With coins = [2,3] and amount = 4, that version returned 0 instead of 2.

Internally, inf is the right value because it composes safely:

dp = [float("inf")] * (amount + 1)
dp[0] = 0

for a in range(1, amount + 1):
    for coin in coins:
        if coin <= a:
            dp[a] = min(dp[a], dp[a - coin] + 1)

return -1 if dp[amount] == float("inf") else dp[amount]

The external API may require -1. That does not mean -1 belongs inside the recurrence. I now separate the algorithm's internal algebra from the return-value protocol, and translate only at the exit.

3. Update order is part of the algorithm

Two adjacent lines are not interchangeable just because both eventually run.

In prefix-sum counting, the map is a record of the past. I must query it before the current prefix joins that past:

prefix += x
answer += seen.get(prefix - k, 0)       # query history
seen[prefix] = seen.get(prefix, 0) + 1  # then become history

If I insert first, the current prefix can match itself. For Subarray Sum Equals K with nums = [1,-1] and k = 0, the correct answer is 1; inserting first counts two zero-length “subarrays” as well and returns 3.

The same ordering constraint reappeared in three different tools:

BFS: mark a node when it enters the queue, not when it leaves. Otherwise several parents can enqueue the same node before its first dequeue.
Union-Find cycle detection: ask whether find(u) == find(v) before union. If I union first, the condition is true for every edge. On [[1,2],[2,3],[3,1],[3,4]], the redundant edge is [3,1], not the first or last edge by accident.
Compressed 0/1 knapsack: iterate capacity backward. With one item 2 and target 4, a forward scan sets dp[2] and then reuses that freshly written value to set dp[4]—using the same item twice.

These are all temporal invariants. “Seen” means seen before now. “Already connected” means connected before this edge. “Previous row” means the state before this item. Moving a line changes that meaning, even when the variables have the same names.

4. Every destructive move needs a proof

Popping a stack entry, advancing a pointer, pruning a branch, and moving a binary-search boundary all destroy candidates. Before doing any of them, I now ask:

What fact proves this candidate can never be the answer?

My rotated-array minimum passed tests with two overlapping safety nets: an early “peek at the previous element” return, plus a final fallback. The loop itself used right = mid - 1 even when mid could still be the minimum. Delete either safety net and the code quietly breaks.

The clean invariant keeps mid whenever it may still be the answer:

left, right = 0, len(nums) - 1
while left < right:
    mid = (left + right) // 2
    if nums[mid] > nums[right]:
        left = mid + 1   # mid is provably not the minimum
    else:
        right = mid      # mid may be the minimum, so keep it
return nums[left]

Container With Most Water gave me the same lesson in greedy form. I tried moving both ends when neither immediate next move improved the current area. On [5,1,100,50], that jumps past the optimal pair (100,50) with area 50.

The valid greedy move is not “take the next thing that looks better.” It is “discard the shorter wall because every narrower container that keeps it is capped by the same short wall.” That is a domination proof.

Longest Increasing Subsequence exposed the distinction between pop and replace. Treating its tails array like a monotonic stack and popping larger endings destroys the historical fact that a subsequence of that length has existed. For [2,3,1,4], popping gives length 2 instead of 3. The correct operation replaces exactly one threshold—the first value greater than or equal to the new number—without erasing longer records.

The syntax is tiny; the proof is the algorithm.

5. Constraints are algorithm instructions

I used to read constraints after understanding the problem. Now I read them as a list of approaches the problem setter is trying to kill.

“O(log n) required” killed my “binary search, then expand left and right” solution for finding a target range. On [8,8,8,8,8], the expansion is linear. The answer needs two boundary searches.
“The array may contain negatives” kills a sum-based sliding window because the window sum is no longer monotone. Prefix sum plus a hash map survives.
“O(n) required” in Longest Consecutive Sequence kills sorted(set(nums)). Sorting gets the right result in O(n log n) but misses the point; scanning only from values whose predecessor is absent gives linear total work.
Coordinates up to 10^9 or 2^31 - 1 kill per-coordinate arrays and loops. I tried a difference array for Interval List Intersections and a coordinate-by-coordinate loop for Skyline. One wants impossible memory; the other can run for billions of empty positions. The right iteration domain is intervals or event points, not the numeric axis.

This question has become part of my pre-code checklist:

Which obvious solution is this constraint designed to exclude?

It catches complexity bugs before a correct-looking implementation makes them harder to notice.

6. When many searches share a destination, reverse the search

Several graph problems became simple only after I stopped searching from every unknown point.

Surrounded Regions is awkward as “which regions are enclosed?” It becomes easy as “which O cells can the boundary reach?” Mark those safe cells, then flip the rest.
Pacific Atlantic Water Flow is expensive as “can this cell flow to each ocean?” Reverse the edges: start from both oceans and climb to cells of equal or greater height. The answer is the intersection of the two reachable sets.
01 Matrix asks every cell for its nearest zero. Put all zeros into one queue at distance 0 and expand once.
Rotting Oranges asks how simultaneous infection spreads. Put all rotten oranges into the initial queue; one BFS layer is one minute.

The shared template is small:

queue = deque(all_states_with_known_answer)
mark_all_as_seen(queue)

while queue:
    state = queue.popleft()
    for nxt in reverse_or_outward_neighbors(state):
        if nxt not in seen:
            answer[nxt] = answer[state] + 1
            seen.add(nxt)
            queue.append(nxt)

Running one BFS per source repeats the same regions. A multi-source BFS is not merely an optimization; it models simultaneous expansion correctly.

I now look for phrases such as “nearest source,” “eventually reaches a boundary,” or “all sources spread at once.” They often mean: start from the states whose answers are already known and propagate outward.

7. The lifetime of `visited` depends on the question

I once treated “mark visited” as a generic graph rule. Backtracking showed why that is incomplete.

For flood fill, the question is about nodes: “which cells belong to this component?” Once a cell is processed, visiting it again has no value. The mark is permanent.

For Word Search or permutations, the question is about paths: “which choices form this particular solution?” A cell or item may be used by a different sibling path. The mark must live only for the current recursive frame:

path.append(choice)
used[i] = True

backtrack()

used[i] = False
path.pop()

Leaving out the restoration does not just leak state; it permanently blocks legal sibling branches and creates false negatives.

I hit the companion bug in Subsets:

answers.append(path)     # stores the same mutable list object

After recursion unwound, every entry referred to the same now-empty list. The fix is a snapshot:

answers.append(path[:])

This also explains why “generate everything, then deduplicate” is usually a smell. My first subset approach generated both [1,2] and [2,1] and tried to filter afterward. A start index makes the invalid ordering impossible to generate. For duplicate values, sorting plus same-level pruning gives each result one canonical path.

The rule I keep now is:

Enumerating nodes: mark permanently.
Enumerating paths: mark on entry, restore on exit.
Saving a mutable path: copy at the moment it becomes a result.

8. A recursive function can return one value and build another answer

Tree diameter initially tempted me into two recursive functions: compute a node's height, then recursively compute diameters and call height again at every node. The logic is correct; on a skewed tree the repeated height work makes it O(n^2).

One postorder traversal can produce two different quantities:

best = 0

def height(node):
    nonlocal best
    if not node:
        return 0

    left = height(node.left)
    right = height(node.right)

    best = max(best, left + right)  # answer using both branches
    return 1 + max(left, right)     # value one parent can extend

The distinction is structural. A path that continues to the parent can use only one child branch. A path whose highest point is the current node can join both.

Maximum Path Sum uses the same skeleton:

return the best one-sided gain to the parent;
update a global best with left + node + right;
clamp negative child gains to zero.

Its smallest counterexample is also its most important initialization test: a one-node tree [-3]. Initializing the global answer to 0 returns a path that does not exist. It must start from a real node value or -inf.

Minimum Depth delivered a related warning about base cases. Replacing max with min in the maximum-depth recurrence fails on [1,None,2]: the nonexistent left branch contributes 0 and wins, producing depth 1 instead of 2. An identity that is harmless under one aggregation can poison another.

I now write down two contracts for recursive tree problems: what this call returns upward, and what candidate answer this node contributes globally.

9. Graph modeling happens before DFS, BFS, or Union-Find

The hardest part of several graph problems was choosing what a node meant.

In Bus Routes, counting station-to-station edges answers the wrong question. The cost is buses boarded, so a BFS layer must represent one additional route. The useful index is stop -> routes containing that stop, with separate visited sets for stops and routes.

In Accounts Merge, my account-to-account model was correct but expensive: compare every pair of accounts for shared email, then union matching accounts. That is O(n^2) before considering set intersections.

The cleaner model makes emails the nodes. Union all emails within each account. If an email appears in two accounts, it is literally the shared node that joins the components. The pairwise comparison disappears.

The same modeling question showed up elsewhere:

Word Ladder nodes are words; edges are one-character changes.
Open the Lock nodes are four-digit strings; edges are one wheel turn.
Clone Graph is primarily an old_node -> new_node mapping problem; DFS versus BFS is secondary.
In grid Union-Find, (row, col) becomes row * width + col, but boundary checks must still happen in two dimensions. Checking only the flattened id lets the right edge wrap into the next row.

Before choosing a traversal, I now ask three questions:

What exactly is a node?
What operation creates an edge?
What does one unit of cost or one BFS layer mean?

Getting those right often makes the algorithm routine. Getting them wrong can make a correct traversal solve a different problem.

10. Keep the unresolved frontier, not the whole history

My best heap improvements came from asking what must be eligible right now.

For Merge K Sorted Lists, I first pushed every node and used the list index as a tuple tie-breaker. One list containing [1,1,2] breaks it: equal values from the same list also share the same list index, so Python eventually tries to compare two ListNode objects and raises TypeError.

The better invariant is:

The heap contains at most one node from each list: that list's smallest unresolved node.

Pop one node, then push its successor. Now the list index is unique within the heap, the heap size stays at most K, and the complexity becomes O(N log K) instead of sorting all N nodes.

IPO exposed the same frontier bug in a different form. I rescanned every project on every round and reinserted affordable projects, so the same project could be completed twice. With k = 2, w = 0, profits [1,2], and capital [0,3], my version earned the first project's profit twice and returned 2; the correct answer is 1.

Sorting projects by capital and advancing a pointer fixes the lifecycle:

each newly affordable project enters the profit heap once;
unchosen affordable projects stay eligible in the heap;
the chosen project leaves once;
the pointer never moves backward because capital never decreases.

The same skeleton later powered offline interval queries and Skyline: sort by the condition that unlocks a candidate, advance one-way, keep active candidates in a heap, and lazily remove candidates only when they can affect the top.

A heap is not a bag of everything I have seen. A monotonic stack is not a bag of previous indices. They are compressed representations of the unresolved frontier. If I cannot state exactly why every stored item is still eligible, I probably do not yet have the invariant.

The meta-lesson

At 46 problems, I thought my bugs lived in edge cases around otherwise-correct algorithms. At 123 tracker slots, I think that distinction is mostly false. The “edge” details are the algorithm:

What does the state mean?
How is the empty state represented?
In what order do reads, writes, marks, and moves happen?
What proof allows a candidate to be discarded?
Which approach does a constraint rule out?
Can the search run backward from known answers?
How long should a visited mark live?
What returns to the parent, and what updates the global answer?
What are the graph's actual nodes and edges?
What exactly belongs in the unresolved frontier?

The most useful part of my notes is still the same as it was in the first version: every problem gets a “bug I actually wrote” and a counterexample. The fix tells me how to pass one test. The counterexample tells me which assumption was false—and that is the part that transfers to the next topic.

What mistake has followed you across the largest number of seemingly unrelated problems?

DEV Community: Xinyang Wu

A Working Map of Computer Networking

Layering is the whole trick

TCP vs UDP

Opening and closing a connection

From URL to pixels

HTTP essentials

DNS resolution

Flow control vs congestion control

Addresses and NAT

Where to go deeper

RAG Beyond the Demo: Pipeline, Citations, Evaluation, and When Not to Bother

What RAG buys you

Two pipelines, not one

Embeddings without the magic

When brute force is the right call

Two-stage retrieval and hybrid search

Citations: soft vs hard

Evaluate the two halves separately

Prompt caching changes the calculus

Self-test

Refactoring OpenTelemetry Metrics: from God Class to Per-Subsystem Modules

The constraint that shapes everything

The anti-pattern: one god class of mixins

The replacement pattern

Metric names are a public API

Cardinality is the bill

Push vs pull instruments

Self-test

OpenTelemetry from First Principles: Three Signals, One Identity

Why OpenTelemetry exists

Three signals, three questions

The vocabulary that makes docs readable

The API/SDK split

Correlation mechanism 1: shared Resource

Correlation mechanism 2: trace context

Correlation mechanism 3: logs that know their trace

Correlation mechanism 4: metrics that point at traces

One gotcha: temporality

Self-test

A Weekend of Kubernetes: FastAPI, Helm, and Breaking Things on Purpose

The setup

Chart anatomy

How a secret becomes config

Small tricks that carry the chart

Breaking things on purpose

What I'd add next

Self-test

An Agent Is a Loop: a Working Mental Model for Agentic Systems

The one-sentence definition

Five layers

Who owns the control flow

Tool calling and the security boundary

Grounding

State

One agent or many

Autonomy and escalation

Agent factories

The whole model in one paragraph

Your SSH Key Isn’t Always the Problem: A Layer-by-Layer Debugging Guide

The connection has seven checkpoints

Stage 0: SSH resolves its configuration

Stage 1: The operating system resolves the hostname

Stage 2: The client opens a TCP connection

Stage 3: SSH negotiates a secure transport

Stage 4: The client verifies the server

Stage 5: The server authenticates the user

Stage 6: SSH opens a channel

Read the error as a stage marker

My three-step first response

Debug the earliest failure, not the loudest theory

10 Cross-Topic Lessons from a 123/138 LeetCode Sprint

1. If I cannot define the state in one sentence, I am not ready to update it

2. The empty state is data, not a special case

3. Update order is part of the algorithm

4. Every destructive move needs a proof

5. Constraints are algorithm instructions

6. When many searches share a destination, reverse the search

7. The lifetime of visited depends on the question

8. A recursive function can return one value and build another answer

7. The lifetime of `visited` depends on the question