DEV Community: Jordan

Why Setting Up Observability Takes Forever (And What To Do About It)

Jordan — Tue, 16 Jun 2026 15:12:07 +0000

Everyone talks about observability like it is a solved problem. Pick a stack, wire it up, done. In practice, every team I have seen attempt this discovers the same thing: observability becomes its own project. A week becomes a month. The stack itself needs babysitting. And at some point you realise half the job is keeping the monitoring alive rather than actually using it.

This is not a tooling problem. It is a platform problem. And until you treat it that way, you will keep having the same fights.

The Real Reasons It Takes So Long

Cardinality kills you before you notice:

Prometheus is the default metrics choice for good reason. It is powerful, well supported, and the ecosystem around it is mature. It is also extremely sensitive to cardinality, and most teams find this out the hard way.

One bad label choice, one team emitting high cardinality identifiers like request IDs or user IDs as metric labels, and you are tuning memory instead of building features. The problem compounds because cardinality issues are invisible until they are not. Everything looks fine until Prometheus OOMs at 2am.

The fix is not a Prometheus configuration change. It is treating high cardinality labels as a platform policy problem. If teams can emit arbitrary labels without guardrails, you will be fighting this forever. Recording rules and metric aggregation at the collector layer before data hits Prometheus buys significant headroom. But this requires a platform decision, not a per-team one.

Log volume costs spiral without intervention

The instinct is to keep everything. The reality is that keeping everything in hot storage is expensive, and teams only discover how expensive when the bill arrives.

The pattern that works is tiered retention with aggressive filtering at the collector level before anything hits storage. Drop debug logs in production at the OTel collector, not at query time. Most teams do it the wrong way around, paying for storage they never query and then making panicked retention cuts that they regret during the next incident.

Deciding what to keep, at what tier, for how long, is not a decision individual teams should be making independently. It is a platform decision with cost, compliance, and operational implications.

The OTel collector YAML problem

OpenTelemetry is the right long term bet. The collector is genuinely powerful. The configuration is also deeply unpleasant until you have been through it a few times.

Receivers, processors, exporters, pipelines, the YAML surface area is large and the blast radius of a misconfiguration is real. The mistake most platform teams make is letting every team write their own collector configuration. You end up with as many collector configs as you have teams, each slightly different, each with its own quirks, none of them easy to maintain at scale.

Treat the collector config as a platform concern. Own the base configuration. Template it. Let teams extend it within defined guardrails. This is the difference between an observability platform and a collection of observability configurations that happen to exist in the same organisation.

Nobody owns the stack

The deeper problem underneath all of these is ownership. Observability stacks assembled from open source components do not own themselves. Somebody has to be responsible for the Prometheus upgrade, the Loki retention policy, the Grafana dashboard standards, the OTel collector base config.

In most organisations nobody explicitly owns this. It grows organically, maintained by whoever has time, which means it is maintained by nobody consistently. The result is a stack that works until it does not, and when it does not, the blame is diffuse and the fix is slow.

What a Sane Stack Looks Like in 2026

For teams starting fresh or looking to reduce the operational burden of what they already have, here is what I would recommend based on what I have seen work in practice.

Metrics: Consider VictoriaMetrics instead of vanilla Prometheus if cardinality is already causing problems. It is drop-in compatible with the Prometheus ecosystem, significantly more memory efficient at scale, and requires less tuning. For smaller deployments Prometheus is still fine, just go in with cardinality guardrails from day one.

Logs: Loki with aggressive retention policies and collector-level filtering. Accept that you will not keep everything and make that decision deliberately rather than reactively.

Traces: Tempo. It integrates cleanly with the rest of the Grafana ecosystem and the operational overhead is reasonable.

Frontend: Grafana as the unified query and visualisation layer across all three signals. The cross-signal correlation capability is where the real value is, being able to jump from a metric spike to the relevant logs to the relevant traces without switching tools.

Collector: OpenTelemetry Collector as the single ingestion layer. Own the base config at the platform level.

The Shift That Actually Fixes It

The teams I have seen get observability right treat it as a product they build and maintain for their engineering organisation, not a set of tools they install and hope for the best.

That means a dedicated owner or team. A roadmap. An adoption process that brings engineering teams onto the platform rather than letting everyone build their own. Standards for what good looks like. And a feedback loop with the teams consuming the platform so you know what is actually useful versus what is just noise on a dashboard nobody looks at.

Observability taking forever to set up is a symptom. The underlying condition is treating it as an infrastructure task rather than a platform capability. Fix the ownership model and the tooling problems become manageable.

I cover observability as a platform discipline in depth in The Comprehensive Guide to Platform Engineering, including reference architectures, retention strategy, and how to build the business case for treating observability as a first class platform investment.

Reference Architectures Are Lying to You

Jordan — Sat, 09 May 2026 18:57:22 +0000

Every cloud provider has one. Every major consultancy sells one.
Every conference talk ends with one on the final slide.

Reference architectures are everywhere. And most of them are making your platform worse.

The problem with reference architectures

A reference architecture is a idealised blueprint for how a system should be structured. In theory, it gives teams a proven starting point, reduces decision fatigue, and encodes best practices from organisations that have already solved the hard problems.

In practice, most organisations use them wrong. They treat the reference architecture as the destination rather than the starting point. They adopt the full stack because the diagram says so, not because they have validated that each component solves a problem they actually have. They optimise for looking like the reference architecture rather than optimising for what their developers actually need.

The result is platforms that are architecturally impressive and operationally painful. Teams running service meshes they do not need. Organisations operating multi-cluster Kubernetes setups at a scale that does not justify the overhead. Platform teams spending more time maintaining the architecture than delivering value through it.

Where reference architectures come from

It is worth understanding what a reference architecture actually represents before adopting one.

A cloud provider reference architecture represents what is possible on that provider's platform, optimised for showcasing their services. It is not a neutral recommendation. It is a product catalogue with arrows between the boxes.

A consultancy reference architecture represents what worked at the clients that consultancy has served, filtered through the biases and preferences of the people who designed it. It may be excellent. It may also encode decisions that made sense in a context completely different from yours.

A CNCF reference architecture represents the consensus view of a community with strong opinions about open source tooling. That community is smart and well-intentioned. It is also not operating your platform or accountable for your on-call rota.

None of these are wrong. All of them require translation before they are useful to your specific organisation.

What to do instead

Start with the problems your developers are actually experiencing, not with the architecture you want to build toward.

If the problem is inconsistent deployment practices across teams, you need a CI/CD opinion and a way to enforce it. You may not need a service mesh, a service catalogue, and a full GitOps implementation on day one.

If the problem is slow onboarding for new teams, you need a repeatable path from zero to first deployment. You may not need a sophisticated internal developer platform before you have validated what that path should look like.

Use reference architectures as a map of the territory, not as a set of instructions. They tell you what exists and what is possible. They do not tell you what you need right now, in what order, or at what cost.

The question worth asking

Before adopting any component from a reference architecture, ask one question: what specific problem does this solve for the engineers who will have to operate it?

If you cannot answer that question concretely, the component is not ready to be adopted. It is being adopted because it is on the diagram.

The best platform architectures I have worked with look nothing like the reference architectures they started from. They look like the specific, considered,incrementally
validated answers to the specific problems that organisation
faced.

That is harder to put on a slide. It is also the only version that actually works.

This is one of the themes I explore in
The Comprehensive Guide to Platform Engineering, particularly around how platform teams make and sequence architectural decisions in practice rather than in theory.
Free sample at Platform Engineering Guide Sample if you want to get a feel for the depth before committing.

Why Most Internal Developer Platforms Fail (And What To Do About It)

Jordan — Wed, 06 May 2026 11:08:29 +0000

I've spent twenty years building and scaling platforms across financial services technology. In that time I've seen internal developer platforms succeed and I've seen them fail. The technical differences between the successes and the failures are smaller than you'd expect.

The organisational differences are enormous.

The adoption problem nobody talks about

Most IDP failures share a common characteristic: the platform team
treated adoption as inevitable rather than earned.

The assumption goes like this, if we build something genuinely better
than what exists, developers will naturally migrate to it. This is
rarely true. Developers are busy, sceptical of platform initiatives
based on past experience, and rational about where they invest their
time.

The teams that get adoption right treat the platform as a product with a go-to-market problem. They identify a first team, make that team successful, and let word of mouth do the rest.

The metrics that actually matter

The industry has converged on DORA metrics as the standard for
measuring engineering performance. They're useful and worth tracking.
But they measure outputs, not platform health.

A platform can have strong DORA metrics and still be quietly failing.

The metrics I've come to care about most:

Time to first deployment for a new team. Not a team that's been
on the platform for two years — a new team starting fresh. If this
is measured in days rather than hours, the platform has an onboarding
problem that deployment frequency won't reveal.

Unplanned dependency rate. How often do developers go outside the
platform to get something done? Every workaround is a product signal.

Platform team toil ratio. What percentage of time is reactive
versus proactive? If this isn't improving quarter on quarter, the
platform is treading water.

Developer NPS. Uncomfortable to measure. Impossible to argue with.

The documentation trap

Platform teams that rely on documentation to drive adoption have
already lost.

If developers need to read three Confluence pages to understand how
to deploy a service on your platform, the platform has a usability
problem. The documentation is papering over the gap between what the
platform is and what it should be.

When I hear a platform team say "we need better documentation," I now
ask a different question: what specifically are developers confused
about, and why does the platform not make that thing obvious?

The answer to that question is a product improvement. Not a new
Confluence page.

Where this comes from

These are some of the themes I explore in depth in
The Comprehensive Guide to Platform Engineering a 550-page practitioner reference I've just published covering the full platform engineering lifecycle, from Kubernetes and GitOps through to IDPs, AI-native infrastructure, and the organisational change required to make any of it stick.

Happy to discuss any of this in the comments, I'm particularly interested in what others are seeing around IDP adoption in practice.