DEV Community: Alexandre Vazquez

Skopeo, Crane, and regctl: Container Image Management Without the Docker Daemon (2026)

Alexandre Vazquez — Thu, 28 May 2026 13:39:03 +0000

The Problem: Docker Is Overkill for Image Operations

You need to copy an image from Docker Hub to your private registry. Or inspect a manifest before pulling. Or delete old tags programmatically. Or sync an entire repository during a migration.

The instinct is to reach for Docker. But Docker requires a running daemon, root access (or group membership that amounts to the same thing), and pulls the entire image to disk just to read its metadata. For CI pipelines, GitOps workflows, and platform tooling, that’s a significant overhead for what should be lightweight registry operations.

This is the problem that daemonless container image tools solve. Skopeo pioneered the category; today it has real competition from crane, regctl, and ORAS — each with different strengths and ideal use cases.

This article gives you the practical comparison to pick the right tool for your workflow.

The Contenders

Tool	Maintainer	Language	Daemon required
Skopeo	Red Hat / containers	Go	No
crane	Google / ko-build	Go	No
regctl	regclient	Go	No
ORAS	CNCF	Go	No
cosign	Sigstore / OpenSSF	Go	No

All five are Go binaries, statically compiled, and work directly against the OCI Distribution Spec. None of them require Docker or any container runtime.

Skopeo

Skopeo was the first major tool to address daemonless image operations, released by Red Hat in 2016 as part of the containers/image ecosystem (alongside Podman and Buildah).

What Skopeo does well

Image inspection without pulling:

skopeo inspect docker://registry.k8s.io/pause:3.9

Returns full image metadata — digest, layers, labels, architecture, OS — without downloading a single layer. Useful in admission controllers, policy checks, and pre-deployment validation.

Cross-registry copying:

skopeo copy \
  docker://docker.io/library/nginx:1.27 \
  docker://harbor.internal/library/nginx:1.27

Copies image manifests and layers directly between registries, bypassing your local machine entirely. The image never touches your disk.

Multi-arch handling:

skopeo copy --all \
  docker://docker.io/library/nginx:1.27 \
  docker://harbor.internal/library/nginx:1.27

The --all flag copies the full manifest list, preserving all architectures (linux/amd64, linux/arm64, etc.). This is critical when mirroring images for multi-arch clusters.

Registry synchronization:

skopeo sync \
  --src docker \
  --dest docker \
  --all \
  docker.io/library/nginx \
  harbor.internal/mirrors/

skopeo sync mirrors an entire repository, including all tags. You can also use a YAML file to define which images and tags to sync — useful for air-gapped environment bootstrapping.

Tag deletion:

skopeo delete docker://harbor.internal/myapp:old-tag

Useful in CI for cleanup pipelines. Note that registry-side deletion requires the registry to have the DELETE method enabled.

Skopeo’s weaknesses

No image modification : Skopeo copies and inspects, but doesn’t build or modify images
Tag listing is verbose : skopeo list-tags returns JSON you need to parse
No retry logic by default : transient network errors in long sync operations require wrapping with retry scripts
Auth configuration : relies on containers/auth.json format, which differs from Docker’s ~/.docker/config.json (though it supports both)

When to use Skopeo

Air-gapped environment image mirroring
CI pipelines that need to copy or inspect images without Docker
Platform teams on Red Hat / OpenShift stacks
Any workflow already using Podman or Buildah

Crane

Crane is Google’s answer to Skopeo, developed as part of the ko project and later extracted into its own tool. It’s simpler, more scriptable, and has a cleaner CLI design.

What Crane does well

Tag listing:

crane ls registry.k8s.io/pause

No JSON parsing needed. One tag per line. Pipe directly into grep, sort, head.

Digest resolution:

crane digest docker.io/library/nginx:1.27

Returns the image digest. Combine with yq or sed to pin image references in Helm values or Kubernetes manifests.

Image copying:

crane cp docker.io/library/nginx:1.27 harbor.internal/library/nginx:1.27

Same capability as Skopeo’s copy, arguably with a cleaner syntax.

Manifest inspection:

crane manifest docker.io/library/nginx:1.27 | jq .

Returns raw manifest JSON. Useful when you need the exact manifest for digest verification or policy enforcement.

Tagging and retagging:

crane tag harbor.internal/myapp:abc123 harbor.internal/myapp:stable

Adds a new tag to an existing image without re-uploading layers. The tag operation is purely a manifest pointer update.

Flattening images:

crane flatten docker.io/library/ubuntu:24.04 -t harbor.internal/ubuntu:flat

Squashes all layers into one. Reduces layer count for images where layer history doesn’t matter.

Crane’s weaknesses

No sync command : unlike Skopeo, crane has no built-in repository sync. You script it yourself with crane ls + crane cp in a loop
Less mature multi-arch support : crane cp supports multi-arch but the UX is less explicit than Skopeo’s --all
No delete command : doesn’t implement registry deletion

When to use Crane

CI/CD scripting where you want clean, pipeable output
Digest pinning workflows
Lightweight image tagging operations
When you’re already in the ko / Google Cloud ecosystem

regctl

Regctl is the least known of the three but arguably the most feature-complete. It’s the CLI for the regclient Go library and covers use cases that Skopeo and crane leave out.

What regctl does uniquely well

Image modification without rebuild:

regctl image mod myimage:tag \
  --label "org.opencontainers.image.version=1.2.3" \
  --replace

You can add/change labels, annotations, and config fields directly on an existing image in the registry — without pulling, rebuilding, or pushing a new image. This is impossible with Skopeo or crane.

Layer operations:

# Remove a specific layer from an image
regctl image mod myimage:tag \
  --layer-rm sha256:abc123... \
  --replace

Useful for removing accidentally included secrets or large unnecessary layers from published images.

OCI artifact support:

regctl artifact put \
  --media-type application/vnd.example.config.v1+json \
  --config config.json \
  file.tar.gz \
  harbor.internal/myartifacts:v1

Regctl has solid OCI artifact support alongside standard image operations.

Formatting and output:

regctl tag list harbor.internal/myapp --format '{{range .}}{{println .}}{{end}}'

Go template formatting throughout. Useful for integrating into shell scripts without jq.

Referrers (OCI 1.1):

regctl manifest get-list harbor.internal/myapp:v1 --referrers

Lists referrers (signatures, SBOMs, attestations) attached to an image via the OCI 1.1 referrers API.

Regctl’s weaknesses

Smaller community : fewer examples, less StackOverflow coverage
Steeper learning curve : more commands, more flags
Less packaging : not in most distro repos by default

When to use regctl

Image post-processing (labels, annotations, layer removal) without rebuild
Advanced manifest and referrer workflows
When you need OCI artifact operations alongside image operations

ORAS

ORAS (OCI Registry As Storage) is a CNCF project focused specifically on OCI artifact management — pushing and pulling arbitrary files to container registries, not necessarily container images.

# Push a Helm chart as an OCI artifact
oras push harbor.internal/charts/myapp:1.0.0 \
  --artifact-type application/vnd.helm.chart.v1+tar \
  mychart.tgz

# Push SBOM
oras push harbor.internal/myapp:v1 \
  --artifact-type application/spdx+json \
  sbom.spdx.json

# Pull
oras pull harbor.internal/charts/myapp:1.0.0

ORAS is not a direct Skopeo replacement — it’s for when your registry is a general-purpose artifact store, not just a container registry. Helm OCI, SBOMs, attestations, and policy bundles all benefit from ORAS.

cosign

Cosign from Sigstore is not a general-purpose image tool — it’s specifically for supply chain security. But it’s increasingly part of any container image workflow.

# Sign an image
cosign sign --key cosign.key harbor.internal/myapp:v1@sha256:abc123...

# Verify
cosign verify --key cosign.pub harbor.internal/myapp:v1

# Attach SBOM
cosign attach sbom --sbom sbom.spdx harbor.internal/myapp:v1

# Keyless signing (Sigstore)
cosign sign harbor.internal/myapp:v1

Cosign integrates with OIDC providers for keyless signing (no key management required), which is the direction the ecosystem is moving. If you’re building a supply chain security practice, cosign is mandatory, not optional.

Side-by-side comparison

Operation	Skopeo	Crane	regctl
Inspect image	`inspect`	`manifest`	`manifest get`
Copy image	`copy`	`cp`	`image copy`
Copy all arches	`copy --all`	`cp` (auto)	`image copy`
Sync repository	`sync`	script it	script it
List tags	`list-tags` (JSON)	`ls` (plain)	`tag list`
Delete tag/image	`delete`	—	`tag delete`
Modify labels	—	—	`image mod`
Remove layer	—	—	`image mod`
OCI artifacts	limited	limited	`artifact`
Referrers (1.1)	—	—	`manifest get-list`

Practical workflows

Mirror images for air-gapped clusters (Skopeo)

# sync-list.yaml
docker.io:
  images:
    library/nginx:
      - "1.25"
      - "1.26"
      - "1.27"
    library/redis:
      - "7.2"
      - "7.4"


skopeo sync \
  --src yaml \
  --dest docker \
  --all \
  sync-list.yaml \
  harbor.internal/mirrors/

Pin image digests in CI (Crane)

#!/bin/bash
# Update image digests in values.yaml
for image in nginx:1.27 redis:7.4; do
  digest=$(crane digest docker.io/library/${image})
  echo "docker.io/library/${image}@${digest}"
done

Combine with yq to update Helm values files automatically, ensuring reproducible deployments.

Retag without re-pushing (Crane or regctl)

# After a successful deploy to staging, promote to production
crane tag harbor.internal/myapp:${GIT_SHA} harbor.internal/myapp:production

No layer transfer. The operation is a metadata update in the registry.

Add OCI annotations post-build (regctl)

regctl image mod harbor.internal/myapp:v1.2.3 \
  --annotation "org.opencontainers.image.source=https://github.com/org/repo" \
  --annotation "org.opencontainers.image.revision=${GIT_SHA}" \
  --replace

Attaches build metadata to an image already in the registry, without a rebuild.

Supply chain security pipeline

# 1. Build and push
docker buildx build --push -t harbor.internal/myapp:${GIT_SHA} .

# 2. Generate SBOM
syft harbor.internal/myapp:${GIT_SHA} -o spdx-json > sbom.spdx.json

# 3. Attach SBOM
cosign attach sbom --sbom sbom.spdx.json harbor.internal/myapp:${GIT_SHA}

# 4. Sign (keyless with OIDC in CI)
cosign sign harbor.internal/myapp:${GIT_SHA}

# 5. Verify in admission controller or deployment pipeline
cosign verify \
  --certificate-identity-regexp="https://github.com/org/repo" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  harbor.internal/myapp:${GIT_SHA}

Installation

All tools install as single static binaries:

# Skopeo (via package manager)
brew install skopeo # macOS
dnf install skopeo # RHEL/Fedora
apt install skopeo # Debian/Ubuntu

# Crane
brew install crane
# or binary release
curl -sL https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar xz crane

# regctl
curl -sL https://github.com/regclient/regclient/releases/latest/download/regctl.linux.amd64 -o /usr/local/bin/regctl
chmod +x /usr/local/bin/regctl

# ORAS
brew install oras

# cosign
brew install cosign

Which tool should you use?

Use Skopeo if: you’re on a Red Hat / OpenShift stack, you need repository sync, or you’re building air-gapped environment pipelines. It’s the most battle-tested and widely packaged.

Use Crane if: you’re scripting image operations in CI and want clean, composable CLI output. crane ls + crane cp + crane digest cover 80% of automation use cases with minimal friction.

Use regctl if: you need to modify images post-build, work with OCI referrers, or want the most complete feature set for a registry client. It has a higher learning curve but can replace both Skopeo and crane for advanced workflows.

Use ORAS if: you’re using a registry to store non-image artifacts — Helm charts, SBOMs, policy bundles, ML models.

Use cosign regardless of which of the above you pick, as soon as supply chain security matters to your organization. It’s not a replacement for the others — it’s a complement.

In practice, most platform teams end up using 2-3 of these tools together. Crane for day-to-day scripting, Skopeo for sync jobs, cosign for signing, ORAS for artifact storage.

FAQ

Can I use these tools with private registries?

Yes. All support standard registry authentication. Crane and Skopeo both read from ~/.docker/config.json. Regctl has its own config file (~/.regctl/config.json) but can import Docker credentials. Set DOCKER_CONFIG to point to your credentials file in CI environments.

Do these work with Docker Hub rate limits?

Yes, and they’re often more efficient than the Docker CLI because they only fetch manifest metadata for inspect operations, not full layers. For heavy pull workloads, authenticate with your Docker Hub credentials to get higher rate limits.

What about ECR, GCR, and Azure Container Registry?

All tools support these with the appropriate credential helpers. For ECR, use docker-credential-ecr-login. Crane has native ECR support via the --platform flag and crane auth commands. Skopeo supports ECR via --creds "AWS:$(aws ecr get-login-password)".

Are these tools safe to run in Kubernetes pods?

Yes. Since they require no daemon and no elevated privileges for read operations, they’re well-suited to run as init containers or sidecar containers in Kubernetes. Skopeo is commonly used in image pre-pulling init containers. Use a dedicated service account with least-privilege registry credentials.

Can I copy a multi-arch image and keep all platforms?

Skopeo: skopeo copy --all. Crane: crane cp copies the index automatically when the source is a manifest list. Regctl: regctl image copy preserves manifest lists by default.

{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "Can I use Skopeo, crane, and regctl with private registries?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. All support standard registry authentication. Crane and Skopeo both read from ~/.docker/config.json. Regctl has its own config file (~/.regctl/config.json) but can import Docker credentials. Set DOCKER_CONFIG to point to your credentials file in CI environments."
}
},
{
"@type": "Question",
"name": "Do these container image tools work with Docker Hub rate limits?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes, and they are often more efficient than the Docker CLI because they only fetch manifest metadata for inspect operations, not full layers. For heavy pull workloads, authenticate with your Docker Hub credentials to get higher rate limits."
}
},
{
"@type": "Question",
"name": "Do Skopeo, crane, and regctl work with ECR, GCR, and Azure Container Registry?",
"acceptedAnswer": {
"@type": "Answer",
"text": "All tools support these registries with the appropriate credential helpers. For ECR, use docker-credential-ecr-login. Crane has native ECR support via crane auth commands. Skopeo supports ECR via --creds \"AWS:$(aws ecr get-login-password)\"."
}
},
{
"@type": "Question",
"name": "Are Skopeo, crane, and regctl safe to run in Kubernetes pods?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. Since they require no daemon and no elevated privileges for read operations, they are well-suited to run as init containers or sidecar containers in Kubernetes. Skopeo is commonly used in image pre-pulling init containers. Use a dedicated service account with least-privilege registry credentials."
}
},
{
"@type": "Question",
"name": "Can I copy a multi-arch image and keep all platforms?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. Skopeo: use skopeo copy --all. Crane: crane cp copies the index automatically when the source is a manifest list. Regctl: regctl image copy preserves manifest lists by default."
}
}
]
}

Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each The container works fine in CI. It deploys successfully to...
ArgoCD Guide: GitOps Continuous Delivery for Kubernetes ArgoCD is the leading GitOps operator for Kubernetes. This guide...
Harbor Registry Explained: Securing Container Images in Kubernetes and DevSecOps Learn how Harbor Registry improves container security by enabling vulnerability...

Transforming XML to JSON and CSV with XSLT

Alexandre Vazquez — Thu, 28 May 2026 09:00:02 +0000

XSLT is usually associated with XML-to-XML transformations, but in integration work you often need JSON or CSV. The good news is that XSLT is perfectly capable of producing non-XML outputs when you design the stylesheet for it. The key is to choose the right output method, control whitespace carefully, and build an intermediate structure if it helps clarify the mapping. This post covers practical patterns for generating JSON and CSV from XML while keeping the stylesheet maintainable.

For JSON, the simplest method is to output text and build the JSON structure manually. This gives you precise control, but it also requires careful escaping and formatting. If you are on XSLT 3.0, use maps and arrays and let the processor serialize to JSON. This reduces string manipulation and makes your transform more robust. If you are on XSLT 1.0 or 2.0, you can still build JSON text safely by using templates that escape quotes, backslashes, and control characters.

A clear pattern is to create a template that takes a string and outputs an escaped JSON string. Then, for each object, output the property names and values with explicit commas. Keep a template to handle comma placement so you do not end up with trailing commas in arrays. This is a good place to use position checks like position() != last() to decide when to emit a comma. While it can look verbose, the logic is deterministic and easy to debug.

CSV output is simpler but comes with its own hazards. You need to wrap fields that contain commas, quotes, or line breaks. The common rule is to wrap the field in quotes and double any interior quotes. Again, a dedicated template to escape fields pays off. Define the column order explicitly and avoid depending on the source document order. This keeps the CSV consistent even if the XML input changes slightly. If you need multiple CSV sections, consider running two passes: one to compute the rows and one to serialize them.

An example CSV field template can look like this in XSLT 1.0:

xml

Designing XSLT transforms with parameters and multiple inputs

Alexandre Vazquez — Mon, 25 May 2026 09:00:00 +0000

Many real-world transformations do not run on a single XML document. You often merge a primary payload with reference data, catalog lookups, or environment configuration. Done well, this results in a clean, predictable transform. Done poorly, it becomes a maze of document() calls and hidden dependencies. The difference is in how you model inputs and parameters from the start. As an integration engineer, I treat input selection and parameter design as first-class API design for the stylesheet.

Start by naming every input. Instead of embedding document('config.xml') in multiple templates, load each external document once near the top of the stylesheet and bind it to a global variable. This makes dependencies explicit and keeps the rest of the code focused on mapping. It also helps with testing, because you can override the URI with a parameter. A clean pattern is to define xsl:param values for input URIs and then bind them to xsl:variable values that hold the parsed documents.

The same clarity applies to parameters. Keep parameters primitive and predictable, and avoid passing in node sets unless you truly need them. A parameter should be an external knob: region, language, a feature flag, or an output format. If you have a complex decision tree, consider using a lookup XML or JSON input and then query it inside the stylesheet. This approach keeps the invocation interface stable while still letting you evolve business rules.

A simple skeleton might look like this:

From there, templates can reference $catalog without worrying about IO or base URIs. You can also define a named template that accepts a parameter for reuse across multiple modes. This is useful when the same output block is needed for several sections of the document but the selection context differs.

When combining multiple inputs, always anchor your lookups to a clear key. If you can, define xsl:key on the external document so lookups are efficient and readable. In XSLT 2.0 or 3.0, xsl:for-each-group and the map types can reduce boilerplate, but the core idea remains: make your joins explicit and deterministic. If you rely on default order or on undocumented assumptions about uniqueness, you will eventually get a hard-to-reproduce bug.

Another important integration pattern is separating parsing from formatting. For example, you might normalize all values from the various inputs into a canonical intermediate structure and then render that structure into the final output. This makes testing easier and supports future outputs such as CSV, JSON, or a secondary XML format. Even in XSLT 1.0, you can emulate this by creating result tree fragments, then processing them in a second pass if needed.

Multiple inputs also raise questions about fallbacks. Decide how you want to behave when optional data is missing. I prefer to centralize defaults in a few named templates or functions and avoid sprinkling xsl:choose blocks everywhere. This keeps the stylesheet readable and makes it obvious how to override the defaults later. Document your fallbacks in the code with short, clear names so a future maintainer does not have to rediscover the rules by reading the entire stylesheet.

Finally, create a small set of inputs that represent common scenarios and run them regularly. For example, have a baseline case, a case with missing reference data, and a case with unexpected elements. These are the cases that reveal poor assumptions about inputs. A fast way to iterate on these scenarios is to run the transform with a tool that lets you swap inputs and parameters quickly.

If you want to try these patterns with real inputs and multiple documents, the online editor at https://xsltplayground.com is built for that workflow. It lets you load multiple XML documents and parameters, see how they interact, and keep your integration logic transparent as it grows.

XSLT performance tuning without losing readability

Alexandre Vazquez — Thu, 21 May 2026 09:00:01 +0000

Performance problems in XSLT are sneaky. The stylesheet looks clean, the output is correct, but the transform slows down as the input grows. Most of the time this is caused by expensive selections that are repeated in loops, or by deep // searches that scan the entire tree more often than you expect. The good news is that you can usually fix these issues without turning the stylesheet into unreadable micro-optimizations.

The first step is to examine where you are traversing the document. XSLT processors are optimized for template matching, so prefer xsl:apply-templates and specific match patterns over xsl:for-each with // in the select. When you do need a search, limit it to the smallest possible subtree. A single // at the top-level becomes a full-tree scan each time it runs. If it runs inside another loop, the cost can explode.

Keys are the most important performance feature, and they also improve readability. When you define an xsl:key, you turn a repeated search into a fast lookup. This is especially critical for join-like operations where you match a reference value to another document or a secondary section of the same document. Build the key once, and then use key('id', $value) everywhere. The intent becomes clear: you are doing a lookup, not a scan. If you only use keys occasionally, it can feel like overkill, but it is often the biggest win.

Modes are another useful tool. If you use the same templates in multiple contexts, you may end up doing extra work or firing templates that you do not need. A dedicated mode lets you create a focused processing pipeline that touches only the nodes relevant to that output section. This can reduce both runtime and mental overhead. It also makes it easier to reason about precedence: within a mode, you can define more specific templates without worrying about side effects on unrelated parts of the transform.

Consider caching computed values in variables. XSLT variables are immutable, so they are safe to reuse without unintended side effects. If you are computing a complex string or a filtered node set repeatedly, store it once per relevant scope. Just be careful not to define a variable at the top of the stylesheet if it depends on the context; keep it as close as possible to where it is used to avoid confusion.

If you are working in XSLT 2.0 or 3.0, you gain access to xsl:for-each-group and higher-order functions. These can be faster and clearer than manual grouping with keys. For XSLT 1.0, the Muenchian grouping pattern is still effective, and when combined with keys it remains a strong choice. Either way, focus on minimizing passes over large node sets.

Also consider the output method. Serializing large outputs can be a significant part of the runtime. If you do not need pretty-printed XML, avoid indentation to reduce the amount of whitespace and processing time. Similarly, if you are generating text or JSON, use method="text" or structured XSLT 3.0 serialization options rather than building a text string node by node.

I recommend using realistic test data when tuning performance. A transform that runs in 100 milliseconds on a tiny input may take seconds on real data. Use a handful of real documents and measure changes as you apply each improvement. This keeps the optimization process grounded and prevents you from making the code worse without a measurable gain.

Finally, keep a balance between speed and clarity. The fastest stylesheet is useless if it is too hard to maintain. Use a few consistent patterns: keys for lookups, modes for pipelines, variables for repeated values, and limited selection scope. With those in place, the performance usually becomes acceptable without heroics.

If you want a quick way to benchmark different approaches with the same input set, try the online editor at https://xsltplayground.com. It is a convenient place to experiment with keys, modes, and alternative match patterns while keeping your transform readable.

XSLT debugging patterns that save hours

Alexandre Vazquez — Mon, 18 May 2026 09:00:00 +0000

XSLT bugs are rarely loud. More often, a template silently matches the wrong node, a predicate filters out a value you needed, or a namespace mismatch turns an element into a ghost. The fastest fix comes from a repeatable debugging workflow that keeps your assumptions visible. Over time you learn the same patterns appear in almost every real project, whether you are cleansing XML feeds, integrating partner payloads, or generating documents. This post walks through the techniques I use as an integration engineer to debug transforms quickly without losing context.

Start by making the matching rules obvious. The majority of issues are caused by using // too freely or relying on default namespaces. Replace broad paths with anchored ones, and when in doubt, print out what the processor thinks the current node is. A simple xsl:message combined with name() and namespace-uri() can reveal a namespace mismatch in seconds. I also add short, temporary templates that match the suspected nodes and output minimal text, which is a fast way to confirm whether the selection is correct.

Next, isolate the failing region by reducing input size. You rarely need the entire input document to debug a single mapping. Extract the smallest fragment that reproduces the issue and run the transform against that. This lets you simplify predicates and remove unrelated templates. When a transform uses xsl:key, add a temporary output that lists the key index for a given value so you can see if the key is being built correctly. The same idea works for variables: output them just once in a deterministic area of the result so you can verify their shape.

A stable debug transform also benefits from deterministic ordering. When you iterate through nodes, add an explicit xsl:sort so the output is predictable. That makes diffs meaningful when you tweak a predicate or update a template priority. If you are mixing modes, ensure the call chain is explicit; a missing mode is a classic way to call a generic template by accident. A related trap is having a high-priority identity template that overrides a specialized one, so watch for priority values and make sure the most specific template wins.

When handling multiple inputs, be clear about document boundaries. Use document() or collection() with explicit base URIs and add messages that show which document node you are iterating. If you are using XSLT 2.0 or 3.0, a quick serialize() to a short string can show you whether the tree is what you expect. If you stick to XSLT 1.0, the same idea works by writing xsl:copy-of into a separate debug result tree and inspecting it.

Here is a short pattern I often add while troubleshooting:



    node=
    ns=

You can drop this at the top of the stylesheet, run a quick transform, and then remove it once the root cause is found. The idea is not to keep noise in production, but to have a fast way to make the invisible visible. For more focused tracing, add the template only for the nodes you suspect are wrong. Debugging gets faster the more you scope down the noise.

Finally, keep a checklist of the classic XSLT footguns: missing namespaces, wrong context node, incorrect @ in attribute selection, and predicates that use 1-based indexes when you thought they were 0-based. I also look for template import precedence issues and for unexpected whitespace handling when the output is textual. These are easy to miss because the transform still runs, it just runs incorrectly.

If you want a fast place to test these patterns with real inputs, use the online editor at https://xsltplayground.com. It is built for rapid iteration with multiple inputs and parameters, which makes debugging much less painful and keeps your feedback loop tight.

XSLT string functions: complete reference with examples

Alexandre Vazquez — Thu, 14 May 2026 09:00:00 +0000

String manipulation is one of the most common tasks in XSLT. Whether you are formatting output, parsing codes, or normalising values from external systems, XPath provides a rich set of string functions. This reference covers the most useful ones with examples you can run in XSLT Playground.

Basic string functions (XSLT 1.0+)

string-length

Returns the number of characters in a string.

substring

Extracts a portion of a string. Arguments: string, start position (1-based), optional length.

substring-before and substring-after

Split a string on a delimiter.

contains, starts-with, ends-with

Test membership without extracting.

...
...

...

concat

Joins strings together. Takes any number of arguments.

normalize-space

Strips leading and trailing whitespace, and collapses internal whitespace to single spaces. Essential for cleaning values from XML sources.

translate

Replaces characters one-for-one. Useful for simple case conversion or character removal.

Advanced string functions (XSLT 2.0+)

upper-case and lower-case

No longer need translate for case conversion.

replace

Regex-based substitution. Much more powerful than translate.

matches

Tests a string against a regex.

tokenize

Splits a string into a sequence using a regex delimiter. Returns a sequence of strings.

string-join

The inverse of tokenize. Joins a sequence with a separator.

format-number

Formats a number as a string with a picture pattern.

format-date and format-dateTime

Format xs:date and xs:dateTime values using picture strings.

Practical patterns

Extract domain from URL:

Pad a number with leading zeros:

Check if a node text is numeric:

All of these work in XSLT Playground. Set the version to 2.0 or 3.0 for the functions that require it.

XSLT grouping with xsl:for-each-group: complete guide

Alexandre Vazquez — Mon, 11 May 2026 09:00:00 +0000

Grouping is one of the most powerful features introduced in XSLT 2.0. Before it, grouping in XSLT 1.0 required the Muenchian method — a clever but verbose technique involving keys and node-set comparisons. In 2.0, xsl:for-each-group makes grouping straightforward.

Basic grouping with group-by

group-by groups nodes that share the same value for a given expression. The result is one iteration per distinct group value.

Input:


  DE120
  US85
  DE200
  FR60
  US140

Stylesheet:

Output:


  2320
  160
  2225

Key functions inside for-each-group:

current-grouping-key() — returns the value that defines the current group
current-group() — returns the sequence of all nodes in the current group

Nested grouping

Groups can be nested. Group orders by country, then within each country by status:

group-adjacent

Groups consecutive nodes that share the same key value. Unlike group-by, it starts a new group when the key changes, even if the same key appeared earlier. This is useful for processing structured text or segmented data.


  Starting
  Processing
  Failed
  Retrying
  Done

This produces three blocks: two INFO (positions 1-2), one ERROR (3-4), one INFO (5). With group-by, the two INFO groups would be merged into one.

group-starting-with and group-ending-with

These group nodes based on a pattern match rather than a key value. Every time a node matches the pattern, a new group starts (or ends).

group-starting-with example — treat every ## as the start of a section:



    <xsl:value-of select="self::h2"/>

group-ending-with example — group lines until a blank line:

Computing aggregates

current-group() returns a sequence, so you can apply any XPath aggregate function directly:

Try it in XSLT Playground

Paste any of the examples above into XSLT Playground with version set to 2.0 or 3.0. Grouping is one of the features that benefits most from live testing — you can immediately see how changing the grouping key or switching between group-by and group-adjacent affects the output structure.

Radar: A New Kubernetes IDE Worth Knowing About (vs OpenLens, FreeLens)

Alexandre Vazquez — Sat, 09 May 2026 22:10:36 +0000

If you’ve been following Kubernetes tooling, you’ve probably already been through the Lens saga: Lens went commercial, OpenLens emerged as the community fork, then FreeLens appeared when OpenLens maintenance slowed. The pattern is familiar — a useful desktop tool, a licensing decision, a fork, another fork.

Radar is not a fork. It’s a different approach to the same problem: giving engineers a useful interface for Kubernetes clusters without the friction of kubectl for every task. Built by Skyhook (YC-backed, Google Cloud Partner), it’s been live since 2025, has 1.7k+ GitHub stars, releases weekly, and the founder reaches out to the community directly. That’s usually a good signal that someone is genuinely building in public.

This article covers what Radar actually does, where it pulls ahead of OpenLens and FreeLens, and when those tools are still the right choice.

The State of Kubernetes Desktop Tooling in 2026

Before getting into Radar specifically, it’s worth naming the landscape clearly:

Lens — the original. Electron-based, polished, now commercial (Mirantis). The free Personal tier is non-commercial only. Pro is ~$22-35/user/month.
OpenLens — the community fork of Lens before Mirantis closed exec/logs/shell in v6.3 (January 2023). Maintenance has slowed significantly. No active release cadence.
FreeLens — a more active community fork, filling the gap left by OpenLens’ decline. Restores the missing features. No commercial backing.
k9s — terminal TUI, fast, keyboard-driven, single-cluster. Different audience.
Headlamp — CNCF Sandbox project, plugin-extensible, web-based.
Radar — Go binary, Apache 2.0, team-oriented, topology and event timeline focused.

The problem with OpenLens and FreeLens is not that they’re bad tools — they’re genuinely useful for the solo developer with one or two clusters. The problem is that they’re single-cluster-at-a-time desktop apps with no concept of team, no persistent state, and no awareness of the modern Kubernetes ecosystem (ArgoCD, Flux, Karpenter, KEDA). As your infrastructure grows, you outgrow them.

What Radar Actually Is

Radar is available in two forms:

Radar OSS — a single ~30MB Go binary, Apache 2.0, free forever. Can run locally (desktop app) or deployed in-cluster via Helm. No sidecars, no feature gates.
Radar Cloud — same binary, adds a hosted control plane with fleet aggregation, 30-day event retention, SSO/SCIM, scoped RBAC, and shared URLs for team incident response. Priced per cluster ($99/cluster/month for Team), not per user.

The per-cluster pricing is a deliberate design decision — teams don’t pay more as they add engineers, only as they add clusters. For a 20-person platform engineering team managing 5 clusters, Radar Cloud runs $495/month. The equivalent Lens Pro seats would cost $2,200-4,200/month.

For most self-hosted environments, the OSS version is sufficient and costs nothing.

Key Features

Topology View

This is the most visually distinctive feature. Radar renders a live service graph for your cluster: deployments, services, ingresses, cross-namespace dependencies, and east-west traffic flows — all in a single view without running kubectl get all -A and stitching the output together mentally.

OpenLens and FreeLens have resource list views. They show you what exists. Radar shows you how things connect — which is what you actually need when debugging why Service A can’t reach Service B.

Persistent Event Timeline

Kubernetes events are ephemeral by default — they expire after approximately one hour. When something breaks at 2am and you’re looking at it at 9am, the events that explain what happened are gone. Logs may still be there if you’re running a log aggregator, but the Kubernetes-level events (pod restarts, scheduling failures, node pressure events, probe failures) are gone.

Radar retains events. The OSS version extends this beyond the default 1-hour cluster retention. The Cloud version retains 30 days. You can rewind the timeline to any point and reconstruct what the cluster looked like at that moment.

Neither OpenLens nor FreeLens have any event retention beyond what the cluster itself provides.

GitOps Integration (ArgoCD + Flux)

Radar auto-detects ArgoCD and Flux and surfaces sync state, drift, and health directly in the UI. You can see whether a deployment is in sync, when it last synced, and whether it drifted from the desired state in Git.

In OpenLens and FreeLens, ArgoCD resources appear as generic Kubernetes custom resources. You can see the CRDs, but there’s no purpose-built understanding of what they mean — no sync status visualization, no diff view, no rollback trigger.

Helm Management

Radar tracks Helm releases with full revision history and supports one-click rollbacks from the UI. This is similar to what OpenLens/FreeLens offer via the Helm releases view, but Radar adds revision diffing — you can see what changed between release 5 and release 6 before deciding to roll back.

Image Filesystem

You can browse container image filesystems through Radar without needing kubectl exec into a running pod or access to the container registry. Useful for security audits and debugging — you can verify what’s actually in an image at rest.

MCP Server (AI Integration)

Radar ships with an MCP (Model Context Protocol) server, which means you can connect Claude, Cursor, or GitHub Copilot directly to your cluster context and ask questions about it in natural language. The MCP server is token-optimized — it doesn’t dump raw YAML at the model, it structures cluster state into meaningful context.

This is something neither OpenLens nor FreeLens have. It’s also something that’s genuinely useful if you’re already using AI assistants for development work.

Cluster Audit

30 built-in best-practice checks — resource requests/limits, RBAC permissions, image pinning, network policies, security contexts. The checks are labeled by compliance framework. This is not a replacement for dedicated security tooling (Trivy, Falco, Polaris), but it’s a useful first-pass audit without leaving the tool you’re already using.

Multi-Cluster Support (Cloud)

The Cloud tier adds fleet-level visibility: a single view across all clusters, cross-cluster search, and drift detection between environments (e.g., staging vs. production). This is the feature that changes the calculus for platform engineering teams managing 5+ clusters.

OpenLens and FreeLens require you to switch cluster context manually. There is no fleet view.

Architecture: Why a Go Binary Matters

OpenLens and FreeLens are Electron apps — Chromium + Node.js wrapped in a desktop shell. This means:

200-500MB install size
1-2 second startup time on a fast machine, more on slower ones
Memory footprint in the hundreds of megabytes
Local kubeconfig required on each engineer’s machine

Radar’s in-cluster deployment is a single Go binary (~30MB) that runs as a Pod with a ServiceAccount. It connects to the hosted control plane over outbound WebSocket + TLS. No inbound firewall rules, no kubeconfig distribution, no per-engineer setup.

The local desktop app is also a lightweight Go binary — 65-second startup was demonstrated on a 322-node cluster. That’s not a typo.

For in-cluster deployment, the architecture means security is handled at the ServiceAccount level, not by distributing kubeconfigs to engineer laptops. That matters for teams with security requirements around credential management.

Feature Comparison

Feature	Radar OSS	Radar Cloud	OpenLens	FreeLens
License	Apache 2.0	Proprietary (hosted)	MIT/GPL	MIT
Maintenance	Active (weekly releases)	Active	Stalled	Active (community)
Architecture	Go binary / in-cluster	In-cluster + hosted	Electron	Electron
Multi-cluster	Basic	Fleet view
Event retention	Extended	30 days	Cluster default (~1h)	Cluster default (~1h)
Topology view
GitOps (ArgoCD/Flux)			CRDs only	CRDs only
Helm management
kubectl exec / logs / shell			(restored)
MCP / AI integration
Cluster audit
SSO / SCIM
Shared incident URLs
Image filesystem browser
Cost tracking	(OpenCost)
Price	Free	$99/cluster/month	Free	Free

When Radar Makes Sense

You’re managing multiple clusters. Even with the OSS version, the topology view and event timeline make Radar more useful than OpenLens/FreeLens at 3+ clusters. The Cloud fleet view is the compelling option at 5+.

Your team uses GitOps. If ArgoCD or Flux is part of your workflow, Radar’s native understanding of sync state and drift is meaningfully better than seeing CRDs in a generic list view.

You need post-mortem capability. If your incident review process involves looking at what the cluster was doing when the alert fired, you need event retention. Radar has it; OpenLens and FreeLens don’t.

You’re adopting AI tooling. The MCP server is the most forward-looking feature here. If you use Claude Code, Cursor, or Copilot for your infrastructure work, having cluster context available to those tools without copy-pasting YAML is a genuine productivity improvement.

You have a platform engineering team. Per-cluster pricing, SSO, SCIM, and shared incident URLs are features that only matter if you have more than one person managing infrastructure.

When OpenLens or FreeLens Still Makes Sense

You’re a solo developer with one or two clusters. OpenLens and FreeLens are familiar, local, and have zero setup overhead. If you don’t need team features, event retention, or topology views, they remain perfectly functional tools.

You’re deeply invested in the Lens UX. The resource tree, the terminal integration, the way Lens presents namespace-scoped resources — if your muscle memory is built around that interface, switching has a real cost. Radar is different, not just better.

You need maximum customization. OpenLens and FreeLens support plugins. Radar does not currently have a plugin system.

Your environment is air-gapped or has strict egress restrictions. Radar OSS can run fully in-cluster, but Radar Cloud requires outbound connectivity to the hosted control plane. OpenLens and FreeLens are fully local.

Getting Started

OSS installation takes about two minutes:

# Homebrew (macOS/Linux)
brew install skyhook-io/tap/radar

# Helm (in-cluster)
helm repo add skyhook https://charts.skyhook.io
helm install radar skyhook/radar \
  --namespace radar \
  --create-namespace \
  --set service.type=ClusterIP

Or download the binary directly from radarhq.io.

Verdict

Radar is the most interesting new entrant in the Kubernetes tooling space in a while — not because it replaces everything else, but because it addresses the specific gap that OpenLens and FreeLens never covered: teams, multiple clusters, and persistent state.

For a solo developer, OpenLens or FreeLens are still completely reasonable choices. For a platform engineering team managing more than two clusters with ArgoCD or Flux, Radar’s feature set is materially better and the OSS version costs nothing.

The active release cadence and the YC backing suggest this isn’t a one-person side project — there’s a team actively working on it. Whether the Cloud pricing sticks long-term is a question only usage will answer, but the Apache 2.0 core with an explicit “always open source” commitment is the right foundation.

Worth evaluating if you haven’t already.

Tested with Radar OSS v0.x on Kubernetes 1.29–1.32. Pricing and feature availability as of May 2026.

XSLT template matching explained with examples

Alexandre Vazquez — Thu, 07 May 2026 09:00:00 +0000

Template matching is the mechanism that drives every XSLT transformation. Understanding how the processor selects templates — and what happens when multiple templates could match — is the difference between a stylesheet that works reliably and one that produces surprising output. This post covers everything you need to know.

How match patterns work

When the processor visits a node, it evaluates every template's match attribute as an XPath pattern. A pattern is a restricted form of XPath that tests properties of a node rather than selecting nodes from a starting point. If the pattern is satisfied, that template is a candidate.

Priority and conflict resolution

More than one template can match the same node. The processor resolves the conflict using priority. Each pattern has a default priority calculated by the spec:

Pattern type	Default priority
`node()` or `*`	-0.5
`element-name`	0
`prefix:element-name`	0
`a/b` (path)	0.5
`a[predicate]`	0.5
`@attr`	0

More specific patterns automatically get higher priority. You can override this with the priority attribute:

If two templates have equal computed priority, the processor signals an error (or picks the last one, depending on implementation — Saxon issues an error by default). Always assign explicit priorities when you have competing templates.

The built-in templates

XSLT has default templates for every node type. If no explicit template matches a node, the built-in fires. For elements and the document root, the built-in calls apply-templates on all children. For text and attribute nodes, it outputs the string value.

This means that without any templates at all, the processor will walk the entire tree and output all text content. Understanding this explains why simple stylesheets can produce unexpected extra text — a text node matched nothing explicit, and the built-in output it.

To suppress text output globally, add:

This overrides the built-in with an empty template, producing no output for text nodes that are not handled elsewhere.

Modes

Modes let you have multiple templates for the same node that serve different purposes. A mode is a named context for a set of templates.

##

Call with mode:

Modes are especially useful when you need to process the same nodes in multiple places in the output with different logic each time.

apply-templates vs for-each

Both iterate over a set of nodes. The difference is that apply-templates dispatches to the best matching template for each node, while for-each stays in the current context and does not do template lookup.

Use apply-templates when you want polymorphism — different node types handled differently. Use for-each when you are doing a simple iteration over a homogeneous set and do not need dispatch.

Testing patterns in XSLT Playground

XSLT Playground is a fast way to experiment with matching rules. Paste a stylesheet with multiple competing templates and check which one fires. Add `` in each template to trace which one the processor picks. The trace panel shows all messages in order so you can follow the dispatch chain.

Solid understanding of template matching pays off every time you work on a complex stylesheet. Once you know how priorities and built-ins interact, most "unexpected output" bugs become obvious.

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Alexandre Vazquez — Tue, 05 May 2026 11:00:00 +0000

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Introduction

Most observability stacks running in production for over a year end up with alerting spread across two systems: Prometheus Alertmanager handling metric-based alerts and Grafana Alerting managing everything else. This creates the "alerting consolidation problem" where on-call teams receive duplicated pages, silencing rules live in two places, and nobody is certain which system is authoritative.

The question is straightforward: should you standardize on Prometheus Alertmanager, move everything into Grafana Alerting, or deliberately run both? The answer depends on your datasource mix, your GitOps maturity, and how your organization manages on-call routing.

Architecture Overview

Prometheus Alertmanager: The Standalone Receiver

Alertmanager is a dedicated, standalone component in the Prometheus ecosystem. It does not evaluate alert rules itself. Instead, Prometheus (or compatible senders like Thanos Ruler, Cortex, or Mimir Ruler) evaluates PromQL expressions and pushes firing alerts to the Alertmanager API. Alertmanager then handles deduplication, grouping, inhibition, silencing, and notification delivery.

# Simplified Prometheus → Alertmanager flow
#
# [Prometheus] --evaluates rules--> [firing alerts]
#        |
#        +--POST /api/v2/alerts--> [Alertmanager]
#                                      |
#                          +-----------+-----------+
#                          |           |           |
#                       [Slack]    [PagerDuty]  [Email]

The entire configuration lives in a single YAML file (alertmanager.yml). This includes the routing tree, receiver definitions, inhibition rules, and silence templates. There is no database, no UI-driven state — just a config file and an optional local storage directory for notification state and silences. This makes it trivially reproducible and ideal for GitOps workflows.

For high availability, you run multiple Alertmanager instances in a gossip-based cluster. They use a mesh protocol to share silence and notification state, ensuring that failover does not result in duplicate or lost notifications.

Grafana Alerting: The Integrated Platform

Grafana Alerting (sometimes called "Grafana Unified Alerting," introduced in Grafana 8) takes a different architectural approach. It embeds the entire alerting lifecycle — rule evaluation, state management, routing, and notification — inside the Grafana server process. Under the hood, it uses a fork of Alertmanager for the routing and notification layer.

# Simplified Grafana Alerting flow
#
# [Grafana Server]
#   ├── Rule Evaluation Engine
#   │     ├── queries Prometheus
#   │     ├── queries Loki
#   │     ├── queries CloudWatch
#   │     └── queries any supported datasource
#   │
#   ├── Alert State Manager (internal)
#   │
#   └── Embedded Alertmanager (routing + notifications)
#           |
#           +-----------+-----------+
#           |           |           |
#        [Slack]    [PagerDuty]  [Email]

The critical distinction is that Grafana Alerting evaluates alert rules itself, querying any configured datasource — not just Prometheus. It can fire alerts based on Loki log queries, Elasticsearch searches, CloudWatch metrics, PostgreSQL queries, or any of the 100+ datasource plugins available in Grafana. Rule definitions, contact points, notification policies, and mute timings are stored in the Grafana database (or provisioned via YAML files and the Grafana API).

Feature Comparison

Feature	Prometheus Alertmanager	Grafana Alerting
Datasources	Prometheus-compatible only (Prometheus, Thanos, Mimir, VictoriaMetrics)	Any Grafana datasource (Prometheus, Loki, Elasticsearch, CloudWatch, SQL databases, etc.)
Rule evaluation	External (Prometheus/Ruler evaluates rules and pushes alerts)	Built-in (Grafana evaluates rules directly)
Routing tree	Hierarchical YAML-based routing with match/match_re, continue, group_by	Notification policies with label matchers, nested policies, mute timings
Grouping	Full support via group_by, group_wait, group_interval	Full support via notification policies with equivalent controls
Inhibition	Native inhibition rules (suppress alerts when a related alert is firing)	Supported since Grafana 10.3 but less flexible than Alertmanager
Silencing	Label-based silences via API or UI, time-limited	Mute timings (recurring schedules) and silences (ad-hoc, label-based)
Notification channels	Email, Slack, PagerDuty, OpsGenie, VictoriaOps, webhook, WeChat, Telegram, SNS, Webex	All of the above plus Teams, Discord, Google Chat, LINE, Threema, Oncall, and more via contact points
Templating	Go templates in notification config	Go templates with access to Grafana template variables and functions
Multi-tenancy	Not built-in; achieved via separate instances or Mimir Alertmanager	Native multi-tenancy via Grafana organizations and RBAC
High availability	Gossip-based cluster (peer mesh, well-proven)	Database-backed HA with peer discovery between Grafana instances
Configuration model	Single YAML file, fully declarative	UI + API + provisioning YAML files, stored in database
GitOps compatibility	Excellent — config file lives in version control natively	Possible via provisioning files or Terraform provider, but requires extra tooling
External alert sources	Any system that can POST to the Alertmanager API	Supported via the Grafana Alerting API (external alerts can be pushed)
Managed service	Available via Grafana Cloud (as Mimir Alertmanager), Amazon Managed Prometheus	Available via Grafana Cloud

Alertmanager Strengths

Alertmanager has been a production staple since 2015. Over a decade of use across thousands of organizations has made it one of the most battle-tested components in the CNCF ecosystem.

Declarative, GitOps-Native Configuration

The entire Alertmanager configuration is a single YAML file. There is no hidden state in a database, no click-driven configuration that someone forgets to document. You check it into Git, review it in a pull request, and deploy it through your CI/CD pipeline like any other infrastructure code.

# alertmanager.yml — everything in one file
global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/T00/B00/XXX"

route:
  receiver: platform-team
  group_by: [alertname, cluster, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-oncall
      group_wait: 10s
    - match_re:
        team: "^(payments|checkout)$"
      receiver: payments-slack
      continue: true

receivers:
  - name: platform-team
    slack_configs:
      - channel: "#platform-alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: ""
  - name: payments-slack
    slack_configs:
      - channel: "#payments-oncall"

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [alertname, cluster]

Every change is auditable. Rollbacks are a git revert away. This matters enormously when you are debugging why an alert did not fire at 3 AM.

Lightweight and Single-Purpose

Alertmanager does one thing: route and deliver notifications. It has no dashboard, no query engine, no datasource plugins. This single-purpose design makes it operationally simple. Resource consumption is minimal — a small Alertmanager instance handles thousands of active alerts on a few hundred megabytes of memory. It starts in milliseconds and requires almost no maintenance.

Mature Inhibition and Routing

Alertmanager's inhibition rules are first-class citizens. You can suppress downstream warnings when a critical alert is already firing, preventing alert storms from overwhelming your on-call team. The hierarchical routing tree with continue flags allows for nuanced delivery: send to the team channel AND escalate to PagerDuty simultaneously, with different grouping strategies at each level.

Proven High Availability

The gossip-based HA cluster has been stable for years. Running three Alertmanager replicas behind a load balancer (or using Kubernetes service discovery) gives you reliable notification delivery without shared storage. The protocol handles deduplication across instances automatically, which is the hardest part of distributed alerting.

Grafana Alerting Strengths

Grafana Alerting has matured considerably since its rocky introduction in Grafana 8. By Grafana 11 and 12, it has become a legitimate production alerting platform with capabilities that Alertmanager cannot match on its own.

Multi-Datasource Alert Rules

This is Grafana Alerting's strongest differentiator. You can write alert rules that query Loki for error log spikes, CloudWatch for AWS resource utilization, Elasticsearch for application errors, or a PostgreSQL database for business metrics — all from the same alerting system. If your observability stack includes more than just Prometheus, this eliminates the need for separate alerting tools per datasource.

# Grafana alert rule provisioning example — alerting on Loki log errors
apiVersion: 1
groups:
  - orgId: 1
    name: application-errors
    folder: Production
    interval: 1m
    rules:
      - uid: loki-error-spike
        title: "High error rate in payment service"
        condition: C
        data:
          - refId: A\            datasourceUid: loki-prod
            model:
              expr: 'sum(rate({app="payment-service"} |= "ERROR" [5m]))'
          - refId: B
            datasourceUid: "__expr__"
            model:
              type: reduce
              expression: A
              reducer: last
          - refId: C
            datasourceUid: "__expr__"
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: gt
                    params: [10]
        for: 5m
        labels:
          severity: warning
          team: payments

This is something Alertmanager simply cannot do. Alertmanager only receives pre-evaluated alerts — it has no concept of datasources or query execution.

Unified UI for Alert Management

Grafana provides a single pane of glass for alert rule creation, visualization, notification policy management, contact point configuration, and silence management. For teams where not every engineer is comfortable editing YAML routing trees, the visual notification policy editor significantly reduces the barrier to entry. You can see the state of every alert rule, its evaluation history, and the exact notification path it will take — all without leaving the browser.

Native Multi-Tenancy and RBAC

Grafana's organization model and role-based access control extend naturally to alerting. Different teams can manage their own alert rules, contact points, and notification policies within their organization or folder scope, without seeing or interfering with other teams. Achieving this with standalone Alertmanager requires either running separate instances per tenant or using Mimir's multi-tenant Alertmanager.

Mute Timings and Richer Scheduling

While Alertmanager supports silences (ad-hoc, time-limited suppressions), Grafana Alerting adds mute timings — recurring time-based windows where notifications are suppressed. This is useful for scheduled maintenance windows, business-hours-only alerting, or suppressing non-critical alerts on weekends. Alertmanager requires external tooling or manual silence creation for recurring windows.

Grafana Cloud as a Managed Option

For teams that want to avoid managing alerting infrastructure entirely, Grafana Cloud provides a fully managed Grafana Alerting stack. This includes HA, state persistence, and notification delivery without any self-hosted components. The Grafana Cloud alerting stack also includes a managed Mimir Alertmanager, which means you can use Prometheus-native alerting rules if you prefer that model while still benefiting from the managed infrastructure.

When to Use Prometheus Alertmanager

Alertmanager is the right choice when the following conditions describe your environment:

Your metrics stack is Prometheus-native. If all your alert rules are PromQL expressions evaluated by Prometheus, Thanos Ruler, or Mimir Ruler, Alertmanager is the natural fit. There is no added value in routing those alerts through Grafana.
GitOps is non-negotiable. If every infrastructure change must go through a pull request and be fully declarative, Alertmanager's single-file configuration model is significantly easier to manage than Grafana's database-backed state. Tools like amtool provide config validation in CI pipelines.
You need fine-grained routing with inhibition. Complex routing trees with multiple levels of grouping, inhibition rules, and continue flags are more naturally expressed in Alertmanager's YAML format. The routing logic has been stable and well-documented for years.
You run microservices with per-team routing. If each team owns its routing subtree and the routing logic is complex, Alertmanager's hierarchical model scales better than UI-driven configuration. Teams can own their section of the config file via CODEOWNERS in Git.
You want minimal operational overhead. Alertmanager is a single binary with minimal resource requirements. There is no database to back up, no migrations to run, and no UI framework to keep updated.

When to Use Grafana Alerting

Grafana Alerting is the right choice when these conditions apply:

You alert on more than just Prometheus metrics. If you need alert rules based on Loki logs, Elasticsearch queries, CloudWatch metrics, or database queries, Grafana Alerting is the only option that handles all of these natively. The alternative is running separate alerting tools per datasource, which is worse.
Your team prefers UI-driven configuration. Not every engineer wants to edit YAML routing trees. If your organization values a visual interface for managing alerts, contact points, and notification policies, Grafana's UI is a major productivity advantage.
You are using Grafana Cloud. If you are already on Grafana Cloud, using its built-in alerting is the path of least resistance. You get HA, managed notification delivery, and a unified experience without running any additional infrastructure.
Multi-tenancy is a requirement. If multiple teams need isolated alerting configurations with RBAC, Grafana's native organization and folder-based access model is significantly easier to set up than running per-tenant Alertmanager instances.
You want mute timings for recurring maintenance windows. If your team regularly needs to suppress alerts during scheduled windows (deploy windows, batch processing hours, weekend non-critical suppression), Grafana's mute timings feature is more ergonomic than creating and managing recurring silences in Alertmanager.

Running Both Together: The Hybrid Pattern

In practice, many production environments run both Alertmanager and Grafana Alerting. This is not necessarily a mistake — it can be a deliberate architectural choice when done with clear boundaries.

Common Hybrid Architecture

The most common pattern looks like this:

Prometheus Alertmanager handles all metric-based alerts. PromQL rules are evaluated by Prometheus or a long-term storage ruler (Thanos, Mimir). Alertmanager owns routing, grouping, and notification for these alerts.
Grafana Alerting handles non-Prometheus alerts: log-based alerts from Loki, business metrics from SQL datasources, and cross-datasource correlation rules.

The key to making this work without chaos is establishing clear ownership rules:

# Ownership boundaries for hybrid alerting
#
# Prometheus Alertmanager owns:
#   - All PromQL-based alert rules
#   - Infrastructure alerts (node, kubelet, etcd, CoreDNS)
#   - Application SLO/SLI alerts based on metrics
#
# Grafana Alerting owns:
#   - Log-based alert rules (Loki, Elasticsearch)
#   - Business metric alerts (SQL datasources)
#   - Cross-datasource correlation rules
#   - Alerts for teams that prefer UI-driven management
#
# Shared:
#   - Contact points / receivers use the same Slack channels and PagerDuty services
#   - On-call rotations are managed externally (PagerDuty, Grafana OnCall)

Both systems can deliver to the same notification channels. The critical discipline is ensuring that silencing and maintenance windows are applied in both systems when needed. This is the primary operational cost of the hybrid approach.

Grafana as a Viewer for Alertmanager

Even if you use Alertmanager exclusively for routing and notification, Grafana can serve as a read-only viewer. Grafana natively supports connecting to an external Alertmanager datasource, allowing you to see firing alerts, active silences, and alert groups in the Grafana UI. This gives you the operational visibility of Grafana without moving your alerting logic into it.

# Grafana datasource provisioning for external Alertmanager
apiVersion: 1
datasources:
  - name: Alertmanager
    type: alertmanager
    url: http://alertmanager.monitoring.svc:9093
    access: proxy
    jsonData:
      implementation: prometheus

Migration Considerations

If you are moving from one system to the other, here are the practical considerations to plan for.

Migrating from Alertmanager to Grafana Alerting

Rule conversion. Your PromQL-based recording and alerting rules defined in Prometheus rule files need to be recreated as Grafana alert rules. Grafana provides a migration tool that can import Prometheus-format rules, but complex expressions may need manual adjustment.
Routing tree translation. Alertmanager's hierarchical routing tree maps to Grafana's notification policies, but the semantics are not identical. Test the notification routing thoroughly — the continue flag behavior and default routes may differ.
Silence and inhibition migration. Active silences are ephemeral and do not need migration. Inhibition rules need to be recreated in Grafana's format. Recurring maintenance windows should be converted to mute timings.
Run in parallel first. The safest migration strategy is to run both systems in parallel for two to four weeks, sending notifications from both, then cutting over when you have confidence in the Grafana setup. Accept the temporary noise of duplicate alerts — it is far cheaper than missing a critical page during migration.

Migrating from Grafana Alerting to Alertmanager

Datasource limitation. You can only migrate alerts that are based on Prometheus-compatible datasources. Alerts querying Loki, Elasticsearch, or SQL datasources have no equivalent in Alertmanager — you will need an alternative solution for those.
Rule export. Export Grafana alert rules and convert them to Prometheus-format rule files. The Grafana API (GET /api/v1/provisioning/alert-rules) provides structured output that can be transformed with a script.
Contact point mapping. Map Grafana contact points to Alertmanager receivers. The configuration format is different, but the concepts are equivalent.
State loss. Alertmanager does not carry over Grafana's alert evaluation history. You start fresh. Plan for a brief period where alerts may re-fire as Prometheus evaluates rules that were previously managed by Grafana.

Decision Framework

If you want a quick decision path, use this framework:

Start here:
│
├── Do you alert on non-Prometheus datasources (Loki, ES, SQL, CloudWatch)?
│   ├── YES → Grafana Alerting (at least for those datasources)
│   └── NO ↓
│
├── Is GitOps/declarative config a hard requirement?
│   ├── YES → Alertmanager
│   └── NO ↓
│
├── Do you need multi-tenancy with RBAC?
│   ├── YES → Grafana Alerting (or Mimir Alertmanager)
│   └── NO ↓
│
├── Are you on Grafana Cloud?
│   ├── YES → Grafana Alerting (path of least resistance)
│   └── NO ↓
│
└── Default → Alertmanager (simpler, lighter, well-proven)

For many teams, the honest answer is "both" — Alertmanager for the Prometheus-native metric pipeline, Grafana Alerting for everything else. That is a valid architecture as long as the ownership boundaries are documented and the on-call team knows where to look.

Frequently Asked Questions

What is the difference between Alertmanager and Grafana Alerting?

Prometheus Alertmanager is a standalone notification routing engine that receives pre-evaluated alerts from Prometheus and delivers them to receivers like Slack, PagerDuty, or email. Grafana Alerting is an integrated alerting platform embedded in Grafana that both evaluates alert rules and handles notification routing. Alertmanager is configured entirely via YAML, while Grafana Alerting offers a UI, API, and file-based provisioning. The fundamental difference is scope: Alertmanager handles only the routing and notification phase, while Grafana Alerting handles the full lifecycle from query evaluation to notification.

Can Grafana Alerting replace Prometheus Alertmanager?

Yes, for many use cases. Grafana Alerting can evaluate PromQL rules directly against your Prometheus datasource, so you do not strictly need a separate Alertmanager instance. However, there are scenarios where Alertmanager remains the better choice: heavily GitOps-driven environments, teams that need Alertmanager's mature inhibition rules, or architectures where Prometheus rule evaluation happens externally (Thanos Ruler, Mimir Ruler) and a dedicated Alertmanager is already in the pipeline. If your only datasource is Prometheus and you value declarative configuration, Alertmanager is still simpler and lighter.

Is Grafana Alertmanager the same as Prometheus Alertmanager?

Not exactly. Grafana Alerting uses a fork of the Prometheus Alertmanager code internally for its notification routing engine, but it is not the same product. The Grafana "Alertmanager" visible in the UI is a managed, embedded component with a different configuration interface (notification policies, contact points, mute timings) compared to the standalone Prometheus Alertmanager (routing tree, receivers, inhibition rules in YAML). Grafana can also connect to an external Prometheus Alertmanager as a datasource, which adds to the confusion.

What are the best alternatives to Prometheus Alertmanager?

The most direct alternative is Grafana Alerting, which can receive and route Prometheus alerts while also supporting other datasources. Beyond that: Grafana OnCall for on-call management and escalation, PagerDuty or Opsgenie as managed incident response platforms, Keep as an open-source AIOps alert management platform, and Mimir Alertmanager for multi-tenant environments running Grafana Mimir.

Should I use Prometheus alerts or Grafana alerts for Kubernetes monitoring?

For Kubernetes monitoring specifically, the kube-prometheus-stack (which includes Prometheus, Alertmanager, and a comprehensive set of pre-built alerting rules) remains the industry standard. These rules are PromQL-based and are designed to work with Alertmanager. If you are deploying kube-prometheus-stack, using Alertmanager for metric-based alerts is the straightforward choice. Add Grafana Alerting on top if you also need to alert on logs (via Loki) or non-metric datasources.

Final Thoughts

The Alertmanager vs Grafana Alerting debate is not really about which tool is better — it is about which tool fits your operational context. Alertmanager is simpler, lighter, and more GitOps-friendly. Grafana Alerting is more versatile, more accessible to UI-oriented teams, and the only option if you need multi-datasource alerting. Running both is perfectly valid when the boundaries are clear.

The worst outcome is not picking the "wrong" tool. The worst outcome is running both accidentally, with overlapping coverage, duplicated notifications, and no clear ownership. Whatever you choose, document the decision, define the ownership boundaries, and make sure your on-call team knows exactly where to go when they need to silence an alert at 3 AM.

Originally published at alexandre-vazquez.com/alertmanager-vs-grafana-alerting

Prometheus Alertmanager Vs Grafana Alerting (2026): Architecture, Features, And When To Use Each

Alexandre Vazquez — Tue, 05 May 2026 10:00:00 +0000

Prometheus Alertmanager Vs Grafana Alerting (2026): Architecture, Features, And When To Use Each

Originally published at alexandre-vazquez.com

Read the full article on my blog: https://alexandre-vazquez.com/alertmanager-vs-grafana-alerting/

Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each

Alexandre Vazquez — Tue, 05 May 2026 08:00:01 +0000

Originally published at alexandre-vazquez.com/debugging-distroless-containers/

The container works fine in CI. It deploys successfully to staging. Then something goes wrong in production and you type the command you always type: kubectl exec -it my-pod -- /bin/bash. The response is immediate: OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory.

You try /bin/sh. Same error. You try ls. Same error. The container image is distroless — it ships only your application binary and its runtime dependencies, with no shell, no package manager, no debugging tools of any kind. This is intentional and correct from a security standpoint. It is also a significant operational challenge the first time you face it in production.

This article covers every practical technique for debugging distroless containers in Kubernetes: kubectl debug with ephemeral containers (the standard approach), pod copy strategy (for Kubernetes versions without ephemeral container support, or when you need to modify the running pod spec), debug image variants (the pragmatic developer shortcut), cdebug (a purpose-built tool that simplifies the process), and node-level debugging (the last resort with the most power). For each technique I will explain what it can and cannot do, what Kubernetes version or RBAC permissions it requires, and in which scenario — developer in local, platform engineer in staging, ops in production — it is the appropriate choice.

Why Distroless Breaks the Normal Debugging Workflow

Traditional container debugging assumes you can exec into the container and use shell tools: ps, netstat, strace, curl, a text editor. Distroless images remove all of this by design. The Google distroless project, Chainguard's Wolfi-based images, and the broader minimal image ecosystem deliberately exclude everything that is not required to run the application. The result is a dramatically smaller attack surface: no shell means no RCE via shell injection, no package manager means no easy escalation path, fewer binaries means fewer CVEs in the image scan.

The tradeoff is operational: when something goes wrong, you cannot use the tools that the process itself is not allowed to run. A Java application in gcr.io/distroless/java17-debian12 has the JRE and nothing else. A Go binary compiled with CGO disabled and shipped in gcr.io/distroless/static-debian12 has literally only the binary and the necessary CA certificates and timezone data. There is no wget to download a debug binary, no apt to install one, no bash to run a script.

Kubernetes solves this at the platform level with ephemeral containers , added as stable in Kubernetes 1.25. The principle is that a debug container — which can have a full shell and any tools you want — can be injected into a running pod and share its process namespace, network namespace, and filesystem mounts without modifying the original container or restarting the pod.

Option 1: kubectl debug with Ephemeral Containers

Ephemeral containers are the canonical solution. Since Kubernetes 1.25 (stable), kubectl debug can inject a temporary container into a running pod. The container shares the target pod's network namespace by default, and with --target it can also share the process namespace of a specific container, allowing you to inspect its running processes and open file descriptors.

The basic invocation is:

kubectl debug -it my-pod \
  --image=busybox:latest \
  --target=my-container

The --target flag is the critical piece. Without it, the ephemeral container gets its own process namespace. With it, it shares the process namespace of the specified container — meaning you can run ps aux and see the application's processes, use ls -la /proc//fd to inspect open file descriptors, and read the application's environment via cat /proc//environ.

For a more capable debug environment, replace busybox with a richer image:

kubectl debug -it my-pod \
  --image=nicolaka/netshoot \
  --target=my-container

nicolaka/netshoot includes tcpdump, curl, dig, nmap, ss, iperf3, and dozens of other network diagnostic tools, making it the standard choice for network debugging scenarios.

What You Can and Cannot Do

Ephemeral containers share the pod's network namespace and, when --target is used, the process namespace. This gives you:

Full visibility into the application's network traffic from inside the pod (tcpdump, ss, netstat)
Process inspection via /proc/ — open files, memory maps, environment variables, CPU/memory usage
Access to the pod's DNS resolution context — exactly the same /etc/resolv.conf the application sees
Ability to make outbound network calls from the same network namespace (testing service endpoints, DNS resolution)

What you do not get with ephemeral containers:

Access to the application container 's filesystem. The ephemeral container has its own root filesystem. You cannot cat /app/config.yaml from the application container's filesystem unless you access it via /proc//root/.
Ability to remove the container once added. Ephemeral containers are permanent until the pod is deleted. This is by design — the Kubernetes API does not allow removing them after creation.
Volume mount modifications via CLI. You cannot add volume mounts to an ephemeral container via kubectl debug (though the API spec supports it, the CLI does not expose this).
Resource limits. Ephemeral containers do not support resource requests and limits in the kubectl debug CLI, though this is evolving.

Accessing the Application Filesystem

The most common surprise for developers new to ephemeral containers is that they cannot directly browse the application container's filesystem. The workaround is the /proc filesystem:

# Find the application's PID
ps aux

# Browse its filesystem via /proc
ls /proc/1/root/app/
cat /proc/1/root/etc/config.yaml

# Or set the root to the application's root
chroot /proc/1/root /bin/sh  # only if /bin/sh exists in the app image

The /proc//root path is a symlink to the container's root filesystem as seen from the process namespace. Because the ephemeral container shares the process namespace with --target, the application's PID is typically 1, and /proc/1/root gives you full read access to its filesystem.

RBAC Requirements

Ephemeral containers require the pods/ephemeralcontainers subresource permission. This is separate from pods/exec, which controls kubectl exec. A common mistake is to grant pods/exec for debugging purposes without realizing that ephemeral containers require an additional grant:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ephemeral-debugger
rules:
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

In production environments, this permission should be tightly scoped: time-limited via RoleBinding rather than permanent ClusterRoleBinding, restricted to specific namespaces, and ideally gated behind an approval workflow. The debug container runs as root by default, which can create privilege escalation paths if the application container runs as a non-root user with shared process namespace — the debug container can attach to the application's processes with higher privileges.

Option 2: kubectl debug -copy-to (Pod Copy Strategy)

When you need to modify the pod's container spec — replace the image, change environment variables, add a sidecar with a shared filesystem — the --copy-to flag creates a full copy of the pod with your modifications applied:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=my-app:debug \
  --share-processes

This creates a new pod named my-pod-debug that is a copy of my-pod but with the container image replaced by my-app:debug. If my-app:debug is your application image built with debug tooling included (or a debug variant from your registry), this lets you interact with the exact same binary in the exact same configuration as the original pod.

A more common use of --copy-to is to attach a debug container alongside the existing application container while keeping the original image unchanged:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  --container=debugger

This creates the copy-pod with both the original containers and a new debugger container sharing the process namespace. Unlike ephemeral containers, this approach supports volume mounts and resource limits, and the debug pod can be deleted cleanly when you are done.

Limitations of the Copy Strategy

The pod copy approach has a critical limitation: it is not debugging the original pod. It creates a new pod that may behave differently because:

It does not share the original pod's in-memory state — if the issue is a goroutine leak or heap corruption that has been accumulating for hours, the fresh copy will not exhibit it immediately
It creates a new Pod UID, which means any admission webhooks, network policies, or pod-level security contexts that depend on pod identity may apply differently
If the original pod is crashing (CrashLoopBackOff), the copy will also crash — this technique does not help for crash debugging unless you also change the entrypoint

For crash debugging specifically, combine --copy-to with a modified entrypoint to keep the container alive:

kubectl debug my-crashing-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  -- sleep 3600

Option 3: Debug Image Variants

The most pragmatic approach — and the one most appropriate for developer workflows — is to maintain a debug variant of your application image that includes shell tooling. Both the Google distroless project and Chainguard provide this pattern officially.

Google distroless images have a :debug tag that adds BusyBox to the image:

# Production image
FROM gcr.io/distroless/java17-debian12

# Debug variant — identical but with BusyBox shell
FROM gcr.io/distroless/java17-debian12:debug

Chainguard images follow a similar convention with :latest-dev variants that include apk, a shell, and common utilities:

# Production (zero shell, minimal footprint)
FROM cgr.dev/chainguard/go:latest

# Development/debug variant
FROM cgr.dev/chainguard/go:latest-dev

If you build your own base images, the recommended approach is to use multi-stage builds and maintain separate build targets:

FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp .

# Production: static distroless image
FROM gcr.io/distroless/static-debian12 AS production
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

# Debug variant: same binary, with shell tools
FROM gcr.io/distroless/static-debian12:debug AS debug
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

In your CI/CD pipeline, build both targets and push my-app:${VERSION} (production) and my-app:${VERSION}-debug (debug variant) to your registry. The debug image is never deployed to production by default, but it exists and is ready to be used with kubectl debug --copy-to when needed.

Security Considerations for Debug Variants

Debug image variants defeat much of the security benefit of distroless if they are used in production, even temporarily. Track usage carefully: log when debug images are deployed, require explicit approval, and ensure they are removed after the debugging session. In regulated environments, consider whether deploying a debug variant to production namespaces is permitted by your security policy — in many cases it is not, and you must use ephemeral containers (which add a debug process to the pod without modifying the application image) instead.

Option 4: cdebug

cdebug is an open-source CLI tool that simplifies distroless debugging by wrapping kubectl debug with more ergonomic defaults and additional capabilities. Its primary value is in making ephemeral container debugging feel like a native shell experience:

# Install
brew install cdebug
# or: go install github.com/iximiuz/cdebug@latest

# Debug a running pod
cdebug exec -it my-pod

# Specify a namespace and container
cdebug exec -it -n production my-pod -c my-container

# Use a specific debug image
cdebug exec -it my-pod --image=nicolaka/netshoot

What cdebug adds over raw kubectl debug:

Automatic filesystem chroot. cdebug exec automatically sets the filesystem root of the debug container to the target container's filesystem, so you browse / and see the application's files — not the debug image's files. This addresses the most common friction point with kubectl debug.
Docker integration. cdebug exec works identically for Docker containers (cdebug exec -it), making it the same muscle memory for local and cluster debugging.
No RBAC complications for Docker-based local development — useful for developer workflows before the code reaches Kubernetes.

The tradeoff: cdebug is a third-party dependency and requires installation. In environments with strict tooling policies (regulated industries, air-gapped clusters), it may not be an option. In those cases, the raw kubectl debug workflow with /proc/1/root filesystem navigation is the baseline.

Option 5: Node-Level Debugging

When everything else fails — the pod is in CrashLoopBackOff too fast to attach to, the issue is a kernel-level problem, or you need tools like strace that require elevated privileges — node-level debugging gives you direct access to the container's processes from the host node.

kubectl debug node/ creates a privileged pod on the target node that mounts the node's root filesystem under /host:

kubectl debug node/my-node-name \
  -it \
  --image=nicolaka/netshoot

From this privileged pod, you can use nsenter to enter the namespaces of any container running on the node:

# Find the container's PID on the node
# (from within the node debug pod)
crictl ps | grep my-container
crictl inspect  | grep pid

# Enter the container's namespaces
nsenter -t  -m -u -i -n -p -- /bin/sh

# Or just the network namespace (for network debugging)
nsenter -t  -n -- ip a

The nsenter approach lets you run tools from the node's or debug container's toolset while operating in the namespaces of the target container. This is how you run strace against a distroless process: strace is not in the application container, but you can run it from the node level while targeting the application's PID.

# Trace all syscalls from the application process
nsenter -t  -- strace -p  -f -e trace=network

RBAC and Security for Node Debugging

Node-level debugging requires nodes/proxy and the ability to create privileged pods, which in most production clusters is restricted to cluster administrators. The debug pod runs with hostPID: true and hostNetwork: true, giving it visibility into all processes and network traffic on the node — not just the target container. This is significant: every process running on the node, including those in other tenants' namespaces, is visible.

This technique should be treated as a break-glass procedure: log the access, require dual approval in production environments, and clean up immediately after the debugging session with kubectl delete pod --selector=app=node-debugger.

Choosing the Right Approach: Access Profile and Environment Matrix

The technique you should use depends on two axes: who you are (developer, platform engineer, ops/SRE) and where the issue is (local development, staging, production). The requirements and constraints differ significantly across these combinations.

Developer — Local or Development Cluster

Goal: Reproduce and understand a bug, inspect configuration, verify network connectivity to services.

Constraints: None material — full cluster admin on local or personal dev namespace.

Recommended approach: Debug image variants or cdebug.

In local development (Minikube, Kind, Docker Desktop), the fastest path is to build the debug variant of your image and deploy it directly. If you are working with another team's service, cdebug exec gives you a shell in the container with automatic filesystem root without any special RBAC. The goal is speed and iteration — reserve the more structured approaches for higher environments.

Developer — Staging Cluster

Goal: Debug integration issues, inspect live configuration, verify environment-specific behavior.

Constraints: Shared cluster — cannot deploy arbitrary workloads to other teams' namespaces, but has pods/ephemeralcontainers in own namespace.

Recommended approach: kubectl debug with ephemeral containers (--target), scoped to own namespace.

Staging is where ephemeral containers earn their keep. You can attach to a running pod without restarting it, without modifying the deployment spec, and without affecting other users of the same cluster. Grant developers pods/ephemeralcontainers in their team's namespaces and they can self-service debug without needing ops involvement.

Platform Engineer / SRE — Production

Goal: Diagnose a live production incident. The pod is behaving unexpectedly — high latency, memory growth, unexpected connections, incorrect responses.

Constraints: Changes to running pods are high-risk. Any debug image deployment must be gated. The issue is live and affecting users.

Recommended approach: kubectl debug with ephemeral containers (ephemeral containers do not restart the pod, do not modify the deployment, and are auditable via API audit logs).

The key production requirements are auditability and minimal blast radius. Ephemeral containers satisfy both: they are recorded in the Kubernetes API audit log (who attached, when, to which pod), they do not modify the running application container, and they are limited to the pod's own network and process namespaces. Document the debug session in your incident ticket: pod name, time, what was observed, who ran the debug container.

The --copy-to strategy is generally inappropriate for production incident response: it creates a new pod that may or may not exhibit the issue, it adds load to the cluster during an incident, and if it is attached to the same services (databases, downstream APIs), it produces additional traffic that complicates forensics.

Platform Engineer — Production, Node-Level Issue

Goal: Diagnose a kernel-level issue, a container runtime problem, a networking issue that spans multiple pods, or a situation where the pod is crashing too fast to attach to.

Constraints: Maximum privilege required. High operational risk.

Recommended approach: Node-level debug pod with nsenter. Treat as break-glass.

For this scenario, create a dedicated RBAC role that grants nodes/proxy access and the ability to create pods with hostPID: true in a dedicated debug namespace. Bind it only to specific users, require a separate authentication step (e.g., kubectl auth can-i check against a time-limited binding), and log all access. This level of access should generate a PagerDuty-style alert so that the security team knows a privileged debug session is active in production.

Common Errors and Solutions

Error: "ephemeral containers are disabled for this cluster"

Ephemeral containers require Kubernetes 1.16+ (alpha, behind feature gate) and are stable from 1.25. If you are on 1.16–1.22, you need to enable the EphemeralContainers feature gate on the API server and kubelet. From 1.23 it was beta and enabled by default. From 1.25 it is stable and always on. On managed Kubernetes services (EKS, GKE, AKS), check the cluster version — versions older than 1.25 may still have it disabled depending on your configuration.

Error: "cannot update ephemeralcontainers" (RBAC)

You have pods/exec but not pods/ephemeralcontainers. Add the grant shown in the RBAC section above. Note that pods/exec and pods/ephemeralcontainers are separate subresources — having one does not imply the other.

Error: "container not found" with -target

The container name in --target must match exactly the container name as defined in the Pod spec — not the image name. Check with kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}' to get the exact container names.

Error: Can see processes but cannot read /proc/1/root

The application container runs as a non-root user (e.g., UID 1000) and the ephemeral container runs as root. The application's filesystem may have files owned by UID 1000 that are not readable by other UIDs depending on permissions. The /proc//root path itself requires CAP_SYS_PTRACE capability. If your cluster's PodSecurityStandards (PSS) are set to restricted, the debug container may not have this capability. Use the Baseline PSS profile for debug namespaces or explicitly add SYS_PTRACE to the ephemeral container's securityContext.

Error: tcpdump shows no traffic

When using nicolaka/netshoot for network debugging, ensure the ephemeral container is created without --target if your goal is to capture all traffic on the pod's network interface (not just the specific container's process). With --target, you share the process namespace but the network namespace is shared at the pod level regardless. Run tcpdump -i any to capture on all interfaces including loopback, which is where inter-container traffic within a pod travels.

Decision Framework

Use this as a starting point to select the right technique for your situation:

Scenario	Technique	Requirement
Active production incident, pod running	kubectl debug + ephemeral container	pods/ephemeralcontainers RBAC, k8s 1.25+
Pod crashing too fast to attach	kubectl debug -copy-to + modified entrypoint	Ability to create pods in namespace
Developer debugging in dev/staging	cdebug exec or kubectl debug	pods/ephemeralcontainers or pod create
Need full filesystem access	kubectl debug -copy-to + debug image variant	Debug image in registry, pod create
Need strace or kernel tracing	Node-level debug with nsenter	nodes/proxy, cluster admin equivalent
Network packet capture	kubectl debug + nicolaka/netshoot	pods/ephemeralcontainers
Local Docker debugging	cdebug exec	Docker socket access
CI-reproducible debug environment	Debug image variant in separate build target	Separate image tag in registry

Production RBAC Design

A clean RBAC design for production distroless debugging separates three roles with different privilege levels:

# Tier 1: Developer self-service in team namespaces
# Allows attaching ephemeral containers, no node access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: distroless-debugger
  namespace: team-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 2: SRE production incident access
# Ephemeral containers across all namespaces
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sre-distroless-debugger
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 3: Break-glass node access
# Only for platform team, time-limited binding recommended
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-debugger
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "get", "list", "delete"]
  # Restrict to debug namespace via RoleBinding, not ClusterRoleBinding

Bind Tier 1 permanently to your developers. Bind Tier 2 to SREs permanently but with audit alerts on use. Bind Tier 3 only on-demand (via a Kubernetes operator that creates time-limited RoleBindings) and never as a permanent ClusterRoleBinding.

Summary

Distroless containers are the correct choice for production workloads. They reduce attack surface, eliminate unnecessary CVEs, and force a cleaner separation between application and tooling. The operational cost is that your traditional debugging workflow — exec into the container, run some commands — no longer works by default.

Kubernetes provides a clean answer with ephemeral containers and kubectl debug: inject a debug container with whatever tools you need into the running pod, sharing its network and process namespaces, without restarting or modifying the application. For scenarios where ephemeral containers are insufficient — filesystem access, crash debugging, kernel-level investigation — the copy strategy and node-level debug fill the remaining gaps.

The key to making this work at scale is not the technique itself but the access model : developers get self-service ephemeral container access in their own namespaces, SREs get cluster-wide ephemeral container access for production incidents, and node-level access is a break-glass procedure with audit trail and time limits. With that model in place, distroless becomes an operational non-issue rather than an obstacle.

DEV Community: Alexandre Vazquez

Skopeo, Crane, and regctl: Container Image Management Without the Docker Daemon (2026)

The Problem: Docker Is Overkill for Image Operations

The Contenders

Skopeo

What Skopeo does well

Skopeo’s weaknesses

When to use Skopeo

Crane

What Crane does well

Crane’s weaknesses

When to use Crane

regctl

What regctl does uniquely well

Regctl’s weaknesses

When to use regctl

ORAS

cosign

Side-by-side comparison

Practical workflows

Mirror images for air-gapped clusters (Skopeo)

Pin image digests in CI (Crane)

Retag without re-pushing (Crane or regctl)

Add OCI annotations post-build (regctl)

Supply chain security pipeline

Installation

Which tool should you use?

FAQ

Related articles:

Transforming XML to JSON and CSV with XSLT

Designing XSLT transforms with parameters and multiple inputs

XSLT performance tuning without losing readability

XSLT debugging patterns that save hours

XSLT string functions: complete reference with examples

Basic string functions (XSLT 1.0+)

string-length

substring

substring-before and substring-after

contains, starts-with, ends-with

concat

normalize-space

translate

Advanced string functions (XSLT 2.0+)

upper-case and lower-case

replace

matches

tokenize

string-join

format-number

format-date and format-dateTime

Practical patterns

XSLT grouping with xsl:for-each-group: complete guide

Basic grouping with group-by

Nested grouping

group-adjacent

group-starting-with and group-ending-with

Computing aggregates

Try it in XSLT Playground

Radar: A New Kubernetes IDE Worth Knowing About (vs OpenLens, FreeLens)

The State of Kubernetes Desktop Tooling in 2026

What Radar Actually Is

Key Features

Topology View

Persistent Event Timeline

GitOps Integration (ArgoCD + Flux)

Helm Management

Image Filesystem

MCP Server (AI Integration)

Cluster Audit

Multi-Cluster Support (Cloud)

Architecture: Why a Go Binary Matters

Feature Comparison

When Radar Makes Sense

When OpenLens or FreeLens Still Makes Sense

Getting Started

Verdict

XSLT template matching explained with examples

How match patterns work

Priority and conflict resolution

The built-in templates