DEV Community: Alexandre Vazquez

Transforming XML to JSON and CSV with XSLT

Alexandre Vazquez — Thu, 28 May 2026 09:00:02 +0000

XSLT is usually associated with XML-to-XML transformations, but in integration work you often need JSON or CSV. The good news is that XSLT is perfectly capable of producing non-XML outputs when you design the stylesheet for it. The key is to choose the right output method, control whitespace carefully, and build an intermediate structure if it helps clarify the mapping. This post covers practical patterns for generating JSON and CSV from XML while keeping the stylesheet maintainable.

For JSON, the simplest method is to output text and build the JSON structure manually. This gives you precise control, but it also requires careful escaping and formatting. If you are on XSLT 3.0, use maps and arrays and let the processor serialize to JSON. This reduces string manipulation and makes your transform more robust. If you are on XSLT 1.0 or 2.0, you can still build JSON text safely by using templates that escape quotes, backslashes, and control characters.

A clear pattern is to create a template that takes a string and outputs an escaped JSON string. Then, for each object, output the property names and values with explicit commas. Keep a template to handle comma placement so you do not end up with trailing commas in arrays. This is a good place to use position checks like position() != last() to decide when to emit a comma. While it can look verbose, the logic is deterministic and easy to debug.

CSV output is simpler but comes with its own hazards. You need to wrap fields that contain commas, quotes, or line breaks. The common rule is to wrap the field in quotes and double any interior quotes. Again, a dedicated template to escape fields pays off. Define the column order explicitly and avoid depending on the source document order. This keeps the CSV consistent even if the XML input changes slightly. If you need multiple CSV sections, consider running two passes: one to compute the rows and one to serialize them.

An example CSV field template can look like this in XSLT 1.0:

xml

Designing XSLT transforms with parameters and multiple inputs

Alexandre Vazquez — Mon, 25 May 2026 09:00:00 +0000

Many real-world transformations do not run on a single XML document. You often merge a primary payload with reference data, catalog lookups, or environment configuration. Done well, this results in a clean, predictable transform. Done poorly, it becomes a maze of document() calls and hidden dependencies. The difference is in how you model inputs and parameters from the start. As an integration engineer, I treat input selection and parameter design as first-class API design for the stylesheet.

Start by naming every input. Instead of embedding document('config.xml') in multiple templates, load each external document once near the top of the stylesheet and bind it to a global variable. This makes dependencies explicit and keeps the rest of the code focused on mapping. It also helps with testing, because you can override the URI with a parameter. A clean pattern is to define xsl:param values for input URIs and then bind them to xsl:variable values that hold the parsed documents.

The same clarity applies to parameters. Keep parameters primitive and predictable, and avoid passing in node sets unless you truly need them. A parameter should be an external knob: region, language, a feature flag, or an output format. If you have a complex decision tree, consider using a lookup XML or JSON input and then query it inside the stylesheet. This approach keeps the invocation interface stable while still letting you evolve business rules.

A simple skeleton might look like this:

From there, templates can reference $catalog without worrying about IO or base URIs. You can also define a named template that accepts a parameter for reuse across multiple modes. This is useful when the same output block is needed for several sections of the document but the selection context differs.

When combining multiple inputs, always anchor your lookups to a clear key. If you can, define xsl:key on the external document so lookups are efficient and readable. In XSLT 2.0 or 3.0, xsl:for-each-group and the map types can reduce boilerplate, but the core idea remains: make your joins explicit and deterministic. If you rely on default order or on undocumented assumptions about uniqueness, you will eventually get a hard-to-reproduce bug.

Another important integration pattern is separating parsing from formatting. For example, you might normalize all values from the various inputs into a canonical intermediate structure and then render that structure into the final output. This makes testing easier and supports future outputs such as CSV, JSON, or a secondary XML format. Even in XSLT 1.0, you can emulate this by creating result tree fragments, then processing them in a second pass if needed.

Multiple inputs also raise questions about fallbacks. Decide how you want to behave when optional data is missing. I prefer to centralize defaults in a few named templates or functions and avoid sprinkling xsl:choose blocks everywhere. This keeps the stylesheet readable and makes it obvious how to override the defaults later. Document your fallbacks in the code with short, clear names so a future maintainer does not have to rediscover the rules by reading the entire stylesheet.

Finally, create a small set of inputs that represent common scenarios and run them regularly. For example, have a baseline case, a case with missing reference data, and a case with unexpected elements. These are the cases that reveal poor assumptions about inputs. A fast way to iterate on these scenarios is to run the transform with a tool that lets you swap inputs and parameters quickly.

If you want to try these patterns with real inputs and multiple documents, the online editor at https://xsltplayground.com is built for that workflow. It lets you load multiple XML documents and parameters, see how they interact, and keep your integration logic transparent as it grows.

XSLT performance tuning without losing readability

Alexandre Vazquez — Thu, 21 May 2026 09:00:01 +0000

Performance problems in XSLT are sneaky. The stylesheet looks clean, the output is correct, but the transform slows down as the input grows. Most of the time this is caused by expensive selections that are repeated in loops, or by deep // searches that scan the entire tree more often than you expect. The good news is that you can usually fix these issues without turning the stylesheet into unreadable micro-optimizations.

The first step is to examine where you are traversing the document. XSLT processors are optimized for template matching, so prefer xsl:apply-templates and specific match patterns over xsl:for-each with // in the select. When you do need a search, limit it to the smallest possible subtree. A single // at the top-level becomes a full-tree scan each time it runs. If it runs inside another loop, the cost can explode.

Keys are the most important performance feature, and they also improve readability. When you define an xsl:key, you turn a repeated search into a fast lookup. This is especially critical for join-like operations where you match a reference value to another document or a secondary section of the same document. Build the key once, and then use key('id', $value) everywhere. The intent becomes clear: you are doing a lookup, not a scan. If you only use keys occasionally, it can feel like overkill, but it is often the biggest win.

Modes are another useful tool. If you use the same templates in multiple contexts, you may end up doing extra work or firing templates that you do not need. A dedicated mode lets you create a focused processing pipeline that touches only the nodes relevant to that output section. This can reduce both runtime and mental overhead. It also makes it easier to reason about precedence: within a mode, you can define more specific templates without worrying about side effects on unrelated parts of the transform.

Consider caching computed values in variables. XSLT variables are immutable, so they are safe to reuse without unintended side effects. If you are computing a complex string or a filtered node set repeatedly, store it once per relevant scope. Just be careful not to define a variable at the top of the stylesheet if it depends on the context; keep it as close as possible to where it is used to avoid confusion.

If you are working in XSLT 2.0 or 3.0, you gain access to xsl:for-each-group and higher-order functions. These can be faster and clearer than manual grouping with keys. For XSLT 1.0, the Muenchian grouping pattern is still effective, and when combined with keys it remains a strong choice. Either way, focus on minimizing passes over large node sets.

Also consider the output method. Serializing large outputs can be a significant part of the runtime. If you do not need pretty-printed XML, avoid indentation to reduce the amount of whitespace and processing time. Similarly, if you are generating text or JSON, use method="text" or structured XSLT 3.0 serialization options rather than building a text string node by node.

I recommend using realistic test data when tuning performance. A transform that runs in 100 milliseconds on a tiny input may take seconds on real data. Use a handful of real documents and measure changes as you apply each improvement. This keeps the optimization process grounded and prevents you from making the code worse without a measurable gain.

Finally, keep a balance between speed and clarity. The fastest stylesheet is useless if it is too hard to maintain. Use a few consistent patterns: keys for lookups, modes for pipelines, variables for repeated values, and limited selection scope. With those in place, the performance usually becomes acceptable without heroics.

If you want a quick way to benchmark different approaches with the same input set, try the online editor at https://xsltplayground.com. It is a convenient place to experiment with keys, modes, and alternative match patterns while keeping your transform readable.

XSLT debugging patterns that save hours

Alexandre Vazquez — Mon, 18 May 2026 09:00:00 +0000

XSLT bugs are rarely loud. More often, a template silently matches the wrong node, a predicate filters out a value you needed, or a namespace mismatch turns an element into a ghost. The fastest fix comes from a repeatable debugging workflow that keeps your assumptions visible. Over time you learn the same patterns appear in almost every real project, whether you are cleansing XML feeds, integrating partner payloads, or generating documents. This post walks through the techniques I use as an integration engineer to debug transforms quickly without losing context.

Start by making the matching rules obvious. The majority of issues are caused by using // too freely or relying on default namespaces. Replace broad paths with anchored ones, and when in doubt, print out what the processor thinks the current node is. A simple xsl:message combined with name() and namespace-uri() can reveal a namespace mismatch in seconds. I also add short, temporary templates that match the suspected nodes and output minimal text, which is a fast way to confirm whether the selection is correct.

Next, isolate the failing region by reducing input size. You rarely need the entire input document to debug a single mapping. Extract the smallest fragment that reproduces the issue and run the transform against that. This lets you simplify predicates and remove unrelated templates. When a transform uses xsl:key, add a temporary output that lists the key index for a given value so you can see if the key is being built correctly. The same idea works for variables: output them just once in a deterministic area of the result so you can verify their shape.

A stable debug transform also benefits from deterministic ordering. When you iterate through nodes, add an explicit xsl:sort so the output is predictable. That makes diffs meaningful when you tweak a predicate or update a template priority. If you are mixing modes, ensure the call chain is explicit; a missing mode is a classic way to call a generic template by accident. A related trap is having a high-priority identity template that overrides a specialized one, so watch for priority values and make sure the most specific template wins.

When handling multiple inputs, be clear about document boundaries. Use document() or collection() with explicit base URIs and add messages that show which document node you are iterating. If you are using XSLT 2.0 or 3.0, a quick serialize() to a short string can show you whether the tree is what you expect. If you stick to XSLT 1.0, the same idea works by writing xsl:copy-of into a separate debug result tree and inspecting it.

Here is a short pattern I often add while troubleshooting:



    node=
    ns=

You can drop this at the top of the stylesheet, run a quick transform, and then remove it once the root cause is found. The idea is not to keep noise in production, but to have a fast way to make the invisible visible. For more focused tracing, add the template only for the nodes you suspect are wrong. Debugging gets faster the more you scope down the noise.

Finally, keep a checklist of the classic XSLT footguns: missing namespaces, wrong context node, incorrect @ in attribute selection, and predicates that use 1-based indexes when you thought they were 0-based. I also look for template import precedence issues and for unexpected whitespace handling when the output is textual. These are easy to miss because the transform still runs, it just runs incorrectly.

If you want a fast place to test these patterns with real inputs, use the online editor at https://xsltplayground.com. It is built for rapid iteration with multiple inputs and parameters, which makes debugging much less painful and keeps your feedback loop tight.

XSLT string functions: complete reference with examples

Alexandre Vazquez — Thu, 14 May 2026 09:00:00 +0000

String manipulation is one of the most common tasks in XSLT. Whether you are formatting output, parsing codes, or normalising values from external systems, XPath provides a rich set of string functions. This reference covers the most useful ones with examples you can run in XSLT Playground.

Basic string functions (XSLT 1.0+)

string-length

Returns the number of characters in a string.

substring

Extracts a portion of a string. Arguments: string, start position (1-based), optional length.

substring-before and substring-after

Split a string on a delimiter.

contains, starts-with, ends-with

Test membership without extracting.

...
...

...

concat

Joins strings together. Takes any number of arguments.

normalize-space

Strips leading and trailing whitespace, and collapses internal whitespace to single spaces. Essential for cleaning values from XML sources.

translate

Replaces characters one-for-one. Useful for simple case conversion or character removal.

Advanced string functions (XSLT 2.0+)

upper-case and lower-case

No longer need translate for case conversion.

replace

Regex-based substitution. Much more powerful than translate.

matches

Tests a string against a regex.

tokenize

Splits a string into a sequence using a regex delimiter. Returns a sequence of strings.

string-join

The inverse of tokenize. Joins a sequence with a separator.

format-number

Formats a number as a string with a picture pattern.

format-date and format-dateTime

Format xs:date and xs:dateTime values using picture strings.

Practical patterns

Extract domain from URL:

Pad a number with leading zeros:

Check if a node text is numeric:

All of these work in XSLT Playground. Set the version to 2.0 or 3.0 for the functions that require it.

XSLT grouping with xsl:for-each-group: complete guide

Alexandre Vazquez — Mon, 11 May 2026 09:00:00 +0000

Grouping is one of the most powerful features introduced in XSLT 2.0. Before it, grouping in XSLT 1.0 required the Muenchian method — a clever but verbose technique involving keys and node-set comparisons. In 2.0, xsl:for-each-group makes grouping straightforward.

Basic grouping with group-by

group-by groups nodes that share the same value for a given expression. The result is one iteration per distinct group value.

Input:


  DE120
  US85
  DE200
  FR60
  US140

Stylesheet:

Output:


  2320
  160
  2225

Key functions inside for-each-group:

current-grouping-key() — returns the value that defines the current group
current-group() — returns the sequence of all nodes in the current group

Nested grouping

Groups can be nested. Group orders by country, then within each country by status:

group-adjacent

Groups consecutive nodes that share the same key value. Unlike group-by, it starts a new group when the key changes, even if the same key appeared earlier. This is useful for processing structured text or segmented data.


  Starting
  Processing
  Failed
  Retrying
  Done

This produces three blocks: two INFO (positions 1-2), one ERROR (3-4), one INFO (5). With group-by, the two INFO groups would be merged into one.

group-starting-with and group-ending-with

These group nodes based on a pattern match rather than a key value. Every time a node matches the pattern, a new group starts (or ends).

group-starting-with example — treat every ## as the start of a section:



    <xsl:value-of select="self::h2"/>

group-ending-with example — group lines until a blank line:

Computing aggregates

current-group() returns a sequence, so you can apply any XPath aggregate function directly:

Try it in XSLT Playground

Paste any of the examples above into XSLT Playground with version set to 2.0 or 3.0. Grouping is one of the features that benefits most from live testing — you can immediately see how changing the grouping key or switching between group-by and group-adjacent affects the output structure.

XSLT template matching explained with examples

Alexandre Vazquez — Thu, 07 May 2026 09:00:00 +0000

Template matching is the mechanism that drives every XSLT transformation. Understanding how the processor selects templates — and what happens when multiple templates could match — is the difference between a stylesheet that works reliably and one that produces surprising output. This post covers everything you need to know.

How match patterns work

When the processor visits a node, it evaluates every template's match attribute as an XPath pattern. A pattern is a restricted form of XPath that tests properties of a node rather than selecting nodes from a starting point. If the pattern is satisfied, that template is a candidate.

Priority and conflict resolution

More than one template can match the same node. The processor resolves the conflict using priority. Each pattern has a default priority calculated by the spec:

Pattern type	Default priority
`node()` or `*`	-0.5
`element-name`	0
`prefix:element-name`	0
`a/b` (path)	0.5
`a[predicate]`	0.5
`@attr`	0

More specific patterns automatically get higher priority. You can override this with the priority attribute:

If two templates have equal computed priority, the processor signals an error (or picks the last one, depending on implementation — Saxon issues an error by default). Always assign explicit priorities when you have competing templates.

The built-in templates

XSLT has default templates for every node type. If no explicit template matches a node, the built-in fires. For elements and the document root, the built-in calls apply-templates on all children. For text and attribute nodes, it outputs the string value.

This means that without any templates at all, the processor will walk the entire tree and output all text content. Understanding this explains why simple stylesheets can produce unexpected extra text — a text node matched nothing explicit, and the built-in output it.

To suppress text output globally, add:

This overrides the built-in with an empty template, producing no output for text nodes that are not handled elsewhere.

Modes

Modes let you have multiple templates for the same node that serve different purposes. A mode is a named context for a set of templates.

##

Call with mode:

Modes are especially useful when you need to process the same nodes in multiple places in the output with different logic each time.

apply-templates vs for-each

Both iterate over a set of nodes. The difference is that apply-templates dispatches to the best matching template for each node, while for-each stays in the current context and does not do template lookup.

Use apply-templates when you want polymorphism — different node types handled differently. Use for-each when you are doing a simple iteration over a homogeneous set and do not need dispatch.

Testing patterns in XSLT Playground

XSLT Playground is a fast way to experiment with matching rules. Paste a stylesheet with multiple competing templates and check which one fires. Add `` in each template to trace which one the processor picks. The trace panel shows all messages in order so you can follow the dispatch chain.

Solid understanding of template matching pays off every time you work on a complex stylesheet. Once you know how priorities and built-ins interact, most "unexpected output" bugs become obvious.

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Alexandre Vazquez — Tue, 05 May 2026 11:00:00 +0000

Prometheus Alertmanager vs Grafana Alerting (2026): Architecture, Features, and When to Use Each

Introduction

Most observability stacks running in production for over a year end up with alerting spread across two systems: Prometheus Alertmanager handling metric-based alerts and Grafana Alerting managing everything else. This creates the "alerting consolidation problem" where on-call teams receive duplicated pages, silencing rules live in two places, and nobody is certain which system is authoritative.

The question is straightforward: should you standardize on Prometheus Alertmanager, move everything into Grafana Alerting, or deliberately run both? The answer depends on your datasource mix, your GitOps maturity, and how your organization manages on-call routing.

Architecture Overview

Prometheus Alertmanager: The Standalone Receiver

Alertmanager is a dedicated, standalone component in the Prometheus ecosystem. It does not evaluate alert rules itself. Instead, Prometheus (or compatible senders like Thanos Ruler, Cortex, or Mimir Ruler) evaluates PromQL expressions and pushes firing alerts to the Alertmanager API. Alertmanager then handles deduplication, grouping, inhibition, silencing, and notification delivery.

# Simplified Prometheus → Alertmanager flow
#
# [Prometheus] --evaluates rules--> [firing alerts]
#        |
#        +--POST /api/v2/alerts--> [Alertmanager]
#                                      |
#                          +-----------+-----------+
#                          |           |           |
#                       [Slack]    [PagerDuty]  [Email]

The entire configuration lives in a single YAML file (alertmanager.yml). This includes the routing tree, receiver definitions, inhibition rules, and silence templates. There is no database, no UI-driven state — just a config file and an optional local storage directory for notification state and silences. This makes it trivially reproducible and ideal for GitOps workflows.

For high availability, you run multiple Alertmanager instances in a gossip-based cluster. They use a mesh protocol to share silence and notification state, ensuring that failover does not result in duplicate or lost notifications.

Grafana Alerting: The Integrated Platform

Grafana Alerting (sometimes called "Grafana Unified Alerting," introduced in Grafana 8) takes a different architectural approach. It embeds the entire alerting lifecycle — rule evaluation, state management, routing, and notification — inside the Grafana server process. Under the hood, it uses a fork of Alertmanager for the routing and notification layer.

# Simplified Grafana Alerting flow
#
# [Grafana Server]
#   ├── Rule Evaluation Engine
#   │     ├── queries Prometheus
#   │     ├── queries Loki
#   │     ├── queries CloudWatch
#   │     └── queries any supported datasource
#   │
#   ├── Alert State Manager (internal)
#   │
#   └── Embedded Alertmanager (routing + notifications)
#           |
#           +-----------+-----------+
#           |           |           |
#        [Slack]    [PagerDuty]  [Email]

The critical distinction is that Grafana Alerting evaluates alert rules itself, querying any configured datasource — not just Prometheus. It can fire alerts based on Loki log queries, Elasticsearch searches, CloudWatch metrics, PostgreSQL queries, or any of the 100+ datasource plugins available in Grafana. Rule definitions, contact points, notification policies, and mute timings are stored in the Grafana database (or provisioned via YAML files and the Grafana API).

Feature Comparison

Feature	Prometheus Alertmanager	Grafana Alerting
Datasources	Prometheus-compatible only (Prometheus, Thanos, Mimir, VictoriaMetrics)	Any Grafana datasource (Prometheus, Loki, Elasticsearch, CloudWatch, SQL databases, etc.)
Rule evaluation	External (Prometheus/Ruler evaluates rules and pushes alerts)	Built-in (Grafana evaluates rules directly)
Routing tree	Hierarchical YAML-based routing with match/match_re, continue, group_by	Notification policies with label matchers, nested policies, mute timings
Grouping	Full support via group_by, group_wait, group_interval	Full support via notification policies with equivalent controls
Inhibition	Native inhibition rules (suppress alerts when a related alert is firing)	Supported since Grafana 10.3 but less flexible than Alertmanager
Silencing	Label-based silences via API or UI, time-limited	Mute timings (recurring schedules) and silences (ad-hoc, label-based)
Notification channels	Email, Slack, PagerDuty, OpsGenie, VictoriaOps, webhook, WeChat, Telegram, SNS, Webex	All of the above plus Teams, Discord, Google Chat, LINE, Threema, Oncall, and more via contact points
Templating	Go templates in notification config	Go templates with access to Grafana template variables and functions
Multi-tenancy	Not built-in; achieved via separate instances or Mimir Alertmanager	Native multi-tenancy via Grafana organizations and RBAC
High availability	Gossip-based cluster (peer mesh, well-proven)	Database-backed HA with peer discovery between Grafana instances
Configuration model	Single YAML file, fully declarative	UI + API + provisioning YAML files, stored in database
GitOps compatibility	Excellent — config file lives in version control natively	Possible via provisioning files or Terraform provider, but requires extra tooling
External alert sources	Any system that can POST to the Alertmanager API	Supported via the Grafana Alerting API (external alerts can be pushed)
Managed service	Available via Grafana Cloud (as Mimir Alertmanager), Amazon Managed Prometheus	Available via Grafana Cloud

Alertmanager Strengths

Alertmanager has been a production staple since 2015. Over a decade of use across thousands of organizations has made it one of the most battle-tested components in the CNCF ecosystem.

Declarative, GitOps-Native Configuration

The entire Alertmanager configuration is a single YAML file. There is no hidden state in a database, no click-driven configuration that someone forgets to document. You check it into Git, review it in a pull request, and deploy it through your CI/CD pipeline like any other infrastructure code.

# alertmanager.yml — everything in one file
global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/T00/B00/XXX"

route:
  receiver: platform-team
  group_by: [alertname, cluster, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-oncall
      group_wait: 10s
    - match_re:
        team: "^(payments|checkout)$"
      receiver: payments-slack
      continue: true

receivers:
  - name: platform-team
    slack_configs:
      - channel: "#platform-alerts"
  - name: pagerduty-oncall
    pagerduty_configs:
      - service_key: ""
  - name: payments-slack
    slack_configs:
      - channel: "#payments-oncall"

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [alertname, cluster]

Every change is auditable. Rollbacks are a git revert away. This matters enormously when you are debugging why an alert did not fire at 3 AM.

Lightweight and Single-Purpose

Alertmanager does one thing: route and deliver notifications. It has no dashboard, no query engine, no datasource plugins. This single-purpose design makes it operationally simple. Resource consumption is minimal — a small Alertmanager instance handles thousands of active alerts on a few hundred megabytes of memory. It starts in milliseconds and requires almost no maintenance.

Mature Inhibition and Routing

Alertmanager's inhibition rules are first-class citizens. You can suppress downstream warnings when a critical alert is already firing, preventing alert storms from overwhelming your on-call team. The hierarchical routing tree with continue flags allows for nuanced delivery: send to the team channel AND escalate to PagerDuty simultaneously, with different grouping strategies at each level.

Proven High Availability

The gossip-based HA cluster has been stable for years. Running three Alertmanager replicas behind a load balancer (or using Kubernetes service discovery) gives you reliable notification delivery without shared storage. The protocol handles deduplication across instances automatically, which is the hardest part of distributed alerting.

Grafana Alerting Strengths

Grafana Alerting has matured considerably since its rocky introduction in Grafana 8. By Grafana 11 and 12, it has become a legitimate production alerting platform with capabilities that Alertmanager cannot match on its own.

Multi-Datasource Alert Rules

This is Grafana Alerting's strongest differentiator. You can write alert rules that query Loki for error log spikes, CloudWatch for AWS resource utilization, Elasticsearch for application errors, or a PostgreSQL database for business metrics — all from the same alerting system. If your observability stack includes more than just Prometheus, this eliminates the need for separate alerting tools per datasource.

# Grafana alert rule provisioning example — alerting on Loki log errors
apiVersion: 1
groups:
  - orgId: 1
    name: application-errors
    folder: Production
    interval: 1m
    rules:
      - uid: loki-error-spike
        title: "High error rate in payment service"
        condition: C
        data:
          - refId: A\            datasourceUid: loki-prod
            model:
              expr: 'sum(rate({app="payment-service"} |= "ERROR" [5m]))'
          - refId: B
            datasourceUid: "__expr__"
            model:
              type: reduce
              expression: A
              reducer: last
          - refId: C
            datasourceUid: "__expr__"
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: gt
                    params: [10]
        for: 5m
        labels:
          severity: warning
          team: payments

This is something Alertmanager simply cannot do. Alertmanager only receives pre-evaluated alerts — it has no concept of datasources or query execution.

Unified UI for Alert Management

Grafana provides a single pane of glass for alert rule creation, visualization, notification policy management, contact point configuration, and silence management. For teams where not every engineer is comfortable editing YAML routing trees, the visual notification policy editor significantly reduces the barrier to entry. You can see the state of every alert rule, its evaluation history, and the exact notification path it will take — all without leaving the browser.

Native Multi-Tenancy and RBAC

Grafana's organization model and role-based access control extend naturally to alerting. Different teams can manage their own alert rules, contact points, and notification policies within their organization or folder scope, without seeing or interfering with other teams. Achieving this with standalone Alertmanager requires either running separate instances per tenant or using Mimir's multi-tenant Alertmanager.

Mute Timings and Richer Scheduling

While Alertmanager supports silences (ad-hoc, time-limited suppressions), Grafana Alerting adds mute timings — recurring time-based windows where notifications are suppressed. This is useful for scheduled maintenance windows, business-hours-only alerting, or suppressing non-critical alerts on weekends. Alertmanager requires external tooling or manual silence creation for recurring windows.

Grafana Cloud as a Managed Option

For teams that want to avoid managing alerting infrastructure entirely, Grafana Cloud provides a fully managed Grafana Alerting stack. This includes HA, state persistence, and notification delivery without any self-hosted components. The Grafana Cloud alerting stack also includes a managed Mimir Alertmanager, which means you can use Prometheus-native alerting rules if you prefer that model while still benefiting from the managed infrastructure.

When to Use Prometheus Alertmanager

Alertmanager is the right choice when the following conditions describe your environment:

Your metrics stack is Prometheus-native. If all your alert rules are PromQL expressions evaluated by Prometheus, Thanos Ruler, or Mimir Ruler, Alertmanager is the natural fit. There is no added value in routing those alerts through Grafana.
GitOps is non-negotiable. If every infrastructure change must go through a pull request and be fully declarative, Alertmanager's single-file configuration model is significantly easier to manage than Grafana's database-backed state. Tools like amtool provide config validation in CI pipelines.
You need fine-grained routing with inhibition. Complex routing trees with multiple levels of grouping, inhibition rules, and continue flags are more naturally expressed in Alertmanager's YAML format. The routing logic has been stable and well-documented for years.
You run microservices with per-team routing. If each team owns its routing subtree and the routing logic is complex, Alertmanager's hierarchical model scales better than UI-driven configuration. Teams can own their section of the config file via CODEOWNERS in Git.
You want minimal operational overhead. Alertmanager is a single binary with minimal resource requirements. There is no database to back up, no migrations to run, and no UI framework to keep updated.

When to Use Grafana Alerting

Grafana Alerting is the right choice when these conditions apply:

You alert on more than just Prometheus metrics. If you need alert rules based on Loki logs, Elasticsearch queries, CloudWatch metrics, or database queries, Grafana Alerting is the only option that handles all of these natively. The alternative is running separate alerting tools per datasource, which is worse.
Your team prefers UI-driven configuration. Not every engineer wants to edit YAML routing trees. If your organization values a visual interface for managing alerts, contact points, and notification policies, Grafana's UI is a major productivity advantage.
You are using Grafana Cloud. If you are already on Grafana Cloud, using its built-in alerting is the path of least resistance. You get HA, managed notification delivery, and a unified experience without running any additional infrastructure.
Multi-tenancy is a requirement. If multiple teams need isolated alerting configurations with RBAC, Grafana's native organization and folder-based access model is significantly easier to set up than running per-tenant Alertmanager instances.
You want mute timings for recurring maintenance windows. If your team regularly needs to suppress alerts during scheduled windows (deploy windows, batch processing hours, weekend non-critical suppression), Grafana's mute timings feature is more ergonomic than creating and managing recurring silences in Alertmanager.

Running Both Together: The Hybrid Pattern

In practice, many production environments run both Alertmanager and Grafana Alerting. This is not necessarily a mistake — it can be a deliberate architectural choice when done with clear boundaries.

Common Hybrid Architecture

The most common pattern looks like this:

Prometheus Alertmanager handles all metric-based alerts. PromQL rules are evaluated by Prometheus or a long-term storage ruler (Thanos, Mimir). Alertmanager owns routing, grouping, and notification for these alerts.
Grafana Alerting handles non-Prometheus alerts: log-based alerts from Loki, business metrics from SQL datasources, and cross-datasource correlation rules.

The key to making this work without chaos is establishing clear ownership rules:

# Ownership boundaries for hybrid alerting
#
# Prometheus Alertmanager owns:
#   - All PromQL-based alert rules
#   - Infrastructure alerts (node, kubelet, etcd, CoreDNS)
#   - Application SLO/SLI alerts based on metrics
#
# Grafana Alerting owns:
#   - Log-based alert rules (Loki, Elasticsearch)
#   - Business metric alerts (SQL datasources)
#   - Cross-datasource correlation rules
#   - Alerts for teams that prefer UI-driven management
#
# Shared:
#   - Contact points / receivers use the same Slack channels and PagerDuty services
#   - On-call rotations are managed externally (PagerDuty, Grafana OnCall)

Both systems can deliver to the same notification channels. The critical discipline is ensuring that silencing and maintenance windows are applied in both systems when needed. This is the primary operational cost of the hybrid approach.

Grafana as a Viewer for Alertmanager

Even if you use Alertmanager exclusively for routing and notification, Grafana can serve as a read-only viewer. Grafana natively supports connecting to an external Alertmanager datasource, allowing you to see firing alerts, active silences, and alert groups in the Grafana UI. This gives you the operational visibility of Grafana without moving your alerting logic into it.

# Grafana datasource provisioning for external Alertmanager
apiVersion: 1
datasources:
  - name: Alertmanager
    type: alertmanager
    url: http://alertmanager.monitoring.svc:9093
    access: proxy
    jsonData:
      implementation: prometheus

Migration Considerations

If you are moving from one system to the other, here are the practical considerations to plan for.

Migrating from Alertmanager to Grafana Alerting

Rule conversion. Your PromQL-based recording and alerting rules defined in Prometheus rule files need to be recreated as Grafana alert rules. Grafana provides a migration tool that can import Prometheus-format rules, but complex expressions may need manual adjustment.
Routing tree translation. Alertmanager's hierarchical routing tree maps to Grafana's notification policies, but the semantics are not identical. Test the notification routing thoroughly — the continue flag behavior and default routes may differ.
Silence and inhibition migration. Active silences are ephemeral and do not need migration. Inhibition rules need to be recreated in Grafana's format. Recurring maintenance windows should be converted to mute timings.
Run in parallel first. The safest migration strategy is to run both systems in parallel for two to four weeks, sending notifications from both, then cutting over when you have confidence in the Grafana setup. Accept the temporary noise of duplicate alerts — it is far cheaper than missing a critical page during migration.

Migrating from Grafana Alerting to Alertmanager

Datasource limitation. You can only migrate alerts that are based on Prometheus-compatible datasources. Alerts querying Loki, Elasticsearch, or SQL datasources have no equivalent in Alertmanager — you will need an alternative solution for those.
Rule export. Export Grafana alert rules and convert them to Prometheus-format rule files. The Grafana API (GET /api/v1/provisioning/alert-rules) provides structured output that can be transformed with a script.
Contact point mapping. Map Grafana contact points to Alertmanager receivers. The configuration format is different, but the concepts are equivalent.
State loss. Alertmanager does not carry over Grafana's alert evaluation history. You start fresh. Plan for a brief period where alerts may re-fire as Prometheus evaluates rules that were previously managed by Grafana.

Decision Framework

If you want a quick decision path, use this framework:

Start here:
│
├── Do you alert on non-Prometheus datasources (Loki, ES, SQL, CloudWatch)?
│   ├── YES → Grafana Alerting (at least for those datasources)
│   └── NO ↓
│
├── Is GitOps/declarative config a hard requirement?
│   ├── YES → Alertmanager
│   └── NO ↓
│
├── Do you need multi-tenancy with RBAC?
│   ├── YES → Grafana Alerting (or Mimir Alertmanager)
│   └── NO ↓
│
├── Are you on Grafana Cloud?
│   ├── YES → Grafana Alerting (path of least resistance)
│   └── NO ↓
│
└── Default → Alertmanager (simpler, lighter, well-proven)

For many teams, the honest answer is "both" — Alertmanager for the Prometheus-native metric pipeline, Grafana Alerting for everything else. That is a valid architecture as long as the ownership boundaries are documented and the on-call team knows where to look.

Frequently Asked Questions

What is the difference between Alertmanager and Grafana Alerting?

Prometheus Alertmanager is a standalone notification routing engine that receives pre-evaluated alerts from Prometheus and delivers them to receivers like Slack, PagerDuty, or email. Grafana Alerting is an integrated alerting platform embedded in Grafana that both evaluates alert rules and handles notification routing. Alertmanager is configured entirely via YAML, while Grafana Alerting offers a UI, API, and file-based provisioning. The fundamental difference is scope: Alertmanager handles only the routing and notification phase, while Grafana Alerting handles the full lifecycle from query evaluation to notification.

Can Grafana Alerting replace Prometheus Alertmanager?

Yes, for many use cases. Grafana Alerting can evaluate PromQL rules directly against your Prometheus datasource, so you do not strictly need a separate Alertmanager instance. However, there are scenarios where Alertmanager remains the better choice: heavily GitOps-driven environments, teams that need Alertmanager's mature inhibition rules, or architectures where Prometheus rule evaluation happens externally (Thanos Ruler, Mimir Ruler) and a dedicated Alertmanager is already in the pipeline. If your only datasource is Prometheus and you value declarative configuration, Alertmanager is still simpler and lighter.

Is Grafana Alertmanager the same as Prometheus Alertmanager?

Not exactly. Grafana Alerting uses a fork of the Prometheus Alertmanager code internally for its notification routing engine, but it is not the same product. The Grafana "Alertmanager" visible in the UI is a managed, embedded component with a different configuration interface (notification policies, contact points, mute timings) compared to the standalone Prometheus Alertmanager (routing tree, receivers, inhibition rules in YAML). Grafana can also connect to an external Prometheus Alertmanager as a datasource, which adds to the confusion.

What are the best alternatives to Prometheus Alertmanager?

The most direct alternative is Grafana Alerting, which can receive and route Prometheus alerts while also supporting other datasources. Beyond that: Grafana OnCall for on-call management and escalation, PagerDuty or Opsgenie as managed incident response platforms, Keep as an open-source AIOps alert management platform, and Mimir Alertmanager for multi-tenant environments running Grafana Mimir.

Should I use Prometheus alerts or Grafana alerts for Kubernetes monitoring?

For Kubernetes monitoring specifically, the kube-prometheus-stack (which includes Prometheus, Alertmanager, and a comprehensive set of pre-built alerting rules) remains the industry standard. These rules are PromQL-based and are designed to work with Alertmanager. If you are deploying kube-prometheus-stack, using Alertmanager for metric-based alerts is the straightforward choice. Add Grafana Alerting on top if you also need to alert on logs (via Loki) or non-metric datasources.

Final Thoughts

The Alertmanager vs Grafana Alerting debate is not really about which tool is better — it is about which tool fits your operational context. Alertmanager is simpler, lighter, and more GitOps-friendly. Grafana Alerting is more versatile, more accessible to UI-oriented teams, and the only option if you need multi-datasource alerting. Running both is perfectly valid when the boundaries are clear.

The worst outcome is not picking the "wrong" tool. The worst outcome is running both accidentally, with overlapping coverage, duplicated notifications, and no clear ownership. Whatever you choose, document the decision, define the ownership boundaries, and make sure your on-call team knows exactly where to go when they need to silence an alert at 3 AM.

Originally published at alexandre-vazquez.com/alertmanager-vs-grafana-alerting

Prometheus Alertmanager Vs Grafana Alerting (2026): Architecture, Features, And When To Use Each

Alexandre Vazquez — Tue, 05 May 2026 10:00:00 +0000

Prometheus Alertmanager Vs Grafana Alerting (2026): Architecture, Features, And When To Use Each

Originally published at alexandre-vazquez.com

Read the full article on my blog: https://alexandre-vazquez.com/alertmanager-vs-grafana-alerting/

Debugging Distroless Containers: kubectl debug, Ephemeral Containers, and When to Use Each

Alexandre Vazquez — Tue, 05 May 2026 08:00:01 +0000

Originally published at alexandre-vazquez.com/debugging-distroless-containers/

The container works fine in CI. It deploys successfully to staging. Then something goes wrong in production and you type the command you always type: kubectl exec -it my-pod -- /bin/bash. The response is immediate: OCI runtime exec failed: exec failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory.

You try /bin/sh. Same error. You try ls. Same error. The container image is distroless — it ships only your application binary and its runtime dependencies, with no shell, no package manager, no debugging tools of any kind. This is intentional and correct from a security standpoint. It is also a significant operational challenge the first time you face it in production.

This article covers every practical technique for debugging distroless containers in Kubernetes: kubectl debug with ephemeral containers (the standard approach), pod copy strategy (for Kubernetes versions without ephemeral container support, or when you need to modify the running pod spec), debug image variants (the pragmatic developer shortcut), cdebug (a purpose-built tool that simplifies the process), and node-level debugging (the last resort with the most power). For each technique I will explain what it can and cannot do, what Kubernetes version or RBAC permissions it requires, and in which scenario — developer in local, platform engineer in staging, ops in production — it is the appropriate choice.

Why Distroless Breaks the Normal Debugging Workflow

Traditional container debugging assumes you can exec into the container and use shell tools: ps, netstat, strace, curl, a text editor. Distroless images remove all of this by design. The Google distroless project, Chainguard's Wolfi-based images, and the broader minimal image ecosystem deliberately exclude everything that is not required to run the application. The result is a dramatically smaller attack surface: no shell means no RCE via shell injection, no package manager means no easy escalation path, fewer binaries means fewer CVEs in the image scan.

The tradeoff is operational: when something goes wrong, you cannot use the tools that the process itself is not allowed to run. A Java application in gcr.io/distroless/java17-debian12 has the JRE and nothing else. A Go binary compiled with CGO disabled and shipped in gcr.io/distroless/static-debian12 has literally only the binary and the necessary CA certificates and timezone data. There is no wget to download a debug binary, no apt to install one, no bash to run a script.

Kubernetes solves this at the platform level with ephemeral containers , added as stable in Kubernetes 1.25. The principle is that a debug container — which can have a full shell and any tools you want — can be injected into a running pod and share its process namespace, network namespace, and filesystem mounts without modifying the original container or restarting the pod.

Option 1: kubectl debug with Ephemeral Containers

Ephemeral containers are the canonical solution. Since Kubernetes 1.25 (stable), kubectl debug can inject a temporary container into a running pod. The container shares the target pod's network namespace by default, and with --target it can also share the process namespace of a specific container, allowing you to inspect its running processes and open file descriptors.

The basic invocation is:

kubectl debug -it my-pod \
  --image=busybox:latest \
  --target=my-container

The --target flag is the critical piece. Without it, the ephemeral container gets its own process namespace. With it, it shares the process namespace of the specified container — meaning you can run ps aux and see the application's processes, use ls -la /proc//fd to inspect open file descriptors, and read the application's environment via cat /proc//environ.

For a more capable debug environment, replace busybox with a richer image:

kubectl debug -it my-pod \
  --image=nicolaka/netshoot \
  --target=my-container

nicolaka/netshoot includes tcpdump, curl, dig, nmap, ss, iperf3, and dozens of other network diagnostic tools, making it the standard choice for network debugging scenarios.

What You Can and Cannot Do

Ephemeral containers share the pod's network namespace and, when --target is used, the process namespace. This gives you:

Full visibility into the application's network traffic from inside the pod (tcpdump, ss, netstat)
Process inspection via /proc/ — open files, memory maps, environment variables, CPU/memory usage
Access to the pod's DNS resolution context — exactly the same /etc/resolv.conf the application sees
Ability to make outbound network calls from the same network namespace (testing service endpoints, DNS resolution)

What you do not get with ephemeral containers:

Access to the application container 's filesystem. The ephemeral container has its own root filesystem. You cannot cat /app/config.yaml from the application container's filesystem unless you access it via /proc//root/.
Ability to remove the container once added. Ephemeral containers are permanent until the pod is deleted. This is by design — the Kubernetes API does not allow removing them after creation.
Volume mount modifications via CLI. You cannot add volume mounts to an ephemeral container via kubectl debug (though the API spec supports it, the CLI does not expose this).
Resource limits. Ephemeral containers do not support resource requests and limits in the kubectl debug CLI, though this is evolving.

Accessing the Application Filesystem

The most common surprise for developers new to ephemeral containers is that they cannot directly browse the application container's filesystem. The workaround is the /proc filesystem:

# Find the application's PID
ps aux

# Browse its filesystem via /proc
ls /proc/1/root/app/
cat /proc/1/root/etc/config.yaml

# Or set the root to the application's root
chroot /proc/1/root /bin/sh  # only if /bin/sh exists in the app image

The /proc//root path is a symlink to the container's root filesystem as seen from the process namespace. Because the ephemeral container shares the process namespace with --target, the application's PID is typically 1, and /proc/1/root gives you full read access to its filesystem.

RBAC Requirements

Ephemeral containers require the pods/ephemeralcontainers subresource permission. This is separate from pods/exec, which controls kubectl exec. A common mistake is to grant pods/exec for debugging purposes without realizing that ephemeral containers require an additional grant:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ephemeral-debugger
rules:
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

In production environments, this permission should be tightly scoped: time-limited via RoleBinding rather than permanent ClusterRoleBinding, restricted to specific namespaces, and ideally gated behind an approval workflow. The debug container runs as root by default, which can create privilege escalation paths if the application container runs as a non-root user with shared process namespace — the debug container can attach to the application's processes with higher privileges.

Option 2: kubectl debug -copy-to (Pod Copy Strategy)

When you need to modify the pod's container spec — replace the image, change environment variables, add a sidecar with a shared filesystem — the --copy-to flag creates a full copy of the pod with your modifications applied:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=my-app:debug \
  --share-processes

This creates a new pod named my-pod-debug that is a copy of my-pod but with the container image replaced by my-app:debug. If my-app:debug is your application image built with debug tooling included (or a debug variant from your registry), this lets you interact with the exact same binary in the exact same configuration as the original pod.

A more common use of --copy-to is to attach a debug container alongside the existing application container while keeping the original image unchanged:

kubectl debug my-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  --container=debugger

This creates the copy-pod with both the original containers and a new debugger container sharing the process namespace. Unlike ephemeral containers, this approach supports volume mounts and resource limits, and the debug pod can be deleted cleanly when you are done.

Limitations of the Copy Strategy

The pod copy approach has a critical limitation: it is not debugging the original pod. It creates a new pod that may behave differently because:

It does not share the original pod's in-memory state — if the issue is a goroutine leak or heap corruption that has been accumulating for hours, the fresh copy will not exhibit it immediately
It creates a new Pod UID, which means any admission webhooks, network policies, or pod-level security contexts that depend on pod identity may apply differently
If the original pod is crashing (CrashLoopBackOff), the copy will also crash — this technique does not help for crash debugging unless you also change the entrypoint

For crash debugging specifically, combine --copy-to with a modified entrypoint to keep the container alive:

kubectl debug my-crashing-pod \
  -it \
  --copy-to=my-pod-debug \
  --image=busybox \
  --share-processes \
  -- sleep 3600

Option 3: Debug Image Variants

The most pragmatic approach — and the one most appropriate for developer workflows — is to maintain a debug variant of your application image that includes shell tooling. Both the Google distroless project and Chainguard provide this pattern officially.

Google distroless images have a :debug tag that adds BusyBox to the image:

# Production image
FROM gcr.io/distroless/java17-debian12

# Debug variant — identical but with BusyBox shell
FROM gcr.io/distroless/java17-debian12:debug

Chainguard images follow a similar convention with :latest-dev variants that include apk, a shell, and common utilities:

# Production (zero shell, minimal footprint)
FROM cgr.dev/chainguard/go:latest

# Development/debug variant
FROM cgr.dev/chainguard/go:latest-dev

If you build your own base images, the recommended approach is to use multi-stage builds and maintain separate build targets:

FROM golang:1.22 AS builder
WORKDIR /app
COPY . .
RUN go build -o myapp .

# Production: static distroless image
FROM gcr.io/distroless/static-debian12 AS production
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

# Debug variant: same binary, with shell tools
FROM gcr.io/distroless/static-debian12:debug AS debug
COPY --from=builder /app/myapp /myapp
ENTRYPOINT ["/myapp"]

In your CI/CD pipeline, build both targets and push my-app:${VERSION} (production) and my-app:${VERSION}-debug (debug variant) to your registry. The debug image is never deployed to production by default, but it exists and is ready to be used with kubectl debug --copy-to when needed.

Security Considerations for Debug Variants

Debug image variants defeat much of the security benefit of distroless if they are used in production, even temporarily. Track usage carefully: log when debug images are deployed, require explicit approval, and ensure they are removed after the debugging session. In regulated environments, consider whether deploying a debug variant to production namespaces is permitted by your security policy — in many cases it is not, and you must use ephemeral containers (which add a debug process to the pod without modifying the application image) instead.

Option 4: cdebug

cdebug is an open-source CLI tool that simplifies distroless debugging by wrapping kubectl debug with more ergonomic defaults and additional capabilities. Its primary value is in making ephemeral container debugging feel like a native shell experience:

# Install
brew install cdebug
# or: go install github.com/iximiuz/cdebug@latest

# Debug a running pod
cdebug exec -it my-pod

# Specify a namespace and container
cdebug exec -it -n production my-pod -c my-container

# Use a specific debug image
cdebug exec -it my-pod --image=nicolaka/netshoot

What cdebug adds over raw kubectl debug:

Automatic filesystem chroot. cdebug exec automatically sets the filesystem root of the debug container to the target container's filesystem, so you browse / and see the application's files — not the debug image's files. This addresses the most common friction point with kubectl debug.
Docker integration. cdebug exec works identically for Docker containers (cdebug exec -it), making it the same muscle memory for local and cluster debugging.
No RBAC complications for Docker-based local development — useful for developer workflows before the code reaches Kubernetes.

The tradeoff: cdebug is a third-party dependency and requires installation. In environments with strict tooling policies (regulated industries, air-gapped clusters), it may not be an option. In those cases, the raw kubectl debug workflow with /proc/1/root filesystem navigation is the baseline.

Option 5: Node-Level Debugging

When everything else fails — the pod is in CrashLoopBackOff too fast to attach to, the issue is a kernel-level problem, or you need tools like strace that require elevated privileges — node-level debugging gives you direct access to the container's processes from the host node.

kubectl debug node/ creates a privileged pod on the target node that mounts the node's root filesystem under /host:

kubectl debug node/my-node-name \
  -it \
  --image=nicolaka/netshoot

From this privileged pod, you can use nsenter to enter the namespaces of any container running on the node:

# Find the container's PID on the node
# (from within the node debug pod)
crictl ps | grep my-container
crictl inspect  | grep pid

# Enter the container's namespaces
nsenter -t  -m -u -i -n -p -- /bin/sh

# Or just the network namespace (for network debugging)
nsenter -t  -n -- ip a

The nsenter approach lets you run tools from the node's or debug container's toolset while operating in the namespaces of the target container. This is how you run strace against a distroless process: strace is not in the application container, but you can run it from the node level while targeting the application's PID.

# Trace all syscalls from the application process
nsenter -t  -- strace -p  -f -e trace=network

RBAC and Security for Node Debugging

Node-level debugging requires nodes/proxy and the ability to create privileged pods, which in most production clusters is restricted to cluster administrators. The debug pod runs with hostPID: true and hostNetwork: true, giving it visibility into all processes and network traffic on the node — not just the target container. This is significant: every process running on the node, including those in other tenants' namespaces, is visible.

This technique should be treated as a break-glass procedure: log the access, require dual approval in production environments, and clean up immediately after the debugging session with kubectl delete pod --selector=app=node-debugger.

Choosing the Right Approach: Access Profile and Environment Matrix

The technique you should use depends on two axes: who you are (developer, platform engineer, ops/SRE) and where the issue is (local development, staging, production). The requirements and constraints differ significantly across these combinations.

Developer — Local or Development Cluster

Goal: Reproduce and understand a bug, inspect configuration, verify network connectivity to services.

Constraints: None material — full cluster admin on local or personal dev namespace.

Recommended approach: Debug image variants or cdebug.

In local development (Minikube, Kind, Docker Desktop), the fastest path is to build the debug variant of your image and deploy it directly. If you are working with another team's service, cdebug exec gives you a shell in the container with automatic filesystem root without any special RBAC. The goal is speed and iteration — reserve the more structured approaches for higher environments.

Developer — Staging Cluster

Goal: Debug integration issues, inspect live configuration, verify environment-specific behavior.

Constraints: Shared cluster — cannot deploy arbitrary workloads to other teams' namespaces, but has pods/ephemeralcontainers in own namespace.

Recommended approach: kubectl debug with ephemeral containers (--target), scoped to own namespace.

Staging is where ephemeral containers earn their keep. You can attach to a running pod without restarting it, without modifying the deployment spec, and without affecting other users of the same cluster. Grant developers pods/ephemeralcontainers in their team's namespaces and they can self-service debug without needing ops involvement.

Platform Engineer / SRE — Production

Goal: Diagnose a live production incident. The pod is behaving unexpectedly — high latency, memory growth, unexpected connections, incorrect responses.

Constraints: Changes to running pods are high-risk. Any debug image deployment must be gated. The issue is live and affecting users.

Recommended approach: kubectl debug with ephemeral containers (ephemeral containers do not restart the pod, do not modify the deployment, and are auditable via API audit logs).

The key production requirements are auditability and minimal blast radius. Ephemeral containers satisfy both: they are recorded in the Kubernetes API audit log (who attached, when, to which pod), they do not modify the running application container, and they are limited to the pod's own network and process namespaces. Document the debug session in your incident ticket: pod name, time, what was observed, who ran the debug container.

The --copy-to strategy is generally inappropriate for production incident response: it creates a new pod that may or may not exhibit the issue, it adds load to the cluster during an incident, and if it is attached to the same services (databases, downstream APIs), it produces additional traffic that complicates forensics.

Platform Engineer — Production, Node-Level Issue

Goal: Diagnose a kernel-level issue, a container runtime problem, a networking issue that spans multiple pods, or a situation where the pod is crashing too fast to attach to.

Constraints: Maximum privilege required. High operational risk.

Recommended approach: Node-level debug pod with nsenter. Treat as break-glass.

For this scenario, create a dedicated RBAC role that grants nodes/proxy access and the ability to create pods with hostPID: true in a dedicated debug namespace. Bind it only to specific users, require a separate authentication step (e.g., kubectl auth can-i check against a time-limited binding), and log all access. This level of access should generate a PagerDuty-style alert so that the security team knows a privileged debug session is active in production.

Common Errors and Solutions

Error: "ephemeral containers are disabled for this cluster"

Ephemeral containers require Kubernetes 1.16+ (alpha, behind feature gate) and are stable from 1.25. If you are on 1.16–1.22, you need to enable the EphemeralContainers feature gate on the API server and kubelet. From 1.23 it was beta and enabled by default. From 1.25 it is stable and always on. On managed Kubernetes services (EKS, GKE, AKS), check the cluster version — versions older than 1.25 may still have it disabled depending on your configuration.

Error: "cannot update ephemeralcontainers" (RBAC)

You have pods/exec but not pods/ephemeralcontainers. Add the grant shown in the RBAC section above. Note that pods/exec and pods/ephemeralcontainers are separate subresources — having one does not imply the other.

Error: "container not found" with -target

The container name in --target must match exactly the container name as defined in the Pod spec — not the image name. Check with kubectl get pod my-pod -o jsonpath='{.spec.containers[*].name}' to get the exact container names.

Error: Can see processes but cannot read /proc/1/root

The application container runs as a non-root user (e.g., UID 1000) and the ephemeral container runs as root. The application's filesystem may have files owned by UID 1000 that are not readable by other UIDs depending on permissions. The /proc//root path itself requires CAP_SYS_PTRACE capability. If your cluster's PodSecurityStandards (PSS) are set to restricted, the debug container may not have this capability. Use the Baseline PSS profile for debug namespaces or explicitly add SYS_PTRACE to the ephemeral container's securityContext.

Error: tcpdump shows no traffic

When using nicolaka/netshoot for network debugging, ensure the ephemeral container is created without --target if your goal is to capture all traffic on the pod's network interface (not just the specific container's process). With --target, you share the process namespace but the network namespace is shared at the pod level regardless. Run tcpdump -i any to capture on all interfaces including loopback, which is where inter-container traffic within a pod travels.

Decision Framework

Use this as a starting point to select the right technique for your situation:

Scenario	Technique	Requirement
Active production incident, pod running	kubectl debug + ephemeral container	pods/ephemeralcontainers RBAC, k8s 1.25+
Pod crashing too fast to attach	kubectl debug -copy-to + modified entrypoint	Ability to create pods in namespace
Developer debugging in dev/staging	cdebug exec or kubectl debug	pods/ephemeralcontainers or pod create
Need full filesystem access	kubectl debug -copy-to + debug image variant	Debug image in registry, pod create
Need strace or kernel tracing	Node-level debug with nsenter	nodes/proxy, cluster admin equivalent
Network packet capture	kubectl debug + nicolaka/netshoot	pods/ephemeralcontainers
Local Docker debugging	cdebug exec	Docker socket access
CI-reproducible debug environment	Debug image variant in separate build target	Separate image tag in registry

Production RBAC Design

A clean RBAC design for production distroless debugging separates three roles with different privilege levels:

# Tier 1: Developer self-service in team namespaces
# Allows attaching ephemeral containers, no node access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: distroless-debugger
  namespace: team-namespace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 2: SRE production incident access
# Ephemeral containers across all namespaces
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: sre-distroless-debugger
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/ephemeralcontainers"]
  verbs: ["update", "patch"]
- apiGroups: [""]
  resources: ["pods/attach"]
  verbs: ["create", "get"]
---
# Tier 3: Break-glass node access
# Only for platform team, time-limited binding recommended
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-debugger
rules:
- apiGroups: [""]
  resources: ["nodes/proxy"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create", "get", "list", "delete"]
  # Restrict to debug namespace via RoleBinding, not ClusterRoleBinding

Bind Tier 1 permanently to your developers. Bind Tier 2 to SREs permanently but with audit alerts on use. Bind Tier 3 only on-demand (via a Kubernetes operator that creates time-limited RoleBindings) and never as a permanent ClusterRoleBinding.

Summary

Distroless containers are the correct choice for production workloads. They reduce attack surface, eliminate unnecessary CVEs, and force a cleaner separation between application and tooling. The operational cost is that your traditional debugging workflow — exec into the container, run some commands — no longer works by default.

Kubernetes provides a clean answer with ephemeral containers and kubectl debug: inject a debug container with whatever tools you need into the running pod, sharing its network and process namespaces, without restarting or modifying the application. For scenarios where ephemeral containers are insufficient — filesystem access, crash debugging, kernel-level investigation — the copy strategy and node-level debug fill the remaining gaps.

The key to making this work at scale is not the technique itself but the access model : developers get self-service ephemeral container access in their own namespaces, SREs get cluster-wide ephemeral container access for production incidents, and node-level access is a break-glass procedure with audit trail and time limits. With that model in place, distroless becomes an operational non-issue rather than an obstacle.

XSLT 3.0 new features: what changed from 2.0

Alexandre Vazquez — Mon, 04 May 2026 09:00:01 +0000

XSLT 3.0 is a significant step beyond 2.0. If you are running Saxon on the backend — as XSLT Playground does — you have access to the full 3.0 feature set today. This post covers the additions that matter most in practice and shows how to try each one directly in the playground.

Streaming

The most impactful change in 3.0 is streaming. In earlier versions, the processor loads the entire source document into memory before any template can run. With 3.0 streaming, selected templates can consume the document as a stream, which drastically reduces memory usage for large inputs.

To enable streaming, declare it on the stylesheet and on the mode:

Not every expression is streamable. Saxon will tell you at compile time if a pattern is not compatible. The key restriction is that you can only visit each node once — no backward axes, no variables that hold nodes for later inspection.

Maps and arrays

XSLT 3.0 adds maps and arrays as first-class values, borrowed from XPath 3.1. A map is a collection of key-value pairs; an array is an ordered sequence that can hold any value including other maps or arrays.

Arrays work similarly:

Arrays use 1-based indexing. Use array:size(), array:get(), and array:append() from the array namespace for common operations.

JSON input and output

XSLT 3.0 can parse and produce JSON natively via json-to-xml() and xml-to-json(). This eliminates the need for a pre-processing step when your source is JSON.

The function converts JSON into a predictable XML representation defined by the W3C. Objects become elements, arrays become, and primitives become typed ,, or `elements. You transform this intermediate XML normally and then serialize it back withxml-to-json()` if needed.

Higher-order functions

You can now pass functions as arguments using xsl:function and the function() type. This enables patterns like map, filter, and fold over sequences without recursion.

`xml

The #1 notation creates a function reference with arity 1.

Packages

XSLT 3.0 introduces packages, which let you split a large stylesheet into independently compiled units that expose explicit interfaces. This is the equivalent of modules or libraries in other languages.

`xml

...

Packages reduce coupling and enable reuse across projects without copy-paste.

Try it in XSLT Playground

All the examples above run in XSLT Playground with version set to 3.0. Maps and JSON support are the quickest to explore. Paste the json-to-xml() example, provide a JSON string as input, and see the intermediate representation immediately.

XSLT 3.0 is available today. If your integration still targets 2.0, the features above are the best reasons to upgrade.

XSL online tester: run XSL and XSLT transforms in your browser

Alexandre Vazquez — Thu, 30 Apr 2026 09:00:00 +0000

XSL (Extensible Stylesheet Language) is an umbrella term that covers three related specifications: XSLT for transformations, XPath for node selection, and XSL-FO for formatting objects. When developers search for an "XSL tester" or "XSL online editor", they are usually looking for a way to run XSLT stylesheets against XML input without a local install. XSLT Playground does exactly that.

XSL vs XSLT — what is the difference?

XSL is the full family of W3C specifications:

XSLT (XSL Transformations) — transforms XML documents into other formats
XPath — the path language used inside XSLT to navigate XML trees
XSL-FO (XSL Formatting Objects) — describes page layout for print and PDF output

In practice, when people say "XSL" in an integration or development context, they almost always mean XSLT. The stylesheet file extension .xsl and .xslt are interchangeable — Saxon and most processors accept both.

Running XSL transforms online

XSLT Playground supports XSLT 1.0, 2.0, and 3.0 via the Saxon processor. To run an XSL transform:

Paste your XML source document in the input panel
Paste your XSL stylesheet in the stylesheet panel
Select the XSLT version (1.0, 2.0 or 3.0)
Click Run

The output appears immediately. If the stylesheet has errors, the error panel shows the exact line and message from Saxon.

Common XSL use cases

XML to HTML — the most common use. An XSL stylesheet walks an XML document tree and emits HTML tags:

XML to XML — reshaping or filtering a document structure:

XML to plain text or CSV — using xsl:output method="text":

XSL file extensions: .xsl vs .xslt

Both .xsl and .xslt are valid. The .xsl extension is older and more common in enterprise systems (SAP, Oracle, IBM DataPower). The .xslt extension is more explicit. Saxon accepts either. XSLT Playground accepts any content regardless of what you call it.

Testing XSL stylesheets online

The main advantage of an online XSL tester is speed of iteration. You can:

Paste a real XML payload from a production system and see what the stylesheet produces
Add parameters and test different code paths
Enable trace mode to see which templates fired and in what order
Export the entire test case (input + stylesheet + parameters) as a JSON workspace to share with a colleague

All of this is available at XSLT Playground without creating an account or installing anything.

XSL transform online vs local Saxon

For most development and debugging tasks, the online tester is faster than running Saxon locally. Use local Saxon when:

You are processing confidential data that cannot leave your network
Your input files are very large (several MB or more)
You need to integrate the transform into a build pipeline

For everything else — prototyping, debugging, sharing test cases — the online XSL tester is quicker.