DEV Community: DevOps Start

Azure SDK for Rust Migration Guide: REST to GA Crates

DevOps Start — Sat, 01 Aug 2026 09:23:29 +0000

If you have been hitting Azure REST endpoints from Rust with hand-rolled reqwest calls and a pile of header-signing code, you can delete most of it now. The Azure SDK for Rust reached general availability in mid-2026, shipping stable 1.0 crates for Core, Identity, Key Vault, and Storage. This guide walks you through replacing raw REST access with the official clients: adding the crates, wiring up DefaultAzureCredential, reading a Key Vault secret, downloading a blob, and turning on retries and tracing. Every step maps a piece of REST plumbing you can retire to the typed client that replaces it.

The migration is not a rewrite. The clients follow the same design patterns as the .NET, Python, Go, and Java SDKs, so the shapes are predictable. What changes is that authentication, retries, and pagination stop being your problem and become the SDK's.

What actually went GA

The GA wave promoted a specific set of crates to stable 1.0. Knowing which ones are production-ready keeps you from pinning a preview crate by accident:

Crate	Purpose
`azure_core`	Shared pipeline: HTTP, retries, auth traits, error types
`azure_identity`	Credential types, including `DefaultAzureCredential`
`azure_security_keyvault_secrets`	Key Vault secrets client
`azure_security_keyvault_keys`	Key Vault keys client
`azure_security_keyvault_certificates`	Key Vault certificates client
`azure_storage_blob`	Blob upload, download, and container operations
`azure_storage_queue`	Queue send and receive
`azure_core_opentelemetry`	Distributed tracing bridge for the pipeline

Two things did not make the GA cut, and you should plan around them. Event Hubs is slated for the next stable wave, and Cosmos DB support is in active development with a stable release expected later in 2026. If your service depends on either, keep your existing REST or preview code for those paths and migrate the rest now. Microsoft's Rust on Azure overview tracks the current crate status if you want to confirm before you pin a version.

Step 1: add the crates

Start with a clean dependency set. From your crate root, let Cargo resolve the latest stable versions rather than guessing patch numbers:

$ cargo add azure_identity azure_security_keyvault_secrets azure_storage_blob tokio

That pulls azure_core transitively, so you rarely add it by hand. If you prefer to pin versions explicitly, your Cargo.toml ends up looking like this:

[dependencies]
azure_identity = "1.0"
azure_security_keyvault_secrets = "1.0"
azure_storage_blob = "1.0"
azure_core = "1.0"
tokio = { version = "1", features = ["full"] }

The SDK is async-first and expects a Tokio runtime, which is why tokio is in the list with the full feature. If your service already runs on async-std or a custom runtime, you will need a compatibility shim, because the clients are built and tested against Tokio.

Verify the tree resolved cleanly before you write any client code:

$ cargo tree -p azure_core

You should see a single azure_core 1.x in the output. If two versions show up, one of your other Azure crates is still on a preview release, and mixing a 1.0 core with a 0.x client is the most common source of trait-mismatch errors during migration.

Step 2: replace your auth code with azure_identity

This is where you delete the most code. If your REST client was fetching tokens from the IMDS endpoint or juggling a client secret from an environment variable, DefaultAzureCredential replaces all of it with one type that tries a chain of sources in order: environment variables, workload identity, managed identity, and your local developer sign-in.

Here is the full pattern for constructing a credential and handing it to a client:

use azure_identity::DefaultAzureCredential;
use azure_security_keyvault_secrets::SecretClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let credential = DefaultAzureCredential::new()?;

    let client = SecretClient::new(
        "https://your-vault-name.vault.azure.net/",
        credential.clone(),
        None,
    )?;

    // client is now ready to make authenticated calls
    Ok(())
}

The credential.clone() is cheap. Credentials are reference-counted internally, so cloning shares the same token cache rather than re-authenticating. Build one credential at startup and clone it into every client you construct.

For local development, DefaultAzureCredential will pick up the identity you signed in with through az login. Confirm that works before you run anything:

$ az login
$ az account show --query user.name -o tsv

If you want to be explicit about using developer tooling and skip the managed-identity probes (which add latency and noisy log lines when you run locally), swap in DeveloperToolsCredential:

use azure_identity::DeveloperToolsCredential;

let credential = DeveloperToolsCredential::new(None)?;

A note on scope. In production, prefer managed identity so there is no secret to leak, and grant the identity only the specific Key Vault and Storage roles it needs. The same discipline that keeps secrets out of your CI pipelines applies here: the fewer long-lived credentials your Rust service holds, the smaller your blast radius. If you run across several Azure subscriptions, the multi-subscription patterns from the Terraform side carry over, because DefaultAzureCredential honors the same AZURE_SUBSCRIPTION_ID and tenant environment variables.

Step 3: migrate a Key Vault secret read

A typical pre-SDK secret fetch was a signed GET against https://your-vault.vault.azure.net/secrets/{name}?api-version=7.4, plus JSON parsing to dig the value out of the response envelope. Replace the whole thing with a typed call:

use azure_identity::DefaultAzureCredential;
use azure_security_keyvault_secrets::SecretClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let credential = DefaultAzureCredential::new()?;
    let client = SecretClient::new(
        "https://your-vault-name.vault.azure.net/",
        credential.clone(),
        None,
    )?;

    let secret = client
        .get_secret("database-password", "", None)
        .await?
        .into_body()
        .await?;

    if let Some(value) = secret.value {
        println!("secret length: {}", value.len());
    }

    Ok(())
}

Three details matter here. First, the empty string for the version argument means "current version," which is what you almost always want. Pass a specific version string only when you need to pin to a historical value. Second, the response is a two-stage unwrap: .await? gives you the HTTP response, and .into_body().await? deserializes it into the typed Secret model. Third, never log the secret value itself. Print its length or a hash if you need a sanity check, as the example does above.

Run it with your vault name substituted in:

$ cargo run

If you get a 403, the problem is almost always RBAC rather than code. Grant your identity the Key Vault Secrets User role:

$ az role assignment create \
    --role "Key Vault Secrets User" \
    --assignee "$(az ad signed-in-user show --query id -o tsv)" \
    --scope "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<vault-name>"

The authentication overview on Microsoft Learn documents the full credential chain and which environment variables each link reads.

Step 4: migrate blob storage access

Blob access follows the same construction pattern. You build a client against the account URL, then operate on containers and blobs. Here is a download that streams the blob body into memory:

use azure_identity::DefaultAzureCredential;
use azure_storage_blob::BlobClient;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let credential = DefaultAzureCredential::new()?;

    let blob_client = BlobClient::new(
        "https://yourstorageaccount.blob.core.windows.net/",
        "reports".to_string(),
        "2026-q3.json".to_string(),
        credential.clone(),
        None,
    )?;

    let response = blob_client.download(None).await?;
    let body = response.into_body().collect().await?;
    println!("downloaded {} bytes", body.len());

    Ok(())
}

The constructor arguments are the account URL, the container name, the blob name, the credential, and an options struct. Passing None for options accepts the defaults, which is the right starting point. The download call returns a response whose body you collect into bytes. For large blobs you would stream chunks rather than collecting the whole body, but collecting is fine for config files and small artifacts.

The same authorization rule applies: the identity needs the Storage Blob Data Reader role (or Contributor if you also write). Assign it the same way you did for Key Vault, swapping the role name and the scope to your storage account.

Uploading is the mirror image. You build the same client and hand it the bytes plus a length:

use azure_core::http::RequestContent;

let data = b"{\"generated_at\":\"2026-07-31\"}".to_vec();
let len = data.len() as u64;

blob_client
    .upload(RequestContent::from(data), true, len, None)
    .await?;

The boolean argument is the overwrite flag. Passing true replaces an existing blob of the same name, and false fails the call if the blob already exists, which is the safer default when you are writing an object that should be created exactly once. As with the download, the options struct is None until you need to set content type, metadata, or an access tier.

Step 5: retries and resilience are already on

One of the quieter wins in the GA release is that resilience is built into azure_core's pipeline and on by default. Transient failures (HTTP 429, 503, and connection resets) are retried automatically with exponential backoff. You do not write retry loops anymore, and you should delete any you carried over from your REST client, because doubling up on retries turns a brief throttle into a much longer stall.

When you do need to tune the behavior, you set it through the client options struct rather than wrapping calls yourself. The pattern looks like this:

use azure_core::http::policies::RetryOptions;

// Construct retry options and pass them through the client's options struct
let retry = RetryOptions::exponential(Default::default());

The GA release also added challenge-based authentication, so the clients work correctly in sovereign and private cloud environments where the token audience is discovered from a challenge response rather than assumed. If you previously special-cased Azure Government or a private cloud, that branch can likely go.

Start with the defaults. The built-in policy is tuned for the common case, and premature retry tuning is a frequent way to make throttling worse rather than better.

Step 6: turn on distributed tracing

If your service already emits OpenTelemetry spans, you can thread Azure SDK calls into the same traces using the azure_core_opentelemetry crate. It bridges the SDK's internal pipeline instrumentation to your OpenTelemetry tracer, so every Key Vault or Storage call shows up as a child span under your request.

Add the crate:

$ cargo add azure_core_opentelemetry

You wire it in by attaching the OpenTelemetry tracer provider to the client options, after which HTTP calls the SDK makes are recorded as spans with the operation name, target, and status. The HTTP logging layer sanitizes secrets by default, so authorization headers and secret values are redacted before anything reaches your log sink. That default matters: it means turning on verbose SDK logging during an incident will not accidentally dump a Key Vault secret into your log aggregator.

If you are new to wiring OpenTelemetry through an application, the mechanics of tracer setup and exporters are the same ones covered in this walkthrough on setting up observability with OpenTelemetry; the Azure crate simply feeds the SDK's own spans into that pipeline instead of you instrumenting each call by hand.

Handle errors with the typed error model

Your REST client probably branched on raw status codes pulled out of a response struct. The SDK gives you a typed azure_core::Error instead, and the useful move during migration is to inspect its HTTP status when you need to distinguish a genuine "not found" from a transient failure the pipeline already gave up retrying.

A common case is treating a missing secret as an expected outcome rather than a hard failure:

use azure_core::http::StatusCode;

match client.get_secret("optional-flag", "", None).await {
    Ok(response) => {
        let secret = response.into_body().await?;
        println!("found: {}", secret.value.unwrap_or_default().len());
    }
    Err(err) if err.http_status() == Some(StatusCode::NotFound) => {
        println!("secret not set, using default");
    }
    Err(err) => return Err(err.into()),
}

The pattern is worth internalizing because it is identical across every GA client. A Key Vault 404, a Storage 404, and an Identity failure all surface through the same azure_core::Error type with the same http_status() accessor. That consistency is a large part of why migrating the second and third service is faster than the first: once you have written error handling for one client, you have written it for all of them.

Resist the urge to match on error strings. The status accessor is stable across releases; the human-readable message is not, and matching on it will break the next time the wire format changes a word.

Deploying to AKS with workload identity

The payoff of DefaultAzureCredential shows up in production, where you want zero secrets in the container. On Azure Kubernetes Service, workload identity federates your pod's service account to an Azure managed identity, and the credential picks it up automatically through environment variables the workload-identity webhook injects. Your Rust code does not change at all between laptop and cluster, which is the point.

The cluster-side wiring is three annotations and a federated credential:

$ az identity federated-credential create \
    --name rust-app-federated \
    --identity-name rust-app-identity \
    --resource-group <rg> \
    --issuer "$(az aks show -g <rg> -n <cluster> --query oidcIssuerProfile.issuerUrl -o tsv)" \
    --subject "system:serviceaccount:<namespace>:<service-account>"

Once the federated credential exists and your pod's service account carries the azure.workload.identity/client-id annotation, the same binary you tested locally authenticates as the managed identity with no code path difference. Grant that identity the same Key Vault and Storage roles you used during local testing, scoped to production resources, and you have a service with no long-lived credential anywhere in the deployment.

A migration checklist

Work through your codebase in this order to keep the change reviewable:

Inventory every place you call an Azure REST endpoint from Rust. Group them by service (Key Vault, Storage, and anything not yet GA).
Add the GA crates for the services you found, and run cargo tree to confirm a single azure_core 1.x in the graph.
Replace token acquisition with a single DefaultAzureCredential built at startup and cloned into each client.
Convert one service at a time. Migrate Key Vault first, since it is usually the smallest surface, then Storage.
Delete your hand-written retry loops and header-signing helpers once the typed client covers that path.
Leave Event Hubs and Cosmos DB on your existing code until their stable crates ship, and tag those spots with a comment so you remember to revisit.
Add azure_core_opentelemetry last, once functionality is proven, so tracing reflects the new call paths.

Migrating incrementally like this is the same principle behind any staged platform move, including the Azure DevOps to GitHub playbook: change one bounded surface, verify it in production, then move to the next. Do not try to flip every service in a single pull request.

Common migration gotchas

A few things trip people up on the first pass:

Mixed crate versions. A preview 0.x client against a 1.0 azure_core produces confusing trait errors. Pin everything to the 1.0 line and re-run cargo tree.
Forgetting .into_body(). The first .await? gives you the response, not the parsed model. The typed value comes from the second .into_body().await?. Skipping it is a frequent compile-time confusion.
Rebuilding credentials per request. Constructing DefaultAzureCredential inside a request handler defeats the token cache and adds latency. Build once, clone many.
RBAC, not code. A 401 usually means the audience or tenant is wrong; a 403 almost always means a missing role assignment. Check az role assignment list before you suspect the SDK.

Where this leaves you

After this migration, your Azure access code in Rust is smaller, typed, and consistent with how the rest of your Azure fleet is written in other languages. Authentication is one credential built at startup. Retries and secret redaction are handled by the pipeline. Tracing plugs into the OpenTelemetry setup you already run. The two gaps to watch are Event Hubs and Cosmos DB, both of which have stable crates on the roadmap, so keep those integration points isolated and ready to swap.

Start with a single non-critical service, prove the pattern end to end in a staging environment, and use the checklist above to roll it out service by service. The crate status page on Microsoft Learn is worth a bookmark, because the GA surface is still expanding and the next wave will let you delete even more REST plumbing.

Manage OTel Collectors at Scale with OpAMP

DevOps Start — Tue, 28 Jul 2026 09:56:39 +0000

If you run more than a handful of OpenTelemetry Collectors, you already know the pain: a config change means SSHing into boxes, redeploying DaemonSets, or babysitting a Git pipeline per cluster, and you never quite trust that every agent is running the config you think it is. OpAMP fixes exactly that. It is a protocol that lets a central server push configuration to a fleet of Collectors, watch their health, and roll changes out in stages, without you touching each host. This post walks through how OpAMP works, the two ways a Collector can speak it, and the config you need to wire one up.

The problem OpAMP solves

A single Collector is easy. A hundred of them, spread across clusters, VMs, and edge nodes, is a fleet-management problem that has nothing to do with telemetry itself. Every observability team eventually builds some version of the same thing: a way to ship a new pipeline config, confirm it actually applied, and back it out when a processor starts dropping spans.

Without a management protocol you end up gluing that together from ConfigMaps, Ansible runs, and dashboards that only tell you an agent is alive, not what config it is actually running. Config drift creeps in. One node keeps an old sampling rate for months because its rollout quietly failed and nobody noticed.

OpAMP, the Open Agent Management Protocol, is the OpenTelemetry answer to this. Splunk donated it to the project in 2022, and it has since become the standard control channel for the Collector. It is worth pairing with a clear-eyed view of what a Collector actually is versus lighter agents; the OpenTelemetry Collector vs Grafana Alloy comparison covers that trade-off if you are still choosing a data plane.

What OpAMP actually is

OpAMP is a client/server network protocol for remote management of large fleets of data-collection agents. It is transport-flexible: agents connect to the server over either plain HTTP or a WebSocket, and the WebSocket path gives you a persistent bidirectional channel so the server can push a new config the moment you save it.

The protocol is deliberately narrow. It handles a specific set of jobs:

Remote configuration: the server sends a config, the agent applies it and reloads.
Health and status reporting: agents report whether they are healthy and what they are doing.
Effective config reporting: agents send back the config they are actually running, so you can detect drift.
Own-telemetry reporting: agents can stream their own metrics, logs, and traces about themselves.
Package and version management: the server can discover an agent's version and, optionally, push binary updates.

Notice what is not in that list: OpAMP does not define what your telemetry pipeline looks like. It carries an opaque config blob to the agent and lets the agent decide what to do with it. For a Collector, that blob is just your normal Collector YAML. OpAMP is the envelope, not the letter.

That separation is the whole design. The OpenTelemetry management docs describe OpAMP as the recommended path for fleet management, and the protocol spec is published at opentelemetry.io/docs/specs/opamp.

Two ways a Collector speaks OpAMP

There are two distinct integration points, and mixing them up is the most common early mistake.

Approach	What it is	What it manages
`opamp` extension	An extension compiled into the Collector	Reports health, effective config, and identity to the server
OpAMP Supervisor	A separate process that wraps the Collector	Full lifecycle: applies remote config, restarts, and reports on the Collector's behalf

The built-in opamp extension is the lightweight option. It lets a Collector announce itself to an OpAMP server and report status, but the extension alone cannot rewrite the Collector's config and restart it, because a running process cannot swap out the config file it booted from and cleanly reload every component.

The Supervisor closes that gap. It is a small parent process that launches the Collector as a child, holds the OpAMP connection itself, and owns the Collector's lifecycle. When a new config arrives, the Supervisor writes it to disk, restarts the Collector against it, and reports the result upstream. For actual remote-configuration-driven fleet management, the Supervisor is the path you want. It lives in the opentelemetry-collector-contrib repository under cmd/opampsupervisor.

Wiring up the Supervisor

The Supervisor takes its own config file, usually supervisor.yaml, which is separate from the Collector config it manages. Here is a representative example:

server:
  endpoint: wss://opamp.example.com:4320/v1/opamp
  tls:
    insecure_skip_verify: false
capabilities:
  accepts_remote_config: true
  reports_effective_config: true
  reports_own_metrics: true
  reports_own_logs: true
  reports_health: true
  reports_remote_config: true
agent:
  executable: /usr/local/bin/otelcol-contrib
storage:
  directory: /var/lib/otelcol-supervisor

Three blocks matter here.

The server block points at your OpAMP backend. The wss:// scheme selects the WebSocket transport, and 4320 is the conventional OpAMP port used across the project's examples. Keep insecure_skip_verify at false in anything real; you are opening a control channel that can change what runs on your hosts, so certificate verification is not optional.

The capabilities block is a set of explicit opt-ins. Nothing is implied. If you want the server to be able to push config, you must set accepts_remote_config: true. If you want drift detection, reports_effective_config: true is what sends the running config back. Turning these on individually means you can start conservative (health only) and add remote config later once you trust the setup.

The agent block tells the Supervisor which binary to run and manage, and storage is where it persists the last-known remote config so a restart does not lose it.

You start the Supervisor, not the Collector, and let it own the child process:

$ otelcol-supervisor --config /etc/otelcol/supervisor.yaml

From here on you never start the Collector directly. The Supervisor connects to the server, sends an AgentDescription that identifies this instance, and waits for config. When you push a new pipeline from the server, the Supervisor lands it on disk and cycles the Collector.

What a remote config flow looks like

Once an agent is connected with accepts_remote_config enabled, the loop is simple to reason about:

You edit a Collector config in the server's UI or API and target a set of agents.
The server sends the config over the open connection.
The Supervisor writes it to its storage directory and restarts the Collector against it.
The Collector boots, and the Supervisor reports back the new effective config and health.
The server marks the rollout applied for that agent, or surfaces an error if the Collector rejected the config.

Step 4 is the part teams underrate. Because the agent reports its effective config, you get a closed loop: the server does not just assume the push worked, it sees the config the Collector is genuinely running. That is how you catch the node that silently kept an old sampling rate. A dashboard built on reports_effective_config shows you real drift instead of a green checkmark that means nothing.

Health, identity, and self-telemetry

The reporting capabilities are worth turning on even before you trust remote config. With reports_health: true, each agent tells the server whether it is up and functioning, which beats inferring liveness from whether metrics happen to be flowing. Health here means the Collector's own view of itself, including whether its pipelines started cleanly.

Identity comes from the AgentDescription message. Every connecting agent sends a set of attributes about itself: hostname, OS, Collector version, and any custom labels you attach. Those labels are the backbone of fleet management, because they are how you target a subset of agents. You push a config to service.namespace=payments and only those Collectors receive it. Getting your labeling scheme right early is more important than the config content itself; without good labels, every rollout is all-or-nothing.

With reports_own_metrics: true, the Collector streams its internal metrics (queue sizes, dropped spans, export failures) as part of the same channel. Feed those into your existing metrics backend. If you scrape them with Prometheus, the Prometheus documentation covers the receiver side, and Grafana's own agent tooling documented at grafana.com shows how a similar management model looks in a different distribution.

Package management, and why to be careful

OpAMP can also push binary updates. The server can discover an agent's version and, with the right capability enabled, deliver a new package so the agent upgrades or downgrades itself. On paper this is the dream: patch a Collector CVE across a thousand hosts from one console.

In practice, treat auto-update as the most dangerous capability in the protocol and turn it on last. A bad config push restarts a Collector; a bad package push replaces the binary on every targeted host at once. Stage it the way you would any other production rollout: a canary group first, watch health and effective-config reporting, then widen. The protocol gives you the mechanism, not the judgment. Keep package management off until your health and config feedback loops are boringly reliable.

Rolling out across a real fleet

Scale is where the labeling discipline pays off. A sane rollout pattern looks like this:

Tag everything. Attach environment, region, and service labels to every agent via its identifying attributes.
Canary by label. Push a config change to a small, clearly labeled canary group first.
Watch effective config. Confirm the canary agents report the new config as their effective config, not just that they acknowledged the push.
Watch health. Give it long enough to catch a pipeline that starts fine but fails under load.
Widen in waves. Expand to the next label group, then the rest, with the same two checks each time.

This is the same staged-rollout thinking you would apply to a Kubernetes deployment, and it composes well with cost and reliability work already in your pipeline. If you are instrumenting application workloads at the same time, the collector fleet you manage with OpAMP is what those pipelines feed into; see How to Set Up LLM Observability with OpenTelemetry and, for cluster-scale patterns, LLM Observability on Kubernetes for the data-plane side of the same system.

Picking a server

OpAMP is only half a system; you also need a server that speaks it. You have two routes. You can run a managed or open-source OpAMP backend such as BindPlane, which grew out of the same observIQ work that seeded the protocol, and get a UI, agent inventory, and config management out of the box. Or you can build against opamp-go, the reference server and client implementation, if you want the control channel wired directly into your own platform.

For most teams, starting with an existing server is the right call. The value of OpAMP is the fleet view and the safe rollout mechanics, and rebuilding those from the reference libraries is a real engineering investment. Start managed, learn the operational patterns, and only build your own server if you have a platform reason the off-the-shelf options cannot meet.

Where this leaves you

OpAMP turns a pile of independently configured Collectors into a fleet you can actually operate. The mental model is small: the Supervisor owns the Collector's lifecycle, the capabilities block is a set of explicit opt-ins, and effective-config reporting is what makes the whole thing trustworthy instead of hopeful. Start with health and effective-config reporting so you can see your fleet, add remote config once you trust the feedback loop, and leave package auto-update for last.

If you are still deciding whether the full Collector is even the right data plane for your fleet, settle that first, then bring OpAMP in to manage whatever you land on. The protocol is agnostic about the pipeline; it just makes running a lot of them survivable.

Validate Kubernetes Manifests with Flux Schema

DevOps Start — Sun, 26 Jul 2026 09:26:07 +0000

If you run GitOps with Flux, a broken manifest usually gets caught the slow way: it merges, the reconciler chokes, and you find out from a failing Kustomization. Flux Schema, the plugin that shipped with Flux 2.9, moves that check left into CI. It validates every YAML document against JSON Schema and CEL rules using the same evaluation logic as the Kubernetes API server, so a bad field fails the pull request instead of the cluster.

Install and run it

Flux Schema is a CLI plugin, not part of the core binary. Install it through the plugin system:

$ flux plugin install schema
$ flux schema --help

Pin a version in CI so a new release never changes your gate's behavior mid-sprint:

$ flux plugin install schema@0.5.0

Point it at a directory of manifests and it validates each document:

$ flux schema validate ./manifests

It ships with built-in schemas for Kubernetes, OpenShift, Gateway API, and the Flux CRDs, so a fresh install already knows your HelmRelease and Kustomization kinds without any setup. Strict validation flags unknown fields, wrong types, and missing required properties as hard errors, which catches the typos kubectl apply --dry-run=client quietly ignores.

What CEL adds over plain schema checks

JSON Schema catches shape problems: a string where an int belongs, a misspelled key. CEL rules catch logic problems. Because Flux Schema runs the x-kubernetes-validations rules embedded in CRDs through the same CEL engine the API server uses, a manifest that violates a cross-field constraint (say, a replica count that must stay below a limit, or two mutually exclusive fields both set) fails in CI with the exact message the cluster would have returned. You are testing against the real admission logic, not a stale copy of it.

Wire it into a config file

Drop a .fluxschema.yml at your repo root to control what gets checked. The file uses the schema.plugin.fluxcd.io/v1beta1 API and a Config kind:

apiVersion: schema.plugin.fluxcd.io/v1beta1
kind: Config
skipKind:
  - Secret
skipJSONPath:
  - "$.metadata.annotations"

skipKind, skipFile, and skipJSONPath let you exclude the things that legitimately fail strict checks, like sealed secrets or generated fields. Then the command reads it automatically:

$ flux schema validate ./manifests --config .fluxschema.yml

Put it in the pull request gate

The real payoff is in CI. Flux Schema ships two composite GitHub Actions: setup installs the CLI on the runner, and validate auto-detects your kustomize overlays, renders them, and validates every rendered document. A minimal gate looks like this:

name: validate-manifests
on: [pull_request]
jobs:
  flux-schema:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: fluxcd/flux-schema/actions/validate@main
        with:
          config: .fluxschema.yml

Because the Ecosystem Catalog behind the plugin refreshes daily from upstream releases, your CI validates against the current API versions rather than whatever was frozen months ago. That matters most right after a Kubernetes minor bump, when a deprecated field you have used for a year suddenly needs to change.

One habit worth keeping: run flux schema validate locally before you push, not just in CI. The feedback loop is a second or two, and it saves a round trip through the runner. If you are still deciding between Flux and Argo CD for this kind of workflow, our Argo CD vs Flux guide compares them for multi-cluster setups, and our GitOps testing strategies piece covers where manifest validation fits in a broader test pyramid.

Read the Flux Schema announcement on fluxcd.io for the full catalog details, then add the action to one repo and watch the first bad manifest fail its PR instead of your cluster.

Fix GitLab CI "dial tcp: lookup docker" no such host error

DevOps Start — Sat, 25 Jul 2026 09:13:31 +0000

The fast fix

If your GitLab CI job dies with error during connect: ... dial tcp: lookup docker on 127.0.0.11:53: no such host, your docker client resolved DOCKER_HOST fine but the DNS name docker does not exist on the job's network. That name is the alias of the docker:dind service, and it only registers when the service container actually starts. The usual fix is to define the service with the -dind image tag and an explicit alias, and point DOCKER_HOST at TLS port 2376:

build:
  image: docker:28.3
  services:
    - name: docker:28.3-dind
      alias: docker
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_CERT_PATH: "/certs/client"
    DOCKER_TLS_VERIFY: "1"
  script:
    - docker info
    - docker build -t my-app .

That covers the common case on the docker executor. If you are on the Kubernetes executor, or the service still refuses to resolve, keep reading. The name resolution is the whole game here, and there are three distinct reasons it fails.

Why this error is not the daemon-connection error

The address in the message tells you exactly how far the client got. This DNS variant is different from the two connection variants:

error during connect: Post "https://docker:2376/v1.44/info":
dial tcp: lookup docker on 127.0.0.11:53: no such host

The client tried to resolve the hostname docker through the container DNS resolver (127.0.0.11 on a Docker bridge network) and got back nothing. That is a name-resolution failure, not a refused connection. Compare it to the tcp://docker:2375 form, where the name resolves but the daemon is unreachable, covered in the Docker daemon connection error write-up, and the unix:///var/run/docker.sock form, where DOCKER_HOST was never set at all, covered in the socket-variant fix. If your message says no such host, the client never reached any daemon because the name pointed at nothing.

The three real causes

1. The dind service never started, so its alias never registered

This is the most common cause and the least obvious. GitLab registers the docker alias on the build network only when the docker:dind service container comes up. If that container fails to start, the alias is missing and every lookup returns no such host. Three things stop it from starting:

You used the plain image, not the -dind tag. docker:28.3 ships only the client. You need docker:28.3-dind, which bundles dockerd. A plain docker service starts, exits immediately (no daemon to run), and takes its alias down with it.
The runner is not in privileged mode. docker:dind runs its own daemon and needs privileged = true in the runner config.toml. Without it the container cannot start the daemon and dies during boot.
The image failed to pull. A registry rate limit or a typo in the tag means the service container never exists. Check the job log's Preparing the "docker" ... service lines near the top.

Pin the runner config and confirm privileged mode:

$ grep -A3 '\[runners.docker\]' /etc/gitlab-runner/config.toml

You want to see privileged = true. If it says false or is absent, that is your problem.

2. A renamed service image broke the derived alias

GitLab derives a service's default alias from its image name. docker:28.3-dind becomes the alias docker. But if you pull the image through a mirror or a private registry, the derived alias changes and docker stops resolving:

services:
  # WRONG: alias becomes "my-mirror.example.com__docker", not "docker"
  - name: my-mirror.example.com/library/docker:28.3-dind

The daemon is running, but under a name your DOCKER_HOST never asks for. Always set the alias explicitly when the image path is anything other than the bare docker name:

services:
  - name: my-mirror.example.com/library/docker:28.3-dind
    alias: docker

3. The Kubernetes executor puts services on localhost

On the docker executor, the service is a linked container with its own alias, so tcp://docker:2376 is correct. The Kubernetes executor is different: every service runs as a container in the same Pod as the build, sharing one network namespace. They all reach each other on localhost, and the docker alias may not resolve at all. If you copied a working docker-executor config onto a Kubernetes runner, this is why it broke.

Set the host to localhost for the Kubernetes executor:

variables:
  DOCKER_HOST: tcp://localhost:2376
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_CERT_PATH: "/certs/client"
  DOCKER_TLS_VERIFY: "1"

Behavior here has shifted across runner versions, so verify against your own version rather than trusting a blog snippet. If tcp://localhost:2376 gives you connection refused instead of no such host, the name resolved and you are back to a daemon-startup problem (see cause 1). The two messages tell you which side of the wall you are on.

A checklist that isolates the cause in under a minute

Work through these in order the next time the job goes red:

Read the address in the error. no such host is DNS. connection refused is a live-but-unreachable daemon. Do not fix the wrong one.
Confirm the -dind tag. Grep your .gitlab-ci.yml for the service image. No -dind, no daemon, no alias.
Check privileged mode in the runner config.toml as shown above.
Match DOCKER_HOST to your executor. tcp://docker:2376 for the docker executor, tcp://localhost:2376 for Kubernetes.
Match the port to TLS. With DOCKER_TLS_CERTDIR set, the daemon listens on 2376. Blank it out and it listens on the plain 2375. A mismatch here surfaces as a connection error once the name resolves.
Scan the top of the job log for the service Preparing/Waiting lines. A service that logs an exit code never registered its alias.

Confirm the daemon is reachable before you build

Once the service is up and named correctly, prove the client can talk to it before your real build steps run. A one-line docker info at the start of script fails fast and loud instead of letting a ten-minute build collapse at the push step:

  script:
    - docker info
    - docker build -t my-app .
    - docker push my-app

The daemon's TCP listener and its 2375-versus-2376 split are documented in the Docker daemon reference; the TLS port only exists because DOCKER_TLS_CERTDIR generated certificates on boot. Keep the port and the TLS variables in sync and the no such host error stays gone. Once your images build cleanly, tightening them is the next job, and multi-stage builds are where that starts.

GitLab CI "Cannot connect to unix:///var/run/docker.sock"

DevOps Start — Thu, 23 Jul 2026 09:39:01 +0000

The fast fix

If your GitLab CI job fails with Cannot connect to the Docker daemon at unix:///var/run/docker.sock, your docker client is looking for a local socket that does not exist inside the job container, because DOCKER_HOST is not set. Point the client at the docker:dind service over TCP and the error goes away:

build:
  image: docker:28.3
  services:
    - name: docker:28.3-dind
      alias: docker
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_CERT_PATH: "/certs/client"
    DOCKER_TLS_VERIFY: "1"
  script:
    - docker info
    - docker build -t my-app .

That is the whole fix for the common case. The rest of this page explains why the socket variant of the error is different from the tcp://docker:2375 variant, and covers the two other setups (socket-mounted runners and the Kubernetes executor) where the same message shows up for a different reason.

Why you get the unix socket variant specifically

This error is not the same as Cannot connect to the Docker daemon at tcp://docker:2375. The address in the message tells you exactly what the client tried:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

When DOCKER_HOST is empty, the Docker CLI falls back to its compiled-in default, the local unix socket at /var/run/docker.sock. Inside a GitLab CI job that uses the docker executor, that socket file simply is not there. The daemon runs in a separate docker:dind service container, not in your job container, so there is nothing listening on the local socket. The client connects, finds no socket, and prints the message above.

The tcp://docker:2375 form is the opposite problem: DOCKER_HOST is set correctly but the dind service is not reachable (missing service, no privileged mode, or a TLS mismatch). If you are seeing that address instead, read the companion write-up on the tcp://docker:2375 form of this error, which walks the service and privileged-mode causes in detail. This page is about the case where the client never even tried the network, because nothing told it to.

The three real causes

1. DOCKER_HOST is unset

This is the usual cause. You added services: - docker:dind but never set DOCKER_HOST, so the client ignores the service container and hits the local socket. Set it as a job or top-level variable:

variables:
  DOCKER_HOST: tcp://docker:2376

Use port 2376 (TLS) when DOCKER_TLS_CERTDIR is set, and 2375 (plain) when you disable TLS with DOCKER_TLS_CERTDIR: "". Mixing them is the second most common mistake, covered next.

2. TLS is half-configured

Docker Engine 19.03 and later turns on TLS between the client and the daemon by default. The dind service generates certificates into the path named by DOCKER_TLS_CERTDIR. If you set the certs directory but point DOCKER_HOST at the plain-text port 2375, or you set neither cert variable, the handshake fails and the client can end up falling back to the socket. Keep the three TLS variables consistent:

variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_CERT_PATH: "/certs/client"
  DOCKER_TLS_VERIFY: "1"

If you would rather skip TLS for an internal runner, disable it cleanly and use port 2375:

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""

Pick one style and set every variable it needs. Do not leave DOCKER_TLS_CERTDIR set while talking to 2375.

3. The runner cannot start dind at all

If DOCKER_HOST is right but the dind container never boots, the client still fails, sometimes reporting the socket address after a retry. The docker:dind service needs privileged mode in the runner's config.toml:

[[runners]]
  executor = "docker"
  [runners.docker]
    privileged = true

With privileged = false or the key absent, dind cannot create its own daemon and no address will work. Confirm this on the runner host before touching your pipeline file.

Socket-mounted runners are the exception

Some self-managed runners deliberately mount the host's Docker socket instead of running dind. In that setup /var/run/docker.sock is supposed to exist inside the job, and the same error means the mount is missing or the path is wrong. The runner's config.toml binds the host socket:

[[runners]]
  executor = "docker"
  [runners.docker]
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

With this style you do not set DOCKER_HOST at all, because the local socket is the daemon. If you get the unix socket error here, check that the host actually has Docker running and that the bind path in volumes matches the real socket location. This approach shares the host daemon with every job, so treat it as a security tradeoff, not a default. The dind service is the safer choice for untrusted pipelines.

The Kubernetes executor needs the same variables

Running GitLab Runner on Kubernetes does not change the fix, but it adds one gotcha. Each dind service runs as a sidecar container in the same Pod, so DOCKER_HOST: tcp://docker:2376 still resolves through the service alias. What breaks people is TLS cert sharing between containers in the Pod. Set an explicit shared volume for the certs directory in the runner's Helm values or pin DOCKER_TLS_CERTDIR to a path both containers can read. If certs land in a directory only the dind container sees, the client falls back to the socket and you get this exact error inside Kubernetes.

Verify the fix

Add a one-line probe to the top of your job and rerun the pipeline:

$ docker info --format '{{.ServerVersion}}'

If that prints a version string, the client reached the daemon and your docker build will work. If it still fails, echo the variable to confirm the pipeline actually applied it:

$ echo "DOCKER_HOST=$DOCKER_HOST"

An empty value here means your variables block is scoped wrong (defined under the wrong job, or shadowed by a group or project variable). Move it to the top level or the specific job that runs Docker.

Prevention checklist

Always set DOCKER_HOST explicitly when you use docker:dind. Never rely on the default socket in a CI job.
Keep the TLS variables consistent: 2376 with a cert dir, or 2375 with DOCKER_TLS_CERTDIR: "". Never mix them.
Pin the image and the dind service to the same tag, for example docker:28.3 and docker:28.3-dind, so client and daemon versions match.
Confirm privileged = true in the runner config.toml before pushing jobs that need dind.
On Kubernetes, give the certs directory a shared volume so both containers in the Pod can read it.

For the daemon flags behind all of this, including the -H host option and the default unix:///var/run/docker.sock binding, see the Docker daemon reference. Once your builds connect reliably, the multi-stage build guide is a good next step for shrinking the images those pipelines produce.

Fix Flux SOPS MAC mismatch in kustomize-controller

DevOps Start — Tue, 21 Jul 2026 09:41:48 +0000

A SOPS MAC mismatch in Flux almost always means one thing: the encrypted file was changed outside of sops. SOPS signs every file with a message authentication code computed over the plaintext at encrypt time. When kustomize-controller decrypts the file and recomputes that code, it no longer matches, so Flux refuses the Secret and stops reconciling. You cannot patch the ciphertext by hand to fix this. The reliable fix is to recover the real values and re-encrypt the file from scratch, which is what the rest of this page walks through.

What the error looks like

The failure shows up on the Kustomization, not the Secret. Check the object status and the controller logs:

$ flux get kustomizations --all-namespaces
$ kubectl -n flux-system logs deploy/kustomize-controller | grep -i "mac mismatch"

You will see a line similar to this:

Kustomization/flux-system/apps: Reconciliation failed after 1.2s:
failed to decrypt secret 'db-credentials': Error getting data key:
Error decrypting tree: MAC mismatch. Expected '9f8c...', got 'a71b...'

The two hashes are the point. "Expected" is the MAC that sops stored when the file was last encrypted correctly. "Got" is the MAC recomputed from the plaintext it just decrypted. They differ, so the content changed since the last clean encryption.

Why it happens

Four situations produce a MAC mismatch in practice.

Cause	Signal	Fix path
File hand-edited outside sops	Recent commit touched the `.enc.yaml` directly	Recover plaintext, re-encrypt
Git merge conflict resolved by hand	Merge commit on the encrypted file	Re-merge from plaintext
`encrypted_regex` / `mac_only_encrypted` changed	`.sops.yaml` edited since last encrypt	Align rules, re-encrypt
Copy-paste truncation or corruption	Value looks short or malformed	Restore from git, re-encrypt

The first row covers most incidents. Opening an encrypted YAML in a text editor and changing a value, a key name, or even reindenting it alters the plaintext that sops will recompute the MAC over. The same goes for resolving a Git merge conflict by editing the encrypted file directly: you end up with a document sops never signed.

The mac_only_encrypted case is subtler. When your .sops.yaml sets mac_only_encrypted: true, only the encrypted values feed the MAC. Flip that flag, or change which fields encrypted_regex selects, and the recomputed MAC covers a different set of values than the stored one, even though nothing looks wrong in the diff.

Fix it step by step

Step 1: Confirm the file is the problem, not the key

Decrypt the file locally with the same age key Flux uses. If you get a MAC mismatch here too, the file is corrupt and the cluster key secret is fine:

$ export SOPS_AGE_KEY_FILE=$HOME/.config/sops/age/keys.txt
$ sops --decrypt clusters/prod/db-credentials.enc.yaml

If instead you see no key could decrypt or a base64 error, that is a different failure. The related Flux SOPS illegal base64 error fix covers the key-encoding case.

Step 2: Recover the real plaintext

You need a trustworthy copy of the values. Pick whichever source you actually trust.

Restore the last known-good version from Git history:

$ git log --oneline -- clusters/prod/db-credentials.enc.yaml
$ git show <good-sha>:clusters/prod/db-credentials.enc.yaml > /tmp/recovered.enc.yaml
$ sops --decrypt /tmp/recovered.enc.yaml > /tmp/plain.yaml

If you trust the current values but only the MAC is stale (for example the file was reindented, not semantically changed), decrypt while skipping MAC verification:

$ sops --decrypt --ignore-mac clusters/prod/db-credentials.enc.yaml > /tmp/plain.yaml

One caveat: --ignore-mac does not work with --in-place, and it will not rescue a file whose ciphertext was actually modified. Use it only to pull known-good plaintext back out so you can re-encrypt it.

As a last resort, if the Secret already applied to the cluster at least once, read the live values back:

$ kubectl -n app get secret db-credentials -o jsonpath='{.data.password}' | base64 -d

Step 3: Re-encrypt from clean plaintext

Encrypt the recovered plaintext into a fresh file with a valid MAC:

$ sops --encrypt --age age1yourclusterpublickey /tmp/plain.yaml > clusters/prod/db-credentials.enc.yaml
$ rm /tmp/plain.yaml

If you keep a .sops.yaml with creation_rules (recommended, so everyone encrypts identically), sops picks up the key and regex automatically:

$ sops --encrypt /tmp/plain.yaml > clusters/prod/db-credentials.enc.yaml

Step 4: Verify before you push

Never push an encrypted file you have not decrypted at least once. This one command is the difference between a clean reconcile and another red Kustomization:

$ sops --decrypt clusters/prod/db-credentials.enc.yaml | head -5

If that returns plaintext with no MAC error, the file is good.

Step 5: Commit and reconcile

$ git add clusters/prod/db-credentials.enc.yaml
$ git commit -m "fix: re-encrypt db-credentials to repair sops mac"
$ git push
$ flux reconcile kustomization apps --with-source

Watch the Kustomization go Ready again with flux get kustomizations.

Can you tell Flux to ignore the MAC?

No. The kustomize-controller always verifies the MAC during decryption and exposes no ignore-mac option, by design: a Secret that fails integrity checks should not be applied to a cluster. Tested with Flux v2.4.0 and sops v3.9.4, there is no spec.decryption field that disables MAC verification. The only durable fix is a file that decrypts cleanly, so treat the mismatch as a signal to re-encrypt rather than something to bypass.

Prevention

Never open a .enc.yaml in a plain text editor. Run sops clusters/prod/db-credentials.enc.yaml, which decrypts into your editor and recomputes the MAC on save.
Add a CI check or pre-commit hook that runs sops --decrypt on every changed encrypted file, so a broken MAC fails the pull request instead of the cluster.
Commit a .sops.yaml with creation_rules so encrypted_regex and key groups are identical for everyone. Inconsistent rules are a quiet source of MAC drift.
Resolve merge conflicts on encrypted files by decrypting both sides, merging the plaintext, and re-encrypting. Do not hand-edit ciphertext to settle a conflict.

For the canonical setup, the official Flux SOPS guide documents the age and key-secret wiring end to end. If you are still deciding how to structure encrypted secrets across environments, the Argo CD vs Flux multi-cluster GitOps guide and the Argo CD GitOps setup tutorial walk through the surrounding reconciliation model that this Secret plugs into.

Fix vLLM CUDA OutOfMemoryError in Kubernetes

DevOps Start — Sun, 19 Jul 2026 09:18:51 +0000

If your vLLM pod dies at startup with torch.cuda.OutOfMemoryError: CUDA out of memory, the model plus its KV cache needs more VRAM than the GPU allocated to that pod can give. The fastest fix is to cap two things: pass --gpu-memory-utilization 0.85 and --max-model-len 4096 on the serve command, then redeploy. If you have more than one GPU in the pod, add --tensor-parallel-size N to shard the weights across them. Those three flags resolve most of these crashes. The rest of this guide explains when each one matters and the Kubernetes-specific traps that make the error worse than it looks.

This is a GPU error, not a pod OOMKill

The first thing to get straight: torch.cuda.OutOfMemoryError is not the same failure as an OOMKilled pod. An OOMKill happens when your container exceeds its host RAM limit and the kernel sends Exit Code 137. A CUDA OOM happens entirely inside the GPU's own memory, which the Linux OOM killer and your pod memory limit know nothing about. You can have gigabytes of free node RAM and still hit this. If you are chasing an Exit Code 137 instead, the diagnosis path is different and covered in debugging OOMKilled pods.

The full error usually looks like this:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 39.38 GiB of which 224.00 MiB is free.

vLLM does a memory profiling run at startup. It loads the weights, runs a forward pass to measure peak activation memory, then claims the remaining GPU memory (up to gpu_memory_utilization) as a static KV cache pool. The crash happens when weights plus activations already exceed what is free, or when the KV cache target overcommits memory that another process on the GPU is holding.

Confirm what is actually on the GPU

Before changing flags, look at the GPU from inside the running or crash-looping pod. Guessing wastes redeploys.

$ kubectl exec -it deploy/vllm-server -- nvidia-smi

Read two numbers from the output: total GPU memory and current used memory. If used memory is already high before vLLM starts, something else is sharing the card. That is common on shared or MIG-partitioned GPUs, where gpu_memory_utilization of 0.9 (the vLLM default) is a fraction of the full physical card, not of your slice, so the target quietly overcommits.

Then confirm how many GPUs the pod was actually granted:

$ kubectl get pod -l app=vllm -o jsonpath='{.items[0].spec.containers[0].resources.limits}'

If this shows nvidia.com/gpu: "1" but you set --tensor-parallel-size 2, vLLM will try to place shards on GPUs that were never scheduled to the pod, and you get an OOM (or a hang) instead of a clean error. GPUs reach the pod through the Kubernetes device plugin, so the limit you request is the hard ceiling vLLM sees. The tensor parallel size must equal the GPU count in the resource limit.

The four flags that fix it

Each flag trades a different resource. Reach for them in this order.

Flag	Default	What it does
`--gpu-memory-utilization`	0.9	Fraction of GPU memory vLLM may claim for weights plus KV cache. Lower it to leave headroom on a shared card.
`--max-model-len`	model max	Caps context length. KV cache size scales with this, so a 128k model capped at 8k frees a large block.
`--max-num-seqs`	256	Max sequences batched at once. Fewer concurrent requests means a smaller KV cache.
`--tensor-parallel-size`	1	Shards model weights across N GPUs in the pod. The main lever for models too big for one card.

Start by trimming the KV cache, since that is where most waste lives:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --gpu-memory-utilization 0.85 \
    --max-model-len 4096 \
    --max-num-seqs 64

If the model weights alone do not fit on one GPU, no amount of KV cache trimming helps. That is when you shard:

$ vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 8192

A rough sizing check: a model in FP16 needs about 2 GB of VRAM per billion parameters just for weights, before any KV cache. A 70B model is roughly 140 GB, so it will not fit on a single 80 GB A100 no matter how you tune the cache. Shard it across GPUs or quantize it with --quantization fp8 to halve the weight footprint. If you are still deciding which serving engine to run, the tradeoffs are compared in choosing an LLM serving engine.

Kubernetes-specific traps

The same flags behave differently under Kubernetes than on a bare workstation. Three traps account for most repeat incidents.

1. Shared memory is too small for tensor parallelism

vLLM uses shared memory for inter-GPU communication when --tensor-parallel-size is greater than 1. Containers default /dev/shm to 64 MB, which is far too small, and the symptom is often a confusing OOM or NCCL hang rather than a clear message. Mount a memory-backed volume:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: "4"
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 8Gi

2. Memory fragmentation on long-running pods

If the error says a large amount is reserved but unallocated, the allocator has fragmented the pool. Set the PyTorch allocator to use expandable segments so freed blocks can be reused:

        env:
        - name: PYTORCH_CUDA_ALLOC_CONF
          value: "expandable_segments:True"

3. CUDA graph capture spikes memory

vLLM captures CUDA graphs at startup for lower latency, and the capture itself needs extra memory. If the OOM lands during capture rather than during the profiling run, disable it with --enforce-eager. You lose some throughput but the pod starts:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --gpu-memory-utilization 0.85 \
    --enforce-eager

Verify the fix

After redeploying, watch the startup logs for the KV cache report. A healthy start prints the number of GPU blocks it allocated:

$ kubectl logs -f deploy/vllm-server | grep -i "kv cache"

If that line appears and the readiness probe passes, the crash is resolved. Then send a real request so a full sequence actually fills the cache, since a crash can still surface under load rather than at boot:

$ kubectl port-forward deploy/vllm-server 8000:8000 &
$ curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "ping", "max_tokens": 16}'

Once serving is stable, keep an eye on GPU memory over time. A slow climb points to fragmentation or a KV cache sized too close to the limit, and wiring GPU metrics into your dashboards early makes that obvious before the next crash. See LLM observability on Kubernetes for the metrics worth tracking, and the top LLMOps tools for the wider serving stack.

The vLLM documentation on conserving memory lists the full set of flags and their interactions. For most Kubernetes deployments, though, the pattern is consistent: cap the KV cache first, shard the weights only when a single GPU genuinely cannot hold the model, and give tensor parallelism the shared memory it needs.

Fix "Resource not accessible by integration" in GitHub Actions

DevOps Start — Sat, 18 Jul 2026 09:00:34 +0000

Your workflow logs a red Error: Resource not accessible by integration and the job dies the moment it tries to write something back: a label, a comment, a commit, a release. The cause is almost always the same. The GITHUB_TOKEN your job runs with is read-only, so any API call that mutates the repository gets a 403. The fix is to grant that token the specific scope the failing step needs, using the permissions key in your workflow. There are three other situations where that fix alone is not enough, and knowing which one you are in saves you an hour of guessing.

What the error actually means

Every workflow run gets a short-lived GITHUB_TOKEN, generated per job and revoked when the job finishes. It authenticates as a bot identity (github-actions[bot]) against the GitHub API. When a step calls the API to change repository state and the token lacks the matching permission, the API answers 403 Resource not accessible by integration. The word "integration" is GitHub API language for the app behind the token, not a hint that some external integration is misconfigured.

So the message is really saying: this token is not allowed to do that. Two things decide what it is allowed to do: the repository or organization default, and any permissions block in your workflow.

Cause 1: the default token is read-only

Since 2023, new repositories default GITHUB_TOKEN to read-only. Many organizations also flip existing repos to read-only as a hardening step, which is the right call. You can confirm the setting under Settings, Actions, General, Workflow permissions. If it says "Read repository contents and packages permissions", the default token cannot write anything.

Do not fix this at the repository level by switching the default back to read/write. That grants every workflow in the repo broad access it does not need. Instead, grant the scope in the one workflow that needs it:

permissions:
  contents: write        # push commits or tags
  pull-requests: write   # comment on or label PRs

jobs:
  release:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ./scripts/tag-release.sh

One rule trips people up constantly. The moment you add a permissions block, GitHub switches that scope from permissive defaults to explicit mode: every permission you do not list becomes none. If your job also needs to read packages or write to the Checks API, you have to list those too. A job that reads and writes typically needs a handful of scopes spelled out, not one.

You can also scope permissions per job, which is stricter and what I reach for by default. A build job gets contents: read, and only the publish job gets contents: write:

jobs:
  build:
    permissions:
      contents: read
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make build

  publish:
    needs: build
    permissions:
      contents: write
      packages: write
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make publish

The full list of scopes and their defaults lives in the GitHub Docs on automatic token authentication. Keep least privilege in mind here; over-scoping the token is one of the quiet ways CI becomes a security liability, a theme covered in GitHub Actions Security: How to Stop Secret Leaks in CI/CD.

Cause 2: the pull request came from a fork

This one catches teams with public repos. When a pull_request event fires from a forked repository, GitHub deliberately hands the workflow a read-only GITHUB_TOKEN and withholds secrets, no matter what your permissions block says. An attacker could otherwise open a PR that runs arbitrary code with write access to your repo. The read-only downgrade is a security boundary, not a bug, and you cannot override it with the permissions key.

If you need to write back on a fork PR (post a comment, apply a label), use the pull_request_target event instead. It runs in the context of the base repository, so the token can be granted write scopes:

on:
  pull_request_target:
    types: [opened, synchronize]

permissions:
  pull-requests: write

Handle pull_request_target carefully. It runs with repository secrets available, so never check out and execute untrusted PR code inside it. Check out the base branch, or only run trusted logic like labeling. The safer pattern for anything that needs the PR's build output is a two-workflow split: an untrusted pull_request job that builds and uploads an artifact, and a trusted workflow_run job that downloads it and writes results back.

Cause 3: the PR was opened by Dependabot

Dependabot PRs look like internal PRs, but since March 2021 GitHub treats workflow runs triggered by Dependabot as if they came from a fork. The GITHUB_TOKEN is read-only and repository secrets are unavailable. Since October 2021 those runs do respect the permissions key, so a workflow that labels or auto-merges Dependabot PRs can work, but you still have to know two things.

First, secrets your job expects are missing on a Dependabot event. Reference secrets.DEPENDABOT_* values from the separate Dependabot secrets store, not the Actions secrets store. Second, the token is still fork-grade, so write operations that GitHub blocks for forks stay blocked. GitHub documents the exact event matrix in Troubleshooting Dependabot on GitHub Actions.

A common working shape for auto-approving patch bumps:

permissions:
  contents: write
  pull-requests: write

jobs:
  automerge:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'
    steps:
      - run: gh pr merge --auto --squash "$PR_URL"
        env:
          PR_URL: ${{ github.event.pull_request.html_url }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Cause 4: the action creates or approves a PR

Some steps hit a separate switch. Actions that open a pull request or approve one (for example, peter-evans/create-pull-request) need both pull-requests: write on the token and a repository setting that is off by default. Under Settings, Actions, General, look for "Allow GitHub Actions to create and approve pull requests" and enable it. Without that box checked, the API returns the same Resource not accessible by integration even when your permissions block looks correct. This is the one case where the scope is right and the error still fires, so check it early if the token clearly has pull-requests: write.

A diagnosis checklist

Work through this in order. The first match is almost always your fix:

Read the failing step. Which API call 403'd (contents, issues, pull-requests, packages, deployments)? That names the scope you are missing.
Is the trigger a fork pull_request or a Dependabot PR? If yes, the token is read-only by design. Move the write logic to pull_request_target or a workflow_run job (fork) or accept fork-grade limits (Dependabot).
Does the repo default to read-only? Add a permissions block granting only the scope from step 1.
Is the step creating or approving a PR? Enable the repository setting for it.
Re-run and confirm.

To see what the token actually carries at runtime, print the scopes near the top of the failing job:

$ echo "$GITHUB_TOKEN" | gh auth login --with-token
$ gh api rate_limit -i 2>&1 | grep -i 'x-oauth-scopes' || echo "using GITHUB_TOKEN, scopes set by permissions block"

If the workflow is a Terraform or infrastructure pipeline that comments plans back on PRs, the same permission model applies; a full working example lives in How to Automate Terraform Reviews with GitHub Actions. And if you are moving a mutating workflow behind a gate before it touches production, the patterns in Testing in Production: Guide to Progressive Delivery pair well with least-privilege tokens.

Verify the fix

After adding the scope, re-run the job and confirm the failing API call now succeeds. A green run is the real signal, but you can also confirm intent by reading the run's permissions in the logs: expand the "Set up job" step, and GitHub prints the resolved GITHUB_TOKEN permissions for that job. If the scope you added shows there and the call still 403s, you are in Cause 2, 3, or 4, not a missing scope. That distinction is the whole game with this error: decide whether the token could carry the permission at all, then whether you actually granted it.

k8s-aibom: Automated AI BOM for Kubernetes Workloads

DevOps Start — Wed, 15 Jul 2026 14:13:43 +0000

If you run a shared Kubernetes cluster, you already have AI workloads you don't know about. Someone shipped a vLLM inference service last sprint, a data team stood up a RAG pipeline behind a plain Deployment, and a contractor left an Ollama pod running in a namespace nobody audits. k8s-aibom, the controller Google Cloud open-sourced this month, exists to find exactly those. It watches your live workloads and writes a CycloneDX 1.6 ML-BOM for every AI system it can identify, so the answer to "what AI is running in this cluster right now" stops being a guess.

That "right now" is the whole point. A build-time SBOM tells you what your CI pipeline thought it was shipping. It says nothing about the pod a teammate kubectl apply'd by hand at 2am, or the image that pulled a new model layer since the last scan. Shadow AI is a runtime problem, and k8s-aibom is a runtime tool.

Why a runtime AI BOM is different

An AI Bill of Materials answers the same question a software BOM does, scoped to AI: which models, frameworks, and inference engines are in play, and where did they come from. The category matters now because regulators started asking. The EU AI Act's logging and transparency obligations, the NIST AI Risk Management Framework's "know what you deployed" controls, and ISO/IEC 42001's inventory clauses all assume you can produce a current, accurate list of your AI systems. You cannot produce that list from a spreadsheet someone updates quarterly.

Build-time and runtime BOMs solve different halves of the problem. Your build pipeline can attest to what it produced, and tools that live there are a good idea. But the pipeline never sees the workload that skipped it. If your supply-chain story stops at the CI system, read our take on why that is not enough in Supply Chain Security Proxy: Move Beyond Vulnerability Scanning. k8s-aibom fills the runtime gap: it reconciles against what the API server actually reports, not what a pipeline claims it built.

What the controller actually watches

k8s-aibom is a standard Kubernetes controller, not a DaemonSet or a privileged agent. It reconciles a set of workload kinds and emits a BOM per workload. The kinds it tracks:

Deployments
StatefulSets
DaemonSets
Jobs and CronJobs
KServe InferenceService resources

That list covers the shapes AI actually takes in a cluster. Inference services and agent stacks run as Deployments, batch training and evaluation runs as Jobs, and model servers packaged for KServe show up as InferenceService objects. Because it reconciles against the API server, a workload created outside your GitOps flow is just as visible as one that went through it. That is the property that makes it useful against shadow AI: you did not have to know the workload existed for the controller to catalog it.

Detection works by pattern-matching signals the workload already carries: container image references, command-line arguments, environment variables such as HF_MODEL_ID, mounted volumes, and workload annotations. From those signals it recognizes a broad set of AI software:

Category	Examples it identifies
Inference runtimes	vLLM, Hugging Face TGI, NVIDIA Triton, Ollama, Ray Serve, SGLang
Agent frameworks	LangChain, LangGraph, AutoGen, CrewAI, Langflow, Flowise
Vector databases	Milvus, Qdrant, Weaviate, Chroma, pgvector
Training frameworks	PyTorch, KubeRay, JAX, Hugging Face Accelerate
Evaluation tools	lm-evaluation-harness, Ragas, Trulens

If you are already running two inference engines and cannot decide whether that is a problem, our comparison Choosing an LLM Serving Engine: vLLM vs TGI covers the tradeoffs k8s-aibom will happily inventory for you.

What detection looks like on a real workload

Concretely, picture a RAG API someone shipped as a plain Deployment. Nothing about the object name says "AI", but the pod spec gives it away:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-api
  namespace: team-ml
spec:
  template:
    spec:
      containers:
        - name: server
          image: vllm/vllm-openai:v0.6.3
          args: ["--model", "meta-llama/Llama-3.1-8B-Instruct"]
          env:
            - name: HF_MODEL_ID
              value: meta-llama/Llama-3.1-8B-Instruct

The controller does not need a label saying this is AI. The vllm/vllm-openai image matches its inference-runtime catalog, the --model arg and HF_MODEL_ID env var name the model, and both get recorded on the BOM. The image tag lands as declared because it is read straight from the spec; the runtime identification lands as inferred because it came from a heuristic. A workload that went out of its way to hide, say a custom image with the model passed through a mounted config file, would still surface the pieces the controller can see and mark the rest unresolved. That gap is visible in the BOM, which is the honest behavior: you get told what the tool is unsure about instead of a confident lie.

Deploying it

Deployment is a Helm install into its own namespace. You build and push the image, then install the chart:

$ git clone https://github.com/GoogleCloudPlatform/k8s-aibom.git
$ cd k8s-aibom
$ export IMG=my-registry.example.com/k8s-aibom:v1.0.0
$ make image
$ make docker-push
$ helm install k8s-aibom ./charts/k8s-aibom \
    --namespace k8s-aibom-system \
    --create-namespace \
    --set image.repository=my-registry.example.com/k8s-aibom \
    --set image.tag=v1.0.0

The controller does not scan every namespace by default, which is the right call on a busy cluster. You opt a namespace in with a label:

$ kubectl label namespace team-ml aibom.k8saibom.dev/enabled=true

This opt-in model is deliberate. On a large platform you probably want to start with the namespaces where AI is likely, confirm the BOMs look right, then widen the net. Rolling it cluster-wide on day one buries you in output before you have tuned anything.

The custom resources live under the aibom.k8saibom.dev/v1alpha1 API group. There are two kinds: AIBOM, a namespace-scoped resource holding the BOM for one workload, and AIBOMControllerConfig, a cluster-scoped singleton that configures where BOMs are sent.

Reading a BOM

Once a namespace is enabled, the controller starts producing AIBOM resources. You read them like any other object:

$ kubectl get aibom -A
$ kubectl describe aibom -n team-ml deployment-rag-api

The output is a CycloneDX 1.6 ML-BOM. If the document is small it lives inline in the resource status; if it is large the status carries a reference to the externalized copy instead, so you are not stuffing megabytes into etcd.

The detail worth understanding is the confidence model. k8s-aibom does not pretend every field is a hard fact. Each attribute is tagged as one of three states:

declared: taken straight from the workload spec or an explicit annotation, so it is authoritative.
inferred: derived from a heuristic, such as recognizing an inference runtime from its image and args.
unresolved: the controller saw a signal but could not pin it down with confidence.

That grading is what makes the output auditable rather than a pile of guesses. When a reviewer asks why a BOM claims a workload runs a particular model, the answer is a field-level provenance tag, not a shrug. If you have been burned by AI tools that state everything with false certainty, our writeup AI Agent Risks: Lessons from Snyk's 10,000 Environment Audit is a good reminder of why that provenance matters.

Sending BOMs somewhere durable

A BOM that only lives in a cluster resource disappears when the workload does, which is useless for an audit trail. AIBOMControllerConfig defines sinks that push each BOM out. Three sink types exist: the always-on CR status (no external egress), a Google Cloud Storage bucket, and a generic webhook. A config with both external sinks looks like this:

apiVersion: aibom.k8saibom.dev/v1alpha1
kind: AIBOMControllerConfig
metadata:
  name: default
spec:
  sinks:
    - name: audit-archive
      type: GCS
      gcs:
        bucket: my-aibom-archive
        pathTemplate: "aibom/{namespace}/{kind}-{name}/{timestamp}.json"
        workloadIdentity: k8s-aibom-controller@my-project.iam.gserviceaccount.com
    - name: graph-ingest
      type: Webhook
      webhook:
        endpoint: https://guac.internal.example.com/ingest
        auth:
          bearerToken:
            secretRef:
              name: graph-ingest-creds
              key: token

The GCS sink has a property worth calling out: writes use a DoesNotExist precondition, so a stored BOM cannot be overwritten once created. That turns the bucket into an append-only historical record. For anyone who has ever tried to reconstruct "what was running when the incident happened" from mutable logs, an immutable, timestamped BOM per workload is a real upgrade. Pair the pathTemplate above with a bucket retention policy and you have a compliance artifact that survives the workload that produced it.

The webhook sink is how you feed a graph database or an SBOM platform. A common pattern is pushing into a supply-chain graph so AI components sit alongside your other software inventory instead of in a separate silo.

Where it fits, and where it doesn't

k8s-aibom is narrow on purpose, and that is a strength. It does not scan for vulnerabilities, enforce policy, or block anything. It builds an accurate inventory of AI workloads and gets it somewhere durable. Everything downstream, such as CVE correlation, policy gates, and drift alerts, is a separate tool consuming the BOM. Trying to make one controller do all of that is how you end up with a privileged agent nobody trusts.

Keep three limitations in mind before you lean on it:

Detection is pattern-based, so it has a coverage frontier. A homegrown inference server with no recognizable image, args, or environment signals may land as unresolved or be missed. The v1alpha1 API group is a fair signal that the detection catalog is still moving. Treat the BOM as a strong lead, not a guarantee of completeness, and watch what shows up unresolved.
It reports, it does not enforce. Finding a shadow workload and doing something about it are different jobs. You still need policy tooling, whether that is an admission controller or a governance layer, to act on what the BOM reveals. If you are building that layer, Governing AI Agents in CI/CD with OPA and MCP covers the policy side.
The GCS sink is Google Cloud native. The webhook sink is portable and works anywhere, but the tightest integration, immutable object writes via Workload Identity, assumes GKE. On other platforms you wire the webhook into your own durable store.

The bottom line

Shadow AI is not going away, and "we think we know what's running" is not an answer an auditor accepts. k8s-aibom gives you a runtime, provenance-tagged inventory of the AI workloads actually live in your cluster, written to an immutable store you can hand to a compliance review. It is early software with a moving detection catalog, so verify its output rather than trusting it blindly. But as a way to turn shadow AI from an unknown into a tracked list, it is a genuinely useful addition to a Kubernetes security stack.

Start small: install it, enable one namespace where you suspect unmanaged AI, and read the first few BOMs. The controller runs as a lightweight reconciler with no privileged access, so there is little downside to letting it watch. For the broader context on how Google frames this problem, their software supply chain security guidance and the how GKE powers AI innovation writeup are worth reading. For the Kubernetes primitives the controller builds on, the upstream controllers documentation covers the reconcile loop it uses.

Fix OpenTofu Ephemeral Value in Non-Ephemeral Context

DevOps Start — Wed, 15 Jul 2026 09:23:01 +0000

You added an ephemeral resource or an ephemeral = true variable in OpenTofu 1.11, wired it into a normal resource argument, and the plan died with Ephemeral value used in non-ephemeral context. The short version: OpenTofu refuses to let a value it promised never to persist flow into a place that would write it to state or plan. The fix is almost always one of three moves: send the value into a write-only (_wo) argument instead of a regular one, mark the output or variable that carries it as ephemeral = true, or keep the whole chain ephemeral so nothing downstream tries to store it. This walks through why the error fires and how to pick the right fix.

What the error is actually telling you

Ephemeral values are OpenTofu's mechanism for handling secrets and other temporary data that must never land in terraform.tfstate or a plan file. An ephemeral resource block, an ephemeral input variable, and an ephemeral output all produce values that exist only during a single operation. OpenTofu tracks that "ephemeral" taint through every expression, and the moment a tainted value reaches a context that persists data, it stops the run rather than silently leaking the secret.

The restricted contexts are consistent and worth memorizing:

Context	Ephemeral value allowed?
A regular (stateful) resource argument	No
A write-only argument (suffix `_wo`)	Yes
A root or child module output	No, unless the output is `ephemeral = true`
A `local` that feeds a non-ephemeral context	No
A provider or provisioner configuration block	Yes
Another ephemeral resource's arguments	Yes

Read that table as one rule: an ephemeral value can only go somewhere that also refuses to persist it. Everything else is a hard error by design. If you have hit persistence problems from the other direction, where values you wanted in state got locked or lost, the mechanics of what OpenTofu keeps and why are covered in Terraform State Locking: A Guide for Growing Teams.

Reproduce it in ten lines

Here is the smallest config that triggers the error. It reads a database password from an ephemeral resource and tries to hand it to a normal argument:

ephemeral "random_password" "db" {
  length = 24
}

resource "aws_db_instance" "main" {
  identifier = "app-db"
  password   = ephemeral.random_password.db.result
}

Run a plan and OpenTofu rejects it before touching the provider:

$ tofu plan

Error: Ephemeral value used in non-ephemeral context

  on main.tf line 7, in resource "aws_db_instance" "main":
   7:   password = ephemeral.random_password.db.result

Ephemeral values cannot be assigned to arguments that OpenTofu persists to
state. Use a write-only argument or mark the destination as ephemeral.

The provider never runs. This is a static check in the language layer, which is why no AWS call is made and no partial state is written. That is the whole point: the guard fires before the value can escape.

Fix 1: send it to a write-only argument

Most real cases are this one. You have a secret and you want it on a managed resource without storing it. That is exactly what write-only arguments exist for. They carry the _wo suffix, accept ephemeral values, and are always written to state and plan as null. Many provider resources expose a _wo twin of their sensitive argument, paired with a _wo_version argument you bump to force a new write.

Rewrite the failing example against a resource that supports write-only arguments:

ephemeral "random_password" "db" {
  length = 24
}

resource "aws_secretsmanager_secret_version" "db" {
  secret_id                = aws_secretsmanager_secret.db.id
  secret_string_wo         = ephemeral.random_password.db.result
  secret_string_wo_version = 1
}

secret_string_wo takes the ephemeral value and never records it. When you rotate the password, you change the value and increment secret_string_wo_version so OpenTofu knows to send the new secret on the next apply. The version integer is the only thing that lands in state. HashiCorp's write-only arguments reference documents the same model OpenTofu implements, including which core arguments pair with a version.

The common mistake here is assuming every argument has a _wo version. They do not. Write-only support is per-argument and per-provider, so check the resource's schema. If the argument you need has no write-only variant yet, that is a provider gap, not something you can force from configuration.

Fix 2: mark the output ephemeral

The second trigger is exporting an ephemeral value. This fails:

output "db_password" {
  value = ephemeral.random_password.db.result
}

An ordinary output is stored, so OpenTofu blocks it. If a parent module genuinely needs to consume this value during the same operation (to feed it into another ephemeral context), mark the output ephemeral:

output "db_password" {
  value     = ephemeral.random_password.db.result
  ephemeral = true
}

An ephemeral = true output can only be consumed by another ephemeral context in the calling module. You cannot mark an output ephemeral and then assign it to a normal resource argument upstream; you would just move the same error one module up. The ephemeral values documentation spells out the propagation rules across module boundaries.

Fix 3: keep the whole chain ephemeral

The subtle version of this error comes through a local. A local value built from an ephemeral expression inherits the ephemeral taint:

locals {
  conn = "postgres://admin:${ephemeral.random_password.db.result}@db:5432"
}

local.conn is now ephemeral. Use it in a provisioner or a provider block and OpenTofu is happy. Assign it to a stored argument and you get the same error, now pointing at the local instead of the resource. The fix is not to launder the value through the local; it is to make sure the local's destination is also ephemeral. If you find yourself wanting to store local.conn, step back, because that means you are trying to persist a secret, which is the exact thing the ephemeral system is stopping.

A full working example

Here is the pattern most teams actually want: pull a secret from a store at apply time and set a database password without ever writing the secret to state.

ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/app/db-password"
}

resource "aws_db_instance" "main" {
  identifier                  = "app-db"
  engine                      = "postgres"
  instance_class              = "db.t3.medium"
  allocated_storage           = 20
  username                    = "appuser"
  password_wo                 = ephemeral.aws_secretsmanager_secret_version.db.secret_string
  password_wo_version         = 3
}

Apply it and check what got stored:

$ tofu apply
$ tofu show -json | grep -c '"password_wo"'

The password value is absent from state; only password_wo_version = 3 is recorded. Rotate by updating the secret in Secrets Manager and bumping the version integer. This is the design working as intended: the secret transits the operation and vanishes, and your state file is safe to store in the same backend as everything else.

Why OpenTofu is this strict

It is tempting to read the error as OpenTofu being pedantic, but the strictness is the feature. State files are the single most common source of leaked infrastructure secrets, because they get committed, copied into CI logs, and shared in backends with loose access. By making it a compile-time error to route an ephemeral value anywhere persistent, OpenTofu removes the entire class of "oops, the password is in the plan output" incidents. The tradeoff is that you have to be explicit about where secrets are allowed to flow, which is a good constraint to have enforced by the tool rather than by a code review someone was too rushed to do. If your review process is where these decisions get made, Terraform Testing Best Practices: Beyond Plan and Pray covers how to catch this class of problem earlier.

Quick triage checklist

When you hit Ephemeral value used in non-ephemeral context, work through this in order:

Read the line the error points to. It names the exact argument, output, or local that broke the rule.
If it is a resource argument, look for a _wo variant of that argument and switch to it, adding the matching _wo_version.
If it is an output, decide whether the consumer is ephemeral. If yes, add ephemeral = true. If no, you are trying to persist a secret and should not.
If it is a local, trace where the local is used and make that destination ephemeral too.
If no write-only variant exists for the argument you need, check the provider version and its changelog. Write-only support is still expanding across providers.

Nine times out of ten it is case two: a value that should have gone into a _wo argument was pointed at the regular one. Fix that and the plan goes green. For a different OpenTofu failure mode that also blocks a clean run, see Fix OpenTofu Registry Timeout Errors.

Azure DevOps to GitHub Migration: An AI-Driven Playbook

DevOps Start — Tue, 14 Jul 2026 09:18:51 +0000

If you are moving repositories from Azure DevOps to GitHub, the tool you want is the gh ado2gh extension of the GitHub CLI, part of GitHub Enterprise Importer (GEI). It migrates Git history, branches, and pull requests from Azure DevOps Services to GitHub Enterprise Cloud in one command per repo. What it does not touch: Azure Boards work items, Azure Pipelines, Azure Artifacts, and wikis. Those need separate tools or a manual plan. Get that split clear before you write a single migration script, because most failed migrations are not technical failures. They are teams that assumed one tool moved everything and found out on cutover day that it did not.

The reason this migration is worth doing in 2026 is not just consolidation. It is that the AI tooling on the GitHub side, specifically the Copilot coding agent, only works against repositories that live on GitHub. If your code sits in Azure Repos, you are locked out of agentic workflows that can now open pull requests, run your tests, and self-review a patch before a human ever looks at it. That is the real incentive, and it changes how you should plan the move.

What GEI actually migrates (and what it drops)

Be precise here, because the gap is where teams get burned. GitHub Enterprise Importer, when pointed at Azure DevOps, moves the following:

Git source, including full commit history and all branches
Pull requests, both active and merged, with their comments
User history and attribution on commits
Work item links that were attached to pull requests
Attachments and branch policies

It does not move:

Azure Boards work items. There is no built-in path from Boards to GitHub Issues. You export and re-import, or you leave history in a read-only Azure project.
Azure Pipelines. These need the separate GitHub Actions Importer tool, which forecasts and partially automates the conversion to GitHub Actions workflows.
Azure Artifacts feeds. You download existing packages, repoint your package source at GitHub Packages, and republish.
Wikis. Manual export, or clone the wiki Git repo and push it as a normal repo.

One more sharp edge: GEI migrates Azure DevOps Services (the cloud product), not Azure DevOps Server (the on-prem product). If you are still on Server, you have to get to Services first or use a different path entirely. And abandoned pull requests whose branches were already deleted do not come across, so if audit history matters to you, snapshot it before cutover.

The official Azure DevOps to GitHub migration docs are the source of truth for the current feature matrix, and it does shift, so check it against your own inventory rather than trusting a blog post's list six months from now.

Step 1: inventory before you touch anything

Do not start with a migration command. Start with an inventory. You cannot plan waves, estimate effort, or spot the repos nobody has committed to in three years without one.

Install the extension and generate a report:

$ gh extension install github/gh-ado2gh
$ export ADO_PAT="your-azure-devops-token"
$ export GH_PAT="your-github-token"
$ gh ado2gh inventory-report --ado-org YOUR_ADO_ORG

The inventory-report command hits the Azure DevOps API and writes CSV files covering organizations, projects, repositories, and pipelines. Read them. You are looking for three things: repos small enough to migrate in the first pilot wave, repos with heavy pipeline coupling that will need Actions conversion, and dead repos you should archive instead of move. The token you use for this needs full access, so create a dedicated one and revoke it when the migration is done.

If you have never run a live cutover before, the GitHub Actions security guide is worth reading first, because the moment your code lands on GitHub, its secrets handling model changes and the old Azure variable groups do not come with it.

Step 2: create the personal access tokens

Both PATs need the right scopes or the migration fails halfway, which is worse than failing at the start. On the Azure side, the ADO_PAT needs read access to code and, if you want work item links and board integration, full access. On the GitHub side, the GH_PAT needs repo, admin:org, and workflow scopes at minimum for a repository migration into an org.

Set them as environment variables so they do not end up in your shell history as literals:

$ read -rs ADO_PAT && export ADO_PAT
$ read -rs GH_PAT && export GH_PAT

The read -rs pattern keeps the token off the screen and out of history. Small thing, but on a migration you are handling org-admin credentials, so treat them like it.

Step 3: generate and run the migration script

For anything beyond one or two repos, do not hand-write migrate-repo calls. Let the tooling generate them:

$ gh ado2gh generate-script --ado-org YOUR_ADO_ORG --github-org YOUR_GH_ORG --all

This produces a PowerShell script, migrate.ps1, with one migration call per repository it found. Open it before you run it. The generated script is a starting point, not gospel, and there are known cases where you need to edit repo name mappings or split it into waves. This is the moment to remove the dead repos you flagged in the inventory and to reorder so your pilot team's repos go first.

A single repository migration, if you want to run one by hand to test the path, looks like this:

$ gh ado2gh migrate-repo --ado-org YOUR_ADO_ORG --ado-team-project YOUR_PROJECT --ado-repo YOUR_REPO --github-org YOUR_GH_ORG --github-repo YOUR_REPO

Migrations are queued server-side and run asynchronously, so the command returns a migration ID rather than blocking until the repo is fully copied. Pull the logs afterward with the importer's log command to confirm history and pull requests came across cleanly. Do not mark a repo done because the command exited zero. Check the log.

Step 4: pipelines are a separate project

This is where teams underestimate the work. Your Azure Pipelines do not migrate with GEI. GitHub Actions Importer is a distinct tool that audits your existing pipelines, forecasts how much converts automatically, and generates draft GitHub Actions workflows.

Expect the audit to tell you that maybe 70 to 80 percent of a typical pipeline converts mechanically, and the rest, custom tasks, marketplace extensions, and complex approval gates, needs hand translation. A converted workflow lands as a YAML file in .github/workflows/:

name: CI
on:
  push:
    branches: [main]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build and test
        run: |
          npm ci
          npm test

Budget real engineering time for the pipeline conversion. If you already run infrastructure-as-code reviews through Actions, the pattern in how to automate Terraform reviews with GitHub Actions shows the shape of a mature Actions pipeline you are converging toward, not the thin line-by-line port the importer gives you first.

Step 5: turn on the AI tooling you migrated for

Here is the payoff. Once a repository lives on GitHub, the Copilot coding agent can work it like a background peer developer. You assign it an issue, it spins up its own environment powered by GitHub Actions, makes changes, runs your tests, and opens a pull request. As of 2026 it also runs Copilot code review against its own diff before it asks for human eyes, so the patch that reaches you has already been through one revision loop.

The guardrails matter and they are sensible defaults. The agent can only push to branches it creates, conventionally copilot/*, so your main and team-managed branches stay untouched. Every pull request still needs independent human review because the agent cannot approve or merge its own work. And CI in GitHub Actions will not run on the agent's PR without your approval. Those three rules are why enterprises can actually turn this on.

You can go further with custom agents. Drop a file under .github/agents/ to codify a repeatable approach, for example a performance agent told to benchmark first, make the change, then measure the difference before opening the PR:

name: perf-optimizer
description: Benchmarks, optimizes a hot path, then verifies the improvement.
instructions: |
  Run the existing benchmark suite and record the baseline.
  Make the smallest change that improves the target metric.
  Re-run the benchmark and include before-and-after numbers in the PR body.

For governing what these agents are allowed to touch across an org, the approach in governing AI agents in CI/CD with OPA and MCP maps cleanly onto GitHub's own AI Controls, which give enterprise admins a central page to enable or disable the coding agent, code review, and custom agents per organization.

If your developers are still deciding which assistant to standardize on inside the editor, the tradeoffs in Cursor vs Copilot vs Cody are worth reading, but note that the coding agent described here is a GitHub-native background worker, not an in-editor completion tool, and it is one of the concrete capabilities you gain by being on GitHub in the first place.

How to sequence the rollout

Do not big-bang a large org. The pattern that works is waves:

Pilot wave. Pick one team with a handful of small, low-risk repos and light pipeline coupling. Migrate their repos, convert their pipelines, and have them work on GitHub for two weeks. Fix what breaks.
Early adopters. Expand to teams that asked for the AI tooling. Their motivation carries them through the rough edges, and their feedback tunes your generated scripts.
Long tail. Everyone else, in batches sized to what your team can support in a week. Archive dead repos instead of migrating them.

Keep the Azure DevOps org in read-only mode for a defined window after each wave rather than deleting it. You will need to reference old work items and abandoned PRs that did not migrate, and a read-only source is cheap insurance against a rollback you did not plan for.

The verdict

The Git migration itself is the easy part. One generate-script, one reviewed run, and your history is on GitHub with pull requests intact. The work that decides whether the migration succeeds is everything GEI does not do: converting pipelines to Actions, deciding what happens to Boards and Artifacts, and rolling out in waves small enough to recover from. Do the inventory first, treat pipelines as their own project, and keep the old org read-only until you are sure.

The reason to spend that effort is not tidiness. It is that the AI development tooling, the coding agent that opens and self-reviews pull requests, only exists for code that lives on GitHub. Migrating the repos is the price of entry to agentic development, and in 2026 that is a price a lot of enterprises have decided is worth paying.

OpenTelemetry Collector vs Grafana Alloy: 2026 Guide

DevOps Start — Mon, 13 Jul 2026 15:14:40 +0000

Which one should you run

If you want a vendor-neutral collector that any backend can consume and any engineer can read, run the OpenTelemetry Collector. If you live inside the Grafana stack or you are migrating off the now-dead Grafana Agent, run Grafana Alloy. Both wrap the same upstream OTel components, so the decision is not about signal support. It is about configuration language, pipeline shape, and how tied you want to be to one vendor's ecosystem.

Grafana Alloy is not a fork of the Collector. It is a separate codebase that bundles OpenTelemetry Collector components and drives them with its own configuration syntax. That single fact explains most of the trade-offs below: you get the same receivers and exporters under the hood, wrapped in a very different operator experience.

Side-by-side comparison

Dimension	OpenTelemetry Collector	Grafana Alloy
Config language	YAML, declarative, receivers/processors/exporters	Alloy syntax (formerly River), HCL-inspired, programmable
Pipeline shape	Linear pipeline per signal (one for metrics, one for logs, one for traces)	Directed graph (DAG); components reference each other's exports
Signals	Metrics, logs, traces (profiles in progress)	Metrics, logs, traces, and profiles (Pyroscope)
Vendor neutrality	Vendor-neutral by design; swap backends via one exporter	OTLP-compatible, but tuned for the Grafana stack
Live UI	None built in	Web UI on port 12345 with a live component graph
Component library	The contrib repo: hundreds of receivers and exporters	Wraps OTel components plus native Prometheus, Loki, Pyroscope blocks
Best fit	Multi-vendor or vendor-agnostic pipelines	Grafana stack shops and Grafana Agent migrations

Configuration: YAML versus a real language

The Collector uses YAML. You declare receivers, processors, and exporters, then wire them into a pipeline per signal. It is boring in the best way. Anyone who has read a Kubernetes manifest can read it.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  batch: {}
exporters:
  otlphttp:
    endpoint: https://backend.example.com
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp]

Alloy replaces YAML with its own syntax, an HCL-inspired language where each block is a component with named inputs and outputs. Components reference each other by their exported fields, so the config describes a graph rather than a fixed list.

otelcol.receiver.otlp "default" {
  grpc {
    endpoint = "0.0.0.0:4317"
  }
  output {
    traces = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    traces = [otelcol.exporter.otlphttp.default.input]
  }
}

otelcol.exporter.otlphttp "default" {
  client {
    endpoint = "https://backend.example.com"
  }
}

The Alloy version is more verbose for this trivial case, and that is the honest trade-off. The payoff shows up when pipelines get complex: one component's output can fan out to several downstream components, and you can express Prometheus scraping, relabeling, and OTLP forwarding in one coherent graph instead of stitching YAML blocks by hand. The cost is a language your team has to learn, and that most tooling does not yet lint or format as well as YAML.

Pipeline architecture: linear versus DAG

The Collector runs a linear pipeline per signal type. Data flows receiver, then processors in order, then exporters. It is simple to reason about and simple to audit. When something drops a span, you walk the line.

Alloy evaluates a directed acyclic graph. Because components reference each other's exports, you build branches and joins directly. That flexibility is real, but it adds a small evaluation cost, and a graph is harder to trace by eye than a straight line when you are debugging at 2 a.m. For most workloads the overhead is negligible; the Collector is the more predictable of the two on memory.

Component ecosystem

The Collector's contrib repository is the center of gravity for OpenTelemetry. Hundreds of receivers, processors, and exporters live there, and most observability vendors ship their own component into it. If a backend exists, an exporter for it almost certainly exists too.

Alloy wraps those same OTel components (its otelcol.* blocks are the upstream components), then adds native Grafana-stack blocks: Prometheus remote write, Loki push, and Pyroscope profiling that are tightly integrated and well tested. If your telemetry ends up in Grafana Cloud or a self-hosted Grafana stack, those native blocks are smoother than the generic OTLP path.

Operational experience

Here Alloy pulls ahead. Run it and hit http://<alloy-host>:12345 for a live graph of every component, its health, and the data moving through it. When a pipeline misbehaves, you see which component is red without grepping logs.

$ alloy run config.alloy --server.http.listen-addr=0.0.0.0:12345

The Collector has no equivalent built-in UI. You get internal telemetry (its own metrics endpoint and zpages), which is capable but nowhere near as approachable when you are onboarding a new engineer or triaging fast. If a visual pipeline view matters to your on-call rotation, that is a point for Alloy.

If you are weighing collectors as part of a broader platform decision, the same trade-offs (vendor lock-in versus integration depth) show up across the space; the Datadog vs AWS Ops Agents comparison walks the SaaS side of that same tension.

The Grafana Agent migration angle

If you are running Grafana Agent in any mode (static, flow, or the operator), this comparison is not academic. Grafana Agent reached end of life on November 1, 2025, and no longer receives security or bug fixes. Grafana's own guidance is to migrate to Alloy, which is the successor to Flow mode and shares its component model. For those teams the choice is effectively made: Alloy is the supported path forward, and its config maps closely from Agent Flow.

For a greenfield deployment with no Grafana Agent history, the field is open again, and the vendor-neutrality argument for the plain Collector carries more weight.

Verdict

Both tools are solid, and both are built on the same OpenTelemetry foundation, so you are not choosing between good and bad telemetry. You are choosing an operator model.

Reach for the OpenTelemetry Collector when portability is the priority: multi-vendor backends, YAML that any engineer can read, and a pipeline you can swap to a new backend by editing one exporter. Reach for Grafana Alloy when you are committed to the Grafana stack, want the live pipeline UI, or are migrating off Grafana Agent before its unpatched code becomes a liability.

A common production pattern uses both: the Collector as a lightweight sidecar in application pods, forwarding OTLP up to Alloy as the cluster aggregator that fans telemetry into the Grafana stack. You do not have to pick one collector for the whole estate.

To go deeper on instrumenting workloads with OpenTelemetry itself, see How to Set Up LLM Observability with OpenTelemetry and, for cluster-scale patterns, LLM Observability on Kubernetes.

Verify the current component list and syntax against the Grafana Alloy documentation and the OpenTelemetry Collector documentation before you commit a production config; both projects move quickly.