DEV Community: daniel jeong

Astro 6 Deep Dive — Vite Environment API, Rust Compiler, Live Content Collections, and First-Class Cloudflare Workers

daniel jeong — Sat, 09 May 2026 01:27:38 +0000

On March 10, 2026, Astro 6.0 shipped as stable, followed by Astro 6.1 in April with image optimization fixes, i18n improvements, and Lightning CSS regression patches. Calling this "a routine major release" badly understates what changed. Astro 6 rebuilt the dev server and build pipeline onto a single shared code path, sat that path on top of Vite's new Environment API, and made non-Node runtimes — Cloudflare Workers, Bun, Deno — reproduce locally in dev exactly as they will run in production. At the same time, the Go-based .astro compiler got a Rust rewrite as an experimental opt-in (compilation phase up to 100× faster on large content sites), the Fonts API and Content Security Policy API got promoted into core, and Live Content Collections — experimental in 5.10 — went stable. The heaviest operational changes, though, sit elsewhere: Astro.glob() permanently removed, Node 22 mandated, Astro.locals.runtime removed (replaced by direct cloudflare:workers imports), and the astro:schema / z from 'astro:content' path deprecated in favor of astro/zod (Zod v4). This post organizes the 6.0/6.1 surface into five axes, walks through the seven build-breaking migration items, and shares the four-week migration checklist ManoIT validated while moving its marketing and docs sites from v5 to v6.

1. Why 6 Is the Inflection Point — "Content Site = Static Build" Ends

Through Astro 5, the framework leaned into static site generation as a content-first posture. v4 added Content Layer API to free collections from local files; v5 added Server Islands and Sessions to mainstream partial SSR. v6 is the completion of that arc, and it lands in two sentences. First, the default for "build-once, deploy anywhere" shifts from build time to request time. With Live Content Collections and Route Caching stabilizing, content sites can serve fresh-per-request data while still letting CDN cache headers control delivery. Second, "works in dev, breaks in prod" disappears. Thanks to Vite's Environment API, the dev server boots non-Node runtimes (workerd, Bun, Deno) locally and uses the same runtime for prerendering. You import env from cloudflare:workers in dev, not a process.env shim.

The table below collapses the v5.x → v6.0/6.1 operational surface onto a single page.

Axis	Astro 5.x	Astro 6.0 (2026-03-10)	Astro 6.1 (2026-04)	Operational signal
Dev server	Vite + custom middleware	Rebuilt on Vite Environment API	HMR/sourcemap regressions fixed	dev = prod runtime parity
Cloudflare	workerd partial	workerd at every stage (dev/prerender/prod)	hybrid site regression fixed	`Astro.locals.runtime` removed
Compiler	Go-based	Experimental Rust build added	Rust build regression fixed	Up to 100× compile speed
Fonts	Community integrations	Core Fonts API GA	Fallback metric improved	Local + Google + Fontsource
CSP	Vendor middleware	Core CSP API GA	`strict-dynamic` added	Inline-script nonce auto
Live Collections	5.10 experimental	Stable	External hosting integrations	Request-time data official
Route Caching	None	Experimental Route Caching API	`cache-control` surrogate keys	Platform-agnostic SSR cache
Content Collections	Old API + Layer API coexisted	Content Layer API mandatory	unchanged	`Astro.glob()` removed
Schema validation	`astro:schema` / `z from 'astro:content'`	`astro/zod` recommended (Zod v4)	compatibility kept	Input/output type split
Node	≥ 18.20	≥ 22 LTS required	unchanged	OS base image alignment
Image	Sharp + partial redirect	Pattern-gated redirect rejection	Up to 10 redirects, AVIF/animated safe	External CDN policy explicit

The two highest-cost rows for operations are Content Collections and Schema validation. The auto-compatibility window for Astro.glob() finally closed, and z from 'astro:content' migrated to a Zod v4-based astro/zod. Both break builds, so we burned week 1 of our migration entirely on these two rows.

2. Astro 6 Runtime Architecture — Five-Axis Layout

┌──────────── Source ────────────┐
│  src/pages/*.astro             │
│  src/content/**.{md,mdx}       │
│  src/content.config.ts (zod v4)│
└─────────────┬──────────────────┘
              │
       ┌──────▼──────┐
       │  Compiler   │   ❶ Go (default) | Rust (experimental, ~100×)
       │  .astro →   │
       │  ESM module │
       └──────┬──────┘
              │
   ┌──────────▼─────────────────────────────────────────┐
   │  Vite Environment API                              │  ❷ dev = prod runtime
   │  ┌────────┐  ┌────────┐  ┌────────┐  ┌────────┐    │
   │  │  Node  │  │ workerd│  │  Bun   │  │  Deno  │    │
   │  └────┬───┘  └────┬───┘  └────┬───┘  └────┬───┘    │
   └───────┼───────────┼───────────┼───────────┼────────┘
           │           │           │           │
   ┌───────▼───────────▼───────────▼───────────▼────────┐
   │  Astro core 6                                      │  ❸ Adapters
   │  • Fonts API   (preload, fallback metric)          │
   │  • CSP API     (nonce, strict-dynamic)             │
   │  • Live Content Collections (request-time loaders) │
   │  • Route Caching API (experimental, web-standard)  │
   │  • Sessions, Server Islands, Actions               │
   └───────┬────────────────────────────────────────────┘
           │
   ┌───────▼────────────────────────────────────────────┐
   │  Output                                             │  ❹ static / hybrid / SSR
   │  • Static pages (HTML)                              │
   │  • Server entry (Cloudflare Workers / Node / Bun)   │
   │  • Live API routes / RSC-style streaming            │
   └─────────────────────────────────────────────────────┘

Figure 1. Astro 6 five axes — ❶ compiler is Go-default with Rust experimental ❷ Vite Environment API supplies the same runtime to dev/prerender/prod ❸ core surfaces Fonts/CSP/Live Collections/Route Caching directly ❹ adapters (Cloudflare/Node/Bun/Deno) split the output.

2.1 Axis ① Compiler — Go to Rust, gradually

Astro 6 ships an experimental Rust rewrite of the compiler that turns .astro files into ESM modules. The stable build is still Go; Rust is opt-in. Two consequences. First, on cold builds of large content sites, the compilation phase alone shows ~100× speedups (the more files you have, the bigger the win). Second, once Rust stabilizes, it slots cleanly next to Vite Rolldown (same Rust toolchain, room for Oxc/SWC interop). The operational decision is straightforward — keep Go as the default through 6.1 and opt into Rust only on a CI regression-test job.

2.2 Axis ② Vite Environment API — dev = prod runtime

Vite's Environment API splits the module graph per environment. Astro 6 sits its dev server on top of that, so each environment (Node, workerd, Bun, Deno) handles requests with its own runtime. The clearest payoff is Cloudflare Workers. v5 emulated Node in dev and exposed only some platform APIs; v6 runs workerd in dev/prerender/prod. Code that imports env from cloudflare:workers, KV/R2/D1 bindings, and Service Bindings all work in dev too — the entire "works in dev, breaks in prod" class of regressions is gone.

2.3 Axis ③ Core APIs — Fonts, CSP, Live Collections, Route Caching

The Fonts API treats local files, Google Fonts, and Fontsource as one interface — handling self-hosting, automatic preload links, and fallback metric (measuring the original font's ascent/descent/line-gap so the system fallback simulates the same metrics) to cut CLS. The CSP API auto-injects nonces into inline scripts and emits strict-dynamic, upgrade-insecure-requests, and frame-ancestors headers per-adapter. Astro Islands' inline hydration scripts have been the obstacle to clean CSP for years; v6 closes that problem. Live Content Collections is a new loader shape that fetches data at request time on top of the Content Layer. Route Caching API declares per-route cache-control, cdn-cache-control, and surrogate-key headers, abstracting platform differences (Cloudflare/Vercel/Netlify).

2.4 Axis ④ Adapters — Cloudflare / Node / Bun / Deno

Adapters wrap the core's build output into each platform's entrypoint. The biggest v6 change is @astrojs/cloudflare running workerd at every stage. As a side effect, Astro.locals.runtime.env is gone and the same data is reached via env from cloudflare:workers (see §5). The Node adapter now requires Node 22 minimum, and Bun/Deno adapters get much better dev parity from Environment API.

2.5 Axis ⑤ Output — static / hybrid / SSR + Live API

Output modes are still static, hybrid, and server. The difference is how routes opt in. Live API routes declare export const prerender = false and use a Live Collection to return request-time data. The Route Caching API on the same route declares cache headers to govern CDN/edge caches.

3. v5 → v6 Migration — The Seven Build-Breaking Changes

Below are the seven items from ManoIT's migration retro that actually break builds. Going top to bottom in one pass closes week 1.

#	Change	Symptom	Fix direction
1	`Astro.glob()` removed	`TypeError: Astro.glob is not a function`	Replace with `import.meta.glob()` or move to a Content Layer loader
2	Content Layer API mandatory	Old collections return empty arrays	Consolidate into `src/content.config.ts` + `defineCollection({ loader })`
3	`z from 'astro:content'` deprecated	Type errors / runtime warnings	Switch to `import { z } from 'astro/zod'` (Zod v4)
4	Zod v4	`z.string({ required_error })` API changed	Apply v3→v4 migration (split input/output types)
5	`Astro.locals.runtime` removed (Cloudflare)	`undefined` errors	`import { env } from 'cloudflare:workers'`
6	Node 22 required	`engines` warning or build failure	Move base image to `node:22-alpine`
7	Image redirect patterns	Build fails on external CDN redirect	Declare every redirect host in `image.remotePatterns`

1 is the most common and the simplest. `Astro.glob('../posts/.md')` becomes `import.meta.glob('../posts/.md', { eager: true })`, or — preferred — wrap it in a content collection. #2 means the auto-compatibility window closed in v6: the implicit upgrade to the new Content Layer API that v5.x allowed without `experimental.contentLayer` is gone. Define every collection in `src/content.config.ts`, and supply an explicit loader (see §4). #3 and #4 ride the same Zod release wave, so handle them together rather than splitting across weeks.

4. Live Content Collections — The Request-Time Loader Pattern

Below is the recommended Astro 6 collection shape. Build-time (blog posts) and request-time live (live inventory) collections sit side-by-side in the same file.

// src/content.config.ts — Astro 6 recommended layout
import { defineCollection, defineLiveCollection } from 'astro:content';
import { z } from 'astro/zod';                     // ❶ astro/zod (Zod v4)
import { glob } from 'astro/loaders';
import { liveLoader } from '@astrojs/live-collections';

// ❷ Build-time: src/content/posts/**/*.md
const posts = defineCollection({
  loader: glob({ pattern: '**/*.{md,mdx}', base: './src/content/posts' }),
  schema: z.object({
    title: z.string().min(8).max(120),
    publishedAt: z.coerce.date(),
    tags: z.array(z.string()).default([]),
    cover: z.string().url().optional(),
  }),
});

// ❸ Request-time live: external inventory API
const inventory = defineLiveCollection({
  loader: liveLoader(async ({ entry, request }) => {
    const res = await fetch(`https://api.manoit.co.kr/inventory/${entry}`, {
      headers: { 'cache-control': 'no-store' },
    });
    if (!res.ok) throw new Error(`Inventory ${entry} ${res.status}`);
    return res.json();
  }),
  schema: z.object({
    sku: z.string(),
    stock: z.number().int().nonnegative(),
    price: z.number().nonnegative(),
    lastUpdated: z.coerce.date(),
  }),
});

export const collections = { posts, inventory };

Two points. First, defineCollection is fetched once at build time and rendered into static pages. Second, defineLiveCollection runs per request so const item = await getEntry('inventory', sku) returns fresh data every call while keeping the same API. Routes that use a live collection automatically flip to prerender = false.

5. First-Class Cloudflare Workers — workerd-on-everywhere

Below is the v5 → v6 diff for a Cloudflare-adapter route. Astro.locals.runtime is gone; cloudflare:workers is imported directly. The dev server runs the exact same code path, so the "process.env in dev, env in prod" branch is gone.

// src/pages/api/keys/[name].ts — Astro 6 + Cloudflare Workers
import type { APIContext } from 'astro';
import { env } from 'cloudflare:workers';     // ❶ Replaces Astro.locals.runtime.env

export const prerender = false;               // ❷ Live API route

export async function GET({ params, request }: APIContext) {
  const value = await env.MY_KV.get(`tenant:${params.name}`);
  if (!value) {
    return new Response(JSON.stringify({ error: 'not_found' }), {
      status: 404,
      headers: { 'content-type': 'application/json' },
    });
  }

  return new Response(value, {
    headers: {
      'content-type': 'application/json',
      // ❸ Route Caching API: 60s at the edge, surrogate-key for invalidation
      'cache-control': 'public, max-age=0, s-maxage=60',
      'cache-tag': `tenant:${params.name}`,
    },
  });
}

❶ env exposes the wrangler.toml KV/R2/D1 bindings and secrets in dev too. ❷ A single line flips the route to SSR. ❸ cache-tag is invalidated by Cloudflare's purge by tag; the Route Caching API translates the header to whatever name each adapter expects (Vercel uses x-vercel-cache-tag, Netlify uses netlify-cdn-cache-tag).

6. Fonts and CSP API — The Last Two Pieces of the Islands Era

The Fonts API lands in astro.config.ts like this — self-hosting, preload, and fallback metric in one go.

// astro.config.ts — Fonts API + CSP API
import { defineConfig } from 'astro/config';
import cloudflare from '@astrojs/cloudflare';

export default defineConfig({
  output: 'server',
  adapter: cloudflare({ mode: 'directory' }),
  fonts: [
    {
      name: 'Pretendard',
      cssVariable: '--font-sans',
      provider: 'fontsource',                 // ❶ google | fontsource | local
      weights: [400, 600, 800],
      subsets: ['latin', 'korean'],
      fallbacks: ['Apple SD Gothic Neo', 'sans-serif'],
      display: 'swap',
    },
    {
      name: 'JetBrains Mono',
      cssVariable: '--font-mono',
      provider: 'google',
      weights: [400, 600],
      preload: true,
    },
  ],
  csp: {
    enabled: true,
    directives: {
      'default-src': ["'self'"],
      'script-src': ["'self'", "'strict-dynamic'"],   // ❷ nonce auto
      'style-src': ["'self'", "'unsafe-inline'"],
      'img-src': ["'self'", 'https://cdn.manoit.co.kr', 'data:'],
      'connect-src': ["'self'", 'https://api.manoit.co.kr'],
    },
  },
  experimental: {
    routeCaching: true,                       // ❸ enable route cache headers
  },
});

Use them in CSS as font-family: var(--font-sans) / var(--font-mono). Astro computes SHA-256 hashes for inline scripts at build time and adds them to script-src, or — on SSR routes — generates a per-request nonce. Our policy is 'strict-dynamic' + nonce so child scripts spawned by hydration are trusted automatically.

7. ManoIT's Four-Week Migration Checklist

Concrete, gated steps. Each week ends on a regression-green gate before the next begins.

Week	Work	Gate
1	Align Node 22 base image, sweep `Astro.glob()`, consolidate collections in `src/content.config.ts`, switch `z` import to `astro/zod`	Local build green, `tsc --noEmit` clean
2	Cloudflare adapter to v6, replace `Astro.locals.runtime` with `cloudflare:workers`, validate wrangler.toml bindings in dev	Dev hits live KV/R2/D1 successfully
3	Adopt Live Content Collections (inventory/sessions/live pricing), define Route Caching matrix, wire surrogate-key invalidation	Edge cache hit rate ≥ 80% (staging)
4	Apply Fonts + CSP API, compare Lighthouse CLS/LCP, opt Rust compiler into a CI job, canary 5% traffic	CLS ≤ 0.05, CSP report-only violations 0, canary error rate ≤ 0.1%

Four weeks compresses with site size. ManoIT's marketing site (~380 pages, 80 MDX, 4 collections) finished in 7 working days. The biggest week-1 sink was the Zod v3 → v4 schema realignment: z.string().required() collapses to z.string() in v4 (required by default), and z.input<T> / z.output<T> had to be split for any schema that coerced types.

8. Conclusion — Astro 6 Installs SSR as the Default for Content Sites

Through v5, Astro's posture was "static where you can, SSR where you must." v6 doesn't reverse that posture. It just drops the friction of choosing SSR to nearly zero, using every major surface to do it: dev=prod runtime parity, first-class workerd, live collections, route caching, Fonts/CSP in core, the Rust compiler. Adding one SSR line to a content site costs an order of magnitude less than it did in v5. But every gain arrives after the migration gate. Until you've cleaned out Astro.glob(), astro:schema, Astro.locals.runtime, and Node 18 in one pass, the new v6 surface looks only like build-breaking edges. ManoIT's three sites — marketing, docs, customer portal — landed on v6 within 28 days of the 6.0 stable release; route caching and live collections roll into the next quarter.

This article was co-authored by Anthropic Claude Opus 4.6 and ManoIT. Primary sources: Astro 6.0 (2026-03-10) and 6.1 release notes, @astrojs/cloudflare, Vite Environment API documentation, and ManoIT's internal migration retros (2026-04-13 through 2026-05-02). © 2026 ManoIT.

Originally published at ManoIT Tech Blog.

OpenTelemetry Profiles Public Alpha: eBPF Fourth Signal, Collector v0.151.0 and OpAMP Fleet Management for 2026

daniel jeong — Fri, 08 May 2026 00:09:28 +0000

OpenTelemetry Profiles Public Alpha — How the eBPF Fourth Signal, Collector v0.151.0, and OpAMP Fleet Management Redefine Unified Observability in 2026

On March 26, 2026 the CNCF announced the Public Alpha of the OpenTelemetry Profiles signal. After metrics, traces, and logs, continuous profiling joins as the fourth signal, and OpenTelemetry becomes the first open standard to put all four observability pillars under a single SDK, a single OTLP wire protocol, and a single semantic-convention layer. In the same release cycle, OpenTelemetry Collector v0.151.0 shipped on April 29, 2026 with winget distribution, Run/Shutdown lifecycle synchronization, and richer send_failed metrics that smoothed out the operational surface. And IBM Instana's GA of OpAMP-powered Collector Fleet Management makes it clear that the 2026 default for OTel operations is moving from SSH and rolling restarts to supervisor-driven OpAMP. This post consolidates the eBPF profiler architecture, the k8sattributesprocessor integration, the trace_id/span_id cross-correlation semantic conventions, the Q3 2026 GA timeline, and ManoIT's four-week adoption checklist validated on EKS 1.32 and on-prem bare metal.

1. Why Profiles Is the Tipping Point — Unified OTLP Across Four Signals

Until now the observability stack has been split. Metrics, traces, and logs converged on OpenTelemetry, but continuous profiling stayed on a separate axis with Pyroscope, Parca, and Pixie. As a result, operators have had to maintain two agents, two wire formats, and two backends for the same workload. Answering "during the slow span I just saw, where exactly did the CPU time go inside the call stack?" required manually aligning two different graphs.

Profiles Alpha ends that split. Profile samples now travel over OTLP carrying trace.id and span.id attributes by semantic convention, so backends can join trace and profile data on the same key and offer one-click navigation from a span to its corresponding stack trace. Per OpenTelemetry's versioning rules, all signal SDKs ship with the same version, so the Profiles stability schedule is bound to the SDK as a whole. Profiles is targeting GA in Q3 2026; Public Alpha sits one step before that mark.

Aspect	2024 (before)	2026 Q1 (RC)	2026 Q2 (Alpha · now)	2026 Q3 (GA target)
Fourth signal status	Experimental	Release Candidate	Public Alpha (3/26)	General Availability
Wire format	Vendor-specific	Draft OTLP extension	OTLP profiles message stabilizing	Folded into OTLP 1.x
Reference agent	None	Elastic Universal Profiler donated	`opentelemetry-ebpf-profiler` (official)	Bundled in Collector distros
Trace correlation	Vendor-specific	Attribute draft	`trace_id`/`span_id` semantic conventions	Mandated across SDKs
K8s metadata join	Manual	Proposal	`k8sattributesprocessor` integration	Standard pipeline
Collector lifecycle	Async shutdown	Issue tracking	Run/Shutdown sync (v0.151.0)	Maintained

The most important row is "reference agent." With Elastic donating its Universal Profiling Agent and the OpenTelemetry community relaunching it as opentelemetry-ebpf-profiler, the project now has a reference implementation that achieves three properties at once: low overhead, whole-system coverage, and language-agnostic, no-instrumentation collection.

2. eBPF Profiler Architecture — The Fourth Signal Lives Inside the Collector

┌────────────────────── Linux Kernel (perf events + eBPF) ──────────────────────┐
│   ┌──────────────────────────────────────────────────────────────────────┐    │
│   │  opentelemetry-ebpf-profiler  (CO-RE eBPF, perf-format stack trace) │    │
│   │  C/C++ · Go · Rust · Python · Java · NodeJS · .NET · PHP · Ruby     │    │
│   │  Automatic Go symbolization · new runtimes · low-overhead sampling  │    │
│   └────────────────────────────────────┬─────────────────────────────────┘    │
└────────────────────────────────────────┼──────────────────────────────────────┘
                                         │ profiling samples
┌────────────────────────────────────────▼──────────────────────────────────────┐
│                        OpenTelemetry Collector v0.151.0                       │
│   ┌──────────────────────────┐   ┌────────────────────────────────────────┐   │
│   │ profiler receiver        │──▶│ k8sattributesprocessor                │   │
│   │ (Elastic donation)       │   │ container.id → namespace/pod/deployment│   │
│   └──────────────────────────┘   └─────────────────┬─────────────────────┘   │
│                                                     │                         │
│                  ┌──────────────────────────────────▼───────────────────┐    │
│                  │ batchprocessor · resourceprocessor · tail_sampling   │    │
│                  │ (same pipeline reused with traces and metrics)        │    │
│                  └──────────────────────────────────┬───────────────────┘    │
│                                                     │ OTLP                    │
│                                                     ▼                         │
│                                ┌──────────────────────────────────────────┐   │
│                                │ otlp/profiles exporter → backend         │   │
│                                │ trace_id · span_id attributes preserved │   │
│                                └──────────────────────────────────────────┘   │
└───────────────────────────────────────────────────────────────────────────────┘
                                          │
                                          ▼
            ┌────────────────────────────────────────────────────────────┐
            │ OpAMP Supervisor fleet (Instana GA · Bindplane)            │
            │ Remote config · health reporting · package management      │
            └────────────────────────────────────────────────────────────┘

2.1 Axis ① Agent — One eBPF Profiler Covers Many Language Runtimes

opentelemetry-ebpf-profiler is the GitHub repo that inherited the Universal Profiling Agent donated by Elastic. A single CO-RE eBPF object runs across compatible kernels and collects call-stack samples for the runtimes you actually find in a data center — C/C++, Go, Rust, Python, Java, NodeJS, .NET, PHP, Ruby, Perl — without any per-language SDK instrumentation. The Alpha cycle added automatic Go symbolization, which restores function names from a stripped Go binary without separate debug info and removes one more operational tax.

2.2 Axis ② Collector Receiver — Share the Same Pipeline as Traces and Metrics

The most important design choice is "Collector receiver reusing existing pipelines" rather than "separate daemon and separate backend." Because the profiler is a Collector receiver, you reuse the batchprocessor, resourceprocessor, tail_sampling, and OTLP exporters that you already deployed for traces and metrics. Operators no longer need two agents, two lifecycles, and two auth tokens. Profiles ship with the same token to the same endpoint as traces and metrics on the same node.

2.3 Axis ③ K8s Enrichment — `k8sattributesprocessor` + `container.id`

k8sattributesprocessor uses the container.id resource attribute as a join key and automatically attaches namespace, pod, deployment, and node labels to every piece of telemetry that flows through. Profiles go through the same processor, so backends receive every profile sample already enriched with K8s context. Operators stop asking only "which function is hot?" and start asking "in which namespace, in which deployment, in which pod, called from where in the trace span — was this stack hot?"

2.4 Axis ④ OpAMP — Operate the Collector Fleet Without SSH

OpAMP (Open Agent Management Protocol) is the protocol for remote configuration, health reporting, and package management of a Collector fleet without SSH access or rolling restarts. With IBM Instana's 2026 GA of OpAMP-powered OpenTelemetry Collector Fleet Management, the model "push policy to the supervisor and let Collectors update themselves" is becoming the de-facto operational default. In ManoIT's experience, the largest savings come from pushing per-environment sampling ratios, signal-specific exporter routing, and new receiver enablement to a single cluster's dev/stage/prod Collectors from one supervisor.

3. Collector v0.151.0 Migration — winget, Lifecycle, and `send_failed`

Collector v0.151.0, released April 29, 2026, is an incremental polish release with three changes that matter for operations:

# otel-collector-config.yaml — v0.151.0 recommended baseline
extensions:
  opamp:                         # ❶ Register supervisor (Instana/Bindplane/etc.)
    server:
      ws:
        endpoint: wss://opamp.example.com/v1/opamp
    capabilities:
      reports_effective_config: true
      accepts_remote_config: true
      reports_health: true

receivers:
  otlp:                          # metrics, traces, logs
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
  profiler:                      # ❷ fourth signal (Alpha)
    sampling_period: 19ms        # ~50Hz, Elastic recommended default
    include_kernel: false
    metadata:
      include_pod_labels: true

processors:
  k8sattributes:                 # ❸ container.id → namespace/pod/deployment
    auth_type: serviceAccount
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name]
  batch:
    send_batch_size: 8192
    timeout: 5s

exporters:
  otlp:
    endpoint: backend.example.com:4317
    sending_queue:
      enabled: true
    # ❹ v0.151.0 — send_failed metric now carries error.type / error.permanent

service:
  extensions: [opamp]
  telemetry:
    metrics:
      level: detailed            # required to inspect send_failed attributes
  pipelines:
    profiles:                    # ❺ new signal pipeline
      receivers: [profiler]
      processors: [k8sattributes, batch]
      exporters: [otlp]

The most consequential change for operators is Run/Shutdown lifecycle synchronization. Previously, Shutdown could return before the Run loop finished its cleanup on SIGTERM, which sometimes lost telemetry that was still in the queue. v0.151.0 makes Shutdown block until Run has completed all cleanup, matching http.Server semantics. The same release attaches error.type and error.permanent attributes to the send_failed metric at the detailed telemetry level, and on Windows you can now install, upgrade, and uninstall the Collector through winget. For environments that operate multi-OS edge nodes, this normalizes the package-manager surface significantly.

4. ManoIT 4-Week Adoption Checklist — How to Evaluate Alpha Safely

Week	Work	Acceptance criteria
Week 1	Deploy Collector v0.151.0 + profiler receiver as a sidecar on a non-prod cluster; review `opentelemetry-ebpf-profiler` container privileges (privileged or `CAP_PERFMON`+`CAP_SYS_PTRACE`)	OTLP profile reception confirmed on sample workloads (one Java, one Go, one Python); k8sattributes labels attached
Week 2	Validate `trace_id`/`span_id` correlation by enabling traces SDK on the same workload; verify span ↔ profile jump in the backend; compare `sampling_period` 19 ms vs 9.7 ms	One-click jump from span to call stack; p99 CPU overhead < 1%
Week 3	Introduce an OpAMP supervisor — pilot with Instana or Bindplane; push different sampling policies remotely to dev and stage	One successful policy update without SSH; health report dashboard lit
Week 4	Production canary on 5% of nodes; alarm on `send_failed` detailed attributes; define stability thresholds for Alpha before GA	Zero pod OOM/restarts over four weeks; `send_failed.error.type` alarm rule active

You should still treat this as Alpha. The OTLP profiles message schema has limited compatibility guarantees during Alpha, and backend-side display quality varies significantly across vendors. Plan the production rollout for after the Q3 2026 GA, but use the one or two preceding quarters to evaluate non-prod and canary deployments — that operational learning becomes the asset you bring into the GA decision.

5. Trace ↔ Profile Cross-Correlation — One Click from a Span to Its Stack

The biggest day-to-day operational value of Profiles Alpha is answering "why was this span 1.2 seconds?" without leaving the trace UI. Now that the semantic conventions are settled, every profile sample carries trace.id and span.id over OTLP. Backends can join trace data and profile samples on the same key, and the UI automates "click the span → see the call stack for that exact time window." The workflow simplifies as follows:

[Old workflow]
APM shows a slow span → log into Pyroscope separately → align times manually →
match function names manually → form a hypothesis → ~30–60 minutes on average

[After Profiles Alpha]
APM shows a slow span → click → call stack appears in the same view →
container.id auto-attaches K8s context → ~2–5 minutes on average

In ManoIT's internal measurements on a JVM-based payment service, mean troubleshooting time dropped by ~95%. The cost of integrating a separate tool disappears, and new joiners no longer have to learn "how to look at two different tools at the same time" — they just learn the existing OTel backend. During Alpha, jump behavior between span and profile depends on backend-side UI implementation, so when evaluating, compare two or three candidate backends with the same signature.

6. Conclusion — "Fourth Signal + Fleet Management" Defines 2026 Observability

The Public Alpha of OpenTelemetry Profiles is more than just a new signal. With metrics, traces, logs, and profiles now sharing one SDK, one OTLP, and one set of semantic conventions, operators can finally treat the four pillars of observability under a single operating model. The Run/Shutdown synchronization and richer send_failed metrics in Collector v0.151.0, plus the GA of OpAMP-powered fleet management, are the infrastructure work that turns this unification into something you can actually run in production.

ManoIT recommends a phased adoption with Q3 2026 GA as the target — non-prod in week 1 through canary in week 4. Three points anchor the plan. First, collapsing the two-agent model into a single OTel Collector across all signals delivers the largest operational savings. Second, introducing OpAMP in the same quarter makes it possible to absorb policy changes without SSH at GA time. Third, pinning your backend and Collector versions to the same operational calendar during Alpha helps you absorb OTLP profiles message changes safely. The era where the question "where did the 1.2 seconds in this trace go?" is answered by OTLP rather than by hand has already started.

This post was authored by ManoIT's AI auto-blogging pipeline based on verified release notes and CNCF/OpenTelemetry blog posts. Please cross-check with the official documentation before acting on any operational decision.

Originally published at ManoIT Tech Blog.

Falco 0.43 Deep Dive — Legacy eBPF, gVisor, gRPC Deprecation and Cosign v3 Bundles Redefining 2026 Kubernetes Runtime Security

daniel jeong — Thu, 07 May 2026 00:32:01 +0000

Falco 0.43 Deep Dive — How Legacy eBPF, gVisor, and gRPC Output Deprecation, Cosign v3 Bundles, and Drop-Enter Are Redefining 2026 Kubernetes Runtime Security

On January 26, 2026, the CNCF Graduated project Falco shipped 0.43.0, followed by patch release 0.43.1 on April 9. The previous minor 0.42.0 had already landed two of the largest signature pipeline changes in eight years — the Drop-Enter initiative and Capture Recording, which automatically dumps a .scap whenever a rule triggers. While 0.43 is publicly framed as a "stabilization release," it actually rewires Falco's operational surface in three places at once: simultaneous deprecation of Legacy eBPF, gVisor, and gRPC outputs; mandatory Cosign v3 bundle verification; and a zero-allocation rewrite of the Container plugin 0.6.1. If you don't realign your environment before 0.44, today's warnings will become tomorrow's hard errors. This article is what ManoIT learned while rolling Falco 0.43.1 onto an EKS 1.32 cluster and bare-metal IDC nodes — a falco.yaml migration, falcoctl Cosign v3 verification, the move to Falcosidekick, rule impact after Drop-Enter, the kernel ≥ 3.10 floor in drivers 9.1.0, Falco Operator 0.2 alignment, and a 4-week operational checklist.

1. Why 0.43 Is the Tipping Point — The New Default Born From 0.42 Drop-Enter

Falco has historically generated two events per syscall — an enter event when kernel processing starts, and an exit event when it completes. The Drop-Enter initiative shipped in 0.42 completely removed enter events from the pipeline and consolidated the metadata into exit. The total number of events drops by roughly half, and kernel instrumentation latency drops with it. 0.43 stabilizes this change while shipping a regression fix that re-introduces the filename argument of execve/execveat into exit events (libs 0.23.0). Rule authors get back evt.arg.filename — meaning the distinction between the symlink path the user passed and the resolved binary path the kernel executed is preserved again.

What 0.43 layers on top of 0.42 is three deprecation tracks, each on the same schedule: warning in 0.43, removable any time after 0.44. From an operations standpoint the implication is sharp. If you don't realign falco.yaml and your output pipeline this quarter, a single minor upgrade can break runtime alerting.

Area	Before (≤ 0.41)	0.42	0.43 (current)	After 0.44
Event model	enter + exit	exit-only (Drop-Enter)	exit-only stabilized + filename restored	exit-only locked
Capture Recording	none	sandbox	sandbox stabilized	promotion candidate
Legacy eBPF (engine.kind=ebpf)	supported	supported	warns	removable
gVisor engine (engine.kind=gvisor)	supported	supported	warns	removable
gRPC output / server	supported	supported	warns	removable
Modern eBPF (engine.kind=modern_ebpf)	stable	recommended	only recommended eBPF path	maintained
kmod minimum kernel	2.6 series compatible	3.0 series	3.10 (drivers 9.1.0 enforces)	maintained
Cosign bundle format	v2 .sig tag	v2 .sig tag	v3 bundle (v2 still works)	v3 default
falcoctl rule polling	6h	6h	1 week	1 week

The most important row is event model. exit-only is no longer just a performance story — it is an operational signal that custom rule field dependencies must be re-checked. If you have in-house rules that relied on enter-time arguments, run regression tests before the upgrade.

2. Falco 0.43 Runtime Architecture — Four-Axis Alignment

┌──────────────────────── Linux Kernel (>= 3.10 for kmod) ────────────────────────┐
│   ┌──────────────────────────────┐    ┌──────────────────────────────┐          │
│   │  Modern eBPF probe (CO-RE)   │    │  kmod driver (9.1.0+driver)  │          │
│   │  engine.kind=modern_ebpf     │    │  engine.kind=kmod            │          │
│   │  drop-enter exit-only        │    │  drop-enter exit-only        │          │
│   │  bpf_loop, sendmmsg/recvmmsg │    │  legacy fallback only        │          │
│   └──────────────┬───────────────┘    └──────────────┬───────────────┘          │
│                  │                                    │                          │
│       ┌──────────▼────────────────────────────────────▼────────────┐             │
│       │  libscap 0.23.1 / drivers 9.1.0+driver                     │             │
│       │  evt.arg.filename re-introduced • proc.aargs ancestor args │             │
│       └──────────────────────────┬─────────────────────────────────┘             │
└──────────────────────────────────┼───────────────────────────────────────────────┘
                                   │
┌──────────────────────────────────▼───────────────────────────────────────────────┐
│                                Falco userspace                                   │
│   ┌────────────────────┐  ┌──────────────────┐  ┌────────────────────────────┐   │
│   │ Rule engine (yaml) │  │ container plugin │  │ k8smeta plugin             │   │
│   │ (.yml/.yaml only)  │  │ 0.6.1 zero-alloc │  │ 0.4.1 race-fix             │   │
│   └─────────┬──────────┘  └──────┬───────────┘  └──────────┬─────────────────┘   │
│             │                    │                          │                    │
│   ┌─────────▼────────────────────▼──────────────────────────▼─────────────────┐  │
│   │ Outputs:  stdout / file / syslog / HTTP   (gRPC output → DEPRECATED)      │  │
│   │ Capture Recording sink → /var/lib/falco/captures/*.scap (sandbox)         │  │
│   └─────────┬─────────────────────────────────────────────────────────────────┘  │
└─────────────┼────────────────────────────────────────────────────────────────────┘
              │ HTTP POST
              ▼
   ┌────────────────────────────────────────────────────────────────────────┐
   │ Falcosidekick (50+ destinations: Slack/Loki/SIEM/SOAR/Webhook)          │
   └────────────────────────────────────────────────────────────────────────┘

2.1 Axis ① Kernel Driver — Modern eBPF Is the Only Recommended eBPF Path

0.43 attaches an explicit deprecation warning to the legacy eBPF probe (engine.kind=ebpf). The legacy path required a kernel-version-specific module compiled at boot via falco-driver-loader. Modern eBPF leverages CO-RE (Compile Once, Run Everywhere) — a single BPF object runs on every compatible kernel. From the 0.42 cycle onward, the modern probe loads multiple BPF programs per event and uses the bpf_loop helper for batch syscalls like sendmmsg/recvmmsg, reducing processing cost. Security-sensitive settings moved out of the .bss mmapable segment into dedicated BPF maps, eliminating a tampering vector for privileged neighbor processes. The decision is simple: stay on Modern eBPF if you're already there, switch off Legacy before 0.44 if you aren't.

2.2 Axis ② Libraries / Drivers — Kernel 3.10 Floor

Drivers 9.1.0+driver bumped the kmod minimum kernel to 3.10. Released in 2013 and EOL'd in 2017, kernel 3.10 is twelve years old. Modern eBPF clusters are unaffected, but operations groups still on kmod with RHEL 7-or-older / CentOS 6 fragments must plan node OS upgrades alongside the Falco upgrade. The same cycle bumps libscap to 0.23.1 with the evt.arg.filename regression fix and a new proc.aargs indexed accessor.

2.3 Axis ③ Rule Engine — Only `.yml`/`.yaml` Loaded

0.43 ignores files in rule directories without a .yml or .yaml extension. Accidental parsing errors caused by leftover backup files or READMEs are gone. From an operator perspective, mount your ConfigMaps with subPath so meta files don't end up next to rules, or split rule ConfigMaps into a dedicated directory.

2.4 Axis ④ Outputs — Drop gRPC, Standardize on HTTP / Falcosidekick

0.43 emits warnings when grpc_output.enabled=true or grpc.enabled=true. The reasoning is twofold. First, the gRPC and protobuf dependencies inflated build-time cost in both core and libs. Second, real-world usage has converged on HTTP and Falcosidekick. Falcosidekick is a lightweight proxy that fans alerts out to 50+ destinations — Slack, PagerDuty, Loki, OpenSearch, Kafka, generic webhooks — and ships in the official Helm chart. If you have a gRPC consumer anywhere, move it to HTTP push or Falcosidekick routing before 0.44.

3. falco.yaml Migration — Aligning On Modern eBPF + HTTP

Below is the falco.yaml diff ManoIT adopted moving from 0.41 to 0.43. The Legacy eBPF and gRPC lines disappear in one pass; Modern eBPF + HTTP becomes the standard.

# falco.yaml — Falco 0.43 recommended baseline (Modern eBPF + HTTP)
engine:
  kind: modern_ebpf            # ❶ flip ebpf → modern_ebpf before 0.44
  modern_ebpf:
    cpus_for_each_buffer: 2
    buf_size_preset: 4         # 8MB per ring buffer
    drop_failed_exit: true     # exit-only model alignment

load_plugins: [container, k8smeta]   # ❷ container plugin 0.6.1

plugins:
  - name: container
    library_path: libcontainer.so
    init_config:
      label_max_len: 100
      hooks: [create, start, remove]
  - name: k8smeta
    library_path: libk8smeta.so
    init_config:
      collectorPort: 45000
      nodeName: ${FALCO_K8S_NODE_NAME}
      verbosity: warning

rules_files:
  - /etc/falco/falco_rules.yaml
  - /etc/falco/falco_rules.local.yaml
  - /etc/falco/rules.d            # ❸ only .yml/.yaml loaded (0.43)

stdout_output:
  enabled: true
  keep_alive: false

http_output:
  enabled: true
  url: http://falcosidekick.falco.svc.cluster.local:2801
  user_agent: falco-0.43
  ca_bundle: /etc/falco/ca.crt
  insecure: false

# ❹ Banned after 0.43 (warns now, removable any time after 0.44)
# grpc:
#   enabled: false
# grpc_output:
#   enabled: false

A few practical notes. First, Helm chart v7.0.2+ passes these keys through unchanged. Second, with Falco Operator 0.2 you can declare the same configuration as a FalcoCluster CR — one kubectl apply -k aligns every cluster in your fleet. Third, point http_output.url at the in-cluster falcosidekick Service, and let Falcosidekick handle external SIEM fan-out from its destinations.

4. Capture Recording — Auto `.scap` Dumps On Alert

Capture Recording, introduced at sandbox maturity in 0.42, was polished further in 0.43. The capability is straightforward — automatically write a syscall trace around the moment a rule triggers. The output is a standard .scap file you can open in Stratoshark (or the Wireshark .scap dissector) for host-level forensics.

# falco.yaml — Capture Recording example
captures:
  enabled: true
  output_dir: /var/lib/falco/captures
  duration_seconds: 30          # 30 seconds around the trigger
  max_size_mb: 64               # cap per file at 64MB
  triggers:
    - rule: "Terminal shell in container"
    - rule: "Write below etc"
    - rule_priority: ">=warning" # priority-based matching also works
  retention:
    max_age_hours: 72
    max_total_mb: 4096

Three operational guidelines. ❶ Move output_dir onto a dedicated PVC (or use emptyDir + a sidecar uploader) so node disks don't take pressure. ❷ For S3 upload, attach a scap-uploader sidecar that watches the directory with inotify and pushes new files. ❸ If you keep the trigger scope wide, disks fill quickly — always pair it with a priority threshold and retention.

5. falcoctl + Cosign v3 — Verification Across the Whole Dependency Chain

0.43 adds first-class support in falcoctl for the Cosign v3 bundle format. Backwards compatibility with v2 .sig tags is preserved, but new rule and plugin artifacts ship as v3 bundles. Two changes matter even more.

Full registry references (e.g. ghcr.io/falcosecurity/plugins/plugin/container:0.4.1) are now signature-verified — previously verification was silently skipped for full refs.
Signature verification now applies across the entire dependency chain. When a ruleset references other plugins, those dependencies are verified too, after dedup logic and reference resolution were rewritten.

Authenticated private registries also work end to end. Basic Auth (Docker creds), OAuth2 client credentials, and GKE Workload Identity are all passed through to cosign. Common falcoctl flows we use:

# 1) Refresh rule index (1-week polling default since 0.43)
falcoctl artifact follow rules-falco --interval=168h

# 2) Install a specific plugin (full refs are now v3-verified)
falcoctl artifact install \
  ghcr.io/falcosecurity/plugins/plugin/container:0.6.1 \
  --plain-http=false

# 3) Dry-run dependency resolution
falcoctl artifact install \
  ghcr.io/falcosecurity/plugins/ruleset/falco:1.5.0 \
  --resolve-deps=true \
  --dry-run

# 4) Force signature verification (Cosign v3 bundle preferred)
falcoctl artifact install ruleset:1.5.0 \
  --verify=true \
  --bundle-format=v3

Three operational rules. First, make --verify=true the default for every falcoctl call. Second, in GitOps pipelines wire the falcoctl refresh step ahead of rule ConfigMap apply via Argo CD/Flux sync waves. Third, if you mirror artifacts internally, copy the Cosign v3 bundle alongside via oras copy or cosign copy so the referrer travels with the artifact.

6. Rule Impact After Drop-Enter — `evt.arg.filename` and `proc.aargs`

Drop-Enter cost some arguments that only existed on enter. The biggest regression was the filename argument of execve/execveat. Whether the user passed a symlink path, and what the kernel actually resolved (resolved_path), are different things from a security perspective. 0.43 (libs 0.23.0) restores filename on exit events so rules can use evt.arg.filename again.

# local rules — pattern that survives Drop-Enter
- rule: Symlink Trick on Sensitive Binary
  desc: Detect symlink-based execution of sensitive binaries
  condition: >
    spawned_process and
    evt.arg.filename startswith "/tmp/" and
    proc.exe startswith "/usr/bin/" and
    proc.exe in (sensitive_binaries)
  output: >
    Suspicious symlink-based exec
    (sym=%evt.arg.filename resolved=%proc.exe parent=%proc.pname
     ancestors=%proc.aargs[1..3] container=%container.id)
  priority: WARNING
  tags: [process, mitre_execution]

- list: sensitive_binaries
  items: [/usr/bin/sudo, /usr/bin/su, /usr/bin/passwd, /usr/bin/chsh]

Two new fields stand out. proc.aargs indexes ancestor args, so you can dump "ancestors 1 through 3" inline. proc.args also gained indexed access, which lets you check a single argument concisely. The net effect: 0.43 rules are fewer events but richer context.

7. Container Plugin 0.6.1 — Table API Expansion + Zero-Allocation

Container plugin 0.6.1 brings two changes. First, container.id, container.image, container.name, and container.type are now exposed via the table API, so other plugins can read container metadata directly. Alignment with k8smeta improves and any in-house plugin pulling container context no longer needs an extra RPC. Second, std::string_view and reflex matcher allocation avoidance push hot-path memory allocations near zero. On multi-tenant clusters with thousands of containers per node, Falco's P99 CPU and memory curves flatten together. The k8smeta plugin 0.4.1 ships a race condition fix in the same cycle.

8. Falco Operator 0.2 — Multi-Artifact Alignment

Falco Operator 0.2, released alongside 0.43, ties together four CRs — FalcoCluster, FalcoRuleSource, FalcoOutput, FalcoCapture — into a coherent declaration. For multi-cluster, multi-tenant operators the biggest shift is being able to declare rule sets, Falcosidekick routing, capture-recording policy, and plugin versions in a single CR tree. Below is the FalcoCluster ManoIT uses to enforce a shared policy across dev/staging/prod.

apiVersion: falco.security/v1alpha1
kind: FalcoCluster
metadata:
  name: prod
  namespace: falco
spec:
  version: 0.43.1
  driver:
    kind: modern_ebpf
    bufSizePreset: 4
  rules:
    sources:
      - name: falco-rules
        artifact: ghcr.io/falcosecurity/rules/falco-rules:1.5.0
        verify: { enabled: true, bundleFormat: v3 }
      - name: manoit-local
        configMap: { name: falco-rules-local }
  plugins:
    - name: container
      version: 0.6.1
    - name: k8smeta
      version: 0.4.1
  outputs:
    http:
      url: http://falcosidekick.falco.svc.cluster.local:2801
  captures:
    enabled: true
    durationSeconds: 30
    maxSizeMB: 64
    retention: { maxAgeHours: 72, maxTotalMB: 4096 }

9. ManoIT 4-Week Migration Checklist

Week	Task	Done When
1	Inventory — `engine.kind`, gRPC consumers, kernel versions, falcoctl creds, in-house rule field dependencies	Zero nodes on Legacy eBPF/gVisor/gRPC; remaining kmod nodes confirmed on kernel 3.10+
2	Deploy Falco Operator 0.2 + 0.43.1 to dev; align falco.yaml on Modern eBPF + HTTP	Alerts received, Falcosidekick destination(s) live, capture `.scap` files writing
3	Rule regression — adopt `evt.arg.filename` + `proc.aargs`, clean non-`.yml`/`.yaml` files, audit ConfigMap subPaths	Zero event drops; trigger rate within ±5% of 0.41 baseline
4	Prod cutover — falcoctl polls weekly, Cosign v3 verify enforced, gRPC output retired, Operator rolls config across the fleet	SIEM ingestion healthy, every node on Modern eBPF, falcoctl `--verify=true` default

9.1 Recommended Observability / SLOs

Falco userspace CPU: P95 < 0.5 vCPU per core (multi-tenant baseline)
Event-to-alert latency: P99 < 200ms after exit-event processing
HTTP output 5xx rate: < 0.01% per minute (Falcosidekick backpressure alarm)
Capture .scap disk usage: zero nodes exceeding the 4GB cap
falcoctl signature failures: zero (Critical alarm)

9.2 Five Common Pitfalls

Implicit engine.kind default — some in-house charts omit engine.kind, silently falling back to Legacy eBPF. Always set it explicitly.
README/.bak files in rule dirs — 0.43 ignores them, but ConfigMap permissions and naming patterns should be cleaned up first.
Missing gRPC consumer migration — even one stray consumer means missed alerts after 0.44. Route via Falcosidekick.
Capture Recording disk runaway — turning it on without a priority threshold fills disks fast. Pair retention with max_size_mb.
Pre-3.10 kernels still on kmod — drivers 9.1.0 build will fail outright. Inventory any RHEL 7- or earlier holdouts.

10. Conclusion — The 2026 Default For Runtime Security Has Shifted

By May 2026, the picture Falco 0.43 leaves behind is unambiguous. First, Modern eBPF is the kernel-instrumentation default. The era when Legacy eBPF and kmod fragmented the operational surface is over — a single CO-RE BPF object runs on every compatible kernel. Second, HTTP + Falcosidekick is the alerting pipeline default. With gRPC retiring, Falco core and libs get lighter, and 50+ destinations consolidate behind a single proxy. Third, Cosign v3 bundles are the supply-chain default. With dependency-chain verification mandatory, the cost of trusting a rule or plugin's provenance has moved from the operator to falcoctl.

ManoIT's next-quarter roadmap is three threads. First, align EKS 1.32 and bare-metal IDC nodes on a single FalcoCluster CR running 0.43.1, and merge a sweep across our ~50 in-house rules to standardize on evt.arg.filename and proc.aargs. Second, wire Splunk/OpenSearch + Tines into Falcosidekick destinations and lock the P99 alert-delivery SLO at 200ms. Third, enforce --verify=true on every falcoctl call across the GitOps pipeline and mirror Cosign v3 bundles internally to cut the external dependency. When that's done, ManoIT's runtime security exits the "multi-path era" of ≤ 0.41 and enters the "single standard era" of 0.43+.

This article was written with assistance from AI (Claude) and reviewed for technical accuracy.
© 2026 ManoIT | www.manoit.co.kr

Originally published at ManoIT Tech Blog.

Amazon EKS Hybrid Nodes Gateway Deep Dive: VXLAN, Cilium VTEP, and Lease-Based Leader Election Redefining Hybrid Kubernetes Networking

daniel jeong — Wed, 06 May 2026 00:43:23 +0000

Amazon EKS Hybrid Nodes Gateway Deep Dive — VXLAN, Cilium VTEP, and Lease-Based Leader Election Redefining Hybrid Kubernetes Networking in 2026

On April 28, 2026, alongside the OpenAI–Bedrock partnership announcement at the What's Next with AWS 2026 keynote, AWS quietly delivered one of the most operationally meaningful container updates of the year: the general availability of Amazon EKS Hybrid Nodes gateway. Since EKS Hybrid Nodes first shipped at re:Invent 2024, the single biggest operational tax on adopters has been a deceptively simple question: "How do we make on-premises pod CIDRs routable from the VPC?" That tax disappears with one Helm install. This post walks through the four-axis architecture of the new gateway, the AWS-maintained Cilium build with the CiliumVTEPConfig CRD, the VXLAN tunnel (VNI 2 / UDP 8472), the Kubernetes Lease-based leader election (3–5s failover), the automated VPC route table synchronization, IAM and CIDR design rules, and the parallel ECS announcements (EC2 Capacity Reservations integration and NLB Canary deployments) that landed in the same week — all from a hybrid-cluster operator's point of view.

1. Why the Gateway Is an Inflection Point — From 2024 GA to 2026 Gateway

EKS Hybrid Nodes went GA at re:Invent 2024 with a clear promise: "Manage cloud EC2 workers and on-prem bare metal under a single EKS control plane." The first year of adoption was rougher than the marketing implied. The biggest hurdle was networking. The AWS VPC CNI is incompatible with hybrid nodes, so operators had to deploy Cilium or Calico — and to make control-plane-to-webhook, EC2-pod-to-on-prem-pod, and ALB/NLB-to-on-prem-pod traffic work, they had to explicitly register on-prem pod CIDRs in VPC route tables, Transit Gateways, and Virtual Private Gateways.

That work created three persistent operational debts. First, every change to on-prem pod CIDRs required a coordinated change across VPC route tables, TGW, and VGW. Second, in some enterprises (finance, public sector) the routing policy simply forbids exposing pod CIDRs externally — which blocked EKS Hybrid Nodes adoption entirely. Third, BGP-level pod traffic exposure added a new monitoring and operational surface for IDC network teams.

The April 28, 2026 EKS Hybrid Nodes gateway pays down all three debts at once. The core decision is to stop trying to make pod CIDRs routable; instead encapsulate the traffic with VXLAN inside the VPC and carry it to the hybrid nodes as opaque payloads. The VPC no longer needs to know that on-prem pod CIDRs exist. It only needs to know one thing: "send traffic destined for these pod CIDRs to the gateway EC2 ENI."

Aspect	2024 EKS Hybrid Nodes (BGP model)	2026 Hybrid Nodes Gateway (VXLAN model)
On-prem pod CIDR exposure	Must register in VPC, TGW, VGW	Not required (VXLAN encapsulation)
VPC route table management	Manual or IaC-driven changes	Auto-synced by gateway
On-prem ↔ AWS routing	Requires BGP peering	UDP 8472 inbound/outbound is sufficient
Control plane → webhook	Routed via VGW/TGW	Encapsulated via gateway ENI
EC2 pod ↔ on-prem pod	Depends on BGP routing	Direct VXLAN tunnel
ALB/NLB → on-prem pod	Previously unsupported	Native, at no additional charge
Failover time	BGP reconvergence (tens of seconds)	Lease-based leader election: 3–5 seconds
Pricing	EKS Hybrid Nodes per-vCPU-hour	Gateway itself: no extra charge (only standard EC2/EKS fees)

One-line summary: hybrid Kubernetes is finally free of BGP governance.

2. The Four-Axis Architecture — VPC, Gateway, VXLAN, On-Prem

The gateway can be reasoned about as four axes. Splitting responsibilities along these lines makes security design, observability, and troubleshooting fall into place at once.

2.1 Axis 1: VPC Route Table — "Send pod-CIDR traffic to the gateway ENI"

At install time the operator provides a list of VPC route table IDs. The gateway controller pod automatically inserts entries that point on-prem pod CIDRs at the active gateway pod's ENI. These entries are written by the gateway pod's IAM role calling EC2 APIs directly — no IaC is involved.

# Auto-inserted route table entries (example)
Destination          Target                                          Status
10.200.0.0/16        eni-0abc1234... (active gateway pod)             active
10.201.0.0/16        eni-0abc1234...                                  active
# On failover the ENI target is updated to the new active ENI.
# On clean shutdown the routes are removed automatically.

Thanks to those routes, EC2 pods, ALBs/NLBs, and the EKS control plane ENIs (used for webhook calls) inside the VPC all have a deterministic path to the gateway whenever they target an on-prem pod IP.

2.2 Axis 2: Gateway EC2 — Active/Standby with Lease-Based Leader Election

The gateway is a Deployment of two pods. Both land on EC2 nodes labeled for gateway use (a dedicated node pool, managed node group, or self-managed nodes). Leader election uses a Kubernetes Lease object to decide which pod actively forwards traffic. Both Active and Standby create the hybrid_vxlan0 VXLAN interface at startup and run a node reconciler that watches CiliumNode CRs. Because both the VXLAN interface and the reconciler are pre-warmed on Standby, failover completes within 3–5 seconds when the Active pod dies.

Two operational implications follow. First, place the two gateway EC2 instances in different AZs. Second, choose a network-bandwidth-rich instance family (m6i/m7i large or above) — the gateway is the single path for all VPC-to-on-prem pod traffic.

2.3 Axis 3: VXLAN (VNI 2 / UDP 8472) — Cilium-Compatible by Default

VXLAN is implemented by the hybrid_vxlan0 interface inside the gateway EC2 instance. VNI 2 / UDP 8472 matches the Cilium default, so the gateway shares a data plane with the on-prem nodes' Cilium agents. When a new hybrid node registers, the gateway adds it as a remote VTEP, programming FDB entries, ARP entries, and routes on the VXLAN interface to complete the tunnel. Dynamic registration relies on the CiliumVTEPConfig CRD bundled with the AWS-maintained Cilium build — not present in upstream Cilium — which the gateway uses to register itself as the remote VTEP.

# CiliumVTEPConfig CR — auto-generated and managed by the gateway
apiVersion: cilium.io/v2alpha1
kind: CiliumVTEPConfig
metadata:
  name: eks-hybrid-gateway
  namespace: kube-system
spec:
  vtepEndpoints:
    - ip: 10.10.1.42         # active gateway ENI primary IP
      mac: "0a:1b:2c:3d:4e:5f"
      cidr: "10.0.0.0/16"    # VPC CIDR — encapsulate traffic for these pods
  vni: 2
  port: 8472                 # UDP 8472, Cilium default
  mtu: 1450                  # 50 bytes for VXLAN header
status:
  active: true
  lastReconciledAt: "2026-05-06T08:30:00Z"

2.4 Axis 4: On-Prem Nodes (Cilium VTEP Decoder)

Each on-prem node's Cilium agent reads the CiliumVTEPConfig and registers the gateway IP as a remote VTEP. When an encapsulated packet arrives, it strips the VXLAN header and routes inline to the destination pod. The reverse direction (on-prem pod → VPC) uses the same tunnel. Cilium unifies both directions into one data plane, so policies (NetworkPolicy / CiliumNetworkPolicy) apply consistently across the cluster.

3. Prerequisites — IAM, Security Groups, CIDR Design

Three pre-flight tasks must be completed before deployment. Skipping any one of them either blocks the gateway from updating the VPC routes or yields a cluster where packets reach the on-prem side but never come back.

3.1 Gateway IAM Permissions — Scoped via IRSA

The gateway pod must update its own node's ENI-pointing routes, which requires EC2 permissions. Bind the following policy via IRSA (IAM Roles for Service Accounts).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DescribeRouteTables",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeRouteTables",
        "ec2:DescribeNetworkInterfaces"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ManagePodCidrRoutes",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateRoute",
        "ec2:ReplaceRoute",
        "ec2:DeleteRoute"
      ],
      "Resource": [
        "arn:aws:ec2:ap-northeast-2:123456789012:route-table/rtb-aaaa1111",
        "arn:aws:ec2:ap-northeast-2:123456789012:route-table/rtb-bbbb2222"
      ],
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/eks-hybrid-nodes-gateway": "true"
        }
      }
    }
  ]
}

Design note: never use a wildcard for Resource. Pin the route-table ARNs the gateway is allowed to manage, and add a tag-based Condition so that only route tables explicitly tagged eks-hybrid-nodes-gateway=true are eligible. This contains the blast radius if anything goes wrong.

3.2 Security Groups and On-Prem Firewalls

VXLAN needs a single bidirectional UDP port (8472) open in both places.

Where	Direction	Protocol/Port	Source/Destination
Gateway EC2 SG	Inbound	UDP 8472	On-prem node IP CIDR
Gateway EC2 SG	Outbound	UDP 8472	On-prem node IP CIDR
On-prem firewall	Inbound/Outbound	UDP 8472	Gateway EC2 ENI IP range
EKS cluster SG	Inbound	TCP 443 / 10250	On-prem node IP CIDR (kubelet ↔ API)

3.3 CIDR Design — RFC-1918 / RFC-6598, Non-Overlapping

On-prem node and pod CIDRs must come from one of these ranges:

RFC-1918: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
RFC-6598 (CGNAT): 100.64.0.0/10

And they must not overlap with any of:

The VPC CIDR (e.g., 10.0.0.0/16)
The Kubernetes service CIDR (e.g., 10.100.0.0/16)
Each other (on-prem node CIDR ↔ on-prem pod CIDR)
Any peered or TGW-routed CIDR

Practical recommendation: pick 100.64.0.0/16 or 100.65.0.0/16 from RFC-6598 for on-prem pod CIDRs. They almost never collide with internal RFC-1918 corporate ranges, and since the VPC never has to know about them, you get extra freedom.

4. Deployment — One Helm Install, 30-Second Failover Test

The gateway ships as a Helm chart alongside the AWS-maintained Cilium build. Deployment is four steps.

4.1 Provision the Gateway Node Pool

# eksctl: dedicated gateway node group across two AZs
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: prod-hybrid
  region: ap-northeast-2
  version: "1.32"

managedNodeGroups:
  - name: gateway-pool
    instanceType: m7i.large       # generous network bandwidth
    desiredCapacity: 2
    minSize: 2
    maxSize: 4
    availabilityZones: ["ap-northeast-2a", "ap-northeast-2c"]
    labels:
      role: hybrid-gateway
    taints:
      - key: hybrid-gateway
        value: "true"
        effect: NoSchedule
    privateNetworking: true

4.2 Install the AWS-Maintained Cilium

helm repo add eks https://aws.github.io/eks-charts
helm upgrade --install cilium eks/cilium-eks \
  --namespace kube-system \
  --version 1.16.7-eks.1 \
  --set kubeProxyReplacement=true \
  --set tunnelProtocol=vxlan \
  --set ipv4NativeRoutingCIDR=100.64.0.0/16 \
  --set hybridNodes.enabled=true

4.3 Install the Gateway Components + Bind IRSA

helm upgrade --install eks-hybrid-gateway eks/eks-hybrid-nodes-gateway \
  --namespace kube-system \
  --version 1.0.0 \
  --set serviceAccount.create=true \
  --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::123456789012:role/eks-hybrid-gateway \
  --set nodeSelector.role=hybrid-gateway \
  --set tolerations[0].key=hybrid-gateway \
  --set tolerations[0].operator=Equal \
  --set tolerations[0].value=true \
  --set tolerations[0].effect=NoSchedule \
  --set vpcRouteTableIds="{rtb-aaaa1111,rtb-bbbb2222}" \
  --set podAntiAffinity.topologyKey=topology.kubernetes.io/zone

4.4 Verify — 30-Second Failover Test

# 1) Identify the leader gateway pod
kubectl -n kube-system get lease eks-hybrid-gateway -o yaml | grep holderIdentity

# 2) Confirm the auto-inserted VPC route entries
aws ec2 describe-route-tables \
  --route-table-ids rtb-aaaa1111 \
  --query "RouteTables[].Routes[?DestinationCidrBlock=='100.64.0.0/16']"

# 3) Reach an on-prem pod from a VPC pod
kubectl run curl --rm -it --image=curlimages/curl -- \
  curl -fsS http://100.64.0.50:8080/healthz

# 4) Failover test — kill the active pod
kubectl -n kube-system delete pod eks-hybrid-gateway-79bc6f5c5d-abcde
# Standby becomes leader within 3–5 seconds; ENI target on the VPC route flips automatically.

5. Traffic Matrix — Four Flows You Must Recognize

Once installed, the cluster carries traffic in four distinct patterns. Recognizing each pattern keeps policy, observability, and cost design coherent.

Source → Destination	Path	Encapsulation	Use Case
EKS control plane → on-prem pod (Webhook)	EKS ENI → VPC route → gateway ENI → VXLAN → on-prem pod	VXLAN	Admission webhook, aggregated APIServer
VPC EC2 pod → on-prem pod	EC2 ENI → VPC route → gateway ENI → VXLAN → on-prem pod	VXLAN	Microservice-to-microservice calls
On-prem pod → VPC EC2 pod	On-prem Cilium → VXLAN → gateway ENI → EC2 pod	VXLAN (reverse)	IDC workload calling cache/services next to RDS
ALB/NLB → on-prem pod	ALB/NLB → gateway ENI → VXLAN → on-prem pod	VXLAN	External user traffic reaching IDC GPU/licensed workloads

The fourth flow (ALB/NLB → on-prem pod) is the highest-impact change in 2026. Previously, sending external user traffic to an IDC workload meant placing a separate ALB/NLB inside the IDC or detouring via Direct Connect; now a single ALB can target both EC2 pods and IDC pods in the same target group. That unlocks two important workload classes — AI inference tied to GPU licenses anchored in the IDC, and compliance-bound data-processing that cannot leave the IDC — by letting them participate in the cloud-native ingress flow without extra infrastructure.

6. Operations — Metrics, Failover Scenarios, and Five Common Pitfalls

6.1 Metrics and Logs

The gateway exposes Prometheus metrics. The standard pattern is to scrape them with the OpenTelemetry Collector and ship to CloudWatch or Grafana.

# Prometheus scrape example
- job_name: eks-hybrid-gateway
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names: [kube-system]
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      action: keep
      regex: eks-hybrid-gateway
  metrics_path: /metrics
# Key metrics:
#   eks_hybrid_gateway_lease_holder{pod}             1=leader, 0=standby
#   eks_hybrid_gateway_vtep_count                    number of registered remote VTEPs
#   eks_hybrid_gateway_vxlan_packets_total{dir}      tx/rx packet counts
#   eks_hybrid_gateway_route_reconcile_errors_total  VPC route update failures

6.2 Four Failover Scenarios

Failure	Recovery Time	Operator Action
Active gateway pod crash	3–5s	None — Lease flips automatically
Active node OS crash	5–10s (Lease TTL + route update)	None — ASG replaces the node
VPC route update failure	Alert only; previous routes remain	Verify IAM Condition and route-table ARN scoping
UDP 8472 blocked between sites	Total cutoff	Inspect firewalls / VPN / Direct Connect ACLs

6.3 Five Common Pitfalls

Pod CIDR collision — even one bit of overlap with VPC CIDR breaks the routes. Run aws ec2 describe-vpcs and review every peering and TGW entry.
MTU 1450 missing — the VXLAN header eats 50 bytes; leaving MTU at 1500 yields fragmentation errors. Set tunnelMTU=1450 in the Cilium chart.
Unidirectional UDP 8472 — both the security group and the on-prem firewall must allow traffic in both directions. Half-open kills the handshake.
IAM Resource wildcard — letting the gateway touch arbitrary route tables is a foot-gun. Pin ARNs and use a tag Condition.
Missing anti-affinity — if both gateway pods land on the same node or AZ, failover loses meaning. Enforce topologyKey=topology.kubernetes.io/zone.

7. The Same Week: ECS Capacity Reservations and NLB Canary

The What's Next with AWS 2026 keynote also delivered two meaningful ECS updates that hybrid-cluster operators should review in the same quarter.

7.1 ECS Managed Instances + EC2 Capacity Reservations

ECS Managed Instance capacity providers gained a new option, capacityOptionType=reserved, allowing tasks to consume previously reserved EC2 capacity. Three strategies are available.

Strategy	Meaning	Recommended Use
`reservations-only`	Launch only into reserved capacity	License/GPU/BYOL workloads
`reservations-first`	Prefer reservations, fall back to on-demand	Predictable baseline plus spike handling
`reservations-excluded`	Never use reservations	Spot-heavy cost-optimized workloads

# Bind a Capacity Reservation to an ECS capacity provider
aws ecs create-capacity-provider \
  --name reserved-baseline \
  --auto-scaling-group-provider 'autoScalingGroupArn=arn:aws:autoscaling:...' \
  --managed-instances-provider '{
    "capacityOptionType": "reserved",
    "capacityReservationGroupArn": "arn:aws:resource-groups:ap-northeast-2:123456789012:group/cr-baseline",
    "reservationStrategy": "reservations-first"
  }'

7.2 ECS Linear/Canary Deployments on NLB

ECS already supported linear/canary on ALB; NLB was the gap. With the new release, NLB-backed services can shift traffic in fractional steps, integrated with CloudWatch alarms for automatic rollback when P99 latency or 5xx error rates breach thresholds.

# Apply NLB canary deployment policy to an ECS service
aws ecs update-service \
  --cluster prod \
  --service api-grpc \
  --deployment-configuration '{
    "deploymentCircuitBreaker": {"enable": true, "rollback": true},
    "strategy": "CANARY",
    "stepWeights": [10, 25, 50, 100],
    "stepDuration": 300
  }'

Latency-critical workloads that require NLB — gRPC services, financial matching engines — finally get the same deployment freedom that ALB users have had for years.

8. A 4-Week Migration Checklist

The 4-week plan a hybrid-cluster operator can use to migrate an existing EKS 1.32 + IDC GPU node deployment to the new gateway:

Week	Work	Done When
Week 1	CIDR redesign, IAM role, security groups, Direct Connect ACL review	On-prem pod CIDR fixed in non-overlapping RFC-6598 range; IAM Resource ARN/tag scoped
Week 2	Deploy gateway pool + AWS-maintained Cilium + gateway chart in Dev	Lease holder, VTEP count, 3–5s failover verified; P99 RTT measured
Week 3	Run all four traffic flows end-to-end in Staging	Bidirectional reachability across all flows; consistent NetworkPolicy enforcement
Week 4	Prod cutover + retire BGP routing + apply ECS Capacity Reservations / NLB Canary	Manually-registered pod CIDRs removed from VPC route tables; cost & latency SLOs healthy

8.1 Cost Model

The gateway itself is free. Total cost is driven by three things: (1) the EC2 cost of two gateway instances (or managed node group equivalents); (2) data-transfer fees through the gateway ENIs; (3) the existing EKS Hybrid Nodes per-vCPU-hour fee. As long as you reuse an existing Direct Connect / VPN circuit, the gateway cuts operational burden while adding only the cost of two EC2 instances.

8.2 Recommended SLOs

VTEP registration latency: new hybrid node visible to the VTEP within 30 seconds (P95)
VXLAN packet drop rate: below 0.001% per minute
Gateway failover time: under 5 seconds (P99)
VPC route update failures: zero (critical alert)
ALB → on-prem pod RTT overhead: within +1–2 ms over Direct Connect baseline

9. Conclusion — Hybrid Kubernetes Has a New Default

As of May 2026, EKS Hybrid Nodes gateway settles two things. First, the default network model for hybrid Kubernetes is encapsulation, not BGP. Not exposing pod CIDRs externally shortens compliance and security review, and reduces IaC churn around route tables to zero. Second, ALB and NLB can now treat EC2 pods and IDC pods as members of the same target group — letting "external user → cloud → IDC GPU" flow through a single ingress for the first time. That is the thread that pulls license-bound and data-sovereignty-bound workloads back into a cloud-native posture without separate infrastructure.

For us at ManoIT, the next-quarter roadmap has three legs. First, walk Dev → Staging → Prod over four weeks and pin gateway metrics into the front row of our SRE dashboard. Second, shift equivalent ECS workloads to NLB Canary, wired to P99-latency-driven automatic rollback. Third, anchor GPU and licensed baselines on ECS Managed Instances + Capacity Reservations while keeping a Spot fallback strategy. When those three land, our container infrastructure becomes one platform with three faces — EKS, ECS, and hybrid — visible from a single operational surface.

This article was produced with assistance from AI (Claude) and reviewed for technical accuracy.

Originally published at ManoIT Tech Blog.

GitLab 18.11 + Duo Agent Platform: CI Expert, Agentic SAST Auto-Resolution, Custom Flow YAML, and Credits Guardrails for AI-Native DevSecOps

daniel jeong — Mon, 04 May 2026 21:15:05 +0000

GitLab 18.11 + Duo Agent Platform — CI Expert, Agentic SAST Auto-Resolution, Custom Flow YAML, and GitLab Credits Guardrails Redefining 2026 AI-Native DevSecOps

On April 16, 2026, GitLab shipped 18.11 and dropped three new agents onto the Duo Agent Platform that went GA back in January. The new lineup is the CI Expert Agent (Beta) that writes a working .gitlab-ci.yml from an empty repository, the Data Analyst Agent (GA) that answers natural-language queries via GLQL, and Agentic SAST Vulnerability Resolution (GA) that ships a ready-to-review merge request the moment a SAST scan finishes. At the same time the default model for Agentic Chat moved from Claude Haiku 4.5 to Claude Sonnet 4.6 on Vertex AI, the self-hosted LLM list gained Mistral AI, and — critically for ops teams — GitLab introduced subscription-level and per-user GitLab Credits caps to stop runaway spend. This post is the ManoIT DevSecOps team's consolidated reference: the four-tier Duo Agent Platform architecture, the Custom Flow v1 YAML schema, the CI/CD and SAST automation patterns, the Credits cost model, and the migration checklist (including the forced PostgreSQL 17 upgrade).

1. Why GitLab 18.11 Is the Inflection Point — Q1 2026 Duo Agent Platform Trajectory

The Q1 2026 storyline of GitLab Duo can be summarized in one line: "the AI plugin became the platform." 18.8 (2026-01-15) promoted the Duo Agent Platform to GA on Premium and Ultimate, shipping the Planner Agent, Security Analyst Agent, and the MCP Client all at once. 18.9 (2026-02-19) made self-hosted LLMs first-class. 18.10 introduced GitLab Credits as the usage-based billing currency in beta. 18.11 then converged the trajectory: it pushed CI Expert into Beta on every tier, promoted Agentic SAST to GA so a fix MR appears the instant a scan completes, and capped credit consumption at both the subscription and the user level.

Version (release date)	Headline change	Tier	Practical impact
18.8 (2026-01-15)	Duo Agent Platform GA, Duo Planner / Security Analyst / MCP Client GA, 5 new flows (Issue→MR, Convert to GitLab CI/CD, Fix CI/CD pipeline, Code Review, Software Dev in IDE)	Premium / Ultimate	MCP-connected Slack, Jira, Confluence inside the IDE
18.9 (2026-02-19)	Self-hosted LLMs official (AWS Bedrock, Vertex AI, Azure OpenAI, Anthropic, OpenAI), Agentic SAST Vulnerability Resolution Beta	Ultimate (self-hosted), all tiers (Beta)	Duo usable in air-gapped finance/government environments
18.10	GitLab Credits usage-based billing (beta)	All tiers	$1 / credit, automated code review at $0.25 flat
18.11 (2026-04-16)	CI Expert Agent Beta, Data Analyst Agent GA, Agentic SAST GA, Custom Flow tool option overrides, Mistral AI self-hosting, Sonnet 4.6 default for Agentic Chat, subscription + per-user credit caps, MR pipeline inputs customization	Free / Premium / Ultimate	"AI writes your `.gitlab-ci.yml` and patches SAST" reaches Free and Self-Managed

The architectural takeaway is that the agents are platform-native. External copilots see only the code in the IDE; Duo agents see the code, issues, MRs, pipelines, vulnerabilities, and runner logs of the same project at the same time. That is why the CI Expert can pick the right cache pattern instead of pasting a generic template — it actually inspects the repository.

2. Duo Agent Platform Architecture — AI Gateway, Workflow Service, Knowledge Graph, Runner

The 18.11 runtime decomposes into four well-defined components. Understanding their responsibilities aligns the self-hosted deployment, observability, and pricing model in one shot.

2.1 AI Gateway — One Entry for Auth, Routing, and Metering

Every Duo call passes through the AI Gateway. It validates the GitLab user token, routes to the configured model provider, and meters GitLab Credits. 18.11 adds two enforceable guardrails. A subscription-level usage cap suspends Duo Agent Platform access for the entire subscription as soon as on-demand credits cross the configured threshold. A per-user usage cap suspends only the offender, leaving everyone else unaffected. When both caps are configured, whichever cap is hit first wins.

2.2 Workflow Service — Home of Custom Flow v1

The Duo Workflow Service consumes YAML definitions written against the Flow Registry v1 specification with the fields version, environment, components, prompts, routers, and flow. In Custom Flow definitions, environment accepts only ambient (the chat and chat-partial values are restricted), the model field inside a prompts entry is not supported (the model is determined by group/instance settings), and v1 top-level name, description, and product_group are also restricted. The 18.11 headline addition is tool option and parameter overrides directly in the flow definition, which lets you pin tool behaviour and enforce guardrails per flow regardless of LLM defaults.

2.3 Knowledge Graph — The Source of "Now"

The Knowledge Graph is a synced graph index over code, issues, MRs, pipelines, commits, and vulnerabilities. External copilots that depend on stale embeddings often answer with last week's context; Duo pulls fresh state from the Knowledge Graph on every call. That is why the CI Expert nails build/test patterns and the Agentic SAST agent can trace a vulnerable function through its callers.

2.4 GitLab Runner 18.11 — Where Remote Flows Actually Run

Remote Flows execute as pipeline jobs on a GitLab Runner; the Rails backend triggers them with start_workflow = true. 18.11 ships two ops-level improvements. First, a concrete helper image with bundled dependencies reduces external image pulls at job startup. Second, the job router feature flag is now read from the runner config file instead of an environment variable, which removes an entire class of incidents where a leaked or overridden env var broke routing silently. Bug fixes in the same release land CONCURRENT_PROJECT_ID collisions, the after_script ordering when pre_build_script fails, pipeline hangs on cache operations, and the silent fallback to job payload credentials when the DOCKER_AUTH_CONFIG credential helper binary is missing.

3. CI Expert Agent — From an Empty `.gitlab-ci.yml` to a Working Pipeline

CI Expert Agent shipped in 18.11 in Beta on every tier (Free, Premium, Ultimate) and on GitLab.com, Self-Managed, Dedicated, and Dedicated for Government. The premise is straightforward: starting from an empty repository or empty .gitlab-ci.yml, the agent inspects the codebase, asks a few guided questions about your build and test process, and emits a reviewable pipeline. It does not paste a template — it identifies language, framework, test runner, and cacheable directories from the actual code.

3.1 Invocation and Output

You invoke CI Expert from the VS Code extension or the Web UI's Agentic Chat:

# Web UI — Agentic Chat
/ci-expert
> Automate the build and test for this project, and deploy to Kubernetes on push to main.

# VS Code (Duo extension)
Cmd+Shift+P → "GitLab Duo: Generate CI Pipeline"

A typical first-cut output for a Node.js + Helm project looks like this:

# .gitlab-ci.yml — first draft from CI Expert
stages:
  - install
  - test
  - build
  - deploy

default:
  image: node:20-bookworm
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/

install:
  stage: install
  script:
    - npm ci --prefer-offline
  artifacts:
    paths:
      - node_modules/
    expire_in: 1 hour

unit-test:
  stage: test
  needs: ["install"]
  script:
    - npm run test:unit -- --reporter=junit
  artifacts:
    when: always
    reports:
      junit: junit.xml

build-image:
  stage: build
  needs: ["unit-test"]
  image: docker:25
  services:
    - docker:25-dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

deploy-prod:
  stage: deploy
  needs: ["build-image"]
  image: alpine/helm:3.16.2
  script:
    - helm upgrade --install web ./chart
        --set image.tag=$CI_COMMIT_SHA
        --namespace prod --atomic --wait
  environment:
    name: production
    url: https://app.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: manual

Each decision comes with a plain-language rationale. For example: "We use npm ci --prefer-offline because package-lock.json exists, and we set it as the cache key so dependencies only reinstall when it changes." That rationale lands in the MR description, which materially reduces reviewer friction.

3.2 Pipeline Failure Triage

CI Expert is not just for greenfield projects — it triages broken pipelines. It reads the failed job log, classifies issues like cache misses, missing tests, or image pull failures, and proposes a patch.

# Trigger directly from a failed job
gitlab-duo ci-expert debug --job-id=42 --project=acme/web

# Sample output
Diagnosis:
  - Cache miss: package-lock.json modified since last successful run.
  - Suggested: pin Node 20.18.1, add --frozen-lockfile guard.
Patch (preview):
  - install:
  -   script:
  -     - npm ci --prefer-offline
  +     - npm ci --prefer-offline --frozen-lockfile
Apply? [y/N]

4. Agentic SAST Vulnerability Resolution — Patch MRs Open the Moment a Scan Closes

Agentic SAST Vulnerability Resolution debuted as a Beta in 18.9 and is GA in 18.11 on Ultimate. The principle is "scan ends → next step happens immediately." When the SAST scan on the main branch completes, false-positive detection runs first; the surviving Critical/High findings are handed to the agent, which uses multi-shot reasoning to traverse the surrounding code and opens a fix MR plus a verification pipeline automatically.

4.1 The Workflow

The SAST scanner finishes on the main branch and writes the vulnerability list.
SAST False Positive Detection runs to drop noisy findings.
The agent analyses each remaining Critical/High vulnerability in context.
If confidence is sufficient, an MR is created and a SAST scan re-runs against the proposed patch to confirm the fix.
The vulnerability detail page exposes a one-click "Apply resolution" button.

4.2 Language Coverage and Triggers

The auto-fix is most reliable for languages directly supported by GitLab Advanced SAST: C, C++, C#, Go, Java, JavaScript, Python, Ruby, and TypeScript. Other languages fall back to the Semgrep-based analyzer with reduced fix confidence. You can also manually trigger the agent for any SAST vulnerability from its detail page.

# .gitlab-ci.yml — Agentic SAST integration (Ultimate)
include:
  - template: Jobs/SAST.gitlab-ci.yml

variables:
  # Auto-fix after SAST (default true in 18.11)
  SAST_DUO_AUTO_RESOLVE: "true"
  # Extend to medium severity if desired
  SAST_DUO_AUTO_RESOLVE_SEVERITY: "critical,high,medium"
  # Branch the auto-fix MR targets
  SAST_DUO_AUTO_RESOLVE_TARGET_REF: "$CI_DEFAULT_BRANCH"

# Extra SAST verification for auto-fix MRs
sast-verify:
  stage: test
  rules:
    - if: $CI_MERGE_REQUEST_LABELS =~ /duo:auto-fix/
  script:
    - echo "Re-running SAST on auto-generated fix MR"

4.3 Troubleshooting — "The MR was created but the diff is empty"

Symptom: the vulnerability page shows "Resolution generated" but the MR diff is empty.

Cause: the agent's confidence assessment fell below threshold. 18.11 will not force a low-quality patch; it leaves an "Insufficient context" reason and a recommended next step (e.g., add the calling functions) in the MR description. Fix: confirm the Knowledge Graph index for the codebase is current, and re-trigger from the vulnerability detail page if needed.

5. Custom Flow YAML Schema — How to Codify Your Own Workflow

For platform engineering teams, the most attractive 18.11 capability is tool option overrides combined with MCP integration. Below is a simplified version of the "security review assistant" Custom Flow that the ManoIT security team uses to pull in Confluence security guidelines as additional context.

# .gitlab/duo/flows/security_review.yml
version: "1"
environment: ambient    # custom flows allow only "ambient"
components:
  - name: collect_context
    type: AgentComponent
    prompt_id: collect_context_prompt
    inputs:
      - from: "context:project_id"
        as: "project_id"
      - from: "context:goal"
        as: "mr_iid"
    # 18.11: tool overrides pin parameters regardless of LLM defaults
    tool_overrides:
      - tool: "gitlab.read_merge_request"
        params:
          include_diff: true
          include_pipelines: true
          max_files: 50
      - tool: "mcp.confluence.search"
        params:
          space: "SECURITY"
          max_results: 10

  - name: review
    type: AgentComponent
    prompt_id: security_review_prompt
    inputs:
      - from: "components.collect_context.output"
        as: "evidence"

prompts:
  - id: collect_context_prompt
    template: |
      Read MR {{ mr_iid }} in project {{ project_id }}.
      Pull related security guidelines from Confluence SECURITY space.
      Output structured evidence.

  - id: security_review_prompt
    template: |
      Given evidence: {{ evidence }}
      Produce a security review with:
      - Risk classification (low/medium/high/critical)
      - Specific code locations of concern
      - Suggested mitigations referencing internal policies

routers:
  - from: collect_context
    to: review
flow:
  - collect_context
  - review

Two things matter here. First, tool_overrides (new in 18.11) pins tool call parameters so the LLM cannot drift from the contract — guardrails travel with the flow. Second, MCP tools (e.g., mcp.confluence.search) are first-class citizens alongside built-in GitLab tools. MCP configuration lives in a separate file gated by Code Owners approval.

// .gitlab/duo/mcp.json — Code Owners approval required
{
  "mcpServers": {
    "confluence": {
      "command": "npx",
      "args": ["-y", "@atlassian/mcp-confluence"],
      "env": {
        "CONFLUENCE_BASE_URL": "https://acme.atlassian.net/wiki",
        "CONFLUENCE_TOKEN": "$CONFLUENCE_API_TOKEN"
      }
    },
    "slack": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-slack"],
      "env": {
        "SLACK_BOT_TOKEN": "$SLACK_DUO_BOT_TOKEN"
      }
    }
  }
}

5.1 Custom Flow Trigger Mapping

A Custom Flow can react to three trigger types, and each delivers a different shape of context:goal. Knowing this prevents flows from behaving inconsistently across triggers.

Trigger	`context:goal` format	Example
Mention event	`Input: <comment_text>\nContext: {<resource_type> IID: <iid>}`	`@ai-security Can you review?` on issue #2
Assign / Assign reviewer	A single integer (the resource IID)	Assigning the service account as reviewer on MR !10 → `10`
Pipeline event	The full pipeline event webhook payload	Auto-analysis of a failed pipeline

6. GitLab Credits — Usage-Based Pricing and the 18.11 Guardrails

Duo Agent Platform's cost model standardizes on GitLab Credits: 1 credit = $1 on-demand list price, billed monthly in arrears. 18.10 introduced automated code reviews at a flat $0.25 per MR, regardless of size. 18.11 added the two enforcement guardrails operations teams had been asking for.

Control	Scope	Behaviour on hit	Notes
Subscription-level cap	Whole subscription	Auto-suspends Duo Agent Platform until the next billing period; admin email notification	Hard guardrail against unexpected overage
Per-user cap	Individual user	Suspends only that user, others unaffected	Stops one user from absorbing the subscription pool
Credits dashboard history	Subscription	Browse past months daily, compare consumption, reconcile with invoices	FinOps standard input

Both caps can run simultaneously, and whichever is hit first applies. Caps reset automatically each billing period. ManoIT's operational guidance:

Set the per-user cap at ~1.5× the average developer's expected usage (e.g., 50 credits/month) to absorb the learning ramp without runaway.
Set the subscription cap conservatively in the first quarter (e.g., 80% of forecast) and tighten or relax it after two cycles of dashboard data.
Agentic SAST runs in the background, so budget separately for it depending on scan cadence (every push vs. nightly).

7. Model Policy — Sonnet 4.6 Default and Mistral for Self-Hosting

Two lines describe 18.11's model policy. Agentic Chat's default model moved from Claude Haiku 4.5 to Claude Sonnet 4.6 hosted on Vertex AI, and Mistral AI joined the supported self-hosted LLM list. Sonnet 4.6 reasons better but uses a higher GitLab Credit multiplier than Haiku. Existing explicit choices are preserved.

7.1 Model Selection Cheatsheet

Workload	Recommended model	Reason
General Agentic Chat	Claude Sonnet 4.6 (default)	Best code reasoning and MR review quality; 18.11 default
High-volume background calls (Agentic SAST)	Claude Haiku 4.5 or self-hosted Mistral	Lower credit multiplier
Air-gapped finance/government	Mistral AI or Vertex AI Private	No external LLM hop; new in 18.11
Long-context codebase analysis	Vertex AI Gemini 2.5 Pro or Sonnet 4.6	Long context plus reasoning depth

# config/gitlab.rb — adding Mistral to self-hosted models (new in 18.11)
gitlab_rails['duo_self_hosted_models'] = [
  {
    'name' => 'mistral-large-latest',
    'provider' => 'mistral',
    'endpoint' => 'https://api.mistral.ai/v1',
    'api_key_env' => 'MISTRAL_API_KEY',
    'use_for' => ['code_completion', 'agentic_chat']
  },
  {
    'name' => 'claude-sonnet-4-6',
    'provider' => 'vertex_ai',
    'endpoint' => 'https://us-central1-aiplatform.googleapis.com/v1',
    'project_id' => 'acme-prod-ai',
    'use_for' => ['agentic_chat', 'agentic_sast']
  }
]

8. 18.11 Migration Checklist — Forced PostgreSQL 17 and MR Pipeline Inputs

Last but not least, the migration checklist the ManoIT operations team wrote up while moving to 18.11. The most consequential change is the forced PostgreSQL 17 path: 19.0's minimum supported version becomes PostgreSQL 17. Instances not running PostgreSQL Cluster will be auto-upgraded to PostgreSQL 17 during the 18.11 upgrade. PostgreSQL Cluster users (and anyone who opts out of the auto-upgrade) must move to PostgreSQL 17 manually before 19.0.

8.1 MR Pipeline Inputs Customization

18.11 lets you override spec:inputs values per merge request pipeline run, so you can rerun the same pipeline against a different environment without code changes.

# .gitlab-ci.yml — 18.11 MR pipeline inputs
spec:
  inputs:
    target_env:
      default: "staging"
      options: ["staging", "uat", "prod"]
    canary_traffic:
      default: 10
      type: number
---
deploy:
  stage: deploy
  script:
    - helm upgrade web ./chart
        --set env=$[[ inputs.target_env ]]
        --set canaryTraffic=$[[ inputs.canary_traffic ]]
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

8.2 Operational Checklist

Backup and rollback plan — full PostgreSQL backup and WAL archive verified before the auto-upgrade.
Runner upgrade — move runners to 18.11 to pick up the concrete helper image and the runner-config-driven job router. Remove any scripts that forced job router via env vars.
Agentic SAST policy — decide whether to enable the GA auto-fix and define the MR label workflow (e.g., duo:auto-fix).
Credits caps — configure both subscription and per-user caps. Compare actual consumption on the Credits dashboard for the first 7 days.
Model policy — assess Sonnet 4.6 default cost impact; route background calls (Agentic SAST etc.) to Haiku or Mistral.
Custom Flow audit — change environment from chat / chat-partial to ambient. Remove fields rejected by 18.11: model in prompts, response_schema_id, stop, top-level name, description, product_group.
MCP governance — .gitlab/duo/mcp.json requires Code Owners approval. Inject tokens via GitLab CI/CD variables or Vault.

9. Conclusion — "AI-Native DevSecOps" Is Now the GitLab Default Surface

By May 2026, GitLab 18.11 makes two things explicit. First, Duo Agent Platform is no longer an option — it is the default surface. The moment CI Expert reaches every tier, "AI writes your pipeline" stops being marketing and becomes the new-project baseline experience. Second, operations finally has hard guardrails. Subscription and per-user caps are not advisory — they cut access the instant the threshold is hit.

ManoIT's roadmap for the next quarter splits into three workstreams. First, enable Agentic SAST GA to compress patch lead time for SOC 2 / ISO reporting into single-digit days. Second, standardize a code-review and release-notes automation pipeline using Custom Flows + MCP to bind Confluence, Jira, and Slack as live context. Third, layer OpenTelemetry-derived metrics on top of the GitLab Credits FinOps dashboard so cost, quality, and lead time live on the same screen. 18.11 is the first GitLab release that makes all three of those workstreams feasible in production.

Originally published at ManoIT Tech Blog.

Google ADK 1.0 + A2A Protocol — Python/Go/Java/TypeScript GA, AgentCard, Task, and SSE Redefining the 2026 Multi-Agent Standard

daniel jeong — Sun, 03 May 2026 23:44:47 +0000

Google ADK 1.0 + A2A Protocol — Python/Go/Java/TypeScript GA, AgentCard, Task, and SSE Redefining the 2026 Multi-Agent Standard

At Google Cloud Next 2026 in April, the Agent Development Kit (ADK) graduated to 1.0 GA across four languages — Python, Go, Java, and TypeScript — simultaneously. In the same window, the Agent2Agent (A2A) protocol crossed 150+ organizations in production under the Linux Foundation, and Anthropic's MCP (donated to the Linux Foundation in December 2025) settled in as the de-facto standard for tool calls. The 2026 multi-agent stack now sits on a clean separation: MCP for tools, A2A for agents, ADK (or equivalent) for orchestration. This post documents the architectural decisions, code patterns, and operational checklist that the ManoIT AI platform team produced while integrating ADK 1.0 + A2A into our internal RAG and orchestration backend.

1. Why May 2026 Is the Inflection Point

The agent landscape in 2025 was a framework war — LangGraph for graph control, CrewAI for ergonomic multi-agent, AutoGen for research, and vendor-specific SDKs from OpenAI and Anthropic. By April–May 2026 that fragmentation collapsed onto two protocols and one cross-language SDK.

Axis	H1 2025	May 2026	Practical impact
Tool calls	Per-framework SDKs	MCP (Linux Foundation)	A tool built once works across Claude, Gemini, GPT, and self-hosted LLMs
Agent-to-agent	Vendor silos	A2A v1.0, 150+ orgs in prod	Salesforce ↔ Vertex ↔ ServiceNow speak one wire format
Code-first SDK	Fragmented	ADK 1.0 GA in Py/Go/Java/TS	One semantic, four languages, model-agnostic
Managed runtime	DIY	Vertex AI Agent Engine + Bedrock AgentCore + Azure AI Foundry	Sessions, memory, observability as a service
Vertex AI brand	Vertex AI Agent Builder	Gemini Enterprise Agent Platform	Catalog of 200+ models including Claude

The bottom line: picking a multi-agent system is no longer a framework decision but a protocol-fit decision. Whichever framework you choose has to consume MCP tools and accept A2A delegations. ADK 1.0 is the first cross-language SDK that treats both as first-class citizens.

2. ADK 1.0 GA — Feature Parity Across Four Languages

The headline of ADK 1.0 is feature parity. Where the LangChain ecosystem in 2025 was Python-first, JS-second, and other languages a distant third, ADK 1.0 explicitly aligned all four runtimes so that an agent prototyped in Python can be re-implemented in Java for production without semantic drift.

Language	Release	Signature 1.0 features	Typical adoption
Python	Continuous (bi-weekly)	Plugin system, Event Compaction, FileArtifactService, Service Registry (`services.yaml`), Vertex AI Agent Engine Sandbox code execution, Anthropic thinking blocks, OTel agentic metrics	RAG, LangGraph replacement, research
Java	1.0.0 (Apr 2026)	GoogleMapsTool, UrlContextTool, ContainerCodeExecutor, VertexAICodeExecutor, ComputerUseTool, HITL ToolConfirmation, Event Compaction, native A2A	Finance, SI, enterprise backends
Go	1.0 (Apr 2026)	YAML agent config, native OpenTelemetry, cross-language parity	High-concurrency microservices, edge
TypeScript	Aligned to 1.0	Same patterns from front-end to BFF	Web chat, BFF, Slack/Teams bots

2.1 ADK Python — Plugins and the Service Registry

The Python 1.0 line collapses into two abstractions worth understanding. First, Plugin is a cross-cutting hook registered once on the Runner and invoked globally before/after every Agent, Model, and Tool call. Second, the Service Registry lets you swap session/artifact/memory backings declaratively in services.py or services.yaml so the same agent can run on in-memory backends locally and on Vertex AI Memory Bank, Firestore sessions, and GCS artifacts in production without code changes.

# agents/customer_support/agent.py
# Note: ADK 1.0 requires Python 3.10+
from google.adk.agents import LlmAgent
from google.adk.tools import google_search
from google.adk.tools.openapi_tool import OpenAPIToolset
from google.adk.plugins import Plugin

# 1) Cross-cutting concerns — PII masking and prompt injection defense as a Plugin
class GuardrailPlugin(Plugin):
    async def before_model_callback(self, ctx, request):
        request.contents = self._mask_pii(request.contents)
        return request

    async def after_tool_callback(self, ctx, tool, result):
        ctx.logger.info(f"tool={tool.name} status={result.status}")
        return result

# 2) Agent — model-agnostic (Gemini, Claude, GPT, Llama all swap freely)
support_agent = LlmAgent(
    name="customer_support",
    model="gemini-2.5-pro",          # or "claude-opus-4-6", "gpt-5", "vllm://..."
    description="ManoIT first-line support agent",
    instruction=(
        "Reply in the user's language. "
        "Delegate billing/refund/account issues to 'billing_agent'."
    ),
    tools=[
        google_search,
        OpenAPIToolset(spec_url="https://api.manoit.co.kr/openapi.json"),
    ],
    sub_agents=["billing_agent"],     # delegated via A2A or in-process
)

# services.yaml — swap backings per environment
sessions:
  type: vertex_ai
  config:
    project: manoit-prod
    location: asia-northeast3

memory:
  type: vertex_ai_memory_bank
  config:
    corpus: projects/manoit-prod/locations/asia-northeast3/ragCorpora/12345

artifacts:
  type: gcs
  config:
    bucket: manoit-agent-artifacts

2.2 ADK Java — Human-in-the-Loop and Computer Use

ADK Java 1.0's signature feature is HITL ToolConfirmation. Risky tools (refunds, permission grants, expensive operations) pause execution before the call, wait for human approval, and resume cleanly. Add ComputerUseTool, ContainerCodeExecutor, and VertexAICodeExecutor and the agent can drive a real browser or run generated code in a sandbox. Together they make "cautious automation" — the most common enterprise requirement — a standard pattern rather than a custom build.

2.3 Event Compaction — Beating the Context Window Honestly

Every long-running LLM agent eventually hits the same wall: context explosion. ADK 1.0's Event Compaction keeps a sliding window of recent events and summarizes the rest, capping the token budget. In ManoIT's internal tests on 12-turn dialogues, compaction cut average token usage by 38% and latency by 18%. The catch: compaction can also drop "decisions you must remember." Use memory_keys to pin user IDs, contract terms, and in-flight payments outside the compactor.

3. A2A Protocol — AgentCard, SkillCard, and the Task Lifecycle

A2A is a wire format for delegating work between opaque agents. You don't need to know the other agent's model, prompt, or memory layout — only that it exposes an AgentCard, that the AgentCard advertises SkillCards, and that calls flow as JSON-RPC 2.0 over HTTP(S).

3.1 AgentCard — The Digital Business Card

{
  "name": "manoit-billing-agent",
  "description": "ManoIT billing, refund, and subscription agent",
  "version": "1.0.0",
  "url": "https://agents.manoit.co.kr/billing/a2a",
  "supported_interfaces": [
    {"protocol": "a2a/1.0", "transport": "https+jsonrpc"}
  ],
  "capabilities": {
    "streaming": true,
    "push_notifications": true,
    "long_running_tasks": true
  },
  "default_input_modes":  ["text/plain", "application/json"],
  "default_output_modes": ["text/plain", "application/json"],
  "skills": [
    {
      "id": "refund.process",
      "description": "Process a refund given a payment ID and reason",
      "tags": ["billing", "refund"],
      "input_modes":  ["application/json"],
      "output_modes": ["application/json"],
      "examples": [
        {"input": {"payment_id": "PAY-...", "reason": "double-charge"}}
      ]
    }
  ],
  "authentication": {
    "schemes": ["oauth2", "api_key"],
    "oauth2": {
      "authorization_url": "https://auth.manoit.co.kr/authorize",
      "token_url": "https://auth.manoit.co.kr/token",
      "scopes": ["billing:refund"]
    }
  }
}

The card lives at /.well-known/agent.json so any other agent can discover capabilities, auth, and transport without prior agreement. Treat it as the OpenAPI of agents — registries and crawlers can build domain-wide capability catalogs from it.

3.2 Task Lifecycle — A First-Class Stateful Unit of Work

A2A is not single-shot RPC; it is stateful Tasks as a first-class abstraction. The client sends tasks/send; the server returns a task ID; the same ID is then driven via tasks/get, tasks/cancel, tasks/sendSubscribe (SSE stream), and tasks/pushNotification/set (webhook). Tasks survive disconnects and are designed to live for seconds or hours.

sequenceDiagram
  autonumber
  participant Caller as Caller Agent (Vertex)
  participant Card as /.well-known/agent.json
  participant Billing as ManoIT Billing (A2A)
  participant Bank as PSP

  Caller->>Card: GET AgentCard
  Card-->>Caller: skills, auth, capabilities
  Caller->>Billing: POST tasks/send (skill=refund.process)
  Billing-->>Caller: task_id=tk_42, state=submitted
  Caller->>Billing: tasks/sendSubscribe (SSE)
  Billing->>Bank: refund call
  Bank-->>Billing: processing
  Billing-->>Caller: artifact: {state: working, progress: 0.4}
  Bank-->>Billing: completed
  Billing-->>Caller: artifact: {state: completed, refund_id: rf_77}

3.3 Two Transports — SSE Streaming vs. Webhook Async

Real-time chat and copilots use SSE for token-level progress. Long-running batches and analyses register a webhook so the client can disconnect and still receive results. A2A allows mixing both transports on the same task ID, which maps cleanly onto graph runtimes like LangGraph that pick a transport per node.

4. ADK × A2A × MCP — The Three-Layer Stack

A well-designed multi-agent system in 2026 separates three concerns: tools (MCP), agents (A2A), and orchestration (ADK or equivalent).

flowchart LR
  subgraph User["User / external system"]
    U[Slack / Teams / Web Chat]
  end

  subgraph Orchestrator["ADK Runner (Python/Java)"]
    Root[Root Agent<br/>customer_support]
  end

  subgraph A2A_Agents["Remote agents over A2A"]
    B[Billing Agent<br/>Java + Spring]
    L[Logistics Agent<br/>SAP Joule]
    H[HR Agent<br/>Workday]
  end

  subgraph MCP_Tools["MCP servers (tools / data)"]
    DB[(Postgres MCP)]
    KB[(Notion KB MCP)]
    GH[GitHub MCP]
  end

  U -->|chat| Root
  Root -->|A2A tasks/send| B
  Root -->|A2A tasks/send| L
  Root -->|A2A tasks/send| H
  Root -->|MCP tools/call| DB
  Root -->|MCP tools/call| KB
  Root -->|MCP tools/call| GH
  B -->|MCP| DB

Layer	Standard	Owns	Forbidden
Tool	MCP	Actions/queries against DBs, SaaS, internal APIs	Business decisions, multi-step reasoning
Agent	A2A	Domain expertise (billing, logistics, HR) + reasoning	Exposing tools directly, leaking schemas
Orchestrator	ADK Runner	Intent, planning, sessions, memory, HITL, observability	Implementing domain logic

Two anti-patterns appear when this separation breaks. Push business decisions into MCP and tools become "hidden agents" — reuse and auditability collapse. Let an A2A agent behave like a tool and you get glorified RPC, losing task lifecycle, retries, and approval flows. ADK 1.0's AgentTool, RemoteA2aAgent, and BaseTool types exist precisely to enforce this boundary at the type level.

5. Vertex AI Agent Engine — Single Process to Managed Multi-Agent

ADK is model- and runtime-agnostic, but Google's preferred operating path is Vertex AI Agent Engine. Where Cloud Run is stateless, Agent Engine absorbs sessions, memory, retries, observability, and scaling as a managed service. As of April 2026, Agent Engine Sandbox code execution ships as an ADK Python integration, letting LLM-generated code run safely in an isolated environment.

# 0) Auth
gcloud auth application-default login

# 1) Scaffold
pip install --upgrade google-adk
adk create my_agent --template=root_agent

# 2) Local eval
adk run my_agent
adk eval my_agent ./evals/golden.json

# 3) Deploy to Vertex AI Agent Engine (Express Mode onboarding — new in 1.0)
adk deploy agent_engine \
  --project=manoit-prod \
  --region=asia-northeast3 \
  --agent_path=./my_agent \
  --runtime=python3.12 \
  --requirements=requirements.txt \
  --service_account=adk-runtime@manoit-prod.iam.gserviceaccount.com

# 4) Or self-host on GKE
adk deploy gke \
  --cluster=adk-prod \
  --namespace=agents \
  --image-tag=v1.4.2

Operationally, Agent Engine wins where it matters: OTel traces flow into Cloud Trace automatically, the Memory Bank wires straight into RAG corpora, and session compaction/retries/multi-step delegation are bundled behind SDK abstractions. Multi-cloud-mandated organizations can run the same ADK code on EKS/AKS and pipe everything through OTel collectors with LiteLLM in front of Anthropic and OpenAI.

6. ManoIT Adoption Roadmap — ADK 1.0 + A2A in 90 Days

The 90-day roadmap the ManoIT AI platform team validated internally. The guiding rule: agree on protocol, permissions, and observability before writing code.

Day 1–14 — Standards alignment. Security, platform, and domain teams agree on the externally exposable tool whitelist (MCP) and the delegatable agent list (A2A). Register the AgentCard schema and a SkillCard naming convention (domain.verb.noun) as an internal standard.
Day 15–30 — Hello A2A. Spin up one ADK Python root agent and one Java A2A remote agent, complete a single end-to-end delegation. Acceptance gate: the OTel trace must be continuous from edge to remote agent.
Day 31–50 — MCP tool alignment. Wire MCP servers to internal catalogs, DBs, Slack, and Notion. Implement PII masking, request-ID propagation, and audit logging cross-cutting via ADK Plugins.
Day 51–70 — HITL and governance. Attach ToolConfirmation to refunds, permission grants, and high-cost operations; integrate the approval UI behind a Slack slash command. Note: any path that reaches an external SaaS LLM must run PII detection, output filtering, and hallucination checks together.
Day 71–90 — Operational stabilization. Memory-key conventions for Event Compaction, per-token cost alerts, Vertex AI Agent Engine autoscale validation, and a retry/recovery game day.

6.1 Operations checklist

Pin versions. Lock google-adk==1.x and a2a-sdk==1.0.x. Auto-merge bi-weekly patches only after staging regression.
Protocol compliance tests. Add the A2A SDK compliance suite plus AgentCard JSON Schema validation to CI.
Observability. OTel service.name is per agent; session.id is the ADK Runner-issued session ID.
Cost. Expose per-call input/output tokens as Prometheus counters in a Plugin. Track unit pricing across Vertex, Bedrock, and Anthropic.
Failure modes. Prefer SSE or webhook over polling tasks/get. Polling loses on cost, latency, and server load.
Rollback. Keep version-tagged Agent Engine deployments and previous ADK lock files. 24-hour rollback must be feasible.

7. Competitive Landscape

Framework	Strength	May 2026 position	Relationship to ADK
LangGraph	Graph control, production reliability	Native A2A	Interop via ADK Plugin/Tool; LangGraph wins on graph runtime
CrewAI	Role-based ergonomics, gentle ramp	Native A2A	Good for lightweight PoCs; production goes to ADK
AutoGen	Research, multi-LLM debate	Community-led	Research → migrate to ADK
OpenAI Agents SDK	OpenAI-model optimized	AGENTS.md convention	Wins inside an OpenAI single-vendor stack
Anthropic Managed Agents	Brain/Hands/Session split, MCP-first	Claude 4.6/4.7 optimized	Wins inside an Anthropic stack; multi-LLM goes to ADK
Google ADK 1.0	Py/Go/Java/TS GA, model-agnostic, A2A + MCP first-class	Reference for protocol-aligned stacks	The reference

Pragmatic guidance: if you're deeply locked into one LLM vendor, that vendor's SDK is the fastest path. But for enterprises facing multi-language, multi-cloud, and multi-LLM demands at once, ADK 1.0 minimizes the cost of standards alignment. In Java- and Go-heavy markets such as Korean SI and finance, ADK lowers the door height that LangChain/LangGraph's Python-first stance raised.

8. Conclusion — From "Picking a Framework" to "Fitting the Protocols"

The 2026 multi-agent stack is no longer about which framework is most elegant. With MCP for tools and A2A for agents locked in, the real evaluation axis is whether your SDK treats both protocols as first-class citizens and offers cross-language parity. Google ADK 1.0 GA is the best-balanced answer on that axis today, and A2A v1.0 has earned the external validation that 150+ production deployments only buy once. ManoIT's recommendation is concrete. First, default new agent builds in Q3 2026 to the three-layer split: MCP tools + A2A agents + ADK Runner. Second, stand up an internal protocol catalog — AgentCards, SkillCards, MCP tool whitelists — co-owned by security, platform, and domain teams in week one. Third, agree on OTel, HITL, and Event Compaction as operational defaults before code is written. Operating model precedes tooling, and whoever standardizes the operating model first goes the farthest.

This article was written by the ManoIT engineering team with assistance from Anthropic Claude AI. Facts and code samples are based on the official Google ADK documentation and 1.0 release notes, the A2A Protocol Specification (a2a-protocol.org), the MCP Specification (modelcontextprotocol.io), and Google Cloud Next 2026 announcements. Some performance and operational figures reflect ManoIT's internal measurements and may vary by environment. Validate in your own environment before adopting. — ManoIT (manoit.co.kr)

Originally published at ManoIT Tech Blog.

Backstage 1.49 Complete Guide — New Frontend System 1.0 RC by Default, Actions Registry, and mcp-actions-backend Make Your IDP AI-Native

daniel jeong — Sun, 03 May 2026 09:57:48 +0000

Backstage 1.49 Complete Guide — New Frontend System 1.0 RC by Default, Actions Registry, and mcp-actions-backend Make Your IDP AI-Native

As of January 2026, Backstage powers internal developer platforms (IDPs) at 3,400+ organizations and 2 million+ developers outside Spotify, commanding 89% market share among IDP frameworks. From that summit, the Spotify and CNCF community has shipped its largest architectural shift in five years. Backstage 1.49 promotes the New Frontend System (NFS) to 1.0 Release Candidate and makes it the default for new apps. It ships the Actions Registry as a first-class governance layer and adds @backstage/plugin-mcp-actions-backend, which exposes every registered action as a tool through the Model Context Protocol. In other words: Claude Code, Cursor, and ChatGPT can now query your catalog, run scaffolder tasks, and grant GitLab group permissions in one natural-language step. This guide walks through the NFS migration, Actions Registry governance, MCP server splitting, and the BUI breaking changes from a production-platform perspective.

1. Why Backstage 1.49 is the Inflection Point

The IDP debate "should we even build a portal?" effectively ended in 2026 — Backstage's 67% overall adoption and 89% framework share answered it. The remaining question is how to operate one, and 1.49 rewrites the operating model on three axes simultaneously.

Axis	≤ 1.48	1.49	Operational Impact
Frontend system	Legacy + `--next` opt-in	NFS 1.0 RC default, `--legacy` opt-out	New apps forced onto NFS, existing apps need 6–12 month migration
Action governance	Scaffolder actions + custom hooks scattered	Single Actions Registry	Unified action catalog, permissions, audit logging
AI integration	External chatbots, separate search index	`mcp-actions-backend` built in	Claude / Cursor drive the IDP directly
BUI transition	MUI-based EntityCards	BUI (Backstage UI) migration in flight	`variant` prop removed, silent layout regressions possible
Catalog query	Equality filters	Predicate filters (`$all`, `$any`, `$not`, `$exists`, `$in`, `$hasPrefix`, `$contains`)	Massive expressiveness for large catalogs

The takeaway is that 1.49 is not a feature drop but the first industry-standard release where platform engineering formally adopts AI. BackstageCon Europe 2026 (Amsterdam, March 26) made the same point: N26, Spotify, and Roadie all argued that Actions Registry plus MCP will define IDP design for the next 18 months.

2. New Frontend System 1.0 RC — `--next` is Gone, `--legacy` is Born

NFS reached 1.0 Release Candidate in 1.49 after alpha (2024) and beta (2025). The most visible change is that the CLI defaults flipped: npx @backstage/create-app now scaffolds an NFS-based app without the --next flag, and you must pass --legacy explicitly to opt back into the old system.

# 1.49 new app — defaults to NFS 1.0 RC
npx @backstage/create-app@latest --path my-portal

# Stay on the legacy MUI system
npx @backstage/create-app@latest --path my-portal --legacy

# Bump existing apps
yarn backstage-cli versions:bump --release latest

# New: Actions Registry CLI
yarn backstage-cli actions list
yarn backstage-cli actions execute scaffolder:catalog:register --input '{"url":"..."}'
yarn backstage-cli actions sources

The essence of NFS is that plugin extension points and routing are decided at app-config time, not at code-build time. Sidebar entries, pages, and catalog widgets toggle on or off purely from app-config.yaml's app.extensions block, which means you can ship the same binary to multiple environments and let configuration decide the plugin surface. In 1.49, PluginWrapperApi graduates from alpha to stable, and a new @backstage/frontend-dev-utils package gives you createDevApp() for spinning up plugin-only dev environments in a single line.

2.1 BUI (Backstage UI) — The Quiet Retirement of MUI Cards

In parallel with NFS 1.0 RC, Spotify is migrating the design system to BUI, a React Aria–based component layer. The 1.49 major bump of @backstage/plugin-catalog rebuilt EntityAboutCard, EntityLinksCard, EntityLabelsCard, GroupProfileCard, and UserProfileCard on BUI, and removed the variant prop in the process. ⚠️ Caution: if your code uses variant="gridItem" it will still compile but quietly break the EntityPage layout. Visual regression on EntityPage must be on the PR review checklist.

3. Actions Registry — Beyond Scaffolder, Toward IDP-Wide Automation Governance

Actions Registry arrived as beta in 1.48 and solidifies in 1.49 as a first-class platform-engineering primitive. The pre-1.48 model only allowed action invocation inside Scaffolder templates, which meant permissions, auditing, and reusability were all tied to template scope. Actions Registry lifts those concerns into a standard action type that plugins register, the permission system governs, and CLI / UI / MCP can all invoke identically.

// plugins/my-platform-backend/src/actions/registerService.ts
import { createBackendModule } from '@backstage/backend-plugin-api';
import { actionsRegistryServiceRef } from '@backstage/backend-plugin-api/alpha';
import { z } from 'zod';

export const registerServiceModule = createBackendModule({
  pluginId: 'platform',
  moduleId: 'register-service',
  register(reg) {
    reg.registerInit({
      deps: { registry: actionsRegistryServiceRef },
      async init({ registry }) {
        registry.register({
          name: 'platform:register-service',
          title: 'Register a new microservice',
          description: 'Creates the catalog entry, GitHub repo, and ArgoCD app in one shot',
          schema: {
            input: z => z.object({
              name: z.string().min(3),
              owner: z.string(),
              tier: z.enum(['tier-1', 'tier-2', 'tier-3']),
            }),
            output: z => z.object({ catalogRef: z.string(), repoUrl: z.string() }),
          },
          // New: who is allowed to invoke this action
          visibilityPermission: { resourceType: 'platform-action', allow: 'platform-admins' },
          async action({ input, credentials, logger }) {
            logger.info(`Registering service: ${input.name}`);
            // ... create GitHub repo, register in catalog, render Argo app
            return { catalogRef: `component:default/${input.name}`, repoUrl: '...' };
          },
        });
      },
    });
  },
});

That single block delivers a major leverage point: a Scaffolder template can call it as action: platform:register-service, a platform engineer can invoke it standalone via yarn backstage-cli actions execute, and mcp-actions-backend exposes it verbatim as an MCP tool. Write the action once, and humans, the CLI, and AI agents all consume it with identical semantics.

3.1 New Built-in Actions in 1.49

backstage:who-am-i — returns the caller's identity and permission context. Critical for AI agents to introspect their own permissions.
catalog:query-entities — predicate-filter catalog queries ($all, $any, $not, $exists, $in, $hasPrefix, $contains).
scaffolder:list-actions — Scaffolder action self-introduction.
scaffolder:get-task-logs — stream logs of running or completed Scaffolder tasks.
scaffolder:list-tasks — list tasks visible under the caller's permissions.

These five compose the meta-toolset an AI agent uses to bootstrap itself the first time it meets an IDP. At session start, calling who-am-i → list-actions → query-entities lets Claude Code learn what the IDP can do and what permissions it has — without human intervention.

4. mcp-actions-backend — The Standard Way to Expose an IDP Over MCP

@backstage/plugin-mcp-actions-backend is the most attention-grabbing addition in 1.49. Its job summarizes in one line: "Expose every action registered in the Actions Registry over the MCP protocol automatically." No additional translation code, no schema mapping. The schema.input/output you already defined in Zod is serialized straight to the MCP tool descriptor.

# app-config.yaml
mcpActions:
  # 1.49 lets you split actions across multiple servers
  servers:
    - name: catalog-readonly
      description: Read-only catalog — safe for external LLMs
      include:
        - catalog:query-entities
        - backstage:who-am-i
      authentication:
        type: oauth2
        clientRegistration: dynamic   # DCR enabled
    - name: platform-admin
      description: Platform admins only — internal Claude Code sessions
      include:
        - platform:*
        - scaffolder:*
      exclude:
        - scaffolder:delete-task   # block dangerous actions
      authentication:
        type: bearer
        longLivedTokens: true

The real strength of 1.49's mcp-actions is server-by-purpose splitting. From a single Backstage instance you can declaratively expose a read-only catalog endpoint to an external SaaS LLM and a full Scaffolder endpoint to authenticated internal Claude Code clients. Combined, include/exclude, dotted-wildcard names (plugin.action), and visibilityPermission give Backstage a small but real API gateway role for AI traffic.

4.1 Hooking Up Claude Code via DCR (Dynamic Client Registration)

Enable OAuth2 DCR in the Backstage auth backend (experimentalDynamicClientRegistration: true).
In Claude Code, run /mcp add backstage https://portal.example.com/api/mcp/catalog-readonly.
Browser callback completes OAuth2 → long-lived token issued (longLivedTokens: true).
Restart Claude Code (the tool catalog only loads at session start).
Natural-language queries like "Group every tier-1 service by owner" translate into catalog:query-entities calls.

ManoIT's platform team wired this into a Tier-1 on-call triage bot. PagerDuty alert → Claude → catalog:query-entities to identify affected services → scaffolder:get-task-logs to inspect recent deployments → first-pass diagnosis report. Mean response time dropped from 11 minutes to 2 minutes 40 seconds.

5. Catalog and Scaffolder Changes — Migration Checklist

Beyond NFS and the Actions Registry, 1.49 ships substantive updates to catalog and scaffolder.

Area	Change	Migration Action
Catalog	Predicate filters (`$all`, `$any`, `$not`, `$exists`, `$in`, `$hasPrefix`, `$contains`)	Existing simple filters keep working; adopt incrementally
Catalog Processor	`AnnotateScmSlugEntityProcessor` and `CodeOwnersProcessor` moved to community plugin	Add `@backstage-community/plugin-catalog-backend-module-scm`
Scaffolder	`secrets.schema` validation introduced, new `gitlab:group:access` action	Add a schema to templates that consume secrets
Scaffolder API	`retry`, `listTasks`, `listTemplatingExtensions`, `dryRun`, `autocomplete` are now required methods	Implement these on any custom `ScaffolderApi` class
Bitbucket	`integrations.bitbucket` removed, `BitbucketUrlReader` removed	Migrate to `bitbucketCloud` / `bitbucketServer`
Slack notifications	Channel DMs deprecated — DMs now only target user entity recipients	Route channel messages through Webhook channels
BUI Provider	`BUIProvider` routing context now mandatory	Wrap custom-page entry points with the provider

5.1 ManoIT Internal IDP Migration Checklist

① Node 20.19 / 22.12+ — verify Yarn Berry + corepack environment.
② Bulk dependency bump — run yarn backstage-cli versions:bump --release latest and regenerate the lockfile.
③ Canary NFS opt-in — seed a new app (no --legacy) on staging; visually regress EntityPage / TechDocs / Scaffolder.
④ EntityCard variant audit — grep for EntityAboutCard variant="gridItem" and migrate to BUI grid props.
⑤ Actions Registry inventory — yarn backstage-cli actions list to discover unpermissioned actions.
⑥ MCP server permission model — agree the externally-exposable action whitelist with security before configuring mcpActions.servers.
⑦ Bitbucket / Slack split — replace single bitbucket key with bitbucketCloud / bitbucketServer; route Slack channels through Webhooks.
⑧ Catalog Processor reinforcement — explicitly install the community SCM Slug / CODEOWNERS processors that left core.
⑨ Rollback plan — keep previous-version lockfiles on a dedicated branch with a 24-hour rollback window.

6. Competitive Landscape — How Does Backstage Hold 89% Against Port, Cortex, and Roadie?

The 2026 IDP market has SaaS-first players (Port, Cortex), managed-Backstage providers (Spotify Portal, Roadie), and the self-hosted Backstage core community all pulling in different directions.

Product	Model	Strength	Relationship to 1.49	Best Fit
Backstage (self-host)	OSS framework	Fastest path to NFS / Actions Registry / MCP	The core itself	Platform teams that can self-operate
Spotify Portal	Managed Backstage	Spotify in-house plugins + DX Insights	NFS / MCP GA delivered in lockstep	Buyers who trust Spotify's lineage
Roadie	Managed Backstage	Automated upgrades, plugin curation	1.49 compatibility shipped automatically	Teams without dedicated platform headcount
Port	Independent SaaS	Low-code data model and blueprints	Not compatible — separate stack	Orgs that include non-developer departments
Cortex	Independent SaaS	Service catalog + scorecards	Not compatible	SRE-led, governance-heavy organizations

The headline is that Actions Registry + MCP put Backstage somewhere Port and Cortex cannot easily reach. Independent SaaS players have to define proprietary protocols and SDKs to expose their data models to AI; Backstage already publishes every action over the standard MCP. On the new evaluation axis of "AI-native IDP compatibility," 1.49 is effectively running unopposed.

7. Conclusion — H2 2026 Forces a Rewrite of Every IDP Operating Model

Backstage 1.49 is not a minor release; it is a turning point in IDP operating models. NFS 1.0 RC becomes the foundation for every new portal. Actions Registry collapses scattered automation into a single governance layer. mcp-actions-backend lifts the IDP into infrastructure that AI agents call routinely. ManoIT's recommendation is twofold. First, align every internal Backstage instance to 1.49 by Q3 2026 and break the NFS migration into PR-sized units to preserve a safety net. Second, agree the MCP-exposed action whitelist with security in advance. Catalog queries surfaced to external LLMs become objects of permissions, audit, and logging — policy precedes code. AI-native IDPs are not a tool change, they are an operating-model change, and Backstage 1.49 is the first release to industrialize that change.

This article was authored by ManoIT's engineering team with assistance from Anthropic Claude AI. Facts and code are based on Backstage's official documentation, the 1.49 release notes, and BackstageCon Europe 2026 sessions; some performance / operational figures and ManoIT internal measurements vary by environment. Validate in your own environment before adoption. — ManoIT (manoit.co.kr)

Originally published at ManoIT Tech Blog.

Vite 8 + Rolldown Complete Guide — Ending the esbuild·Rollup Dual-Bundler Era with a Single Rust Bundler and the Oxc Toolchain

daniel jeong — Sat, 02 May 2026 00:17:39 +0000

Vite 8 + Rolldown Complete Guide — Ending the esbuild·Rollup Dual-Bundler Era with a Single Rust Bundler and the Oxc Toolchain

On March 12, 2026, Evan You's VoidZero released Vite 8.0 as GA. After seven years, Vite's dual-bundler architecture — esbuild for dev + Rollup for production — finally retires, replaced by Rolldown, a Rust-native bundler. This is not a cosmetic dependency swap. The parser (Oxc Parser), transformer (Oxc Transformer), bundler (Rolldown), CSS minifier (Lightning CSS), and linter (Oxlint) all collapse into a single end-to-end Rust toolchain maintained by the same team. Production builds are 10–30× faster than Rollup, React Refresh transforms run 40× faster than Babel, and dev/prod output is now bit-for-bit identical. This guide breaks down Vite 8's architecture, the Environment API + ModuleRunner SSR migration, plugin-react v6 with Babel removed, breaking config changes such as build.rolldownOptions, and a production-grade migration checklist drawn from real ManoIT projects.

1. Why Vite 8 Is a Game Changer — The End of the 7-Year Dual-Bundler Era

When Vite 1.0 shipped in 2021, Evan You made two pragmatic bets: esbuild (Go) for dev speed, and Rollup (JS) for production output quality. The split worked, but it created a "dual-bundler debug hell": code-splitting, tree-shaking, and ESM/CJS interop subtly diverged between dev and prod, producing bugs that only reproduced after deployment. Vite 8 unifies both ends behind Rolldown, a Rust-native bundler that is 100% compatible with the Rollup plugin API while running 10–30× faster than Rollup and roughly 1.5–2× faster than esbuild through native multithreading.

The deeper change is the integrated toolchain: Vite (build tool) → Rolldown (bundler) → Oxc (parser, transformer, minifier, linter), all maintained inside VoidZero. Dev and prod now share the exact same parser, resolver, and transformer, eliminating an entire class of "works in dev, breaks in prod" issues.

Aspect	Before Vite 7 (2024–2026 Q1)	Vite 8 (2026-03 GA)	Operational Impact
Dev bundler	esbuild (Go)	Rolldown (Rust)	dev/prod parity
Prod bundler	Rollup (JS)	Rolldown (Rust)	10–30× faster builds
JS/TS parser	esbuild + acorn	Oxc (3× faster than SWC)	Consistent syntax handling
React Refresh	Babel	Oxc (40× faster than Babel)	80% lower HMR latency
CSS minifier	esbuild (default) / Lightning CSS (opt-in)	Lightning CSS (default)	Better modern CSS accuracy
SSR module loader	`ssrLoadModule`	`ModuleRunner` + Environment API	Multi-runtime / edge-ready
Install footprint	~50 MB	~18 MB	Faster CI cache + cold start

Vite 7 (June 2025) shipped Rolldown as the opt-in rolldown-vite package and gave SvelteKit, React Router v7, Storybook, Astro, and Nuxt six months of beta exposure to surface compatibility regressions. The result is that Vite 8 is widely viewed as a "safe major version jump" rather than a risky migration.

2. Rolldown Architecture — Rollup API + Rust Multithreading + Oxc Backend

Rolldown's core design principle is "keep the Rollup API, replace the engine with Rust." Existing Vite/Rollup plugins keep working in over 99% of cases — resolveId, load, transform, and renderChunk hooks behave identically. The internals decompose into five layers:

Layer	Component	Role	Replaces
1	Oxc Parser	JS/TS/JSX → AST (native Rust)	acorn, esbuild parser
2	Oxc Resolver	Module resolution, tsconfig paths, aliases	enhanced-resolve
3	Oxc Transformer	JSX, TS, React Refresh, decorators	Babel, SWC
4	Rolldown Bundler	Dependency graph, code splitting, tree shaking	Rollup, esbuild
5	Oxc Minifier + Lightning CSS	JS/CSS compression	Terser, esbuild minifier

The most visible win of this unification is the disappearance of dependency pre-bundling. Vite 1–7 spent the first dev startup converting node_modules from CJS to ESM with esbuild; Vite 8 lets Rolldown handle ESM graphs directly, so the warm-up step nearly vanishes. On a project with 800+ dependencies, dev server first-start drops from roughly 20 seconds to under 2 seconds.

A second non-obvious change is code-splitting precision. Rollup could only split along ESM module boundaries, generating a chunk per dynamic import() site. Rolldown adds output.codeSplitting, which gives webpack-style granular chunk control: vendor chunks, route-level chunks with prefetch hints, and shared-component chunks can all be declared up front. Mature SPAs that previously needed manual chunking hacks can now express the same policy declaratively.

2.1 Performance Benchmarks — Official + Community Numbers

These are measured build times on a fixed project (React 19, 320 dependencies, 120k LOC). Real-world numbers vary ±30% depending on graph shape and hardware.

Scenario	Vite 7 + Rollup	Vite 8 + Rolldown	Speedup
Cold prod build	72.4s	3.1s	23×
Warm prod build (cache)	18.6s	1.4s	13×
Dev server first start	21.3s	1.9s	11×
HMR (single React component)	320 ms	62 ms	5×
`npm install` (slimmer deps)	48s	22s	2×

The 5× HMR win is the direct result of @vitejs/plugin-react v6 routing React Refresh through Oxc rather than Babel, eliminating per-file Babel parsing overhead.

3. Migration — Packages, Config, and Breaking Changes

Migrating to Vite 8 is more than a dependency bump. Renamed options, removed dependencies, and a new Environment API need to be handled together. The following four steps cover the safe path.

3.1 Step 1 — Update `package.json`

{
  "devDependencies": {
    "vite": "^8.0.3",
    "@vitejs/plugin-react": "^6.0.0",
    "@rolldown/plugin-babel": "^1.0.0"
  },
  "engines": {
    "node": ">=20.19.0 || >=22.12.0"
  }
}

Vite 8 drops Node 18 support and requires Node 20.19+ or 22.12+. Update CI Docker base images to match.

3.2 Step 2 — Rename `vite.config.ts` Options

// vite.config.ts — Vite 8-compatible config
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'

export default defineConfig({
  plugins: [react()],
  build: {
    // ❌ build.rollupOptions  →  ✅ build.rolldownOptions
    rolldownOptions: {
      output: {
        manualChunks: {
          'react-vendor': ['react', 'react-dom'],
          'apollo': ['@apollo/client', 'graphql']
        }
      }
    },
    // Lightning CSS is now the default minifier (opt back into esbuild explicitly if needed)
    cssMinify: true,
    target: 'es2022'
  },
  worker: {
    // ❌ worker.rollupOptions  →  ✅ worker.rolldownOptions
    rolldownOptions: {}
  }
})

⚠️ import.meta.hot.accept(new URL('./module.js', import.meta.url)) no longer works. You must pass a module ID string: import.meta.hot.accept('./module.js'). This pattern shows up frequently in react-router and vue-router route splitting — git grep "hot.accept(new URL" to catch them all in one pass.

3.3 Step 3 — Remove plugin-react v6 + Babel Dependencies

# Remove Babel-related packages
pnpm remove @babel/core @babel/preset-env @babel/preset-react \
  babel-plugin-styled-components babel-plugin-react-compiler

# Only projects using React Compiler need to reinstall
pnpm add -D @rolldown/plugin-babel babel-plugin-react-compiler

For projects that use React Compiler, opt in explicitly through plugin-react v6's reactCompilerPreset helper:

import react, { reactCompilerPreset } from '@vitejs/plugin-react'

export default defineConfig({
  plugins: [
    react({
      babel: reactCompilerPreset({ target: '19' })
    })
  ]
})

3.4 Step 4 — Migrate Custom SSR to Environment API + ModuleRunner

Users of Next.js, Nuxt, SvelteKit, and Astro do not need to touch this — the framework handles it. But projects that built custom SSR servers (e.g. Express + Vite middleware) must migrate away from server.ssrLoadModule():

// BEFORE — Vite 7
import { createServer } from 'vite'
const vite = await createServer({ server: { middlewareMode: true } })
const mod = await vite.ssrLoadModule('/src/entry-server.tsx')

// AFTER — Vite 8 (Environment API + ModuleRunner)
import { createServer, createServerModuleRunner } from 'vite'
const vite = await createServer({ server: { middlewareMode: true } })
const runner = createServerModuleRunner(vite.environments.ssr)
const mod = await runner.import('/src/entry-server.tsx')

This is more than a rename. ModuleRunner is designed to execute modules in workers, separate processes, or even non-Node edge runtimes — Cloudflare Workers, Deno Deploy — through the same API.

4. ManoIT Case Study — Migrating Four Production Projects

In late April 2026 the ManoIT engineering team migrated four production projects (LMS Admin Web, Instructor Portal, Admin Dashboard, Mobile PWA) from Vite 7 to Vite 8 in a single sprint. Results:

Project	Dependencies	LOC	Vite 7 prod build	Vite 8 prod build	CI speedup
LMS Admin Web	412	180k	108s	5.2s	21×
Instructor Portal	287	92k	54s	2.8s	19×
Admin Dashboard	198	63k	38s	2.1s	18×
Mobile PWA	156	41k	27s	1.6s	17×

Across the full GitHub Actions pipeline, the average run time dropped from 9m 12s to 3m 47s, monthly Actions minutes fell ~58%, and as a side effect, daily merge throughput per developer rose by an average of 1.4 PRs.

4.1 Three Migration Pitfalls We Hit

The path was mostly smooth, but three real-world incompatibilities surfaced:

vite-plugin-svgr regression — pre-1.0 versions had a transform hook that conflicted with Rolldown's load hook ordering. Upgrade to v1.0.0 or later.
Apollo Client codegen watch mode — its chokidar watcher collided with Rolldown's worker thread handles. Switch to --noWatch and run a separate nodemon.
Storybook 8.x static builds — Storybook 9+ is required. 8.x is incompatible with Rolldown's plugin hook ordering. We bumped to Storybook 9.4 in the same migration.

4.2 Production Migration Checklist

The checklist below is extracted from the ManoIT internal migration playbook. Each item maps to a separate PR so rollback cost stays bounded.

① Node runtime — bump Dockerfile / CI base images to node:20.19-alpine or newer.
② Pre-scan deps — run npx vite-deprecation-check to surface deprecated vite.config options.
③ Regenerate lockfile — Babel disappears from the dep graph; recreate pnpm-lock.yaml for a clean tree.
④ Canary deploy — ship to staging first, monitor 24h before promoting to prod.
⑤ Bundle-size diff — run vite-bundle-visualizer before/after; investigate any chunk that grows >5%.
⑥ Source map verification — confirm Sentry / Datadog still resolves stack traces post-deploy.
⑦ Rollback plan — keep a Vite 7 lockfile branch ready for one-command rollback within 24h.

5. Competitive Landscape — Turbopack, Bun, and Rspack

Several Rust-based bundlers compete for the post-esbuild/post-Rollup slot in 2026. They aim at the same target, but ecosystem reach and operational maturity differ sharply.

Bundler	Language	Rollup compatible	Primary deployment	Strengths	Constraints
Rolldown (Vite 8)	Rust	99% plugin parity	Vite, Nuxt, SvelteKit, Astro	Unified toolchain + ecosystem breadth	Some Rollup-only plugins regress
Turbopack	Rust	None (own API)	Next.js only	Deep Next.js integration	Vercel-locked, closed ecosystem
Bun bundler	Zig	None	Bun runtime + fullstack	Runtime + bundler in one	Limited Node API parity
Rspack	Rust	webpack compatible	Webpack monorepo migrations	Friendly to large legacy webpack codebases	Separate from Vite ecosystem
esbuild	Go	Limited	Lightweight library bundling	Stable, simple	Code-splitting / HMR limits

The decisive differentiator is "can existing Vite/Rollup plugins keep working?" Turbopack is locked to Next.js, so Vue/Svelte/Astro users cannot adopt it. Rspack targets webpack compatibility and is therefore expensive for current Vite users. Rolldown lands in the gap between them and takes the most conservative-yet-powerful path: Rollup API + Rust speed.

6. The Next Six Months — Vite+ and What to Expect

Alongside Vite 8, VoidZero introduced Vite+, a commercial add-on. Vite itself remains permanently free and open source; Vite+ targets enterprise teams with distributed build cache, professional Vite DevTools features, and prioritized support. With Bun, Deno, and Turbopack each rolling their own bundlers, Vite's pitch is clear: standardize the engine, differentiate through DevTools and infrastructure.

The pragmatic recommendation: start every new project on Vite 8, and put existing Vite 6/7 projects on a six-month migration plan. Compatibility regressions were burned out during the rolldown-vite opt-in window, the risk surface is small, and the ROI — build speed, CI cost, HMR latency — is immediate. ManoIT's internal standard is updated as of May 2026: "Vite 8 is the default bundler for all new frontend projects."

This article was written by the ManoIT engineering team with the assistance of Anthropic Claude AI. Facts and code samples were checked against official documentation and release notes; some benchmark numbers and ManoIT internal measurements may vary by environment. Validate in your own setup before applying changes in production. — ManoIT (manoit.co.kr)

Originally published at ManoIT Tech Blog.

AWS Lambda S3 Files — The Complete Guide: EFS-Backed S3 Mounts Are Rewriting Serverless Architecture and the Agentic AI Workload Standard

daniel jeong — Wed, 29 Apr 2026 00:19:08 +0000

On April 21, 2026, AWS announced general availability of Lambda S3 Files — the ability to mount an Amazon S3 bucket as a standard POSIX file system inside a Lambda function. This is not a convenience feature. The change demolishes both of the foundational Lambda constraints of the past decade in a single move: the 512MB–10GB /tmp ephemeral storage ceiling, and the mandatory statelessness imposed by S3's GET/PUT object semantics. A week later, in the April 28 "What's Next with AWS" keynote, AWS bundled S3 Files with Lambda Durable Functions, Bedrock Managed Agents, and Amazon Quick — positioning S3 Files as "the default data plane for the agentic AI era." This article unpacks S3 Files from an enterprise production angle: internals, IAM trust model, CloudFormation/SAM templates, Durable Functions composition patterns, cost/latency trade-offs, and a migration checklist for legacy file-based workloads.

1. Why S3 Files Is a Game Changer — Two Lambda Constraints, Dissolved

For 10+ years Lambda has been bounded by two structural limits. First, per-invocation ephemeral /tmp grew to 10 GB in 2022 but remained an isolated, volatile sandbox. Second, S3 is object storage — partial reads, partial writes, directory traversal are awkward, and every interaction pays a serialization cost through boto3 or the AWS SDK. The consequence: file-bound workloads like ML weight loading, BAM/VCF genomics analysis, video transcoding intermediates, and AI agent memory ended up on EC2, ECS, Step Functions — or required a complex VPC + EFS direct mount that few teams enjoyed.

S3 Files removes that detour. Internally it is a managed gateway that borrows the Amazon EFS distributed file system engine and overlays a POSIX layer on an S3 bucket. Lambda receives /mnt/<path> as an automounted directory; mutations sync as S3 objects in the background; every Lambda mounted on the same file system sees the same directory tree in real time. Serverless no longer has to be stateless.

Constraint	Before S3 Files (2024–2026 Q1)	After S3 Files (GA 2026-04)	Real-world impact
Storage ceiling	`/tmp` ≤ 10GB, volatile	Entire S3 bucket as POSIX (effectively unlimited)	Direct handling of large model weights, BAM files
Multi-function data sharing	S3 GET/PUT or external queues	Concurrent mount on the same FS	Parallel agent workspaces
VPC requirements	EFS mount needs VPC, +1s cold start	VPC still needed but mount target auto-created	90% less setup work
File semantics	S3 objects (inefficient partial writes/rename)	POSIX read/write/seek/rename	Drop-in for legacy code
Cost model	Per-request GET/PUT + transfer	S3 + EFS Throughput mode	Workload-specific analysis required

The core promise: the simplicity of a file system with the durability and cost profile of S3. That doesn't make every Lambda a migration target — section 5 revisits the cost/latency math.

2. Architecture — A 5-Layer Chain: Bucket → File System → Mount Target → Access Point → Lambda

S3 Files inherits the same abstraction stack that EFS uses. Every layer controls permissions, isolation, and VPC routing independently.

Layer	Resource	Role	Key controls
1	S3 Bucket	Object durability (11×9)	Versioning, encryption, lifecycle
2	S3 File System	POSIX gateway over the bucket	S3↔POSIX mapping, metadata cache
3	Mount Target	ENI endpoint inside a VPC subnet/AZ	Security groups, subnet routing
4	Access Point	POSIX UID/GID, root path, permissions	UID/GID, root path, 0755
5	Lambda Function	VPC attachment + local mount path	`/mnt/<name>`

When you add S3 Files to a Lambda for the first time and no Access Point exists, one is created automatically (UID/GID 1000:1000, root /lambda, perms 755). If Access Points already exist, you must explicitly select one — a deliberate guardrail against accidental cross-tenant data exposure in shared workspaces.

2.1 What changes inside the function code

The local mount path must start with /mnt/. From the function's point of view it's just a directory. Compare:

# BEFORE — download object via S3 SDK
import boto3, tempfile, os
s3 = boto3.client("s3")

def lambda_handler(event, context):
    bucket = event["bucket"]
    key = event["key"]
    # WARNING: /tmp is per-invocation, capped at 10GB
    with tempfile.NamedTemporaryFile(delete=False) as f:
        s3.download_fileobj(bucket, key, f)
        local_path = f.name
    process(local_path)
    s3.upload_file(local_path + ".out", bucket, key + ".out")
    os.unlink(local_path)

# AFTER — mounted via S3 Files, plain POSIX I/O
def lambda_handler(event, context):
    base = os.environ["S3FILES_MOUNT"]   # e.g. /mnt/workspace
    src = os.path.join(base, event["key"])
    dst = src + ".out"
    process(src, dst)                    # download/upload code is gone
    return {"output": dst}

Line counts drop ~50%, and so do download time (especially during cold starts), peak memory, and the entire failure-handling surface (partial download retry, multipart upload backoff). More importantly: the next invocation can immediately reuse the same file.

3. Production Setup — IAM, CloudFormation, and SAM

To attach S3 Files to a Lambda you need four things in alignment: ① the file system + mount target + access point exist in a single VPC, ② the Lambda is wired to the same VPC and an AZ-aligned subnet, ③ the execution role carries s3files:ClientMount (read) and/or s3files:ClientWrite (write), and ④ the security group allows NFS (TCP 2049).

3.1 IAM execution role — least privilege

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowS3FilesMount",
      "Effect": "Allow",
      "Action": [
        "s3files:ClientMount",
        "s3files:ClientWrite",
        "s3files:DescribeMountTargets"
      ],
      "Resource": "arn:aws:s3files:ap-northeast-2:123456789012:file-system/fs-0abcd1234efgh5678"
    },
    {
      "Sid": "AllowVPCNetworking",
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "*"
    }
  ]
}

Warning: never grant s3files:ClientWrite to a read-only job. POSIX semantics mean a single misbehaving rm -rf from one Lambda can wipe a shared workspace. In multi-agent setups, also isolate root paths per agent (/agents/{agent-id}/) at the Access Point layer.

3.2 SAM template — full Lambda + S3 Files stack

# template.yml — AWS SAM
# Lambda + S3 Files integrated stack
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  BucketName:
    Type: String
  VpcId:
    Type: AWS::EC2::VPC::Id
  SubnetIds:
    Type: List<AWS::EC2::Subnet::Id>

Resources:
  AgentBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Ref BucketName
      VersioningConfiguration: { Status: Enabled }
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault: { SSEAlgorithm: AES256 }

  AgentFileSystem:
    Type: AWS::S3Files::FileSystem    # New resource type, GA 2026-04
    Properties:
      BucketArn: !GetAtt AgentBucket.Arn
      ThroughputMode: ELASTIC          # Auto-scaling
      PerformanceMode: GENERAL_PURPOSE

  AgentMountTarget:
    Type: AWS::S3Files::MountTarget
    Properties:
      FileSystemId: !Ref AgentFileSystem
      SubnetId: !Select [0, !Ref SubnetIds]
      SecurityGroups: [!Ref AgentSG]

  AgentAccessPoint:
    Type: AWS::S3Files::AccessPoint
    Properties:
      FileSystemId: !Ref AgentFileSystem
      PosixUser: { Uid: 1000, Gid: 1000 }
      RootDirectory:
        Path: /lambda
        CreationInfo: { OwnerUid: 1000, OwnerGid: 1000, Permissions: '0755' }

  AgentSG:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId: !Ref VpcId
      GroupDescription: NFS for S3 Files
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 2049
          ToPort: 2049
          CidrIp: 10.0.0.0/8

  AgentFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: python3.13
      Handler: app.handler
      Architectures: [arm64]
      MemorySize: 2048
      Timeout: 300
      VpcConfig:
        SubnetIds: !Ref SubnetIds
        SecurityGroupIds: [!Ref AgentSG]
      FileSystemConfigs:
        - Arn: !GetAtt AgentAccessPoint.Arn
          LocalMountPath: /mnt/workspace
      Environment:
        Variables:
          S3FILES_MOUNT: /mnt/workspace
      Policies:
        - Statement:
            - Effect: Allow
              Action:
                - s3files:ClientMount
                - s3files:ClientWrite
              Resource: !GetAtt AgentFileSystem.Arn

This is the minimum viable stack we run for the AI code review agent in our internal LMS. The single most important choice: ThroughputMode: ELASTIC, which absorbs the IOPS spikes that hit the moment many Lambdas mount simultaneously. For steady-state workloads, BURSTING is cheaper.

4. Composition with Durable Functions — The New Multi-Agent Standard

S3 Files alone is powerful, but the real leverage shows up when paired with Lambda Durable Functions (GA 2025-12). Durable Functions express multi-step workflows up to one year long inside Lambda code itself, with automatic checkpointing. S3 Files becomes the data plane of those workflows.

A canonical example: an orchestrator clones a Git repo into a shared workspace, then security/style/coverage agents read the same directory in parallel and write JSON results back to the same place.

# orchestrator.py — Lambda Durable Function
from aws_lambda_powertools.utilities.durable import durable, step, parallel
import os, subprocess, json
WS = os.environ["S3FILES_MOUNT"]

@durable
def review_workflow(ctx, repo_url: str, commit_sha: str):
    workdir = os.path.join(WS, "runs", ctx.execution_id)
    # 1. Clone — checkpoint saved automatically
    yield step(clone_repo, repo_url, commit_sha, workdir)
    # 2. 3 agents in parallel — all share the same workdir
    results = yield parallel([
        step(security_agent,   workdir),
        step(style_agent,      workdir),
        step(coverage_agent,   workdir),
    ])
    # 3. Merge results
    final = yield step(merge_results, workdir, results)
    return final

@step
def clone_repo(repo_url, sha, workdir):
    os.makedirs(workdir, exist_ok=True)
    subprocess.run(["git", "clone", repo_url, workdir], check=True)
    subprocess.run(["git", "-C", workdir, "checkout", sha], check=True)

@step
def security_agent(workdir):
    # Separate Lambda — same workdir mounted
    out = os.path.join(workdir, "security.json")
    # ... LLM call ...
    json.dump({"findings": [...]}, open(out, "w"))
    return out

Three properties make this pattern hard to beat: ① data exchange between agents bypasses the Step Functions 8KB payload cap and the 256KB SQS message ceiling, ② if any Lambda dies mid-step the workspace survives so retry is naturally idempotent, and ③ external runtimes like Bedrock Managed Agents can mount the same directory to participate in the workflow.

5. Cost & Performance Trade-offs — Not Every Lambda Should Migrate

S3 Files is powerful, but applying it indiscriminately balloons cost. Three variables matter most: EFS Throughput + Storage charges are added on top, VPC Lambda cold-start penalty (~600ms–1.2s) still applies, and POSIX metadata sync overhead adds roughly 5–15ms per object.

Workload pattern	S3 Files fit	Why	Alternative
Large-model inference (50GB+ weights)	★★★★★	Bypasses `/tmp`, no per-cold-start download	—
Multi-agent code review	★★★★★	Shared workspace is the whole point	—
Per-event short transformations	★★☆☆☆	VPC cold start dominates	S3 Object Lambda
High-frequency metadata lookups	★★☆☆☆	POSIX `stat` overhead accumulates	DynamoDB
Static asset serving	★☆☆☆☆	S3 + CloudFront wins	CloudFront
Legacy file-based ETL	★★★★☆	Drop-in code migration	—

Rough cost model: 10M invocations/month, 50MB avg I/O, 100GB resident → typically +18–25% cost vs. plain S3, but if download/upload time savings cut Lambda execution by ~30% on average, the total bill drops 5–12% net. ROI is highest for I/O-heavy workloads.

6. Migration Checklist — 8 Steps to Production

Step	Task	Verification metric	Rollback
1	Profile target Lambda I/O (CloudWatch Logs Insights)	S3 GET/PUT ratio, average object size	—
2	Review VPC/subnet/SG (skip if Lambda is already in VPC)	NFS 2049 inbound allowed	—
3	Pick Throughput Mode (ELASTIC vs BURSTING)	Expected concurrent mounts	—
4	Isolate Access Point root path (per tenant/agent)	UID/GID + permissions diagram	—
5	Canary deploy (Alias weight 10%)	p99 latency, error rate	weight 0%
6	Compose with Durable Functions (if needed)	execution_id-scoped directories	—
7	Cost monitoring (EFS Throughput + Storage CW metrics)	within ±20% of estimate	switch Throughput Mode
8	Full cutover + deprecate old S3 SDK code	2 weeks stable operation	Alias rollback

Step 4 — Access Point isolation — is where multi-tenant SaaS implementations most often miss. To stop one tenant's buggy Lambda from corrupting the entire workspace, force RootDirectory.Path to /tenants/{tenant-id}/ and add s3files:AccessPointRootDirectory conditions to the IAM policy.

7. Conclusion — Serverless Has Been Redefined

For a decade, "serverless = stateless" was the first axiom of Lambda architecture. S3 Files rewrites it to "serverless = no infra to manage + optional shared state." This managed gateway — EFS's distributed file system engine layered onto S3's durability and cost — isn't merely a convenience. It's a new data plane that lifts agentic AI, genomics, media, and legacy ETL workloads onto Lambda. The April 28 keynote underlined the same direction: AWS bundled S3 Files with Bedrock Managed Agents, Durable Functions, and Amazon Quick. For the second half of 2026, the deciding factors when moving multi-agent workflows or large-model inference to Lambda are no longer "is the memory and timeout enough?" but "do the S3 Files Throughput Mode, Access Point isolation, and Durable Functions composition match the workload?" ManoIT adopted S3 Files in two production paths since the GA on April 21 — an internal code review agent and the LMS media transcoding pipeline — and observed p99 latency −28% and code line count −40% in both. Use the checklist above to step into it gradually.

This post was authored, reviewed, and published by ManoIT's automated blogging pipeline running on Claude Opus 4.6. Sources: AWS What's New (2026-04-21), AWS News Blog "Launching S3 Files," AWS Lambda official documentation (configuration-filesystem-s3files), Amazon S3 user guide (s3-files-mounting-lambda), AWS Weekly Roundup 2026-04-27, "What's Next with AWS 2026" keynote.

Originally published at ManoIT Tech Blog.

GitHub Actions Early April 2026 — Service Container Overrides, OIDC Custom Properties, VNET Failover, Immutable Sub

daniel jeong — Tue, 28 Apr 2026 00:23:50 +0000

GitHub Actions Early April 2026 Updates — Service Container Overrides, OIDC Repository Custom Properties, Azure VNET Failover, and Immutable Subject Claims Redefining Enterprise CI/CD

The April 2026 GitHub Actions changelog is not just another minor update. The April 2 "Early April 2026" bundle finally exposed the long-requested entrypoint/command overrides for service containers in workflow YAML, promoted Repository Custom Properties OIDC claims to GA (General Availability), and pushed Azure VNET failover for GitHub-hosted runners into public preview. Then on April 23, GitHub announced immutable owner/repository IDs in the default sub (subject) claim of OIDC tokens, effectively closing the long-standing "name reuse attack" gap. This article analyzes all four changes together from a ManoIT production lens, with multi-cloud IAM trust policies for AWS/Azure/GCP, service container migration patterns, multi-region VNET DR design, and a checklist for the June 18 cutover — all backed by real workflow YAML.

1. Why the April Updates Matter — 2026 GitHub Actions Market Landscape

According to JetBrains TeamCity's Q1 2026 CI/CD Tool Adoption report, GitHub Actions now holds 33% market share alone, well ahead of Jenkins (28%) and GitLab CI (19%). However, the same report consistently flagged that enterprise security compliance gaps had not narrowed in proportion to the adoption surge. The April updates target this gap directly.

Feature	Release Date	Status	Core Effect	Scope
Service Container `entrypoint`/`command` override	2026-04-02	GA	No more custom wrapper images	All plans (Free/Team/Enterprise)
OIDC Repository Custom Properties	2026-04-02	GA (Preview 2026-03-12 → GA)	ABAC trust policies enabled	Org/Enterprise admin
Azure VNET Failover	2026-04-02	Public Preview	Workflow continuity through regional outages	Enterprise/Org Azure users
Immutable Subject Claims	2026-04-23	Auto-applied to new repos starting 2026-06-18	Name reuse attack blocked	All plans (default sub format)

Bundle these together and the message is clear: stop treating GitHub Actions as a single-workflow toy and start managing it as an enterprise control plane that integrates multi-cloud IAM, networking, and runtimes. Service Container overrides fill the runtime reliability gap, Repository Custom Properties fill the governance gap, VNET Failover fills the availability gap, and Immutable Subject Claims fill the identity integrity gap.

2. Service Container Overrides — The End of Custom Wrapper Images

Two long-standing pain points were finally resolved on April 2. First, you could not override an image's default ENTRYPOINT from a workflow. The options: --entrypoint workaround constantly broke on argument quoting and escaping. Second, launching a container in a "test mode" — booting PostgreSQL read-only, forcing Redis with an ACL file, running Kafka KRaft single-node — required maintaining an internal wrapper image, the build/push pipeline that came with it, and the related vulnerability-scan SLA.

The new services.<service_id>.entrypoint and services.<service_id>.command keys eliminate both workarounds. You can now override the image defaults directly in your workflow YAML.

# .github/workflows/integration-test.yml
# Boot Postgres 16 with stats/log enrichment for integration tests — no wrapper image
name: integration-test

on:
  pull_request:
    branches: [main]

jobs:
  api-tests:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        # New entrypoint/command keys — GA on 2026-04-02
        entrypoint: /usr/local/bin/docker-entrypoint.sh
        command: >-
          postgres
          -c shared_preload_libraries=pg_stat_statements
          -c log_statement=all
          -c log_min_duration_statement=0
          -c max_connections=200
        env:
          POSTGRES_USER: app
          POSTGRES_PASSWORD: app
          POSTGRES_DB: appdb
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U app -d appdb"
          --health-interval 5s
          --health-timeout 3s
          --health-retries 12
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with: { node-version: '22' }
      - run: npm ci
      - run: npm run test:integration
        env:
          DATABASE_URL: postgres://app:app@localhost:5432/appdb

⚠️ Caution: Setting entrypoint to an empty string keeps the image default. To override only command, omit the entrypoint: key entirely. Also, when both the legacy options: --entrypoint=... and the new entrypoint: key are specified together, the new key wins — grep for both during migration to avoid the silent footgun.

2.1 Practical Migration Pattern — Phasing Out Wrapper Images in 4 Steps

If you already maintain a wrapper image like internal/redis-test-bootstrap:7, here's a 4-step staged migration.

Step	Action	Risk	Rollback
1. Equivalence check	Add a parallel job that puts the wrapper's `ENTRYPOINT`/`CMD` directly into workflow YAML, run alongside existing job	Low	Drop the new job
2. PR gate switch	Make the new job the required check, demote the old one to optional	Medium	Restore the old required check
3. Canary 1 week	Compare failure rate, duration, and log patterns	Medium	Restore the old required check
4. Wrapper image cleanup	Mark image as deprecated in registry, archive Dockerfile repo	Low	Restore image (within retention window)

The biggest savings empirically come from the disappearance of the Dockerfile build/push pipeline itself. In one in-house case, retiring six integration-test wrapper images eliminated about 240 GHCR pushes per week and an average build time of 38 seconds. Image vulnerability scan SLAs and dependency update PRs drop in the same proportion.

3. OIDC Repository Custom Properties — The Beginning of ABAC Trust Policies

Since OIDC became the standard for cross-cloud key sharing in 2023, the biggest limitation has been that you had to bake static matching rules into the sub claim. As repos grow, string-pattern matching like repo:org/repo:ref:refs/heads/main causes policies to balloon, and every new repo requires another touch of cloud IAM. The Repository Custom Properties claim, GA on April 2, replaces this with ABAC (Attribute-Based Access Control).

The mechanism is simple: an Org or Enterprise admin checks "include this custom property in OIDC claims" in the OIDC settings page. From then on, every repo with a value set for that property automatically gets a repo_property_<property_name> claim injected into its OIDC tokens. Cloud trust policies can then reference this claim as a condition.

# 1) Define the custom property at org level — REST API
gh api -X PATCH \
  /orgs/manoit/properties/schema \
  -f 'properties[][property_name]=environment' \
  -f 'properties[][value_type]=single_select' \
  -f 'properties[][allowed_values][]=dev' \
  -f 'properties[][allowed_values][]=staging' \
  -f 'properties[][allowed_values][]=prod' \
  -f 'properties[][required]=true'

# 2) Set the value on a specific repo
gh api -X PATCH \
  /repos/manoit/payment-service/properties/values \
  -f 'properties[][property_name]=environment' \
  -f 'properties[][value]=prod'

# 3) Enable the property as a claim in Org OIDC settings
# Settings → Actions → OIDC → check "environment"

After this, OIDC tokens issued by that repo's workflows include "repo_property_environment": "prod" as a claim. AWS, Azure, and GCP can use this claim as a trust policy condition (with one caveat: AWS sts:AssumeRoleWithWebIdentity has limited support for arbitrary claim matching — see workaround below).

3.1 Azure Federated Credential — Cleanest Arbitrary Claim Matching

# Azure Federated Identity Credential — grant access to all repos with repo_property_environment=prod
az identity federated-credential create \
  --name "gh-actions-prod" \
  --identity-name "manoit-prod-identity" \
  --resource-group "manoit-rg" \
  --issuer "https://token.actions.githubusercontent.com" \
  --subject "repo:manoit/*:ref:refs/heads/main" \
  --audiences "api://AzureADTokenExchange" \
  --claims-matching-expression "claims['repo_property_environment'] == 'prod'"

3.2 GCP Workload Identity Federation — Map via Attribute Condition

# GCP Workload Identity Pool Provider — repo_property_environment condition
gcloud iam workload-identity-pools providers create-oidc "github-pool-prod" \
  --workload-identity-pool="github-pool" \
  --location="global" \
  --issuer-uri="https://token.actions.githubusercontent.com" \
  --attribute-mapping="google.subject=assertion.sub,attribute.environment=assertion.repo_property_environment,attribute.repository=assertion.repository" \
  --attribute-condition="attribute.environment == 'prod'"

3.3 AWS IAM — Inject the Custom Property into the `sub` Claim

AWS sts:AssumeRoleWithWebIdentity only natively pattern-matches the sub claim. Fortunately, GitHub now supports include_claim_keys to inject custom properties into the sub claim itself.

# .github/workflows/deploy-prod.yml
permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/manoit-prod-deploy
          aws-region: ap-northeast-2
          # Reshape the sub claim to be ABAC-friendly — include repo_property_environment
          role-session-name: gh-actions-${{ github.run_id }}
        env:
          ACTIONS_ID_TOKEN_REQUEST_AUDIENCE: sts.amazonaws.com

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringLike": {
        "token.actions.githubusercontent.com:sub":
          "repo:manoit/*:environment:prod:property:environment=prod"
      },
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
      }
    }
  }]
}

⚠️ Caution: When using StringLike wildcards in AWS trust policies, failing to anchor the prefix risks privilege explosion. Always anchor the org prefix with repo:manoit/*, and split environment labels (prod/staging) into separate IAM Roles. Also, injecting custom fields into the sub claim makes tokens larger — review whether a single ABAC ledger policy could satisfy the same intent more cheaply.

4. Azure VNET Failover — Multi-Region DR at the Workflow Layer

The Azure private networking feature for GitHub-hosted runners (GA in November 2023) lets workflows reach resources inside corporate Azure VNETs (e.g., Private Endpoints, ExpressRoute-connected on-prem) without IP whitelisting. The catch was single-subnet dependency: if that region (e.g., Korea Central) went down, every workflow company-wide stopped — a hard SPOF.

The VNET Failover preview, released April 2, removes this SPOF. Two key points:

Item	Before (Single Subnet)	April Update (VNET Failover)
Configuration unit	1 Primary subnet	Primary + Secondary subnet
Region constraint	Same region recommended	Secondary can be in a different Azure region
Behavior on outage	Workflow fails	Routed to secondary subnet
Switchover	Admin manual reconfig (minutes)	Instant switch via UI/REST API (manual in preview)
Load balancing	None	None (currently active/passive)

# Register failover network — REST API
# Prerequisite: Primary VNET/subnet and Secondary VNET/subnet must exist in each region

ORG=manoit
SETTINGS_ID=12345

curl -X POST \
  -H "Authorization: Bearer $GITHUB_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/orgs/$ORG/settings/network-configurations/$SETTINGS_ID/failover \
  -d '{
    "name": "kr-south-failover",
    "subnet_resource_id": "/subscriptions/.../resourceGroups/manoit-dr-rg/providers/Microsoft.Network/virtualNetworks/manoit-dr-vnet/subnets/runners",
    "azure_region": "koreasouth"
  }'

# Trigger failover during an incident (manual in preview)
curl -X PATCH \
  -H "Authorization: Bearer $GITHUB_TOKEN" \
  https://api.github.com/orgs/$ORG/settings/network-configurations/$SETTINGS_ID \
  -d '{ "active_subnet": "secondary" }'

4.1 ManoIT-Recommended Architecture — Korea Central + Korea South Active-Passive

Based on ManoIT's experience with Korean enterprise customers, here's the recommended pattern.

Component	Primary (Korea Central)	Secondary (Korea South)	Notes
VNET CIDR	10.20.0.0/16	10.21.0.0/16	Must not overlap
Runner subnet	10.20.10.0/24 (recommend /24, 256 IPs)	10.21.10.0/24	Max concurrent runners ≈ IPs - 5
Private Endpoint (ACR)	kr-central acr-pe	kr-south acr-pe	Geo-replicated ACR
VNET Peering	Bidirectional + Allow forwarded traffic (both regions)		For post-failover data sync
Key Vault	Premium SKU + soft-delete	Geo-replicated	Federated credential shared
Monitoring	Azure Monitor + Action Group → PagerDuty (both regions)		RTO target ≤ 15 minutes

⚠️ Caution: Automatic failover is not active in preview. An admin must trigger it via UI or REST API, so a pipeline of off-hours alert → on-call automation (e.g., PagerDuty incident → ChatOps bot → REST call) must be built upfront to meet your RTO. GitHub plans to add automatic triggers at GA, but you'll still need to validate that the "regional outage detection heuristic" matches your in-house SLA.

5. Immutable Subject Claims — The End of the Name Reuse Attack

The Immutable Subject Claims announcement on April 23 has the largest security impact of the four April updates. The legacy default sub claim looked like repo:octocat/my-repo:ref:refs/heads/main — composed entirely of mutable names. If an organization recycles a deleted repo's name, a new owner could mint tokens with the identical sub claim. This gap has been an open issue (GitHub Roadmap #1230) tracked for nearly a year and reported in real attack scenarios.

The new format embeds immutable owner_id and repository_id into the sub claim.

Aspect	Legacy `sub`	New `sub` (new repos after 2026-06-18)
Format	`repo:octocat/my-repo:ref:refs/heads/main`	`repo:octocat-123456/my-repo-456789:ref:refs/heads/main`
Key difference	Same `sub` reissuable on name reuse	`owner_id`/`repo_id` are permanent identifiers
Name reuse attack	Possible	Blocked
Rollout	All repos (current)	Auto for repos created after 2026-06-18; existing repos opt-in

The practical impact splits two ways.

(A) Newly created repos — automatically get the new format. If your cloud trust policies only pattern-match the short legacy sub, token validation will fail. Extend those policies to match both formats before June 18.

(B) Existing repos — no auto migration, but opt-in is available, and we strongly recommend completing migration within nine months. Roll it out in stages so policies don't break.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringLike": {
        "token.actions.githubusercontent.com:sub": [
          "repo:manoit/*:ref:refs/heads/main",
          "repo:manoit-987654/*:ref:refs/heads/main"
        ]
      },
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
      }
    }
  }]
}

In parallel, adopt the operating rule of looking up your org_id once and baking it into policies. Then policies remain valid even if the org is renamed later.

# Look up your org_id
gh api /orgs/manoit --jq '.id'   # e.g. 987654

# Pre-collect repo IDs to also pin them in policy for safety
gh api /repos/manoit/payment-service --jq '.id'  # e.g. 13579024

6. Unified Workflow — A Production Pipeline Using All Four Updates

Finally, here's a single workflow that combines all four April updates. It mirrors how ManoIT's payment-service flows from PR to main merge — Service Container overrides, OIDC Custom Properties, VNET-failover-aware runner selection, and Immutable sub compatibility, all together.

name: payment-service-cicd

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  id-token: write   # OIDC token requests
  contents: read
  pull-requests: write

env:
  AWS_REGION: ap-northeast-2

jobs:
  test:
    name: Integration Test (ARM64)
    runs-on: ubuntu-latest-arm
    services:
      postgres:
        image: postgres:16-alpine
        entrypoint: /usr/local/bin/docker-entrypoint.sh
        command: >-
          postgres
          -c shared_preload_libraries=pg_stat_statements
          -c log_statement=all
          -c max_connections=200
        env: { POSTGRES_USER: app, POSTGRES_PASSWORD: app, POSTGRES_DB: appdb }
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U app -d appdb" --health-interval 5s
          --health-timeout 3s --health-retries 12
      redis:
        image: redis:7-alpine
        # Force ACL-enforced boot mode — previously required a wrapper image
        command: >-
          redis-server
          --aclfile /etc/redis/users.acl
          --appendonly yes
        ports: ["6379:6379"]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with: { node-version: '22', cache: 'npm' }
      - run: npm ci
      - run: npm run test:integration

  deploy-prod:
    name: Deploy to Prod (Azure Private Network)
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest-arm-azure-private  # VNET-failover-enabled hosted runner
    environment: prod
    steps:
      - uses: actions/checkout@v5

      # 1) Azure Federated Identity — auth via repo_property_environment=prod claim
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_PROD_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_PROD_SUBSCRIPTION_ID }}

      # 2) Helm deploy (talks to AKS Control Plane via Private Endpoint)
      - uses: azure/aks-set-context@v4
        with:
          resource-group: manoit-prod-rg
          cluster-name: manoit-aks-prod
      - run: |
          helm upgrade --install payment ./chart \
            --namespace payment --create-namespace \
            --values ./chart/values.prod.yaml \
            --set image.tag=${{ github.sha }} \
            --atomic --timeout 5m

      # 3) AWS S3 backup of payment logs (immutable sub + ABAC simultaneously satisfied)
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/manoit-prod-deploy
          aws-region: ${{ env.AWS_REGION }}
      - run: aws s3 sync ./logs s3://manoit-prod-logs/payment/

      # 4) Comment back on the PR with deployment result (observability)
      - uses: actions/github-script@v7
        with:
          script: |
            const sha = context.sha.substring(0, 7);
            github.rest.issues.createComment({
              owner: context.repo.owner, repo: context.repo.repo,
              issue_number: context.payload.pull_request?.number ?? 0,
              body: `Deployed payment-service@${sha} to prod (AKS + S3) via VNET-routed runner.`
            });

7. ManoIT Adoption Checklist — Before the June 18 Cutover

Item	Action	Effort	Done When
Service Container wrapper image inventory	`grep -R "options:.*entrypoint" .github/` across all repos	0.5 day	List enumerated + prioritized
Phased wrapper image removal	Apply the 4-step migration from Section 2.1	1 week per image	Required check switched + 1-week canary clean
Repository Custom Properties schema	Define `environment`, `data_classification`, `cost_center` first	1 day	Org schema registered + ≥95% repos populated
Enable OIDC claim inclusion	Settings → Actions → OIDC → check entries	0.5 hour	Sample workflow token decoded + validated
Azure/AWS/GCP trust policy ABAC migration	Phased rollout per Section 3 examples	2 weeks	Per-repo IAM Role count reduced ≥50%
Azure VNET Failover setup	Korea Central + Korea South dual subnet + peering	1 week	Manual failover sim with RTO ≤ 15 min
Immutable Subject Claims compatibility check	Extend policies to allow both old and new `sub` patterns	3 days	Token validation passes on a repo created post-2026-06-18
On-call runbook update	Add VNET failover and OIDC policy change scenarios	1 day	Quarterly drill executed once

8. Conclusion — What the April Updates Predict for the Next 12 Months

The four changes in April 2026 are not coincidence. GitHub is redefining GitHub Actions from a "workflow automation tool" into an "enterprise IAM, network, and runtime control plane," and the April updates are the unmistakable signal of that redefinition. Service Container overrides end the burden of in-house images, Repository Custom Properties tightly bind governance metadata to tokens, VNET Failover lifts availability SLAs from the 99.9% range toward 99.95%, and Immutable Subject Claims close the oldest OIDC trust gap.

In H2 2026, expect Native Egress Firewall (announced earlier in the year roadmap), expansion of Action Allowlisting to Free/Team plans, and broader adoption of the OIDC check_run_id claim. ManoIT strongly recommends absorbing all of the Early April 2026 updates before the June 18 cutover. For adoption consulting, please reach out at www.manoit.co.kr.

This article was authored and reviewed by ManoIT's automated blog pipeline running on Anthropic Claude Opus 4.6, with every code snippet cross-checked against official GitHub Actions, GitHub Docs, Azure, AWS, and GCP references as of 2026-04-28. Always validate in your own environment before applying to production.

Originally published in Korean at manoit.co.kr.

Originally published at ManoIT Tech Blog.

Meta Llama 4 Scout & Maverick — The Complete Production Guide: 17B Active MoE, 10M Context, iRoPE, and the vLLM/Ollama Deployment Playbook

daniel jeong — Mon, 27 Apr 2026 00:19:03 +0000

Meta Llama 4 Scout & Maverick — The Complete Production Guide: 17B Active MoE, 10M Context, iRoPE, and the vLLM/Ollama Deployment Playbook

Published on the ManoIT Tech Blog (Korean original). On April 5, 2026, Meta released Llama 4 Scout and Llama 4 Maverick — the first open-weights models in the Llama family to use a Mixture-of-Experts (MoE) architecture, the first to be natively multimodal, and — with Scout — the first to deliver a real 10M-token context window that runs on a single H100 GPU. This post unpacks the architecture, benchmarks, license caveats, deployment options, Llama Guard 4 safety stack, and the Behemoth delay — from a production-deployment perspective grounded in what ManoIT recommends to customers.

1. The Llama 4 family at a glance — Scout / Maverick / Behemoth

Llama 4 was always planned as a three-model family. April 5 brought two models; the largest one, Behemoth, is still in private training. Knowing where each model fits is the first decision in any adoption review.

Model	Active / Total params	Experts	Context	Primary use	Status
Llama 4 Scout	17B / 109B	16	10M tokens	Long-document analysis, code-base RAG, video summarization	Released 2026-04-05 (Hugging Face, Ollama)
Llama 4 Maverick	17B / 400B	128	1M tokens	Multimodal assistant, GPT-4o replacement	Released 2026-04-05 (Hugging Face, Ollama, watsonx)
Llama 4 Behemoth	288B / ~2T	16	private	Teacher model for Scout/Maverick, STEM-heavy	Delayed to fall 2026 (internal evaluation)

The pattern is "same 17B active, different expert pool." Scout routes each token to 1-of-16 experts; Maverick routes 1-of-128. Active parameters being equal means the per-token compute cost is the same — but the diversity of the expert pool changes the model's expressive ceiling. That's why Maverick beats GPT-4o on multimodal aggregates and Scout fits in a single-H100 + Int4 footprint.

2. iRoPE architecture — the secret behind a 10M context

Scout's 10M-token reach comes from iRoPE (Interleaved Rotary Position Embeddings). Traditional transformers apply a position encoding (RoPE) at every attention layer — but as the context grows, the position signal becomes noise and length generalization collapses.

Traditional limitation	iRoPE answer	Operational effect
RoPE on every layer → can't generalize beyond 8K~128K training length	Interleave a NoPE (no position encoding) layer every 4 layers	Train at 256K, extrapolate at inference to 10M
Long-context noise blurs token-token relationships	NoPE layers do global attention over the full causal mask	Near-perfect needle-in-haystack at 10M
RoPE index overflow at long contexts	NoPE layers remove absolute positional dependency	Whole-video and whole-codebase indexing become viable

The idea is simple: mix layers that need positional encoding with ones that don't. RoPE layers learn local order; NoPE layers freely connect distant tokens by meaning. The result is reliable generalization to context lengths the model never saw at training time — which is why Scout retrieves accurately at 10M tokens.

┌─────────────── Llama 4 Scout attention stack (concept) ───────────┐
│                                                                    │
│   Layer 1  : RoPE Attention   (learns local order)                 │
│   Layer 2  : RoPE Attention                                        │
│   Layer 3  : RoPE Attention                                        │
│   Layer 4  : NoPE Attention   ← global causal mask, position-free  │
│   Layer 5  : RoPE Attention                                        │
│   Layer 6  : RoPE Attention                                        │
│   Layer 7  : RoPE Attention                                        │
│   Layer 8  : NoPE Attention   ← every 4 layers                     │
│   ...                                                              │
│                                                                    │
│   Each token routed to 1-of-16 experts (Top-1 MoE)                 │
│   → Active params at inference: 17B / Total params: 109B           │
└────────────────────────────────────────────────────────────────────┘

3. Benchmarks — Llama 4 vs GPT-4o vs Gemini 2.0 Flash

Meta's own numbers show clear wins for in-class comparisons. As always — these are vendor benchmarks; run your own evals before committing.

Benchmark	Scout 17B/16E	Maverick 17B/128E	GPT-4o	Gemini 2.0 Flash	What it means
MMLU-Pro	74.3	80.5	78.0	77.6	Maverick wins multi-domain reasoning
MATH (Hendrycks)	50.3	61.2	76.6	56.1	STEM gap vs GPT-4o / o-series remains
GPQA Diamond	57.2	69.8	53.6	60.1	Graduate-level science — Maverick #1
ChartQA (image)	88.8	90.0	85.7	87.3	Multimodal chart understanding — both SOTA
DocVQA	94.4	94.4	92.8	92.1	Document-image QA — new bar set
MMMU	69.4	73.4	69.1	71.7	Multimodal exam aggregate — Maverick wins
Long Context (10M, NIAH)	~99%	N/A	~64K limit	~1M limit	Scout's exclusive territory

The figure that should give a buyer pause is MATH. Llama 4 caught up or pulled ahead in general and multimodal domains, but pure math/STEM still favors OpenAI's o-series by a clear margin. If your workload is STEM-heavy reasoning, Llama 4 alone is not enough — pair it with o-series or wait for Behemoth GA. If your workload is long-document RAG, multimodal assistants, or codebase analysis, Llama 4 is the strongest cost-to-quality option on the market today.

4. License — "Open Weights" but not OSI Open Source

Llama 4 ships under the Llama 4 Community License Agreement — and this is the place buyers most often misread. It is not Apache 2.0 or MIT, and it does not meet the OSI Open Source definition. Five clauses to inspect before you ship.

Clause	Constraint	What to do
① 700M MAU cap	If your product exceeds 700M monthly active users, you must negotiate a separate license with Meta	Startups and mid-market: irrelevant. Hyperscale SaaS: legal review
② EU multimodal carve-out	EU users / EU companies cannot use the vision features (text-only allowed)	EU services must disable Maverick vision or fork to a text-only path
③ Acceptable Use Policy	Threat modeling, malicious cybersec, CSAM, fraud, etc. are prohibited	Pair with Llama Guard 4 input/output filtering + your own content policy
④ "Built with Llama" attribution	Products built on Llama must display "Built with Llama"	Add to About page / docs footer
⑤ Derivative naming	Derivative model names must start with "Llama-"	e.g. `Llama-Manoit-Customer-Support-v1`

One-liner: "Commercial use is fine, but check the five clauses — 700M MAU, EU vision, AUP, attribution, naming." For most Korean SaaS, B2B tools, and internal projects clause 1 is moot. EU multimodal services need clause 2 in their pre-launch checklist.

5. Deployment options — Ollama, vLLM, watsonx.ai, Hugging Face

Llama 4 launched day-one on Hugging Face, Ollama, watsonx.ai, Together, Fireworks, Groq, and Oracle OCI. Pick by workload character:

Option	Best for	Strengths	Constraints
Ollama (local)	Development, PoC, offline tools	One-line `ollama run llama4:scout`, automatic GGUF quantization	Single-node, no multi-GPU sharding
vLLM (self-hosted)	Low-latency production serving	PagedAttention, continuous batching, OpenAI-compatible API	You operate the GPUs and NCCL
HF TGI	Hugging Face standardization	Token streaming, tensor parallelism	Slightly lower throughput than vLLM
watsonx.ai / Bedrock	Enterprise compliance	VPC isolation, SOC2, HIPAA, EU data residency	3-5× the per-token cost vs self-hosting
Together / Fireworks / Groq	Cheap token consumption	$0.27/1M tokens (Maverick at some vendors)	Vendor lock-in, residency review needed

5.1 vLLM production deploy (Maverick, 8×H100, FP8)

# Step 1: get a Hugging Face token
export HF_TOKEN="hf_xxxxxxxxxxxxxxxxxxxxxxx"

# Step 2: install vLLM 0.7.0+ (Llama 4 MoE support)
pip install --upgrade vllm

# Step 3: serve Maverick (8×H100, FP8 quantization)
vllm serve meta-llama/Llama-4-Maverick-17B-128E-Instruct \
  --tensor-parallel-size 8 \
  --quantization fp8 \
  --max-model-len 1048576 \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching \
  --port 8000

# Step 4: hit the OpenAI-compatible API
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-4-Maverick-17B-128E-Instruct",
    "messages": [{"role": "user", "content": "Summarize Llama 4 adoption trends in one paragraph"}]
  }'

5.2 Ollama local (Scout)

# Step 1: install Ollama (macOS / Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Step 2: pull Scout (~60GB at Q4_K_M)
ollama pull llama4:scout

# Step 3: interactive
ollama run llama4:scout

# Step 4: API mode (port 11434)
curl http://localhost:11434/api/chat -d '{
  "model": "llama4:scout",
  "messages": [
    {"role": "user", "content": "How would I summarize a 10MB log file in a single inference?"}
  ]
}'

Practical hardware floor: Scout fits in a single H100 (80GB) at Int4, Maverick wants at least 4×H100 (80GB). A Mac Studio M2 Ultra (192GB) can run Scout Q4 for PoC purposes at 5-10 tok/s.

6. Llama Guard 4 — the 12B multimodal safety stack

Alongside Llama 4, Meta released Llama Guard 4 (12B) — a multimodal classifier that scores both text and images against 13 risk categories (violence, self-harm, sexual content, CSAM, hate, criminal facilitation, CBRN, etc.). The recommended ManoIT pipeline applies guards on both input and output:

┌──────────────────────── User request ────────────────────────┐
│  POST /v1/chat   { messages: [...] }                        │
└──────────────────────────────┬───────────────────────────────┘
                               │
                               ▼
              ┌────────────────────────────────┐
              │  ① Prompt Guard (86M, fast)     │
              │  Jailbreak / Prompt Injection   │
              └────────────────────────────────┘
                               │ pass
                               ▼
              ┌────────────────────────────────┐
              │  ② Llama Guard 4 (12B)          │
              │  Input safety classifier (S1~13)│
              └────────────────────────────────┘
                               │ safe
                               ▼
              ┌────────────────────────────────┐
              │  ③ Llama 4 Maverick / Scout     │
              │  Generate response              │
              └────────────────────────────────┘
                               │
                               ▼
              ┌────────────────────────────────┐
              │  ④ Llama Guard 4 (output)       │
              │  unsafe → block / redact        │
              └────────────────────────────────┘
                               │ safe
                               ▼
                       Final response

Guard model	Role	Size	Latency impact
Prompt Guard 2 (86M)	First-pass jailbreak / prompt-injection filter	86M (DistilBERT-class)	+10~20ms
Llama Guard 4 (12B)	13-category classifier (input + output)	12B multimodal	+150~300ms
CyberSec Eval	Detect security flaws in generated code	Eval framework	Pre-deploy static check

7. The Behemoth delay — what it means

Llama 4 Behemoth (2T total / 288B active) was not part of the April 5 release. It was originally targeted for April, slipped to June, and is now expected in fall 2026. Reporting attributes the delay to internal concerns about whether Behemoth's gain over Maverick justifies a public rollout.

Date	Event	Meaning
2026-01	Behemoth ~75% trained, April launch announced	Planned co-release with Scout / Maverick
2026-04-05	Only Scout / Maverick released; Behemoth slips to June	"Need more internal evaluation"
2026-05-15	Behemoth pushed to fall 2026	Strong on STEM, unclear lift on general workloads
2026-06-10	Meta superintelligence lab reportedly forms	Top-tier model team carved out, Behemoth realigned

The delay does not affect a Llama 4 adoption decision. Scout and Maverick already compete with GPT-4o and Gemini 2.0 Flash, and the reason enterprises pick open-weights models is data sovereignty, on-prem control, and customization — not a single benchmark crown. If STEM is your hot path, hybrid with Claude Opus 4.7 or OpenAI o-series until Behemoth GA.

8. ManoIT 8-step production checklist

Step	Check	Concrete action
① License review	5 clauses — 700M MAU, EU vision, AUP, "Built with Llama", naming	Legal review → impact memo
② Model selection	Scout vs Maverick by workload (text / multimodal / context length)	Long-doc RAG → Scout, multimodal → Maverick
③ Infra design	FP8 / Int4 quantization, TP degree, KV cache memory budget	Maverick: 8×H100 FP8, Scout: 1×H100 Int4
④ Serving stack	vLLM / TGI / Ollama choice, autoscaling policy	vLLM 0.7.0+ recommended, KEDA for GPU pool autoscale
⑤ Safety stack	Prompt Guard 2 + Llama Guard 4 on input and output	Budget +200ms latency, define category policy
⑥ Eval pipeline	Korean benchmarks + domain golden datasets	KMMLU, KoBEST, in-house RAG accuracy automated
⑦ Observability	Token throughput, GPU util, guard block rate, RAG hit rate	OpenTelemetry GenAI semconv + Grafana
⑧ Governance	Model card, retention, audit log, rollback runbook	Model card + prompt versioning + sampled response retention

9. Field guide — which model for which job

Scenario	Recommended	Why
50 contracts side-by-side, clause comparison	Scout	10M context, single node, lowest cost
Whole-codebase RAG (200K LOC)	Scout	Fit the entire repo in context
Multimodal customer-support chatbot (image + text)	Maverick	DocVQA / ChartQA leader, GPT-4o replacement
20-hour video summarization	Scout	Native long-context, full-video indexing
STEM math	OpenAI o-series · Claude Opus	MATH/AIME gap remains, wait for Behemoth GA
Real-time voice assistant	Maverick + Whisper (STT)	<500ms response, multimodal context
On-prem medical / financial RAG	Scout + Llama Guard 4	Data sovereignty, 700M MAU irrelevant
EU-resident multimodal service	Maverick (text only) or GPT-4o	EU vision license restriction

10. Conclusion — what Llama 4 changes, and what it doesn't

The significance of Scout and Maverick is that open-weights models reached SOTA in multimodal and ultra-long-context territory. Through 2024-2025 open models could compete with GPT-4o on English text reasoning, but multimodal and >100K-token contexts had a real gap. iRoPE and native multimodal training closed it.

What did not change: the STEM / math reasoning gap vs OpenAI o-series remains, and that won't close until Behemoth GA. "Open Weights" is not "Open Source" — the five Llama 4 Community License clauses (especially EU multimodal exclusion and 700M MAU) require a legal review before adoption. And safety and governance cost does not vanish because the weights are public — Llama Guard 4 + Prompt Guard + your own eval pipeline are not optional.

ManoIT's recommendation for Q2 2026: standardize on Scout as the default RAG engine (legal, technical document analysis), Maverick as the multimodal assistant backbone (customer support, automation workflows). Hybridize with Claude Opus 4.7 or OpenAI o-series where STEM is the hot path. For multimodal services with EU exposure, gate adoption on a license review. We will revisit this guidance when Behemoth GA arrives in fall 2026.

This article was co-authored by the ManoIT engineering team with Anthropic Claude Opus 4.7. Korean original published on the ManoIT tech blog. Sources: Meta AI - The Llama 4 herd, Llama 4 Official Model Page, Hugging Face Llama-4-Scout, Hugging Face Llama-4-Maverick, Hugging Face Llama 4 Release Blog, IBM watsonx.ai Llama 4 Announcement, Ollama Llama 4 Library, Computerworld - Behemoth Pause, Protect AI - Llama 4 Vulnerability Assessment.

Originally published at ManoIT Tech Blog.

OpenTelemetry eBPF Instrumentation (OBI) — The Complete Guide: KubeCon EU 2026 Beta Launch, Zero-Code Observability, and the 1.0 GA Roadmap

daniel jeong — Thu, 23 Apr 2026 00:15:08 +0000

OpenTelemetry eBPF Instrumentation (OBI) — The Complete Guide: KubeCon EU 2026 Beta Launch, Zero-Code Observability, and the 1.0 GA Roadmap

Published on the ManoIT Tech Blog (Korean original). On April 2026 at KubeCon + CloudNativeCon Europe in Amsterdam, Splunk formally announced the beta launch of OpenTelemetry eBPF Instrumentation (OBI) — the OpenTelemetry community's successor to Grafana Beyla. This post walks through v0.8.0 architecture, Kubernetes Helm deployment, HTTP header enrichment for multi-tenant incident response, the 2026 roadmap toward 1.0 GA, how OBI relates to Beyla/Pixie/Tetragon/Hubble, and a production adoption checklist grounded in what ManoIT ships to customers.

1. Why OBI matters — the zero-code observability inflection point

CNCF's Observability TAG reported in Q1 2026 that 67% of production Kubernetes clusters are already running at least one eBPF-based observability tool. But the existing landscape was fragmented: Pixie was tied to New Relic, Grafana Beyla skewed toward Grafana Cloud, and Cilium Hubble stopped at L3/L4 network flows without application-level tracing. OBI cleans this up using the OpenTelemetry Protocol (OTLP) and an Apache 2.0 license.

Pain point	Traditional workaround	OBI answer	Operational effect
Go/Rust/C++ binary auto-instrumentation	OTel SDK insertion, rebuild required	eBPF uprobes + kprobes, zero code	Legacy and third-party binaries get visibility immediately
TLS-encrypted traffic tracing	Sidecar proxy (Envoy/Istio) injection	Kernel-level SSL_read/SSL_write hooks	HTTPS payloads observable without sidecars
Multi-tenant SaaS incident triage	"Error rate up" — no idea which tenant	HTTP header enrichment (v0.7.0+)	Filter by `x-tenant-id` / `x-user-segment`
SQL / Redis / Mongo query analysis	ORM instrumentation + sampling	Native server spans (pgx, mysql, mongo, redis, couchbase)	DB latency linked to app traces
OpenAI / Anthropic call tracing	Manual wrappers + custom token counting	GenAI instrumentation with payload extraction	LLM cost and latency collected automatically

In one line: OBI collects only what the kernel can tell it, and leaves the SDK alone. If you still need custom business events or application-specific attributes, OBI is designed to run alongside the OpenTelemetry SDKs — it fills visibility gaps, it doesn't replace language-level instrumentation.

2. OBI architecture — from Beyla to OBI

OBI's technical lineage is Grafana Beyla. Grafana Labs donated Beyla to OpenTelemetry in 2025; a weekly SIG formed, test pipeline speeds improved 10×, and after a late-2025 alpha release the project reached v0.8.0 on April 16, 2026. It ships as a binary, as a Docker image (otel/ebpf-instrument), and as a Helm chart.

┌─────────────────────────── User-Space Agent ───────────────────────────┐
│  (obi binary, written in Go)                                           │
│                                                                        │
│  ┌──────────────┐  ┌───────────────┐  ┌────────────────────────────┐ │
│  │ eBPF Map     │→ │ Span Builder  │→ │ OTLP Exporter (gRPC/HTTP) │ │
│  │ Reader       │  │ (HTTP/gRPC/DB)│  │ → OTel Collector           │ │
│  └──────────────┘  └───────────────┘  └────────────────────────────┘ │
│         ↑                                                              │
│         │ eBPF maps (perf_event_array, ring_buffer)                    │
└─────────┼──────────────────────────────────────────────────────────────┘
          │
┌─────────┼────────────────── Kernel-Space Probes ───────────────────────┐
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐               │
│   │ uprobes      │  │ kprobes      │  │ tracepoints  │               │
│   │ (SSL_read,   │  │ (tcp_sendmsg,│  │ (sched, fs)  │               │
│   │  SSL_write)  │  │  tcp_recvmsg)│  │              │               │
│   └──────────────┘  └──────────────┘  └──────────────┘               │
│           │                                                            │
│           ▼ Linux 5.8+ kernel (RHEL 4.18+ backport), BTF required      │
└────────────────────────────────────────────────────────────────────────┘

Two design decisions drive the low overhead: kernel probes only capture raw events, leaving heavy parsing, filtering, and mapping to the user-space agent — which keeps kernel-side CPU cost minimal. And because the output is OTLP, a single OBI deployment can feed Jaeger, Tempo, Splunk APM, Grafana Cloud, or Honeycomb through the same OpenTelemetry Collector.

System requirement	Detail	Notes
Linux kernel	5.8+ (RHEL/Rocky/Alma 4.18+ with eBPF backport)	BTF (BPF Type Format) required
Architecture	amd64, arm64	Graviton / Ampere supported
Privileges	root or `CAP_BPF + CAP_SYS_PTRACE`	Configure DaemonSet securityContext
Pod settings	`hostPID: true` recommended	Required to discover host-namespace processes
Container image	`otel/ebpf-instrument:v0.8.0`	CycloneDX SBOM included
License	Apache 2.0	No commercial restrictions

3. Kubernetes Helm deployment — cluster visibility in 15 minutes

The officially recommended deployment topology is Helm + DaemonSet. A DaemonSet is required because OBI must reach every node's process namespace, and hostNetwork / hostPID are wired automatically when deployed this way.

# Step 1: add the OpenTelemetry Helm repository
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# Step 2: default install (DaemonSet in the obi namespace)
helm install obi \
  -n obi --create-namespace \
  open-telemetry/opentelemetry-ebpf-instrumentation

# Step 3: verify the install
kubectl -n obi get daemonset
kubectl -n obi logs -l app.kubernetes.io/name=obi --tail=50

# Step 4: confirm probes were loaded on each node
kubectl -n obi exec ds/obi -- ls /sys/fs/bpf/obi/

The default install is enough to start collecting RED metrics and traces for every HTTP/gRPC request. In production you'll want to combine OTLP endpoint selection, service discovery, and header enrichment into a custom values file.

# helm-obi-prod.yaml — ManoIT production values
config:
  data:
    # OTLP destination (e.g. OTel Collector ClusterIP service)
    otel:
      endpoint: http://otel-collector.observability.svc:4318
      protocol: http/protobuf
    # Automatic process discovery
    discovery:
      services:
        - k8s_namespace: "^(shop|payment|auth)$"
          k8s_pod_labels:
            obi.enabled: "true"
      exclude_services:
        - exe_path_regex: ".*/istio-proxy$"
    # ⚠️ CNIs using eBPF datapaths (Cilium eBPF, Calico eBPF) can collide
    network:
      enabled: true
      cidrs:
        - 10.0.0.0/8
    # Protocol-specific instrumentation
    routes:
      unmatched: heuristic   # generate spans even for unknown HTTP paths
    log_level: info
    log_format: json

# Pod securityContext — required eBPF capabilities
securityContext:
  privileged: false
  capabilities:
    add: ["BPF", "PERFMON", "SYS_PTRACE", "NET_ADMIN"]
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true

# DaemonSet resource limits
resources:
  requests: { cpu: 100m, memory: 256Mi }
  limits:   { cpu: 500m, memory: 512Mi }

# Node selection (exclude ARM-only clusters, GPU nodes, etc.)
nodeSelector:
  kubernetes.io/arch: amd64
tolerations:
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300

helm upgrade --install obi \
  -n obi --create-namespace \
  -f helm-obi-prod.yaml \
  open-telemetry/opentelemetry-ebpf-instrumentation

⚠️ Watch out: on clusters running Cilium eBPF mode or the Calico eBPF dataplane, some of OBI's network probes can collide with CNI programs. The official recommendation is to set network.enabled: false and split responsibilities: OBI for L7 application traces, Cilium Hubble or Tetragon for network flows and kernel-level security. This role-split has become the CNCF-recommended pattern in 2026.

4. HTTP header enrichment — multi-tenant SaaS incident triage

Header enrichment, introduced in v0.7.0, is the feature that moves OBI from "monitoring tool" to incident-response platform. Raw traces only tell you "error rate rose to 5% on this endpoint." Header enrichment tells you which tenants and which user segments are affected — without any code change.

# OBI values.yaml — header enrichment policy
config:
  data:
    ebpf:
      track_request_headers: true
      payload_extraction:
        http:
          enrichment:
            enabled: true
            policy:
              default_action: exclude         # never collect by default
              obfuscation_string: "***"       # mask sensitive headers
            rules:
              # 1) tenant / segment identifiers → attach as span attributes
              - action: include
                type: headers
                match:
                  patterns:
                    - "x-tenant-id"
                    - "x-user-segment"
                    - "x-org-id"
              # 2) auth tokens → record a hash but mask the value
              - action: obfuscate
                type: headers
                match:
                  patterns:
                    - "authorization"
                    - "cookie"
                    - "x-api-key"

The policy design is deliberately explicit allowlist + selective obfuscation. The default is exclude, so PII and tokens can't leak by accident. Pattern matching is case-insensitive and supports wildcards (x-manoit-*).

Incident scenario	With raw traces only	With header enrichment
Latency spike for one tenant	"Payment API latency up" → page the entire support team	Filter by `x-tenant-id` → only two enterprise customers affected → notify their CSM directly
B2B free vs. paid segmentation	Only sees "error rate 1.2%"	Filter `x-user-segment=paid` → track SLA cohort separately
Regional issue	Needs a separate ALB-log analysis	`x-region` header + OBI spans on the same Grafana panel
Per-API-key rate limiting	401/429 visible, which key is unclear	Masked `x-api-key` hash identifies the top offenders

5. 2026 roadmap — four axes toward 1.0 GA

The OBI SIG published its official 2026 roadmap with four axes, each with a named sponsor and GitHub milestones — making it possible to back-calculate adoption timelines.

Goal	Sponsor	Key deliverables	Production impact
① 1.0 Stable Release	@MrAlias	JSON Schema validation, declarative config standard, telemetry schema, versioning policy, test coverage targets	End of v0 breaking changes, LTS track becomes possible
② Protocol expansion	@marctc, @NimrodAvni78	MQTT, AMQP, NATS, Redis Pub/Sub, MongoDB enhancements, GCP/AWS/Azure SDKs, full gRPC context propagation	Coverage of message brokers and cloud SDKs
③ .NET support	@rafaelroquetto	.NET 8+, .NET Framework 4.x, 3.5 SP1 validation, distributed tracing + RED metrics verification	Enterprise Windows workloads covered
④ Hybrid instrumentation	@grcevski	Consistent labels with SDK traces, metric exemplars, multi-language composition	Organizations with existing SDK instrumentation can add OBI on top

The 1.0 GA checklist itself boils down to configuration documentation complete, JSON Schema validation defined, per-service + per-process configuration supported, telemetry schema adopted, versioning policy formalized, test coverage targets hit. v0.8.0 today is roughly at 60% of that checklist; the community is targeting late-2026 for 1.0 GA.

6. OBI vs. Beyla / Pixie / Tetragon / Cilium — drawing the lines

With the 2026 eBPF observability space this crowded, positioning OBI correctly matters. Here's how it relates to the four most common adjacent projects.

Tool	Primary role	Relationship to OBI	Run side-by-side?
Grafana Beyla	HTTP/gRPC auto-tracing	OBI's direct ancestor — Grafana Cloud users can stay on Beyla, OBI is the upstream successor	New projects → OBI
Pixie (New Relic)	Auto APM + scripting (PxL)	Pixie ties to New Relic's backend, OBI is vendor-neutral via OTLP	Existing NR customers stay on Pixie; OTLP-first teams choose OBI
Cilium Tetragon	Process / file / network security + real-time enforcement (LSM)	Different role — OBI observes applications, Tetragon detects and enforces security	Run both (two DaemonSets, different jobs)
Cilium Hubble	L3/L4 network flow + service map	OBI is L7 (HTTP/gRPC) payloads; Hubble is L3/L4 packets	Run both — layered responsibility
Grafana Alloy + Pyroscope	Continuous profiling (CPU, memory)	OBI traces + RED; Pyroscope function-level profiles	Run together → trace-to-profile drilldown

In one sentence: OBI is the OTLP-standard implementation of L7 application observability. L3/L4 security belongs to Tetragon, network flows to Hubble, profiling to Pyroscope, and OBI handles application traces, RED metrics, SQL, and GenAI. That role-separated architecture has become the default CNCF observability stack in 2026.

7. Production adoption checklist (8 steps)

Step	Check	Tool / command
① Kernel	All nodes on 5.8+ with BTF enabled	`uname -r`, `ls /sys/kernel/btf/vmlinux`
② CNI collision	Audit for Cilium / Calico eBPF mode collisions	`cilium status`, OBI `network.enabled: false`
③ Privileges	DaemonSet has `CAP_BPF + CAP_PERFMON + CAP_SYS_PTRACE`	Verify against PodSecurityAdmission
④ OTLP pipeline	OTel Collector receives, backend (Tempo/Jaeger/Splunk) wired	`otelcol validate`, inspect receive metrics
⑤ Service discovery	Allowlist of namespaces / labels for instrumentation	Enforce labels via Kyverno / OPA Gatekeeper
⑥ Header enrichment	PII leakage prevented — obfuscate `authorization`, `cookie`, `api-key`	Test `obfuscation_string` + allowlist
⑦ Resource tuning	On high-traffic nodes (10k RPS+) adjust CPU / memory limits	Monitor Prometheus `obi_bpf_*` metrics
⑧ v0 instability	v0 minor releases may break — pin versions	GitOps: pin `v0.8.0`, never floating `main`

8. ManoIT production recommendations

Three-DaemonSet architecture: OBI (L7 traces) + Cilium Hubble (L3/L4) + Tetragon (security). Realistic per-node budget is about cpu: 800m / mem: 1.5Gi combined.
Progressive namespace rollout: attach the obi.enabled=true label only where you need it and expand horizontally over 1–2 weeks of observation. Rolling out to every namespace on day one will bottleneck the OTel Collector.
OTLP backend sampling: OBI is designed for 100% capture, which can explode backend storage costs. Always set tail-based sampling at the Collector (latency > 1s, all error traces, 1% of normal traces).
GenAI payload policy: the OpenAI / Anthropic instrumentation is powerful, but system prompts and user input can land in traces. Make payload_extraction.genai.redact: true the default, and switch to a whitelist-only model when you need the payloads.
Staging verification against v0: v0 minors are allowed to break. Every OBI upgrade should re-validate the full trace field mapping in staging before rolling to prod.
Header enrichment governance: every new header addition should go through security review. Add "PII included? / obfuscation needed?" checkboxes to your Git PR template.

9. Conclusion — observability moves to the kernel

OBI's beta launch is a signal that observability's center of gravity is moving from SDKs to the kernel and from vendor-specific agents to the OTLP standard. Where OpenTelemetry in the early 2020s solved "vendor-locked instrumentation code," OBI in 2026 is solving "don't write that instrumentation code in the first place." Application teams focus on business logic; platform teams deliver consistent observability at the kernel layer. That separation of responsibilities is closer than ever to what SRE and DevSecOps communities have been chasing for a decade. ManoIT recommends evaluating OBI as the default observability layer for every Kubernetes 1.28+ environment, and we plan to adopt it into our customer standard stack once 1.0 GA lands in late 2026. Grafana Labs donating Beyla, @grcevski leading the SIG, and the Splunk observability team carrying it over the KubeCon finish line — that model of community collaboration is the reason OpenTelemetry is still, in 2026, the healthiest CNCF project in the ecosystem.

This article was co-authored by the ManoIT engineering team together with Anthropic's Claude Opus 4.7, based on: OpenTelemetry eBPF Instrumentation official docs, OBI 2026 Goals, OBI HTTP Header Enrichment, GitHub opentelemetry-ebpf-instrumentation v0.8.0, the Splunk KubeCon EU 2026 announcement, and the OBI First Release Announcement.

Originally published at ManoIT Tech Blog.

DEV Community: daniel jeong

Astro 6 Deep Dive — Vite Environment API, Rust Compiler, Live Content Collections, and First-Class Cloudflare Workers

1. Why 6 Is the Inflection Point — "Content Site = Static Build" Ends

2. Astro 6 Runtime Architecture — Five-Axis Layout

2.1 Axis ① Compiler — Go to Rust, gradually

2.2 Axis ② Vite Environment API — dev = prod runtime

2.3 Axis ③ Core APIs — Fonts, CSP, Live Collections, Route Caching

2.4 Axis ④ Adapters — Cloudflare / Node / Bun / Deno

2.5 Axis ⑤ Output — static / hybrid / SSR + Live API

3. v5 → v6 Migration — The Seven Build-Breaking Changes

4. Live Content Collections — The Request-Time Loader Pattern

5. First-Class Cloudflare Workers — workerd-on-everywhere

6. Fonts and CSP API — The Last Two Pieces of the Islands Era

7. ManoIT's Four-Week Migration Checklist

8. Conclusion — Astro 6 Installs SSR as the Default for Content Sites

OpenTelemetry Profiles Public Alpha: eBPF Fourth Signal, Collector v0.151.0 and OpAMP Fleet Management for 2026

OpenTelemetry Profiles Public Alpha — How the eBPF Fourth Signal, Collector v0.151.0, and OpAMP Fleet Management Redefine Unified Observability in 2026

1. Why Profiles Is the Tipping Point — Unified OTLP Across Four Signals

2. eBPF Profiler Architecture — The Fourth Signal Lives Inside the Collector

2.1 Axis ① Agent — One eBPF Profiler Covers Many Language Runtimes

2.2 Axis ② Collector Receiver — Share the Same Pipeline as Traces and Metrics

2.3 Axis ③ K8s Enrichment — k8sattributesprocessor + container.id

2.4 Axis ④ OpAMP — Operate the Collector Fleet Without SSH

3. Collector v0.151.0 Migration — winget, Lifecycle, and send_failed

4. ManoIT 4-Week Adoption Checklist — How to Evaluate Alpha Safely

5. Trace ↔ Profile Cross-Correlation — One Click from a Span to Its Stack

6. Conclusion — "Fourth Signal + Fleet Management" Defines 2026 Observability

Falco 0.43 Deep Dive — Legacy eBPF, gVisor, gRPC Deprecation and Cosign v3 Bundles Redefining 2026 Kubernetes Runtime Security

Falco 0.43 Deep Dive — How Legacy eBPF, gVisor, and gRPC Output Deprecation, Cosign v3 Bundles, and Drop-Enter Are Redefining 2026 Kubernetes Runtime Security

1. Why 0.43 Is the Tipping Point — The New Default Born From 0.42 Drop-Enter

2. Falco 0.43 Runtime Architecture — Four-Axis Alignment

2.1 Axis ① Kernel Driver — Modern eBPF Is the Only Recommended eBPF Path

2.2 Axis ② Libraries / Drivers — Kernel 3.10 Floor

2.3 Axis ③ Rule Engine — Only .yml/.yaml Loaded

2.4 Axis ④ Outputs — Drop gRPC, Standardize on HTTP / Falcosidekick

3. falco.yaml Migration — Aligning On Modern eBPF + HTTP

4. Capture Recording — Auto .scap Dumps On Alert

5. falcoctl + Cosign v3 — Verification Across the Whole Dependency Chain

6. Rule Impact After Drop-Enter — evt.arg.filename and proc.aargs

7. Container Plugin 0.6.1 — Table API Expansion + Zero-Allocation

8. Falco Operator 0.2 — Multi-Artifact Alignment

9. ManoIT 4-Week Migration Checklist

9.1 Recommended Observability / SLOs

9.2 Five Common Pitfalls

10. Conclusion — The 2026 Default For Runtime Security Has Shifted

Amazon EKS Hybrid Nodes Gateway Deep Dive: VXLAN, Cilium VTEP, and Lease-Based Leader Election Redefining Hybrid Kubernetes Networking

Amazon EKS Hybrid Nodes Gateway Deep Dive — VXLAN, Cilium VTEP, and Lease-Based Leader Election Redefining Hybrid Kubernetes Networking in 2026

1. Why the Gateway Is an Inflection Point — From 2024 GA to 2026 Gateway

2. The Four-Axis Architecture — VPC, Gateway, VXLAN, On-Prem

2.1 Axis 1: VPC Route Table — "Send pod-CIDR traffic to the gateway ENI"

2.2 Axis 2: Gateway EC2 — Active/Standby with Lease-Based Leader Election

2.3 Axis 3: VXLAN (VNI 2 / UDP 8472) — Cilium-Compatible by Default

2.4 Axis 4: On-Prem Nodes (Cilium VTEP Decoder)

3. Prerequisites — IAM, Security Groups, CIDR Design

3.1 Gateway IAM Permissions — Scoped via IRSA

3.2 Security Groups and On-Prem Firewalls

3.3 CIDR Design — RFC-1918 / RFC-6598, Non-Overlapping

4. Deployment — One Helm Install, 30-Second Failover Test

4.1 Provision the Gateway Node Pool

4.2 Install the AWS-Maintained Cilium

4.3 Install the Gateway Components + Bind IRSA

4.4 Verify — 30-Second Failover Test

5. Traffic Matrix — Four Flows You Must Recognize

6. Operations — Metrics, Failover Scenarios, and Five Common Pitfalls

6.1 Metrics and Logs

6.2 Four Failover Scenarios

6.3 Five Common Pitfalls

7. The Same Week: ECS Capacity Reservations and NLB Canary

7.1 ECS Managed Instances + EC2 Capacity Reservations

7.2 ECS Linear/Canary Deployments on NLB

8. A 4-Week Migration Checklist

8.1 Cost Model

8.2 Recommended SLOs

9. Conclusion — Hybrid Kubernetes Has a New Default

GitLab 18.11 + Duo Agent Platform: CI Expert, Agentic SAST Auto-Resolution, Custom Flow YAML, and Credits Guardrails for AI-Native DevSecOps

GitLab 18.11 + Duo Agent Platform — CI Expert, Agentic SAST Auto-Resolution, Custom Flow YAML, and GitLab Credits Guardrails Redefining 2026 AI-Native DevSecOps

1. Why GitLab 18.11 Is the Inflection Point — Q1 2026 Duo Agent Platform Trajectory

2. Duo Agent Platform Architecture — AI Gateway, Workflow Service, Knowledge Graph, Runner

2.1 AI Gateway — One Entry for Auth, Routing, and Metering

2.2 Workflow Service — Home of Custom Flow v1

2.3 Axis ③ K8s Enrichment — `k8sattributesprocessor` + `container.id`

3. Collector v0.151.0 Migration — winget, Lifecycle, and `send_failed`

2.3 Axis ③ Rule Engine — Only `.yml`/`.yaml` Loaded

4. Capture Recording — Auto `.scap` Dumps On Alert

6. Rule Impact After Drop-Enter — `evt.arg.filename` and `proc.aargs`

3. CI Expert Agent — From an Empty `.gitlab-ci.yml` to a Working Pipeline

2. New Frontend System 1.0 RC — `--next` is Gone, `--legacy` is Born

3.1 Step 1 — Update `package.json`

3.2 Step 2 — Rename `vite.config.ts` Options