DEV Community: iapilgrim

Tutorial: Role-Based Access Control for a Node/Express API, Backed by Keycloak

iapilgrim — Tue, 21 Jul 2026 03:00:20 +0000

This is a hands-on walkthrough for kc-rbac-api — module 04 in a series
building up Keycloak features one small, runnable project at a time. By the
end you'll have a Node/Express API with three roles enforced at the
endpoint level, two interchangeable ways of validating incoming tokens, and
a full test suite proving the denials work as intended, not just the happy
path.

Everything below assumes Docker + Docker Compose, Node.js, curl, and
python3 (used by the setup script for JSON parsing — nothing else).

What you're building

An API with three roles — viewer, editor, admin — where each endpoint
enforces a specific minimum role, not just "logged in or not":

Endpoint	viewer	editor	admin
`GET /api/documents`	✅	✅	✅
`POST /api/documents`	❌	✅	✅
`PUT /api/documents/:id`	❌	✅	✅
`DELETE /api/documents/:id`	❌	❌	✅
`GET /api/admin/stats`	❌	❌	✅
`GET /api/whoami`	✅	✅	✅

Notice DELETE breaks the simple "editor and up" pattern — it's
admin-only. That's deliberate, and it's the detail worth testing carefully
later on.

Step 1 — Start Keycloak and the API

make start   # docker compose up -d --build: Keycloak + this API's container

Give Keycloak about 30 seconds to finish booting, then provision everything:

make setup

This runs keycloak/setup.sh, which — entirely through Keycloak's Admin
REST API, no clicking through the console — creates:

a realm called rbac-demo
three realm roles: viewer, editor, admin
a confidential client kc-rbac-api with an audience mapper (more on that below)
three demo users, one per role, and writes the client secret into .env

Username	Password	Role
alice	alice	viewer
bob	bob	editor
carol	carol	admin

Confirm the API is up:

curl http://localhost:3000/health
# {"status":"ok","validationMode":"local"}

Step 2 — Understand the two ways to validate a token

Every protected route sits behind one of two middlewares, and which one is
active is a config toggle (TOKEN_VALIDATION_MODE in .env) — not a code
change:

src/middleware/jwtAuth.js (local) verifies the token's RS256
signature locally, against Keycloak's JWKS endpoint (cached via
jwks-rsa), then checks iss, aud, and exp:

jwt.verify(token, getSigningKey, {
  algorithms: ["RS256"],
  issuer: config.issuer,
  audience: config.clientId,
}, (err, payload) => { /* ... */ });

Fast — one JWKS fetch, then it's pure CPU per request after that. The
tradeoff: if Keycloak revokes the token or the user logs out server-side,
this middleware has no way to know. The token stays "valid" here until it
naturally expires.

src/middleware/introspectAuth.js (introspect) instead POSTs the
token to Keycloak's /token/introspect endpoint (RFC 7662) on every
request:

const { data } = await axios.post(config.introspectionUrl, params, {
  headers: { "Content-Type": "application/x-www-form-urlencoded" },
});
if (!data.active) {
  return res.status(401).json({ error: "token_inactive", ... });
}

Slower — one extra network round trip per call — but revocation is
immediate: Keycloak reports active: false the moment a token is no
longer valid, regardless of its exp. This mode needs its own client
credentials, since introspection is itself an authenticated endpoint —
only clients that own or trust the token's audience get to ask Keycloak
about it.

Neither one is "the right answer" on its own — it's a latency-vs-freshness
tradeoff. Short access-token lifespans (this realm is set to 300 seconds)
are the usual way to bound local validation's blind spot without paying
the introspection cost on every single call.

Try switching modes and watching /api/whoami reflect it:

TOKEN_VALIDATION_MODE=introspect npm start

TOKEN=$(curl -s -X POST http://localhost:8080/realms/rbac-demo/protocol/openid-connect/token \
  -d "grant_type=password&client_id=kc-rbac-api&client_secret=$CLIENT_SECRET&username=carol&password=carol" \
  | python3 -c "import sys,json;print(json.load(sys.stdin)['access_token'])")

curl http://localhost:3000/api/whoami -H "Authorization: Bearer $TOKEN"
# "authMode": "introspect"

Step 3 — Enforce roles with two middleware shapes

src/middleware/rbac.js gives you two ways to gate a route, and picking
the right one matters:

// Exact allow-list — use when roles AREN'T hierarchical for this endpoint
requireRole("admin")

// Hierarchical — viewer(1) < editor(2) < admin(3)
requireMinRole("editor")

requireMinRole is what powers most of the table above — POST/PUT
use requireMinRole("editor") since editor-and-up should both succeed. But
DELETE uses requireRole("admin") explicitly, not
requireMinRole("admin") treated as "the top of the chain" by accident —
it's named as an exact requirement on purpose. If you ever add a role that
doesn't nest cleanly into the hierarchy (say, an auditor who can read
things editor can't but can't write anything), model it with
requireRole() allow-lists rather than forcing it into the rank table.

Roles themselves come out of the token via src/utils/tokenClaims.js,
which merges Keycloak's realm_access.roles (realm-wide roles) and
resource_access.<clientId>.roles (client-scoped roles) into one flat set
— route middleware doesn't care which bucket a role happened to live in.

Step 4 — Understand why the audience (`aud`) check exists

keycloak/setup.sh adds an oidc-audience-mapper so tokens for this
client carry aud: kc-rbac-api, and both middlewares reject anything
where that's missing:

const aud = Array.isArray(data.aud) ? data.aud : [data.aud];
if (!aud.includes(config.clientId)) {
  return res.status(401).json({ error: "invalid_audience", ... });
}

Without this check, a token minted for a different API in the same
realm — one with broader service-account scope, say — would pass
signature and issuer checks just fine and get treated as valid here too.
That's the "confused deputy" problem: a token issued for one purpose
getting replayed against a resource server it was never meant for. aud
is the check that closes that gap.

Step 5 — Run the full allow/deny matrix

make test

This runs tests/test.sh, a self-contained bash suite: it fetches real
tokens for alice/bob/carol, hits every endpoint as every role, and asserts
both the HTTP status and the JSON error code — so a 403 for the wrong
reason still fails the test.

── 6. DELETE /api/documents/:id (admin only) ──
✓ PASS  [DENY] viewer cannot delete   (got 403, error=insufficient_role)
✓ PASS  [DENY] editor cannot delete   (got 403, error=insufficient_role)
✓ PASS  [ALLOW] admin can delete      (got 204)

════════════════════════════════════════════════
  Results: 25 passed, 0 failed
════════════════════════════════════════════════

It's mode-aware: one assertion checks for invalid_token under local
validation but token_inactive under introspection, since Keycloak's
introspection endpoint doesn't distinguish "malformed" from
"merely-unrecognized" — both are a correct 401, just different
vocabulary. Exits non-zero on any failure, so it's safe to drop straight
into CI.

Prefer clicking through requests instead of reading a script? The same
matrix is in http/requests.http (VS Code REST Client — token requests
save into named variables the later requests reuse automatically) and as
a Postman collection + environment in postman/, with pm.test
assertions on every request.

Step 6 — Poke around the Keycloak Admin Console

http://localhost:8080

Username: admin / Password: admin
Realm: rbac-demo
Client kc-rbac-api → Roles tab has viewer/editor/admin → Client scopes → Dedicated scopes → Mappers has the audience mapper you just saw enforced in code

Worth clicking into a user (say, carol) and looking at Role mapping
to see the realm role assignment that ends up in realm_access.roles on
her token.

Before you take this to production

A few deliberate simplifications worth knowing about:

This demo uses Resource Owner Password Credentials (grant_type=password) purely so setup.sh, the .http file, and Postman can fetch tokens with one curl call per user. Browser-facing apps should use Authorization Code + PKCE instead — keep directAccessGrantsEnabled: false for those clients.
The in-memory documents array resets on restart. Swap it for a real datastore before this goes anywhere near real traffic.
requireMinRole assumes a strictly linear hierarchy. It works great for viewer < editor < admin, but don't force a non-nesting role into it — use requireRole() allow-lists instead.

Where the pieces live

File	Purpose
`keycloak/setup.sh`	Realm, roles, client + audience mapper, 3 demo users — all via Admin REST API
`src/middleware/jwtAuth.js`	Local JWT validation (JWKS)
`src/middleware/introspectAuth.js`	RFC 7662 introspection validation
`src/middleware/rbac.js`	`requireRole()` / `requireMinRole()`
`src/routes/documents.js`	The CRUD resource from the table above
`src/routes/admin.js`	Admin-only endpoint
`src/routes/whoami.js`	Debug endpoint — inspect resolved roles/claims for your own token
`tests/test.sh`	The 25-case bash suite from Step 5
`http/requests.http` / `postman/*.json`	Same matrix, interactive

Recap

                          ┌─────────────┐
  client ── access_token ─►  kc-rbac-api │
                          └──────┬──────┘
                                 │
                validationMode = local          validationMode = introspect
                                 │                            │
                    verify sig vs JWKS              POST /token/introspect
                    check iss/aud/exp               (client_id + client_secret)
                    read roles from claims           check active + aud
                                 │                            │
                                 └─────────► requireMinRole() / requireRole() ◄─────────┘
                                                       │
                                              200 / 201 / 204  or  403 insufficient_role

Two validation strategies, one audience check standing between them and
"any signed token gets in," and a role hierarchy that's explicit about
where it stops being hierarchical. That's the whole module.

Next up in the series: module 05 — Identity Brokering, wiring up two
Keycloak realms in one Compose stack so a "corp" realm can broker into a
"partner" realm via OIDC, without needing an actual external IdP to test
against.

Keycloak from Scratch, Part 2 — Authorization Code + PKCE, Built by Hand

iapilgrim — Mon, 20 Jul 2026 12:42:11 +0000

In Part 1 we logged in by sending a username and password directly
to Keycloak. That's fine for a script, but no browser-facing app should
ever do that — it means the app itself sees the raw password, and it
can't participate in single sign-on. What every real login button should
do instead is redirect the user to Keycloak and get a token handed
back. That's the Authorization Code flow, and for anything that can't
keep a secret (a browser app, a mobile app), it needs a companion called
PKCE.

We're going to implement it by hand — no keycloak-js, no library — so
every step is visible instead of hidden behind keycloak.login().

What we're building

A public Keycloak client (no secret — a browser can't keep one)
A plain HTML + vanilla JS page that performs the full redirect flow
A small Flask API that validates the resulting token independently, without ever talking to Keycloak per request

Prerequisites

Everything from Part 1 (Keycloak running via Docker Compose)
Python 3 with pip install flask flask-cors "pyjwt[crypto]"
A browser

1. Create a new realm and a public client

Admin Console → Create realm → pkce-demo
Clients → Create client → Client ID: pkce-spa-client → Next
Client authentication: Off — this is what makes it a public client. There's no secret to steal because there isn't one.
Capability config: check Standard flow, uncheck Direct access grants (we don't want this client falling back to Part 1's flow) → Save
Settings tab:
- Valid redirect URIs: http://localhost:5173/*
- Web origins: http://localhost:5173
Advanced tab → scroll to Proof Key for Code Exchange Code Challenge Method → set to S256

This last setting is the one that actually enforces PKCE server-side.
Without it, Keycloak will still accept a PKCE challenge if the client
sends one, but it won't require it — leaving the door open for a
client that skips PKCE entirely.

2. Create a role and a user

Realm roles → create spa-user
Users → Add user → username dave, first/last name Dave/Demo (see Part 1 if you're wondering why that matters)
Credentials tab → password dave, Temporary: Off
Role mapping → assign spa-user

3. The concept: what PKCE actually proves

A confidential client proves its identity with a secret. A public client
can't do that — so PKCE proves something different: that the app
redeeming the authorization code is the same app that started the login.

Browser                                          Keycloak
  │  code_verifier = random(64 bytes)                │
  │  code_challenge = SHA256(code_verifier)          │
  ├──── GET /auth?code_challenge=... ───────────────►│
  │◄─── redirect back with ?code=... ─────────────────┤
  ├──── POST /token                                   │
  │        code=..., code_verifier=... ──────────────►│  (recomputes SHA256(code_verifier),
  │◄─── access_token, id_token ───────────────────────┤   compares to the original challenge)

The code_verifier never leaves the browser tab until the final token
request. If an attacker somehow intercepts the redirect and grabs the
code (say, via a malicious app registered against the same custom
URL scheme on mobile), they still can't redeem it — they don't have the
verifier, and they can't derive it from the challenge, because SHA-256
doesn't run backwards.

4. Generating the challenge (app.js)

function base64UrlEncode(bytes) {
  return btoa(String.fromCharCode(...new Uint8Array(bytes)))
    .replace(/\+/g, "-").replace(/\//g, "_").replace(/=+$/, "");
}

function randomString(length = 64) {
  const arr = new Uint8Array(length);
  crypto.getRandomValues(arr);
  return base64UrlEncode(arr).slice(0, length);
}

async function sha256(input) {
  return crypto.subtle.digest("SHA-256", new TextEncoder().encode(input));
}

Nothing exotic — crypto.subtle (native to every modern browser) does
the hashing, and the verifier is just cryptographically random bytes.

5. Kicking off the login

async function login() {
  const codeVerifier = randomString(64);
  const codeChallenge = base64UrlEncode(await sha256(codeVerifier));
  const state = randomString(16);

  // This never leaves the browser until step 6.
  sessionStorage.setItem("pkce_code_verifier", codeVerifier);
  sessionStorage.setItem("pkce_state", state);

  const params = new URLSearchParams({
    client_id: "pkce-spa-client",
    redirect_uri: window.location.origin + "/index.html",
    response_type: "code",
    scope: "openid",
    state,
    code_challenge: codeChallenge,
    code_challenge_method: "S256",
  });

  window.location.href =
    `http://localhost:8080/realms/pkce-demo/protocol/openid-connect/auth?${params}`;
}

The state parameter is a separate, unrelated safety net: a random value
checked on the way back to guard against CSRF — someone tricking your
browser into completing an authorization flow you didn't start.

6. Handling the redirect back

Keycloak sends the browser back to your redirect_uri with ?code=...&state=...
in the URL. This runs on page load:

async function handleCallback(code, returnedState) {
  const expectedState = sessionStorage.getItem("pkce_state");
  if (returnedState !== expectedState) {
    throw new Error("state mismatch — possible CSRF, aborting");
  }

  const codeVerifier = sessionStorage.getItem("pkce_code_verifier");

  const body = new URLSearchParams({
    grant_type: "authorization_code",
    client_id: "pkce-spa-client",
    redirect_uri: window.location.origin + "/index.html",
    code,
    code_verifier: codeVerifier,
  });

  const resp = await fetch(
    "http://localhost:8080/realms/pkce-demo/protocol/openid-connect/token",
    { method: "POST", headers: { "Content-Type": "application/x-www-form-urlencoded" }, body }
  );

  const tokens = await resp.json();
  sessionStorage.setItem("access_token", tokens.access_token);
  sessionStorage.setItem("id_token", tokens.id_token);
  window.history.replaceState({}, document.title, window.location.origin + "/index.html");
}

Keycloak re-derives SHA256(code_verifier) server-side and compares it
to the code_challenge it received in step 5. If they match, and the
code hasn't already been redeemed or expired, you get tokens back.

7. Validating the token on the API side

The API never talks to Keycloak per request — it fetches Keycloak's
public signing keys once (JWKS) and verifies signatures locally:

from flask import Flask, request, jsonify
from flask_cors import CORS
import jwt
from jwt import PyJWKClient

ISSUER = "http://localhost:8080/realms/pkce-demo"
CLIENT_ID = "pkce-spa-client"
jwks_client = PyJWKClient(f"{ISSUER}/protocol/openid-connect/certs")

app = Flask(__name__)
CORS(app, origins=["http://localhost:5173"])

@app.route("/api/me")
def me():
    auth = request.headers.get("Authorization", "")
    token = auth.split(" ", 1)[1] if auth.startswith("Bearer ") else None
    if not token:
        return jsonify({"error": "missing bearer token"}), 401

    signing_key = jwks_client.get_signing_key_from_jwt(token)
    claims = jwt.decode(
        token, signing_key.key, algorithms=["RS256"],
        issuer=ISSUER, options={"verify_aud": False},
    )
    # Keycloak's default audience is "account", not the client id — check
    # azp (authorized party) instead, which is always the requesting client.
    if claims.get("azp") != CLIENT_ID:
        return jsonify({"error": "unexpected azp"}), 401

    return jsonify({
        "preferred_username": claims.get("preferred_username"),
        "roles": claims.get("realm_access", {}).get("roles", []),
    })

Because Keycloak signs tokens asymmetrically (RS256), the API only ever
needs the public half of the key pair — it can validate tokens without
ever holding a shared secret with Keycloak.

8. Run it

python3 api/server.py &            # :5001
python3 -m http.server 5173 &      # :5173, serving the SPA

Open http://localhost:5173, click Log in, authenticate as
dave/dave, and you should land back on the page with decoded ID
token and access token claims printed, plus a working call to
/api/me.

What actually happened here

Public client, no secret — PKCE substitutes possession of a one-time verifier for a shared secret the client can't keep anyway.
state and code_verifier are solving different problems — CSRF vs. authorization-code interception — don't conflate them.
Asymmetric JWT validation means your API and your identity provider never need to share a secret at all, unlike the client-secret approach from Part 1.

Next up, Part 3: back to server-to-server auth — client credentials
grants, service accounts, and when Direct Grant (Part 1) is and isn't
the right tool for machine identities.

Keycloak from Scratch, Part 1 — Realms, Roles, Clients & Users (No Docker Magic, Just Clicks)

iapilgrim — Mon, 20 Jul 2026 12:40:51 +0000

If you've ever inherited a Keycloak instance and had no idea what a
"realm" actually isolates, or why some clients have secrets and others
don't, this post is the ground floor. We're building one small realm by
hand, in the Admin Console — no automation scripts, no realm-export JSON
to skim past. Just enough clicking to make every concept stick.

By the end you'll have:

A dedicated realm
Three roles representing a basic permission tier (admin, editor, viewer)
A confidential client configured for direct login
Three users, each mapped to one role
A working curl command that logs in as each user and shows you their role inside a decoded JWT

Prerequisites

Docker + Docker Compose
curl and python3 (for a one-line JWT decode — no library needed)

1. Start Keycloak

# docker-compose.yml
services:
  keycloak:
    image: quay.io/keycloak/keycloak:26.7.0
    container_name: keycloak-fundamentals
    command: start-dev
    environment:
      KEYCLOAK_ADMIN: admin
      KEYCLOAK_ADMIN_PASSWORD: admin
    ports:
      - "8080:8080"

docker compose up -d

Give it about 30 seconds, then open http://localhost:8080 and log in
with admin / admin.

start-dev is a dev-mode flag — it skips TLS and some production
checks so you can iterate quickly. Never run this flag in production.

2. Create a realm

A realm is Keycloak's top-level tenancy boundary — its own users,
roles, clients, and even its own login theme, fully isolated from every
other realm on the same server. Think of it as a separate mini
directory service you can spin up per environment, per product, or per
customer.

Top-left realm dropdown → Create realm
Realm name: fundamentals
Create

3. Create three realm roles

Roles are how you express "what can this identity do" without hardcoding
usernames into your authorization logic. We're using realm roles
here — visible to every client in the realm, as opposed to client
roles, which are scoped to one specific client. (We'll hit client
roles in a later post when we build a proper RBAC API.)

Left sidebar → Realm roles → Create role
Create three, one at a time: admin, editor, viewer

4. Create a confidential client

Left sidebar → Clients → Create client
Client ID: kc-fundamentals-app → Next
Client authentication: toggle On

This is the setting that makes a client "confidential" — it gets a
secret, and Keycloak requires that secret before it'll hand out a
token. A public client (toggle off) has no secret at all, because
anything shipped to a browser or mobile app can't actually keep one
secret. We'll build a public client in Part 2.

Capability config: check Direct access grants, leave Standard flow on if you like (harmless either way for this demo) → Save
Go to the Credentials tab and copy the Client secret — you'll paste it into the curl command later

5. Create three users

Repeat this for alice, bob, and carol:

Users → Add user
Username: alice
First name: Alice, Last name: Demo

Don't skip this. Keycloak has a built-in "Update Profile" required
action that checks profile completeness dynamically, at login
time — not something you'll see if you inspect the user's stored
data beforehand. An empty first/last name silently blocks login later
with a generic "Account is not fully set up" error that has nothing
to do with the password you set. This one cost me an embarrassing
amount of debugging the first time — save yourself the trouble.

Create
Credentials tab → Set password → alice → toggle Temporary: Off → Save

"Temporary" means the user is forced to change it on next login —
fine for humans clicking through a browser flow, but it'll break a
scripted or API-driven login every time, since there's no browser to
serve the "set a new password" screen. Off, for this demo.

Role mapping tab → Assign role → pick admin

Repeat for bob → editor and carol → viewer.

6. One more thing to check: OTP

Some Keycloak versions ship the built-in direct grant authentication
flow with its conditional-OTP branch misconfigured as Required instead
of Conditional. If that's the case on your instance, every login
attempt from a user without an OTP device configured — which is all
three of ours — gets rejected before the password is even checked.

Authentication (left sidebar) → Flows tab → select direct grant
Find the OTP-related row at the top level (often labeled something like "Direct Grant - Conditional OTP")
If it's not already Conditional or Disabled, set it to Disabled

7. Log in and inspect the token

This is a Direct Access Grant (also called Resource Owner Password
Credentials, or ROPC) — the client trades a username and password
directly for a token, no browser redirect involved. It's convenient for
this fundamentals demo and for machine-to-machine scripts, but it means
the client sees the raw password, so it's a poor fit for anything
user-facing. (Part 2 covers the flow you actually want for a browser app:
Authorization Code + PKCE.)

curl -s -X POST http://localhost:8080/realms/fundamentals/protocol/openid-connect/token \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=password" \
  -d "client_id=kc-fundamentals-app" \
  -d "client_secret=PASTE_YOUR_SECRET_HERE" \
  -d "username=alice" \
  -d "password=alice" | python3 -m json.tool

If that returns an access_token, decode its payload to see the role
claim in the flesh:

curl -s -X POST http://localhost:8080/realms/fundamentals/protocol/openid-connect/token \
  -d "grant_type=password" -d "client_id=kc-fundamentals-app" \
  -d "client_secret=PASTE_YOUR_SECRET_HERE" -d "username=alice" -d "password=alice" \
  | python3 -c "
import sys, json, base64
token = json.load(sys.stdin)['access_token']
payload = token.split('.')[1]
payload += '=' * (-len(payload) % 4)
print(json.dumps(json.loads(base64.urlsafe_b64decode(payload)), indent=2))
"

Look for:

"realm_access": {
  "roles": ["admin", "default-roles-fundamentals", "offline_access", "uma_authorization"]
}

admin is the one we assigned. The other three are Keycloak's own
housekeeping roles every user gets by default.

Repeat with bob/bob and carol/carol — you should see editor
and viewer respectively.

What actually happened here

Realm isolation: fundamentals never sees or touches any other realm's users, clients, or roles — that's the whole point of realms.
Confidential client + Direct Grant: fine for this demo and for trusted server-side/CLI tooling, wrong for a browser SPA (that's Part 2).
Role claims propagate automatically: once a realm role is assigned to a user, Keycloak embeds it in every token that user gets, with zero extra configuration — this is what your API or gateway will check against later.

Troubleshooting: "Account is not fully set up"

If you hit this, it's one of exactly two causes:

OTP required in the direct grant flow (see step 6 above)
Incomplete user profile — missing first/last name (see step 5). This one is genuinely invisible from the outside: it's evaluated live at login, not stored anywhere you can query beforehand.

Check both. They're independent, and fixing only one can still leave you
stuck.

Next up, Part 2: we swap Direct Grant for the flow every real
browser app should use — Authorization Code with PKCE — and build a
vanilla-JS SPA that implements the whole thing by hand, no auth library,
so every step of the exchange is visible.

Master the ECR Lifecycle: Automating Image Cleanup

iapilgrim — Mon, 23 Mar 2026 11:29:23 +0000

Why Lifecycle Policies?

Every time your CI/CD pipeline runs docker push, you're adding roughly 200MB–1GB of data to your AWS bill. Without a policy, that data sits there forever. Lifecycle policies allow you to define rules like:

"Keep only the last 10 images."
"Delete anything older than 14 days."
"Expire untagged images immediately."

The "Perfect" Policy for Dev/Staging

For most teams, the best balance between "safety" and "savings" is a two-rule policy.

1. The "Untagged" Rule (Priority 1)

When you push a new image with the same tag (like :latest), the old image becomes "untagged." These are orphaned layers that serve no purpose. Delete them after 24 hours.

2. The "Age" Rule (Priority 2)

Delete any image that hasn't been pushed in the last 30 days. This ensures that even if you stop a project, its storage costs don't haunt you for years.

The JSON Configuration

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Cleanup orphaned/untagged images",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 1
      },
      "action": { "type": "expire" }
    },
    {
      "rulePriority": 2,
      "description": "Delete images older than 30 days",
      "selection": {
        "tagStatus": "any",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 30
      },
      "action": { "type": "expire" }
    }
  ]
}

Critical Concepts: `sinceImagePushed` vs. `imageCountMoreThan`

sinceImagePushed: Best for time-based compliance. (e.g., "We only keep 30 days of history").
imageCountMoreThan: Best for storage predictability. (e.g., "I only ever want to pay for 10 images per repo").

How to Test Without Breaking Production

The biggest fear is deleting an image that a production server might need during an auto-scaling event.

The "Lifecycle Policy Preview" is your best friend.

Navigate to your ECR Repo -> Lifecycle Policy.
Click Actions -> Create Preview.
AWS will generate a list of exactly which images would be deleted if the policy ran right now. Check this list against your current production version before hitting save.

The 24-Hour Rule

Remember: ECR policies are not instantaneous. After you click "Save," AWS schedules a background task. It usually takes 24 hours for the images to actually disappear from your console and for the storage metrics to drop in CloudWatch.

Pro-Tip: Infrastructure as Code (Terraform)

Don't click around the console for 50 repos. Add this to your Terraform module:

resource "aws_ecr_lifecycle_policy" "default" {
  repository = aws_ecr_repository.app.name

  policy = <<EOF
{
    "rules": [
        {
            "rulePriority": 1,
            "description": "Keep last 30 images",
            "selection": {
                "tagStatus": "any",
                "countType": "imageCountMoreThan",
                "countNumber": 30
            },
            "action": {
                "type": "expire"
            }
        }
    ]
}
EOF
}

Deploying a Base Sepolia Node with Docker

iapilgrim — Sat, 21 Mar 2026 07:22:07 +0000

We're using QuickNode for Base Sepolia. This post is how to setup our own node.

1. Prerequisites

Docker & Docker Compose installed.
L1 RPC Endpoint: A synced Ethereum Sepolia node (e.g., Geth + Lighthouse).
L1 Beacon Endpoint: Required for post-Canyon/Ecotone consensus.
Hardware: Minimum 16GB RAM and 1.5TB+ NVMe SSD.

2. The Automated Setup Script

We use a wrapper script to handle directory creation, JWT generation, and repository patching. This ensures your local paths are correctly mapped into the Docker containers.

Save as setup-base-sepolia.sh:

#!/usr/bin/env bash
set -e

echo "🚀 Initializing Base Sepolia Setup..."

BASE_DIR="/node-data/testnet/base-sepolia"
REPO_DIR="/opt/base-node"

# 1. Create directory layout
echo "📁 Creating data directories..."
mkdir -p ${BASE_DIR}/{op-geth,op-node,shared}

# 2. Generate the JWT Secret (The "Handshake" Key)
if [ ! -f "${BASE_DIR}/shared/jwt.txt" ]; then
  echo "🔑 Generating 32-byte hex JWT..."
  # CRITICAL: No 0x prefix, no newlines.
  openssl rand -hex 32 | tr -d "\n" > ${BASE_DIR}/shared/jwt.txt
  chmod 644 ${BASE_DIR}/shared/jwt.txt
fi

# 3. Clone the official Base Node Repo
echo "📦 Cloning base-org/node..."
if [ ! -d "${REPO_DIR}" ]; then
  git clone https://github.com/base-org/node.git ${REPO_DIR}
fi

cd ${REPO_DIR}

# 4. Create .env.sepolia
echo "⚙️ Writing environment configuration..."
cat > .env.sepolia <<EOF
# L1 Endpoints (Replace with your actual L1 IPs)
OP_NODE_L1_ETH_RPC=http://testnet:8585
OP_NODE_L1_BEACON=http://testnet-lighthouse:5052

# L2 Execution & Auth
OP_NODE_L2_ENGINE_RPC=http://execution:8551
OP_NODE_L2_ENGINE_AUTH=jwt
OP_NODE_L2_ENGINE_JWT_SECRET=/shared/jwt.txt

# Network & Sync
OP_NODE_NETWORK=base-sepolia
OP_NODE_SYNC_MODE=snap
EOF

# 5. Patch docker-compose.yml for custom volumes
echo "🛠 Patching docker-compose volumes..."
# Map op-geth and op-node to host paths
sed -i 's|${HOST_DATA_DIR}:/data|/node-data/testnet/base-sepolia/op-geth:/data|g' docker-compose.yml
# Inject the shared JWT volume into both services
sed -i '/volumes:/a \      - /node-data/testnet/base-sepolia/shared:/shared' docker-compose.yml

# 6. Start the stack
echo "🚀 Starting containers..."
docker compose --env-file .env.sepolia up -d

3. Critical Configuration Check

For the node to function, your entrypoint scripts inside the repo must use the correct environment variables.

execution-entrypoint: Ensure --authrpc.jwtsecret="$OP_NODE_L2_ENGINE_JWT_SECRET" is set.
op-node-entrypoint: Ensure --l2.jwt-secret="$OP_NODE_L2_ENGINE_JWT_SECRET" is set.
Networking: Ensure --rpc.addr=0.0.0.0 is set in both to allow external monitoring.

4. Troubleshooting & Mastery 🔍

If your node isn't syncing, check these three layers in order:

Layer 1: The JWT Handshake

If op-node cannot talk to op-geth, you will see 401 Unauthorized.

Check: docker exec execution-1 cat /shared/jwt.txt
Fix: Ensure the file is exactly 64 characters long. If it contains "true" or a file path instead of hex, your script logic is overwriting the secret.

Layer 2: L1 Connectivity

The op-node derives L2 blocks from L1 data. If L1 is down, L2 stops.

Check: docker exec node-1 curl -s http://testnet:8585
Fix: Ensure your L1 node is fully synced and reachable on the Docker network.

Layer 3: P2P Peering

In snap sync mode, you need peers to download the state.

Check: curl -s -d '{"id":1,"jsonrpc":"2.0","method":"opp2p_peerStats"}' http://localhost:9645 | jq
Fix: Ensure ports 30303 (Geth) and 9222 (Node) are open on your firewall.

5. Monitoring Your Progress

The most important command for a node operator is the Sync Status. It tells you exactly where you are in history.

The "Scoreboard" Command:

watch -n 5 'curl -s -X POST -H "Content-Type: application/json" --data "{\"jsonrpc\":\"2.0\",\"method\":\"optimism_syncStatus\",\"params\":[],\"id\":1}" http://localhost:9645 | jq ".result.local_safe_l2.number, .result.local_safe_l2.timestamp"'

First Number: Current L2 Block Height.
Second Number: Unix Timestamp of that block (compare this to the current time to see how many days/months you are behind).

Summary

Generate a clean JWT.
Align your volumes so both containers see the same /shared/jwt.txt.
Point to a healthy L1 RPC and Beacon node.
Monitor via optimism_syncStatus.

Automating Container Image Updates with FluxCD (Hands-On Tutorial)

iapilgrim — Sat, 21 Mar 2026 05:06:50 +0000

Modern GitOps workflows aim to keep your Kubernetes cluster fully synchronized with what is defined in Git. One powerful feature of Flux is Image Automation, which automatically updates container image tags in your Git repository whenever a new image becomes available.

In this tutorial, we walk through how image automation works and how to troubleshoot a common issue involving Git authentication.

1. What is Flux Image Automation?

FluxCD provides several components that work together to automate image updates.

The workflow looks like this:

Container Registry
↓
ImageRepository (scans tags)
↓
ImagePolicy (selects allowed tags)
↓
ImageUpdateAutomation (commits updates to Git)
↓
Flux Kustomization deploys updated workload

Instead of manually updating image tags in Git, Flux automatically commits the new tag when a matching image appears in your container registry.

2. Repository Structure Used in This Lab

Example GitOps layout:

flux-minikube-lab
├── apps
│   └── web-server
│       ├── web-server.yaml
│       ├── sealed-db-pass.yaml
│       └── kustomization.yaml
│
└── clusters
    └── my-cluster
        ├── image-reflect.yaml
        ├── image-policy.yaml
        ├── web-server-sync.yaml
        └── kustomization.yaml

The cluster Kustomization references infrastructure and application sync resources.

3. Application Deployment

The web server manifests are defined in:

apps/web-server/

Example kustomization.yaml:

resources:
  - web-server.yaml
  - sealed-db-pass.yaml

A Flux Kustomization sync object deploys the app:

kind: Kustomization
metadata:
  name: web-server-sync
  namespace: flux-system
spec:
  interval: 1m
  path: ./apps/web-server
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: engineering

This instructs Flux to deploy the web server into the engineering namespace.

4. Image Automation Setup

Three resources enable automated updates:

Image Repository

This scans container registry tags.

kind: ImageRepository

Image Policy

Defines which tags are acceptable (e.g., semver or latest).

kind: ImagePolicy

Image Update Automation

Updates the Git repo when a new tag matches the policy.

kind: ImageUpdateAutomation

5. Triggering Automation

To manually trigger automation:

flux reconcile image update flux-system

Flux will:

Check registry tags
Apply the image policy
Update manifests in Git
Commit and push the change

6. Troubleshooting a Common Error

While testing automation, the following error appeared:

failed to update source:
failed to push to remote:
ERROR: The key you are authenticating with has been marked as read only

Checking the automation status:

flux get image update

Output:

READY: False
MESSAGE: failed to push to remote

7. Root Cause

ImageUpdateAutomation needs write access to the Git repository to commit updated image tags.

However, the repository authentication key was configured as read-only.

This allowed Flux to pull manifests but prevented it from pushing updates.

8. Fix: Recreate the Flux Git Secret with Write Access

Step 1 — Generate a New SSH Key

ssh-keygen

Step 2 — Add the Public Key to GitHub

Navigate to the repository settings:

Settings → Deploy Keys

Add the new public key and enable:

Allow write access

Step 3 — Recreate the Flux Git Secret

flux create secret git flux-system \
  --url=ssh://git@github.com/pilgrim2go/flux-minikube-lab \
  --private-key-file=$PWD/fluxcd-test \
  -n flux-system

This updates the authentication credentials used by Flux.

Step 4 — Re-run Automation

flux reconcile image update flux-system

Verify:

flux get image update

Expected result:

READY: True
MESSAGE: committed and pushed update

A new commit should appear in the repository with the updated image tag.

9. Helpful Debug Commands

Check Flux resources:

flux get all

Inspect automation status:

flux get image update

View controller logs:

kubectl logs -n flux-system deploy/image-automation-controller

Trigger Git sync:

flux reconcile source git flux-system

10. Best Practices

When using Flux image automation:

• Use a dedicated deploy key or bot account
• Ensure Git credentials allow read and write access
• Keep Git as the single source of truth
• Avoid manual cluster changes outside GitOps

Conclusion

Flux image automation eliminates manual image updates and ensures that your Kubernetes workloads always run the latest approved container images.

With the proper Git authentication setup, Flux can automatically:

• Detect new container images
• Update manifests in Git
• Trigger GitOps deployments

This enables a fully automated and auditable Kubernetes delivery pipeline.

FluxCD Image Automation Error Troubleshooting

iapilgrim — Sat, 21 Mar 2026 05:05:02 +0000

Problem

Running:

flux reconcile image update flux-system

Result:

failed to update source: failed to push to remote
ERROR: The key you are authenticating with has been marked as read only

And:

flux get image update

shows:

READY: False
MESSAGE: failed to push to remote

Root Cause

FluxCD ImageUpdateAutomation needs to commit and push updates to the Git repository when it updates container image tags.

Pipeline:

Container Registry
        ↓
ImageRepository
        ↓
ImagePolicy
        ↓
ImageUpdateAutomation
        ↓
Git Commit + Push
        ↓
Flux Kustomization deploys update

If the Git credential is read-only, the push fails.

Diagnosis Steps

1. Check Image Automation Status

flux get image update

Look for:

READY: False
failed to push to remote

2. Inspect the Automation Object

kubectl get imageupdateautomation -A

Example:

STATUS: failed to update source

3. Check the Git Source

flux get sources git

This confirms Flux can read the repo.

But pushing still fails if the key is read-only.

4. Confirm Git Authentication

Check the Git secret:

kubectl get secret flux-system -n flux-system

This secret contains the SSH key Flux uses.

Fix Implemented

You recreated the Git authentication secret with a write-enabled SSH key.

1️⃣ Generate SSH Key

ssh-keygen

2️⃣ Add Public Key to GitHub

Go to repo:

Settings → Deploy Keys

Add:

fluxcd-test.pub

Enable:

Allow write access

3️⃣ Recreate Flux Git Secret

flux create secret git flux-system \
  --url=ssh://git@github.com/pilgrim2go/flux-minikube-lab \
  --private-key-file=$PWD/fluxcd-test \
  -n flux-system

This updates the Git credential used by Flux.

4️⃣ Trigger Automation

flux reconcile image update flux-system

Expected Result

flux get image update

READY: True
MESSAGE: committed and pushed update

You should also see a commit in Git like:

flux: update image tag

Useful Debug Commands

Check full Flux status:

flux get all

Check automation logs:

kubectl logs -n flux-system deploy/image-automation-controller

Test Git sync:

flux reconcile source git flux-system

Best Practice

Use a dedicated Flux deploy key with:

read + write

instead of personal access tokens when using SSH Git repositories.

FluxCD journey with Minikube

iapilgrim — Fri, 20 Mar 2026 14:33:52 +0000

🚀 Phase 1: The Manual Foundation

Goal: Set up the cluster and deploy a "Hello World" app the old-fashioned way to understand what we are automating.

🛠️ Step 1: Install Tools

# Install the Big Three (macOS example)
brew install minikube kubectl fluxcd/tap/flux

🏗️ Step 2: Start Minikube

minikube start --cpus 2 --memory 4096 --driver=docker
minikube addons enable ingress

📂 Step 3: Directory Layout

Create this structure on your local machine:

flux-lab/
└── base/
    ├── kustomization.yaml
    └── web-server.yaml

📄 Step 4: The Manifests

flux-lab/base/web-server.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-server
  namespace: engineering
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80

flux-lab/base/kustomization.yaml

resources:
  - web-server.yaml

🚀 Step 5: Deploy Manually

kubectl create namespace engineering
kubectl apply -k flux-lab/base/
kubectl get pods -n engineering

🤖 Phase 2: The Great Automation (FluxCD)

Goal: Connect GitHub to Minikube. From this point on, we never use kubectl apply again.

🛠️ Step 1: Environment Setup

export GITHUB_TOKEN=your_personal_access_token
export GITHUB_USER=your_github_username

🏗️ Step 2: Bootstrap Flux

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=flux-minikube-lab \
  --branch=main \
  --path=clusters/my-cluster \
  --personal

📂 Step 3: Final Git Directory Layout

Clone your new repo and organize it exactly like this:

flux-minikube-lab/
├── apps/
│   └── web-server/
│       ├── kustomization.yaml
│       └── web-server.yaml
└── clusters/
    └── my-cluster/
        ├── flux-system/         # (Auto-generated)
        └── web-server-sync.yaml

📄 Step 4: Create the "Sync" Instruction

clusters/my-cluster/web-server-sync.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: web-server-sync
  namespace: flux-system
spec:
  interval: 1m
  path: ./apps/web-server
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  targetNamespace: engineering

🚀 Step 5: Push and Pray (The GitOps Way)

git add .
git commit -m "Onboard web-server to GitOps"
git push origin main

# Force immediate sync
flux reconcile kustomization flux-system --with-source

🔐 Phase 3: The Secret Sauce (Sealed Secrets)

Goal: Store passwords in GitHub securely using encryption.

🏗️ Step 1: Install Infrastructure

Place these files in infrastructure/sources/ and infrastructure/controllers/.

clusters/my-cluster/infra-sync.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-sync
  namespace: flux-system
spec:
  interval: 1h
  path: ./infrastructure
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

📂 Step 2: Final Phase 3 Directory Layout

flux-minikube-lab/
├── apps/
│   └── web-server/
│       ├── kustomization.yaml   # (Update to include sealed-db-pass.yaml)
│       ├── web-server.yaml
│       └── sealed-db-pass.yaml  # (Generated)
├── clusters/
│   └── my-cluster/
│       ├── infra-sync.yaml
│       └── web-server-sync.yaml
└── infrastructure/
    ├── controllers/
    │   └── sealed-secrets.yaml
    └── sources/
        └── sealed-secrets.yaml

🔐 Step 3: Create an Encrypted Secret

# 1. Create a raw secret (DO NOT PUSH TO GIT)
kubectl create secret generic mwd-db-pass \
  --from-literal=password=SuperSecret123 \
  --namespace engineering \
  --dry-run=client -o yaml > temp.yaml

# 2. Encrypt it using the cluster's key
kubeseal \
  --controller-name sealed-secrets \
  --controller-namespace flux-system \
  --format yaml < temp.yaml > apps/web-server/sealed-db-pass.yaml

# 3. Clean up
rm temp.yaml

🚀 Step 4: Deploy

git add .
git commit -m "Add sealed secret"
git push origin main
flux reconcile kustomization infra-sync --with-source
flux reconcile kustomization web-server-sync --with-source

🧠 Summary of Progress

Phase 1: Learned Kubernetes resources.
Phase 2: Learned FluxCD automation and the "Pull Model."
Phase 3: Learned Security and encryption in Git.

🧠 Migrating BSC Testnet Data to a New Disk (KVM + Docker)

iapilgrim — Thu, 19 Mar 2026 03:44:50 +0000

Running blockchain nodes at scale quickly turns into a storage management problem.
In this guide, I’ll walk through a real-world migration of a BSC testnet dataset (~243GB) to a new disk — without breaking the node — and share key lessons learned.

📊 Initial Situation

Inside the VM:

/dev/vdb1 → /node-data (6.3T disk, almost full)

Breakdown:

Mainnet: ~5.8TB
Testnet: ~243GB

Problem:

Disk usage: 100%
Free space: ~41GB ⚠️

👉 Risk: node crash, DB corruption, sync failure

🎯 Goal

Move testnet data to a new disk
Keep mainnet untouched
Avoid complex Docker changes
Minimize downtime

🏗️ Architecture Before

vdb (6.3T)
└── /node-data
    ├── mainnet (~5.8T)
    └── testnet (~243G)

🚀 Architecture After

vdb → mainnet
vdc → testnet

/bsc-testnet → real mount
/node-data/testnet → bind mount → /bsc-testnet

Docker still uses:

/node-data/testnet:/bsc/node

👉 No container config change needed.

⚡ Step-by-Step Migration

1️⃣ Attach new disk (on host)

qemu-img create -f qcow2 /path/bsc-testnet.qcow2 400G

virsh attach-disk bsc \
  /path/bsc-testnet.qcow2 \
  vdc \
  --targetbus virtio \
  --subdriver qcow2 \
  --persistent

2️⃣ Prepare disk (inside VM)

Skip partitioning (simpler):

mkfs.ext4 /dev/vdc
mkdir /bsc-testnet
mount /dev/vdc /bsc-testnet

3️⃣ Copy data

rsync -avh /node-data/testnet/ /bsc-testnet/

4️⃣ Switch using bind mount (no Docker change)

docker stop testnet

mv /node-data/testnet /node-data/testnet-old
mkdir /node-data/testnet

mount --bind /bsc-testnet /node-data/testnet

docker start testnet

5️⃣ Cleanup (after verification)

rm -rf /node-data/testnet-old

6️⃣ Make persistent

Edit /etc/fstab:

UUID=<vdc-uuid> /bsc-testnet ext4 defaults 0 0
/bsc-testnet /node-data/testnet none bind 0 0

Test:

mount -a

🔥 Key Lessons Learned

1️⃣ You don’t need to touch Docker

Instead of:

editing volumes
recreating containers

👉 Just move the filesystem underneath

This is safer and faster.

2️⃣ Bind mounts are extremely powerful

/node-data/testnet → /bsc-testnet

Acts like:

transparent redirect
zero config change
instant rollback

3️⃣ “target is busy” = you are inside the directory

Classic mistake:

umount /node-data/testnet
→ target is busy

Cause:

cwd = /node-data/testnet

Fix:

cd /

4️⃣ Duplicate mounts can happen easily

Running mount --bind multiple times creates stacked mounts.

Check with:

mount | grep testnet
findmnt /node-data/testnet

Fix:

umount /node-data/testnet (repeat until gone)

5️⃣ Never unmount while container is running

Even if it “works”:

DB writes can fail
corruption risk
node may resync from scratch

👉 Always:

docker stop → umount → remount → start

6️⃣ Partitioning is optional

For dedicated disks:

mkfs.ext4 /dev/vdc

is simpler than:

fdisk → /dev/vdc1

7️⃣ Blockchain nodes will ALWAYS outgrow your disk

After migration:

mainnet still ~92% full

👉 This is temporary relief.

You must plan:

disk expansion (qemu-img resize)
or data rebalancing

📈 Results

After migration:

Freed ~243GB on main disk
Isolated IO between mainnet/testnet
No Docker reconfiguration
Minimal downtime (~1–2 minutes)

🧠 Final Takeaway

The key mindset shift:

Don’t move applications — move the filesystem underneath them.

This approach:

reduces risk
simplifies operations
scales better for large datasets (TBs)

🚀 What’s Next

For production-grade setups:

Separate disks per chain
Use virsh blockcopy for live migration
Implement storage balancing strategy across NVMe pools

vLLM Request Lifecycle (Where TTFT is measured)

iapilgrim — Wed, 11 Mar 2026 13:12:54 +0000

Observing LLM Latency: Monitoring Time-To-First-Token in vLLM

Large language model APIs often feel fast even when generating long responses.
The key reason is streaming tokens — the response starts quickly while the rest of the answer is still being generated.

The most important metric for this experience is Time-To-First-Token (TTFT).

This tutorial explains:

what TTFT is
how vLLM exposes TTFT metrics
how Prometheus calculates percentiles
how to build a Grafana dashboard

What is Time-To-First-Token (TTFT)?

TTFT measures how long it takes from request arrival to the first generated token.

LLM inference pipeline:

Request arrives
      ↓
Prompt tokenization
      ↓
Prefill (model processes prompt)
      ↓
FIRST TOKEN GENERATED  ← TTFT measured here
      ↓
Token streaming
      ↓
Response complete

Users perceive the system as fast if TTFT is small.

Typical targets:

System	Good TTFT
Chat UI	<2s
GPU inference	<4s
Heavy batching	<8s

vLLM TTFT Metric

vLLM exports a Prometheus histogram metric:

vllm:time_to_first_token_seconds_bucket

Example metrics:

vllm:time_to_first_token_seconds_bucket{le="1"} 1
vllm:time_to_first_token_seconds_bucket{le="2.5"} 73
vllm:time_to_first_token_seconds_bucket{le="10"} 103
vllm:time_to_first_token_seconds_bucket{le="20"} 104

These represent histogram buckets.

Example interpretation:

TTFT	Requests
≤1s	1
≤2.5s	73
≤10s	103
≤20s	104

Buckets are cumulative counters.

Common Causes of High TTFT

If TTFT increases, check:

GPU saturation

Too many requests in queue.

Large prompts

Prefill phase takes longer.

Batch scheduling delay

Large batch sizes increase wait time.

KV cache limits

Cache misses slow inference.

Recommended vLLM Observability Metrics

Monitor these together:

Metric	Meaning
TTFT	time until response starts
tokens/sec	generation speed
request latency	full response time
active requests	queue pressure
GPU utilization	hardware bottleneck

Summary

Monitoring TTFT helps maintain a responsive LLM system.

Pipeline:

vLLM metrics
     ↓
Prometheus histogram
     ↓
PromQL percentile query
     ↓
Grafana visualization

With proper monitoring, you can detect:

GPU saturation
batching issues
prompt bottlenecks

before users notice degraded performance.

Troubleshooting: Fixing k6 GPG Key Errors in Ubuntu

iapilgrim — Wed, 11 Mar 2026 03:41:07 +0000

Setting up k6 on Ubuntu should be a straightforward process, but GPG key errors can occasionally turn a "five-minute task" into a troubleshooting marathon. If you’ve encountered the dreaded NO_PUBKEY or unsupported filetype errors, this guide will walk you through the fix and the proper installation.

Modern Linux distributions have moved away from apt-key add (which is now deprecated) toward specific keyrings located in /usr/share/keyrings. This "sandboxes" the security keys so that the k6 key can only be used to verify k6 packages, making your overall system much more secure.

🛠️ Troubleshooting: Fixing k6 GPG Key Errors

When you see errors like "The following signatures couldn't be verified" or "unsupported filetype," it means your system doesn't trust the k6 repository or the key file you downloaded is corrupted (usually saved as plain text instead of binary).

Step 1: Clean Up Old Keys

If you have a broken keyring file, apt will continue to complain until it's gone. Start by removing any existing k6 keyring:

sudo rm /usr/share/keyrings/k6-archive-keyring.gpg

Step 2: Download and "Dearmor" the Key

The key must be in a binary format for modern Ubuntu versions. We use gpg --dearmor to convert the text-based key from k6 into the required binary format.

curl -fsSL https://dl.k6.io/key.gpg | sudo gpg --dearmor -o /usr/share/keyrings/k6-archive-keyring.gpg

Step 3: Set Correct Permissions

System services need to be able to read this file. Set the permissions to 644:

sudo chmod 644 /usr/share/keyrings/k6-archive-keyring.gpg

📦 Setting Up the Repository

Now that the security "handshake" is ready, you need to tell Ubuntu exactly where to find k6 and which key to use for verification.

Step 4: Add the k6 Source List

Run this command to create (or overwrite) the source file. Note the signed-by tag—this is the secret sauce that links the repo to the key we just created.

echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list

🚀 Final Installation

With the repository correctly configured and the key verified, you can now update your package list and install k6.

Step 5: Update and Install

sudo apt-get update
sudo apt-get install k6

Step 6: Verify Success

Confirm k6 is installed and ready to go by checking the version:

k6 version

Overlay vs Underlay Networking Across Regions with Azure Kubernetes Service

iapilgrim — Mon, 09 Mar 2026 03:17:28 +0000

In this tutorial we build a complete AKS networking lab to demonstrate:

AKS Overlay networking
AKS Underlay (Azure CNI) networking
Cross-region VNet peering
Pod-to-pod communication across clusters
Access to Azure SQL via Private Endpoint
Service exposure via NodePort

Final architecture:

Region 1 (Southeast Asia)
VNet: 10.0.0.0/16
--------------------------------

AKS Overlay Cluster
Nodes: 10.0.x.x
Pods : 192.168.x.x

Private Endpoint
Azure SQL
10.0.4.4

Private DNS Zone
privatelink.database.windows.net

Region 2 (East Asia)
VNet: 10.1.0.0/16
--------------------------------

AKS Underlay Cluster
Nodes: 10.1.x.x
Pods : 10.1.x.x

VNet Peering
10.0.0.0/16  <---->  10.1.0.0/16

1. Create the Overlay AKS Cluster

Overlay networking allows pods to use non-VNet IP ranges, which prevents subnet exhaustion.

Create resource group

RG_NETWORK=rg-aks-network
LOCATION=southeastasia

az group create \
  --name $RG_NETWORK \
  --location $LOCATION

Create VNet

VNET_NAME=vnet-aks-overlay

az network vnet create \
  --resource-group $RG_NETWORK \
  --name $VNET_NAME \
  --address-prefix 10.0.0.0/16 \
  --subnet-name aks-subnet \
  --subnet-prefix 10.0.1.0/24

Create AKS overlay cluster

AKS_OVERLAY=aks-overlay

SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RG_NETWORK \
  --vnet-name $VNET_NAME \
  --name aks-subnet \
  --query id -o tsv)

az aks create \
  --resource-group $RG_NETWORK \
  --name $AKS_OVERLAY \
  --location $LOCATION \
  --node-count 2 \
  --network-plugin azure \
  --network-plugin-mode overlay \
  --pod-cidr 192.168.0.0/16 \
  --vnet-subnet-id $SUBNET_ID \
  --generate-ssh-keys

Connect kubectl

az aks get-credentials \
  --resource-group $RG_NETWORK \
  --name $AKS_OVERLAY \
  --context aks-overlay

Verify overlay pod networking

Deploy a test pod.

kubectl run overlay-test \
  --image=busybox \
  --restart=Never \
  -- sleep 3600

Check pod IP:

kubectl get pods -o wide

Example:

overlay-test   192.168.0.4

Overlay pods use 192.168.x.x, not VNet IPs.

2. Create Azure SQL with Private Endpoint

Create SQL server:

SQL_SERVER=aks-lab-sql-31445

az sql server create \
  --name $SQL_SERVER \
  --resource-group $RG_NETWORK \
  --location $LOCATION \
  --admin-user sqladmin \
  --admin-password <password>

Create private endpoint

az network private-endpoint create \
  --resource-group $RG_NETWORK \
  --name sql-pe \
  --vnet-name $VNET_NAME \
  --subnet aks-subnet \
  --private-connection-resource-id $(az sql server show --name $SQL_SERVER --resource-group $RG_NETWORK --query id -o tsv) \
  --group-id sqlServer \
  --connection-name sql-pe-connection

Create Private DNS zone

az network private-dns zone create \
  --resource-group $RG_NETWORK \
  --name privatelink.database.windows.net

Link overlay VNet

az network private-dns link vnet create \
  --resource-group $RG_NETWORK \
  --zone-name privatelink.database.windows.net \
  --name link-overlay-vnet \
  --virtual-network $VNET_NAME \
  --registration-enabled false

Test SQL access from overlay pod

kubectl exec -it overlay-test -- sh

nslookup <sql-server>.database.windows.net

Expected:

10.0.x.x

3. Create Underlay AKS Cluster in Another Region

Underlay clusters assign real VNet IPs to pods.

Create resource group

RG_UNDERLAY=rg-aks-underlay
UNDERLAY_LOCATION=eastasia

az group create \
  --name $RG_UNDERLAY \
  --location $UNDERLAY_LOCATION

Create VNet

VNET_UNDERLAY=vnet-aks-underlay

az network vnet create \
  --resource-group $RG_UNDERLAY \
  --name $VNET_UNDERLAY \
  --location $UNDERLAY_LOCATION \
  --address-prefix 10.1.0.0/16 \
  --subnet-name aks-subnet \
  --subnet-prefix 10.1.1.0/24

Create underlay AKS cluster

SUBNET_UNDERLAY=$(az network vnet subnet show \
  --resource-group $RG_UNDERLAY \
  --vnet-name $VNET_UNDERLAY \
  --name aks-subnet \
  --query id -o tsv)

az aks create \
  --resource-group $RG_UNDERLAY \
  --name aks-underlay \
  --location $UNDERLAY_LOCATION \
  --node-count 2 \
  --network-plugin azure \
  --vnet-subnet-id $SUBNET_UNDERLAY \
  --service-cidr 172.17.0.0/16 \
  --dns-service-ip 172.17.0.10 \
  --generate-ssh-keys

Connect kubectl

az aks get-credentials \
  --resource-group $RG_UNDERLAY \
  --name aks-underlay \
  --context aks-underlay

Verify pod IP

kubectl --context aks-underlay run underlay-test \
  --image=busybox \
  --restart=Never \
  -- sleep 3600

Check IP:

10.1.x.x

Pods now consume VNet IPs.

4. Peer the VNets

Overlay and underlay clusters must communicate.

Overlay → Underlay

az network vnet peering create \
  --resource-group $RG_NETWORK \
  --name overlay-to-underlay \
  --vnet-name $VNET_NAME \
  --remote-vnet /subscriptions/<sub>/resourceGroups/$RG_UNDERLAY/providers/Microsoft.Network/virtualNetworks/$VNET_UNDERLAY \
  --allow-vnet-access

Underlay → Overlay

az network vnet peering create \
  --resource-group $RG_UNDERLAY \
  --name underlay-to-overlay \
  --vnet-name $VNET_UNDERLAY \
  --remote-vnet /subscriptions/<sub>/resourceGroups/$RG_NETWORK/providers/Microsoft.Network/virtualNetworks/$VNET_NAME \
  --allow-vnet-access

5. Link Private DNS to Underlay VNet

Without this step, the underlay cluster resolves public SQL endpoints.

az network private-dns link vnet create \
  --resource-group $RG_NETWORK \
  --zone-name privatelink.database.windows.net \
  --name link-underlay-vnet \
  --virtual-network /subscriptions/<sub>/resourceGroups/$RG_UNDERLAY/providers/Microsoft.Network/virtualNetworks/$VNET_UNDERLAY \
  --registration-enabled false

6. Communication Testing Scenarios

Scenario 1 — Overlay Pod → Underlay Pod

Get underlay pod IP:

10.1.1.18

From overlay pod:

kubectl --context aks-overlay exec -it overlay-test -- sh

ping 10.1.1.18

Success proves cross-region VNet routing works.

Scenario 2 — Underlay Pod → Azure SQL Private Endpoint

Enter underlay pod:

kubectl exec -it underlay-test -- sh

nslookup <sql-server>.database.windows.net

Result:

10.0.4.4

Test SQL port:

nc -zv <sql-server>.database.windows.net 1433

Expected:

Connection succeeded

Scenario 3 — Overlay Pod → Underlay Service

Deploy nginx in underlay cluster:

kubectl create deployment nginx --image=nginx

Expose via NodePort:

kubectl expose deployment nginx \
  --type NodePort \
  --port 80 \
  --name nginx-nodeport

Check NodePort:

80:31904/TCP

Get node IP:

10.1.1.33

Test from overlay pod:

kubectl --context aks-overlay exec -it overlay-test -- sh

wget -qO- http://10.1.1.33:31904

Result:

Welcome to nginx!

Key Networking Differences

Feature	Overlay	Underlay
Pod IP	Overlay CIDR	VNet subnet
VNet IP usage	No	Yes
Subnet exhaustion risk	Low	High
Direct VNet reachability	No	Yes

Final Result

This lab demonstrates:

Overlay networking
Underlay networking
Cross-region VNet peering
AKS pod communication
Azure SQL Private Endpoint connectivity
Kubernetes service exposure

DEV Community: iapilgrim

Tutorial: Role-Based Access Control for a Node/Express API, Backed by Keycloak

What you're building

Step 1 — Start Keycloak and the API

Step 2 — Understand the two ways to validate a token

Step 3 — Enforce roles with two middleware shapes

Step 4 — Understand why the audience (aud) check exists

Step 5 — Run the full allow/deny matrix

Step 6 — Poke around the Keycloak Admin Console

Before you take this to production

Where the pieces live

Recap

Keycloak from Scratch, Part 2 — Authorization Code + PKCE, Built by Hand

What we're building

Prerequisites

1. Create a new realm and a public client

2. Create a role and a user

3. The concept: what PKCE actually proves

4. Generating the challenge (app.js)

5. Kicking off the login

6. Handling the redirect back

7. Validating the token on the API side

8. Run it

What actually happened here

Keycloak from Scratch, Part 1 — Realms, Roles, Clients & Users (No Docker Magic, Just Clicks)

Prerequisites

1. Start Keycloak

2. Create a realm

3. Create three realm roles

4. Create a confidential client

5. Create three users

6. One more thing to check: OTP

7. Log in and inspect the token

What actually happened here

Troubleshooting: "Account is not fully set up"

Master the ECR Lifecycle: Automating Image Cleanup

Why Lifecycle Policies?

The "Perfect" Policy for Dev/Staging

1. The "Untagged" Rule (Priority 1)

2. The "Age" Rule (Priority 2)

The JSON Configuration

Critical Concepts: sinceImagePushed vs. imageCountMoreThan

How to Test Without Breaking Production

The 24-Hour Rule

Pro-Tip: Infrastructure as Code (Terraform)

Deploying a Base Sepolia Node with Docker

1. Prerequisites

2. The Automated Setup Script

3. Critical Configuration Check

4. Troubleshooting & Mastery 🔍

Layer 1: The JWT Handshake

Layer 2: L1 Connectivity

Layer 3: P2P Peering

5. Monitoring Your Progress

Summary

Automating Container Image Updates with FluxCD (Hands-On Tutorial)

1. What is Flux Image Automation?

2. Repository Structure Used in This Lab

3. Application Deployment

4. Image Automation Setup

Image Repository

Image Policy

Image Update Automation

5. Triggering Automation

6. Troubleshooting a Common Error

7. Root Cause

8. Fix: Recreate the Flux Git Secret with Write Access

Step 1 — Generate a New SSH Key

Step 2 — Add the Public Key to GitHub

Step 3 — Recreate the Flux Git Secret

Step 4 — Re-run Automation

9. Helpful Debug Commands

10. Best Practices

Conclusion

FluxCD Image Automation Error Troubleshooting

Problem

Root Cause

Diagnosis Steps

1. Check Image Automation Status

2. Inspect the Automation Object

Step 4 — Understand why the audience (`aud`) check exists

Critical Concepts: `sinceImagePushed` vs. `imageCountMoreThan`