DEV Community: ChungWei Wei

[AI] LM Studio Couldn't Run This Model, So I Got It Running On My Own Mac

ChungWei Wei — Sun, 27 Sep 2026 17:02:25 +0000

Situation

Been Looking At New Models Lately, And Came Across Muse Glimmer, A 30B Model That’s Supposed To Be Really Good For Personal Use. Wanted To See If It Could Replace Gemma-4 As My Log-Summarizer Backend, Plus A Similar Role On My Personal Assistant. LM Studio Was Already Running On This Machine, So I Just Tried Loading It There. It Failed Immediately:

No module named 'mlx_vlm.speculative.drafters.muse_glimmer'

After Actually Checking, LM Studio’s Bundled MLX Runtime Was Already The Latest On Both Stable And Beta Channels. There Was No Update Coming To Fix This. The Model Was Just Too New For What LM Studio Was Shipping.

Result First:


Trigger	Wanted To Test Whether A New 30B MLX Model Could Replace Two Existing Local Models
Root Cause	LM Studio’s Bundled MLX Runtime Doesn’t Support The New Architecture Yet, No Update Available
Fix	Raw `mlx_vlm` Pip Package’s In-Process API, Wrapped In A Small OpenAI-Compatible Shim Server
Outcome	Model Now Serves Both Consumers Through The Same `/v1/chat/completions` Contract They Already Spoke To LM Studio With

The Runtime Gap Turned Out To Be The Easy Part. The Real Surprise Was Finding Out MLX Itself Can’t Run Anywhere Except On The Machine In Front Of Me.

How It Broke (Root Cause)

The Runtime Is Frozen, The Model Isn’t

LM Studio Ships Its Own MLX Runtime As A Versioned Package (mlx-llm-mac-arm64-apple-metal-advsimd, Currently 1.11.0). New Model Architectures On Hugging Face Show Up Faster Than LM Studio Can Ship Runtime Updates For Them. Muse Glimmer Uses A Speculative-Decoding Module Layout The Bundled Runtime’s Loader Simply Doesn’t Know About. Not A Config Problem, Not A Quantization Mismatch, Just Code That Isn’t There Yet.

I Confirmed This Wasn’t Something I Could Fix By Updating LM Studio: Checked lms runtime ls Against Both The Stable And Beta Channels, Same Version On Both. Nothing To Update To.

Wanting To Centralize It Ran Into A Harder Wall

My First Instinct Was To Run This As A Service On My Home Kubernetes Cluster (rke2-home) So It’d Be Managed The Same Way As Everything Else. Checked The Node List:

rke-m01 / rke-n01–n05 all Ubuntu 20.04 x86 Linux

No Apple Silicon Anywhere. MLX Is An Apple Silicon + Metal Framework. It Doesn’t Run On Linux At All, And Critically, It Doesn’t Run Inside A Docker Container Even On A Mac, Because Docker Desktop For Mac Runs A Linux VM Underneath With No Metal GPU Passthrough. This Isn’t A Performance Tradeoff You Can Configure Around. If You Want MLX, It Runs As A Native macOS Process On Apple Silicon, Full Stop.

That Ruled Out The Tidy Kubernetes Answer. The Model Had To Live On The Mac, Which Meant The “Service” Part Had To Come From macOS’s Own Tools.

The Fix

Skip The Runtime, Use The Library Directly

The Raw mlx_vlm Pip Package (0.7.1) Supports Architectures The Bundled LM Studio Runtime Doesn’t. It’s The Same Underlying Library LM Studio Uses, Just Not Frozen At An Old Version. Its In-Process API Is Exactly What I Needed:

from mlx_vlm import load, generate

model, processor = load("mlx-community/Muse-Glimmer-30B-4bit") # load once
result = generate(model, processor, prompt, max_tokens=4096, temperature=0.2)

load() Only Needs To Run Once. Everything After That Is A generate() Call Against An Already-Resident Model, Which Is The Whole Trick For Turning This Into A Server Instead Of A CLI That Reloads 19GB Of Weights Every Invocation.

Wrap It So Nothing Downstream Has To Change

Both Consumers (A Go Service And A Node-Based Personal Assistant) Already Spoke To LM Studio Through The Same Contract: POST /v1/chat/completions, OpenAI Request/Response Shape. So The Shim Just Had To Speak That Same Contract:

@app.post("/v1/chat/completions")
async def chat_completions(request: Request):
    body = await request.json()
    history = to_mlx_history(body["messages"]) # OpenAI shape -> mlx_vlm's chat history shape
    prompt = apply_chat_template(processor, model.config, history, num_images=0)
    result = await anyio.to_thread.run_sync(lambda: generate(
        model, processor, prompt,
        max_tokens=body.get("max_tokens", 4096),
        temperature=body.get("temperature", 0.2),
    ))
    return {"choices": [{"message": {"role": "assistant", "content": result.text}}]}

Neither Client Needed A Single Line Changed. They Just Point At A Different Port.

Make It Actually Stay Up

launchd Handles The “This Needs To Survive A Crash Or A Reboot” Part macOS-Natively. No Docker, No Extra Layer:

<key>KeepAlive</key><true/>
<key>RunAtLoad</key><true/>

Killed The Process With kill -9 To Check This Actually Works Instead Of Trusting The Plist. New PID Came Up Within Seconds, Model Reloaded, /health Went Green Again.

One Serialization Detail That Matters

MLX Generation Isn’t Safe To Call Concurrently Against The Same Model State. A Single threading.Lock Around The generate() Call Keeps Requests Queued Instead Of Corrupting Each Other. That’s Fine For A Single-User Local Shim, But It Would Need Real Batching For Anything With More Than One Caller At A Time.

A Week Later: Staying Up Isn’t The Same As Staying Fast

KeepAlive Solved The Crash Case. It Does Nothing For The Case Where The Process Is Alive But Barely Moving.

A Week After Setup, I Noticed Every Alert Summary Had Quietly Moved To The Cloud Fallback. From The Outside The Shim Looked Fine: /v1/models Answered In Under 0.1 Seconds, So The Health Probe Passed Every Time. The Log Told A Different Story:

2026-09-22 16:13:40 INFO generated 1053 tokens in 162.3s (14.6 tok/s), prompt_tokens=573
2026-09-25 17:25:35 INFO generated 145 tokens in 10867.7s (6.3 tok/s), prompt_tokens=23484
2026-09-27 05:19:04 INFO generated 966 tokens in 6000.9s (0.8 tok/s), prompt_tokens=387

Same Model, Same Machine, 10 To 20 Times Slower. One Request Took 100 Minutes, And Earlier A Single 23K-Token Prompt Held The Generation Lock For Three Hours, So Anything That Arrived Behind It Had To Wait Too.

The Most Likely Cause Is Memory Pressure. This Is A 32GB Mac Holding A Roughly 19GB Model Next To Everything Else I Run, And Swap Was Sitting At 6.7GB Of 8GB. That’s As Far As I Can Prove It. What I Did Confirm Is That A Restart Brought It Straight Back To About 15 Tokens/Sec.

And While Retesting, I Tripped Over My Own Note Below: Sent max_tokens=80, Got An Empty Reply, And For A Minute Thought The Shim Was Broken. The Reasoning Had Simply Eaten The Whole Budget.

Notes

A Runtime Version Being “Latest” Doesn’t Mean It Supports The Latest Models. LM Studio’s Runtime And The Models You Can Download Through Hugging Face Move On Independent Schedules. When Loading A Brand-New Architecture Fails With A Python Import Error Rather Than A Normal LM Studio Error Dialog, That’s The Tell: It’s Not Rejecting The Model, It Genuinely Doesn’t Have The Code Path.
MLX Only Exists On The Metal Side Of The Fence. Not Slower On Linux, Not Emulatable, Not Containerizable On The Same Mac It’s Running On. Anyone Planning To Put An MLX Workload Behind Kubernetes Needs To Know This Before Drawing The Architecture Diagram, Not After.
The In-Process API Was Already There, Just Not Documented As “Build A Server With This.” Every Example I Found Was CLI-Shaped (python -m mlx_vlm.generate), Which Reloads The Model Every Call. The load()/generate() Split Underneath That CLI Is What Makes A Persistent Server Possible, And It’s One Import Away.
First Quality Test Looked Bad For The Wrong Reason. Ran It Once With max_tokens=50 And Greedy Decoding And Got A Leaked Internal Channel Token Plus A Truncated Reasoning Trace. That Wasn’t The Model. It’s A Reasoning Model, And 50 Tokens Isn’t Enough Room For It To Finish Thinking Before The Budget Runs Out. Bumping To max_tokens=4096 And temperature=0.2 With A Realistic Prompt Produced Clean, Correctly-Structured Output Every Time After.
The Real End-To-End Test Is The One That Matters, Not The Curl Test. Manually Curling The Shim Looked Perfect. Only Firing An Actual Alert Through The Full Production Pipeline (Log Query → Retrieval → This Shim → Escalation Logic) Confirmed It Behaved Correctly Under Real Conditions, Including Correctly Recognizing When It Had No Real Evidence To Work With And Asking To Escalate Instead Of Guessing. Generation Took 98 Seconds For 1,125 Tokens On This Machine (~14.5 Tokens/Sec For A 30B Model), Which Is The Kind Of Number You Only Get By Actually Running It Against Real Load, Not A Short Test Prompt.
A Health Check That Answers Isn’t A Model That Works. /v1/models Proves The Process Is Alive, Not That It Can Generate At A Usable Speed. Tokens/Sec Per Request Is Already In The Log, And That’s The Number Worth Watching.

How To Prevent It

Scenario	What To Do
A New Model Fails To Load In LM Studio With A Python Import Error	Check `lms runtime ls` Against Both Channels First. If It’s Already Latest, The Runtime Genuinely Doesn’t Support The Architecture Yet, Waiting Won’t Fix It
You Need An MLX Model Reachable By Multiple Services	Wrap It In A Minimal OpenAI-Compatible Shim With `load()` Once + `generate()` Per Request, Not A CLI Invocation Per Call
You’re Tempted To Put MLX Behind Kubernetes Or Docker	Don’t. It’s Apple Silicon + Metal Native Only, No Container Path Exists, Plan For A `launchd`-Managed Native Process Instead
You’re Judging Model Quality From A Short Test	Use A Realistic Prompt And Enough `max_tokens` For The Model To Actually Finish. A Truncated Reasoning Model’s Output Tells You Nothing About Its Real Quality
A Local Model Endpoint Passes Health Checks But Downstream Keeps Falling Back	Look At Per-Request Tokens/Sec In The Shim Log, Not The Health Endpoint. Cap Prompt Size So One Huge Request Can’t Hold The Lock For Hours, And Watch Swap On The Host

Reference:

[GCP]GCP Weekly Digest Report

ChungWei Wei — Thu, 17 Sep 2026 16:00:18 +0000

Situation

Had A Working GCP Weekly Digest Running On Google Apps Script For Weeks — Pulls Release Notes And Blog RSS Every Friday, Hands Them To Gemini To Write A Categorized Digest, Emails It Out, Archives A Copy To Google Drive. The Original Plan Was To Leave It Alone Until It Hit A Real Limit. Then A Different Reason To Migrate Showed Up: Wanted To Write This Up Next To The AWS Version As An “AWS vs GCP Serverless Digest Bot” Comparison, And Apps Script’s GmailApp/DriveApp/ScriptApp.getOAuthToken() Assume Every Reader Already Lives Inside A Google Account — Rebuilt It On Cloud Run Instead, Skipping The Original Migration Threshold On Purpose.

That Rebuild Alone Found Three Real Surprises And One Near-Miss. Then, With The Digest Itself Stable, Added The Other Half Of The Idea: An Opt-In Section That Points Google Cloud’s Own Recommender API At This Account And Has Gemini Turn The Findings Into Actual Advice — Deliberately Not Porting The AWS Sibling Project’s Hand-Written Detection Logic, Because GCP Already Ships A Better Version Of That Detection For Free. Getting It From “Written Against The Docs” To “Actually Tested” Surfaced Three More Problems, One Of Them The Kind That Looks Exactly Like Success Until You Know What To Check. And Finally, When SendGrid — The Email Provider This Whole Pipeline Depended On — Permanently Banned The Account, Replacing It With Gmail API Domain-Wide Delegation Found Two Last Blockers Neither Of Which Was A Bug In Any Line Of Code.

Result First:


Setup	Cloud Run Job (Python 3.12) + Gemini 3.1 Flash-Lite (digest) + Cloud Recommender → Gemini (account advice) + Gmail API With Domain-Wide Delegation (delivery), Deployed From Source
Real Surprise 1	The SDK The Original Plan Assumed (`google-cloud-aiplatform`’s generative modules) Was Already Fully Retired — Not Deprecated-Soon, Actually Gone Two Months Before This Deploy
Real Surprise 2	The Model The V1 Version Used (`gemini-2.5-flash-lite`) Retires In Under Two Months — Copying It Forward Would’ve Broken A “Runs Every Friday” Job By Mid-October
Real Surprise 3	The GCP Blog RSS URL The Old Version Used Doesn’t Serve RSS Anymore — It 200s, But With An HTML Page, Not XML
Near-Miss	Almost Ran A Local Gemini Auth Test Using Cached Credentials That Turned Out To Belong To A Completely Different Google Account And Project
Advice Bug 1	The Recommender API’s location wildcard `"-"` doesn’t work — a clean `400 InvalidArgument`, not a permissions problem
Advice Bug 2	The first real verification run had every one of 42 (recommender × location) combinations fail with 403 (unset quota project + a disabled API) — and the script still reported “fetch succeeded, 0 results,” because a permission failure and a genuinely empty result look identical unless you specifically catch them differently
Advice Bug 3	The default location list was missing the one zone holding this project’s only VM and only disk — silently zero compute-related advice, ever, for that entire category
Mail Blocker 1	SendGrid permanently banned the account; org policy `constraints/iam.disableServiceAccountKeyCreation` then refused every attempt to download a Service Account key for the Gmail API replacement
Mail Blocker 2	With IAM, code, and Workspace delegation all correct, the first real send still came back 403 — `Gmail API has not been used in project ... or it is disabled`
Outcome	All three pieces confirmed end-to-end on real infrastructure: digest generated and archived, account advice correctly flagging real findings, email delivered via Gmail API keyless delegation with a confirmed message ID

None Of The First Three Surprises Were Bugs In The Old Apps Script Code — They Were The Ground Having Moved Under A Ten-Week-Old Plan. The Advice Bugs And The Mail Blockers Are A Different Shape Entirely: Neither Came From Watching Something Break In Production. Both Came From Deliberately Verifying Assumptions Before Trusting Them — Because On This Particular Project, At This Particular Scale, Most Of These Failure Modes Would Otherwise Have Looked Exactly Like Success.

How It Broke (Root Cause)

Surprise 1: The SDK The Migration Plan Assumed No Longer Exists

The Original V2 Sketch Assumed google-cloud-aiplatform. Went To Actually Write The Call And Found:

google-cloud-aiplatform’s generative AI modules (vertexai.generative_models, etc.) were deprecated June 24, 2025, and fully removed June 24, 2026.

Two Months In The Past By The Time This Got Built. The Officially Recommended Path Is A Different Package Entirely — google-genai — Unifying Access To Both The Gemini Developer API And What Google Now Calls The “Gemini Enterprise Agent Platform” (Confirmed Independently Via gcloud services list --enabled, Which Lists aiplatform.googleapis.com Under The Title “Agent Platform API” , Not “Vertex AI API”). Then A Second, Smaller Trap: Two Web Searches Both Insisted vertexai=True “Does Not Appear Anywhere In The Document.” Installed The Package And Read It Directly Instead Of Trusting Either Search:

import inspect
from google import genai
print(inspect.signature(genai.Client. __init__ ))
# (self, *, enterprise=None, vertexai=None, api_key=None, credentials=None,
# project=None, location=None, debug_config=None, http_options=None)

Both Parameters Exist. enterprise Is Current, vertexai Is Documented As A Legacy Alias. The Search Wasn’t Lying — It Just Answered With The New Name And Left Out That The Old One Still Quietly Works. Went With enterprise=True.

Surprise 2: The Model From The Old Version Retires In Under Two Months

gemini-2.5-flash-lite Is What The Apps Script Version Used, With Explicit Instructions To Keep Using It Unless There Was A Good Reason Not To. Checked Anyway:

Gemini 2.5 Flash-Lite shuts down October 16, 2026 (Gemini Developer API) / October 20, 2026 (Agent Platform API).

Seven Weeks Of Runway On A Job Meant To Run Indefinitely, Once A Week, Unattended. Switched To gemini-3.1-flash-lite, Currently Google’s Own “Most Cost-Efficient Gemini Model” Description — A Better Match For This Job’s Actual Usage Pattern Anyway.

Surprise 3: The Blog RSS Feed The Old Code Points At Just… Isn’t RSS Anymore

curl -sL -o /dev/null -w "%{http_code} %{content_type}\n" 'https://cloud.google.com/blog/rss/'
# 200 text/html; charset=utf-8

200, So Nothing In The Old Error Handling Would Have Flagged This — The Content Type Had Silently Swapped At Some Point After V1 Was Written. Found The Real Feed Instead Of Guessing At URL Variations:

curl -s 'https://cloudblog.withgoogle.com/rss/' | head -c 500
# <?xml version="1.0" encoding="utf-8"?>
# <rss version="2.0" xmlns:atom="..." xmlns:content="..." xmlns:dc="..." xmlns:media="...">

This Feed Carries The Same Namespaced Tags (dc:, content:, media:) That Broke The AWS Sibling Project’s Hand-Rolled Namespace-Stripping Regex. Didn’t Repeat That Bug. The Document Is Well-Formed XML With Its Namespaces Correctly Declared — ElementTree Handles That Natively. The Only Reason The AWS Version Needed String Surgery Was That It Manually (And Incompletely) Stripped The Declarations First. Skipped That Step Entirely; The Whole Bug Class Never Had A Chance To Exist Here.

Near-Miss: Testing Gemini Auth With The Wrong Google Account’s Cached Credentials

The Machine Already Had Application Default Credentials Cached From Unrelated Work On A Different Project. Checked First Instead Of Assuming:

TOKEN=$(gcloud auth application-default print-access-token)
curl -s "https://www.googleapis.com/oauth2/v1/tokeninfo?access_token=${TOKEN}"
# "email": "the-other-account@company.example"

Confirmed Wrong Account. Skipped The Local Test, Validated The Real Production Auth Path Instead — Through The Cloud Run Job’s Own Dedicated Service Account.

Building The Advice Half: Detection Belongs To Google, Not To Me

With The Digest Stable, Added An Opt-In Section: Point Google Cloud Recommender At This Project (Idle Static IPs, Idle Disks, Idle VMs, Machine-Type Rightsizing, Committed-Use Discounts, Near-Zero Project Utilization) And Have Gemini Turn The Findings Into Prioritized Advice. The AWS Sibling Project Had To Hand-Write String-Matching Detection Logic Against Cost Explorer Usage-Type Names, Because AWS Has No Free Equivalent (Trusted Advisor’s Cost Checks Need Business Support). GCP Ships That Detection For Free And More Completely Than Any Hand-Written Pattern List Would — So The Code Comment In account_context.py States The Principle Directly: detection belongs to whatever’s deterministic; the model only explains. On AWS, “Deterministic” Meant Hand-Written String Matching. On GCP, It’s Google’s Own API. Porting The AWS Approach Over Would Have Meant Rebuilding A Worse Version Of Something Google Already Gives Away.

Advice Bug 1: The Location Wildcard That Looks Documented But Isn’t

Most Recommenders Are Scoped Per Zone Or Per Region, And The Official API Reference Doesn’t Say Whether A Wildcard Location Works. Wrote The Code To Try - First And Fall Back To Enumerating Locations On Failure — Then Actually Tested It:

list(client.list_recommendations(
    parent=f'projects/{PROJECT}/locations/-/recommenders/{probe}'))


400 InvalidArgument: Invalid location: -

Clean Rejection, Not A Permissions Problem — So The Fallback Path Became The Only Path, And The Speculative Wildcard-First Branch Was Deleted Rather Than Left In As Dead Code Nobody Would Ever Trust Again.

Advice Bug 2: 42 Silent 403s That Reported Themselves As Success

First Real Verification Run Against The Actual Project. Every Single One Of 42 (Recommender × Location) Combinations Failed — An Unset Quota Project Plus A Not-Yet-Enabled Recommender API — And The Original Code Caught All Exceptions The Same Way, So The Result Was A Clean Empty List. The Script’s Own Log Line Read:

[advice] 掃過 6 個 recommender × 9 個 location，42 個組合不適用，取得 0 筆建議

“0 Results” And “42 Silent Permission Failures” Produce The Exact Same Output Shape If Every Exception Gets Caught The Same Way. That’s The Trap: A Genuinely Empty Recommendation List Is A Real, Expected, Even Good Outcome For A Personal Project — So The Code Cannot Simply Alarm On Empty. It Has To Tell The Difference Between “Nothing To Report” And “Never Actually Asked” Without A Human Reading Every Response. The Fix Splits Exceptions Into Two Classes:

_SKIPPABLE = (gexc.InvalidArgument, gexc.NotFound) # this combination doesn't apply — expected
_FATAL = (gexc.PermissionDenied, gexc.Unauthenticated, gexc.ResourceExhausted) # environment is broken — must surface

Plus A Last-Resort Guard: If Every Combination Gets Skipped, That’s Not “No Recommendations” — It’s “Never Successfully Asked Once,” Almost Certainly A Misconfigured Location List, And It Now Raises Instead Of Returning An Empty List That Would Have Rendered As “No Open Recommendations This Week” — Indistinguishable From The Genuinely Good Outcome It’s Impersonating.

Advice Bug 3: The Location List That Silently Excluded The Only Resources That Existed

Fixing Bug 2 Meant The Fetch Actually Succeeded — And Then Returned Zero Compute-Related Recommendations Anyway, For A Different Reason. The Default Location List Didn’t Include The One Zone Holding This Project’s Only VM And Only Disk. Every Compute-Related Recommender Query Landed On Locations With Nothing In Them — Technically Successful, Structurally Guaranteed To Find Nothing. A Location List Missing One Entry Produces Exactly The Same Output As A Project With No Idle Resources. There’s No Error For “You Forgot A Zone” — Only The Absence Of Findings That Should Have Existed. Caught Only By Cross-Checking The List Against Where This Project’s Actual Resources Live, Not By Anything The API Itself Could Have Signaled.

Mail Blocker 1: SendGrid Banned The Account, Then The Org Wouldn’t Let Anyone Download A Key

SendGrid Permanently Banned The Account — Two Delivery Deferrals At Exactly 72.01 Hours Each, Followed By A Reactivation Request Denied For Good. Since kmp.tw’s MX Records Already Point At Google, Gmail API With Domain-Wide Delegation Looked Like The Natural Replacement. Tried The Standard Recipe First — Dedicated Mailer Service Account, Domain-Wide Delegation Enabled, Download The JSON Key, Mount Through Secret Manager. gcloud iam service-accounts keys create Refused Outright:

“If the iam.disableServiceAccountKeyCreation constraint is enforced for your organization, then you can’t create keys for any service accounts in your organization.”

Not This Service Account — Any Service Account, Org-Wide. The Keyless Alternative Uses The Cloud Run Job’s Own Runtime Identity, Granted roles/iam.serviceAccountTokenCreator Scoped Only To The Mailer SA Resource (Not Project-Wide). At Send Time, The IAM Credentials API’s signJwt Signs An RFC 7523 JWT-Bearer Assertion (Subject wei@kmp.tw — This Is The Actual Delegation Step) Which Gets Exchanged At oauth2.googleapis.com/token For A Genuine, Short-Lived Access Token. No Private Key Is Ever Generated, Downloaded, Or Written To Disk Anywhere In The Chain.

Mail Blocker 2: 403 After Every Other Piece Was Already Right

Once Workspace Admin Console Authorization Was Done, The Token Exchange — Which Had Been Failing With 401 unauthorized_client The Whole Time Delegation Wasn’t Authorized — Started Returning Real Tokens. The Very Next Call, users.messages.send, Came Back 403:

Gmail API has not been used in project 354846332057 before or it is disabled.
Enable it by visiting
https://console.developers.google.com/apis/api/gmail.googleapis.com/overview?project=354846332057
then retry. If you enabled this API recently, wait a few minutes for the
action to propagate to our systems and retry.

The Entire Identity Chain Was Already Working. The Only Thing Wrong Was That gmail.googleapis.com Had Simply Never Been Enabled On This Project Before, Because Nothing In It Had Ever Called Gmail API Until This Line Of Code:

gcloud services enable gmail.googleapis.com --project=amp-project-161400

Reran The Job. Real Send, Real Log Line:

[Gmail] 已送出，message id：1a067facf2c866b7
Email 已寄出
完成！

Worth Noting: The Project Identifier In That URL (354846332057) Is The Project Number , Not The Project ID (amp-project-161400) — Searching Logs For The Project ID String Won’t Find This Error At All.

Notes

A rebrand can be real even when it sounds like a hallucination. aiplatform.googleapis.com Titled “Agent Platform API,” Vertex AI’s Own Docs Saying They’re “No Longer Being Updated” — Both Confirmed Independently Through gcloud services list, Not Assumed From A Search Summary.
When a web search and the actual installed package disagree, install the package. Ten Seconds Of inspect.signature() Settled What Two Searches Got Confidently Wrong.
“It returns 200” is not the same as “it still works.” True Of A Dead RSS Feed Silently Serving HTML, And True Of An API Call That Reports “Success” While Having Caught Every Failure The Same Way.
The scariest bugs in this project all share one shape: a genuinely good outcome and a broken one produce identical output. An Empty Recommendation List Because There’s Nothing To Flag, And An Empty List Because Every Call 403’d — Both Print The Same Sentence Unless The Code Is Specifically Built To Tell Them Apart. A Missing Location In A Config List Produces The Same Silence As A Project With No Idle Resources.
Domain-wide delegation fails at two independent points that don’t look alike. Missing Workspace Authorization Fails At Token Exchange With 401 unauthorized_client. An API Never Enabled On The GCP Project Fails Later, At The Actual Call, With 403 SERVICE_DISABLED. Fixed In Two Completely Different Consoles By Two Completely Different People.
Porting a sibling project’s workaround without asking why it existed there imports the wrong lesson. The AWS Version Needed Hand-Written Detection Because AWS Has No Free Equivalent. Copying That Pattern To GCP Would Have Meant Reinventing A Worse Version Of Something Google Already Provides.

How To Prevent It

Scenario	What To Do
Migrating code that calls a managed AI SDK, based on a plan written more than a few weeks ago	Re-check the SDK’s current deprecation status before writing a single line
A model ID was chosen previously with instructions to “keep using it unless there’s a reason not to”	Check the model’s specific retirement date, not just whether it works today
An RSS/feed URL worked when a project doc was first written	Re-verify with `curl -sL -o /dev/null -w "%{http_code} %{content_type}"` before reusing it
Porting a workaround from a sibling project on a different cloud	Check whether the reason it existed still applies — sometimes the platform already solved it natively
A machine has cached credentials for multiple accounts across unrelated projects	Verify identity (`tokeninfo`, etc.) before the first real API call in a new session
Catching every exception from a bulk API scan the same way	Split “this combination doesn’t apply” from “the environment is broken” — an empty result and a wall of silent permission failures must not look identical
A config list enumerates locations/regions/zones to scan	Cross-check it against where resources actually live — a missing entry produces silence, not an error
An org enforces `constraints/iam.disableServiceAccountKeyCreation`	Don’t request an exception — go keyless via IAM Credentials API `signJwt` with a narrowly-scoped `serviceAccountTokenCreator` grant
A Google API call fails with `SERVICE_DISABLED` / “has not been used in project”	Check `gcloud services list --enabled` before assuming it’s another auth problem

Reference:

[K8s] I Watched A VPA Recommendation For A Month — It Never Moved, Because It Was Never Really A Recommendation

ChungWei Wei — Sat, 12 Sep 2026 13:43:25 +0000

Situation

Part 1 Ended With Two Unresolved Questions: A 250Mi Memory Target For A Dashboard Container Actually Using Only 24Mi, Sitting Suspiciously Close To The 256Mi Limit Already On The Deployment, With No Citable Reason Why. And A CPU Number That Had Never Once Caught My Attention, The One I’d Waved Through Without Checking. The Plan Was To Let It Accumulate History And Come Back To See Which Way The Memory Target Went. A Month Later It Went A Third Way I Hadn’t Considered, The Answer Had Nothing To Do With The Limit, And It Closed Both Questions At Once.

Result First:


Observation Window	2026-08-06 To 2026-09-07, One VPA Object In `updateMode: "Off"`, Untouched
What I Expected	Target Converges Down Toward Real Usage, Or Stays Anchored Near The 256Mi Limit
What Happened	Target Never Moved. `lowerBound`, `upperBound` And `uncappedTarget` All Converged Onto It
Actual Root Cause	`--pod-recommendation-min-memory-mb`, Default `250`. The Target Was The Floor, Not A Calculation
The 256Mi Limit Resemblance	Coincidence. The Floor Is 250MiB Regardless Of What The Container’s Limit Is
The CPU Number Part 1 Left Open	Same Story. `--pod-recommendation-min-cpu-millicores` Defaults To `25`

What A Month Of History Actually Changed

Same Object, Same updateMode: "Off", Nothing Touched In Between:

	Day 0 (2026-08-06)	Day 32 (2026-09-07)
Actual Usage	2m / 24Mi	1m / 26Mi
`target`	25m / 250Mi	25m / 250Mi
`lowerBound`	25m / 250Mi	25m / 250Mi
`upperBound`	640m / 2013Mi	25m / 250Mi

The Only Number That Moved Is The Upper Bound, And It Moved A Long Way. That Part Matches What Part 1 Predicted: The Confidence Multiplier Inflates The Upper Bound When There’s Barely Any History Behind It, And Decays Toward 1 As Real Observation Accumulates. A Month In, It’s Done Decaying.

What Part 1 Got Wrong Was Assuming The Target Was A Calculation That Would Also Move Once Confidence Resolved. It Wasn’t, So It Didn’t.

The Tell Is In The Full Status, Which I Should Have Read More Carefully The First Time:

status:
  conditions:
  - lastTransitionTime: "2026-08-06T03:51:58Z"
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: headlamp
      lowerBound: { cpu: 25m, memory: 250Mi }
      target: { cpu: 25m, memory: 250Mi }
      uncappedTarget: { cpu: 25m, memory: 250Mi }
      upperBound: { cpu: 25m, memory: 250Mi }

Four Fields. One Value. A 50th Percentile Estimate, A 90th Percentile Estimate And A 95th Percentile Estimate Do Not Naturally Land On The Same Number For A Workload With Real Variance In It. When They Do, You’re Not Looking At Three Estimates. You’re Looking At Something Clamping All Three To The Same Place.

The Answer To Part 1’s Open Question

The Recommender On This Cluster Runs With No Resource-Related Flags At All:

kubectl get deploy -n vpa-system vpa-vertical-pod-autoscaler-recommender \
  -o jsonpath='{.spec.template.spec.containers[0].args}'
# ["--v=4","--stderrthreshold=info","--leader-elect=true", ...leader election only...]

Which Means Every Recommendation It Produces Is Shaped By Defaults. Two Of Those Defaults Are The Whole Story:

Flag	Default	Help Text
`--pod-recommendation-min-cpu-millicores`	`25`	“Minimum CPU Recommendation For A Pod”
`--pod-recommendation-min-memory-mb`	`250`	“Minimum Memory Recommendation For A Pod”

The Flag Is Named mb, But The Recommender Multiplies It By 1024*1024, Not 1000*1000:

minMemory := model.ScaleResource(model.MemoryAmountFromBytes(r.minMemoryMb*1024*1024), fraction)

250 × 1024 × 1024 Is Exactly 250Mi. Not “Approximately”, Not “Close To”. The Number VPA Reported Is The Default Floor Rendered Back Verbatim.

So The Actual Chain For This Container Is: Take The 90th Percentile Of Peak Memory Per 24h Window Over 8 Windows (Roughly 26Mi Here), Add The 15% Safety Margin (--recommendation-margin-fraction, Default 0.15, Getting Us To Around 30Mi), Then Raise It To The Minimum Because 30Mi Is Below 250Mi. Every Estimator In The Chain Gets The Same Floor Applied, Which Is Why The Lower And Upper Bounds Sit There Too Once Their Confidence Multipliers Stop Distorting Them.

And The Part I Spent Real Time Being Suspicious About In Part 1, The 250Mi Target Sitting Six Mebibytes Under The Container’s 256Mi Limit, Is A Coincidence. The Floor Is 250MiB Whether The Container’s Limit Is 256Mi, 4Gi, Or Absent Entirely. Two Numbers Landed Near Each Other And I Read A Relationship Into It That Isn’t There.

Notes

target == lowerBound == upperBound Is A Signal, Not A Coincidence. It Means Every Estimate In The Chain Fell Below The Configured Minimum And Got Clamped. The Recommendation Is Telling You About VPA’s Defaults, Not About Your Workload.
A Month Of History Made The Recommendation More Confident Without Making It More Useful. Confidence And Usefulness Are Different Axes. The Bounds Collapsing Onto The Target Looks Like Convergence, And It Is, But It Converged Onto A Constant.
The Floor Is Per-Pod And Adjustable. If You Want VPA To Say Something Meaningful About Workloads Smaller Than 250Mi, Those Two Flags Are Where To Look. Worth Knowing Before Concluding VPA “Doesn’t Work” On Small Containers.

The Number I Waved Through In Part 1

Part 1 Closed With An Admission: I Had Gone Digging Into The Memory Recommendation And Never Checked The CPU One, For No Better Reason Than That 25m Looked Harmless Next To 2m Of Actual Usage. Here Is What Was Sitting Behind It.

25m Is --pod-recommendation-min-cpu-millicores, Default 25. The Same Floor, Doing The Same Thing To The Same Kind Of Small Workload. Both Halves Of That First Recommendation Were Defaults. Neither Was A Measurement.

The Sorting Mechanism Is The Part Worth Keeping. Both Targets Were About 10x Actual Usage. One Of Them Looked Alarming, So It Got A Source Dive. The Other Looked Plausible, So It Got A Sentence Of Explanation And A Pass. Plausibility Is Not Verification, And A Number That Matches What You Expected Is Precisely The One Nobody Checks.

Notes

Check Whether A Recommendation Sits Exactly On A Documented Default Before Explaining Why It’s Reasonable. It Takes One Lookup, And “Reasonable-Looking” Is Precisely The Condition Under Which Nobody Bothers.

Real-World Application

Scenario	What To Do
Reading Any VPA Recommendation For A Small Container	Compare The Numbers Against `25m` And `250Mi` First. If They Match Exactly, You’re Reading Floors And The Recommendation Carries No Information About That Workload
All Three Bounds Report The Same Value	Not A Converged, High-Confidence Answer. It Means Every Estimate Landed Below The Minimum And Got Clamped To It
Running VPA Across A Cluster Of Mostly Small Workloads	Applying Recommendations Wholesale Will Inflate Requests Toward 250Mi Each. Either Lower `--pod-recommendation-min-memory-mb` Or Treat Small Workloads As Out Of Scope
Evaluating Whether VPA Has “Learned” Your Workload Yet	Watch The Upper Bound, Not The Target. The Upper Bound Is Where The Confidence Multiplier Lives, So It’s The Field That Actually Reflects Accumulated History
Inheriting A VPA Install Someone Else Set Up	Read The Recommender’s Args Before Interpreting Any Output. Every Number You’re Looking At Is Shaped By Flags That Are Invisible In The VPA Object Itself

Reference:

Part 1: VPA Isn’t A Cloud Feature — I Put It On A Bare RKE2 Cluster
VPA Recommender Config Flags (v1.7.1 source)
VPA Recommender Logic (v1.7.1 source)
Real Setup: Home RKE2 Cluster, VPA 1.7.1, Observed 2026-08-06 → 2026-09-07

[K8s]Vertical Pod Autoscaler Isn't A Cloud Feature — I Put It On A Bare RKE2 Cluster And Its First Answer Was 10x Reality

ChungWei Wei — Mon, 07 Sep 2026 17:09:14 +0000

Situation

Had Already Set Up VPA On A GKE Workload (A ClickHouse Backend, After It OOM’d Twice) And Assumed — Wrongly — That Vertical Pod Autoscaler Was Something GKE Provided As A Managed Feature. Turns Out The “Enable” Button On GKE Is Just Google’s Wrapper Around The Same Open-Source kubernetes/autoscaler Project Anyone Can Install Anywhere. Wanted To Confirm That For Real, So I Put It On A 6-Node Homelab RKE2 Cluster With No Cloud Provider In Sight.

It Went In Clean. What Wasn’t Clean Was The First Recommendation It Handed Back For A Live Workload — Ten Times The Container’s Actual Memory Usage, Within Minutes Of Being Installed.

Result First:


Goal	Confirm VPA Works On Vanilla Self-Managed Kubernetes, Not Just Cloud-Managed Clusters
Prerequisite Check	Metrics Server — Already There. RKE2 Ships `rke2-metrics-server` As A Default Addon, Unlike Vanilla `kubeadm` Clusters
Install Method	Official In-Repo Helm Chart, Not The `vpa-up.sh` Shell Script — Skips Hand-Rolling Webhook Certs With `openssl`
Real Footprint	6 Pods (2x Recommender / Updater / Admission-Controller), ~8m CPU + ~59Mi Memory Combined — A Rounding Error On 2-vCPU Nodes
The Surprise	Pointed It At A Real, Low-Stakes Workload (Headlamp). Actual Usage: 24Mi. VPA’s First Target: 250Mi

Turns Out The Gap Isn’t A Bug — It’s What VPA Looks Like When It’s Being Deliberately, Structurally Cautious With A Number It Doesn’t Have Enough Data To Trust Yet. Which Is Its Own Kind Of Interesting, Once You Go Looking For Why.

Confirming VPA Is Cloud-Agnostic

The Claim To Verify: GKE’s --enable-vertical-pod-autoscaling Flag Isn’t A GKE-Native Feature — It’s A Managed Install Of The Same kubernetes/autoscaler Sub-Project (vertical-pod-autoscaler) That Ships On GitHub For Anyone To Run.

Two Prerequisites, Checked On The RKE2 Side Before Assuming Anything:

kubectl get pods -n kube-system | grep metric
# rke2-metrics-server-56b57b8dfb-ggnb9 1/1 Running

kubectl get apiservices | grep metrics
# v1beta1.metrics.k8s.io kube-system/rke2-metrics-server True

Metrics Server Was Already Running — RKE2 Bundles It By Default (Vanilla kubeadm Doesn’t, And That’s Usually The First Blocker People Hit Installing VPA On A DIY Cluster). One Fewer Thing To Set Up.

Two Install Paths, Picked The Less Manual One

The Official Repo Offers Both:

hack/vpa-up.sh — Clones The Whole Monorepo, Shells Out To openssl To Generate The Admission Webhook’s TLS Cert By Hand, Applies Raw Manifests
charts/vertical-pod-autoscaler — An Actual Helm Chart Living In The Same Repo, Where Cert Generation Is A Helm Hook Job Instead Of A Shell Script Step You Run Yourself

git clone --depth 1 https://github.com/kubernetes/autoscaler.git
helm install vpa autoscaler/vertical-pod-autoscaler/charts/vertical-pod-autoscaler \
  --namespace vpa-system --create-namespace

Six Pods Come Up: Recommender x2, Updater x2, Admission-Controller x2 (Chart Default Replica Count, For HA). Combined Real-World Footprint After A Few Minutes Settled:

NAME CPU(cores) MEMORY(bytes)
vpa-...-admission-controller-cx99z 1m 10Mi
vpa-...-admission-controller-pjjlm 1m 10Mi
vpa-...-recommender-qsng2 2m 6Mi
vpa-...-recommender-zzgdv 1m 14Mi
vpa-...-updater-hn74p 1m 13Mi
vpa-...-updater-kww9b 2m 6Mi

~8m CPU, ~59Mi Memory, Total, Across All Six. On Nodes With 2 vCPUs Each, That’s Nothing. Whatever Hesitation I Had About Homelab Node Headroom Before Installing Turned Out To Be Unfounded.

Notes

Installing VPA Does Nothing By Itself. The Components Sit Idle Until You Create An Actual VerticalPodAutoscaler Object Pointing At A Workload. No Object, No Recommendation, No Resizing — Just Six Quiet Pods.

The Part I Didn’t Expect: The First Real Recommendation

Picked Headlamp (A Kubernetes Dashboard Deployment Already Running In The Cluster) As A Deliberately Low-Stakes Target — It Collects A Lot Of Cluster State Continuously, So It Seemed Like A Reasonable Thing To Watch. Attached A VPA Object In updateMode: "Off" — Recommendation-Only, Never Touches The Running Pod:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: headlamp-vpa
  namespace: headlamp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: headlamp
  updatePolicy:
    updateMode: "Off"

Checked Actual Usage Right Before Applying It:

NAME CPU(cores) MEMORY(bytes)
headlamp-fb548559f-zs8h2 2m 24Mi

Configured Resources On The Deployment At The Time: requests: 50m/64Mi, limits: 250m/256Mi. Already Comfortably Over-Provisioned Relative To Real Usage, By Eye.

Under A Minute After The VPA Object Landed:

status:
  recommendation:
    containerRecommendations:
      - containerName: headlamp
        lowerBound: { cpu: 25m, memory: 250Mi }
        target: { cpu: 25m, memory: 250Mi }
        upperBound: { cpu: 640m, memory: "2013Mi" }

Both Targets Land At Roughly 10x Actual Usage: 25m Against 2m Of CPU, 250Mi Against 24Mi Of Memory. Only One Of Them Bothered Me. 250Mi , Sitting Six Mebibytes Under The Container’s Existing 256Mi Limit, Looked Like Something That Needed Explaining, And An Upper Bound North Of 2Gi For A Dashboard App Idling At 24Mi Made It Look Worse. 25m Looked Harmless, And Was Even Lower Than The Container’s Current 50m Request, So I Read It As A Sane Safety Margin And Moved On.

Noting That Deliberately, Because It Matters Later: I Went Digging Into The Memory Number And Never Checked The CPU One, For No Better Reason Than That It Looked Reasonable.

Why Memory Recommendations Are Built To Run High

Went Digging Into The Recommender’s Own Source Rather Than Guess:

Memory Targets Are Computed From The 90th Percentile Of Peak Usage Per Aggregation Window (Default 8 Days), Not Average Usage. CPU Is Compressible — Throttling A Container That Briefly Needs More CPU Just Slows It Down. Memory Isn’t — Exceed The Limit And The Kernel OOM-Kills The Container. The Recommender’s Math Is Deliberately Asymmetric For This Reason: It Would Rather Overestimate Memory Than Cause An OOM.
With Very Little Observation History, The Confidence-Interval Math Pulls The Bounds Apart On Purpose, Never Toward A Smaller Number. The Lower Bound’s Confidence Multiplier Collapses Toward The Target Itself When History Is Thin (Effectively: “Don’t Suggest Shrinking Something You Haven’t Watched Long Enough To Trust”). The Upper Bound’s Multiplier Does The Opposite — Balloons Toward An Enormous Number When History Is Thin, Which Is Exactly The ~2Gi Ceiling Showing Up Here. Both Directions Of “I’m Not Sure Yet” Get Resolved The Same Way: Toward Not Under-Provisioning.

What That Doesn’t Fully Explain, And What I’m Deliberately Not Papering Over: Why The Target Itself Landed At Almost Exactly 250Mi — Suspiciously Close To The Container’s Existing 256Mi Limit — Rather Than Somewhere Nearer The 24Mi It’s Actually Using Right Now. Didn’t Find A Clean, Citable Answer To That One In The Time I Spent On It. Possible It’s An Artifact Of How The Very First Recommendation Gets Computed Before A Full Aggregation Window Has Elapsed; Possible It’s Something Else Entirely. Leaving It Genuinely Open Rather Than Inventing A Tidy Explanation That Sounds Right But Isn’t Verified.

Notes

A Confident-Looking Number Isn’t The Same As A Confident Number. VPA Handed Back A Specific-Looking 250Mi, Not A Vague Range — But It Had Observed The Workload For Under A Minute. The Precision Of The Output Format Doesn’t Tell You Anything About How Much To Trust It Yet.
Memory And CPU Are Computed By Deliberately Different Maths. Memory Targets A High Percentile Of Peak Usage Specifically To Avoid OOMs; CPU Is Comparatively Relaxed Because Contention Only Throttles. Worth Knowing As Background — Though It Does Not, On Its Own, Explain The Specific Numbers Above.
The Number You Don’t Question Is The One To Watch. Both Targets Here Are About 10x Actual Usage, But Only The Memory One Looked Alarming Enough To Investigate. That Asymmetry Is In The Reader, Not In The Data.

What Happens Next

This Is Deliberately Posted Before The Story’s Finished. The Whole Point Of Confidence-Weighted Recommendations Is That They’re Supposed To Get Better With More History — VPA’s Own LowConfidence Condition Exists Specifically To Flag When A Recommendation Shouldn’t Be Trusted Yet. Letting This Run Against Real (Light) Traffic, Then Coming Back To See Whether The Memory Target Actually Converges Down Toward The 24Mi It’s Really Using, Or Whether It Stays Anchored Near The Configured Limit Regardless Of Real Usage. And While I’m There, Checking The CPU Number I Waved Through. Part 2 Covers Both.

Real-World Application

Scenario	What To Do
Deciding Whether VPA Is Worth Installing Outside A Cloud-Managed Cluster	It’s Fully Open-Source And Cloud-Agnostic — The Only Real Prerequisite Is Metrics Server. Check For It First; Some Distros (RKE2) Bundle It, Some Don’t
Installing VPA On Any Self-Managed Cluster	Use The In-Repo Helm Chart Over `vpa-up.sh` Unless You Have A Specific Reason Not To — Same Components, Less Manual Cert Handling
Reading A Brand-New VPA Recommendation	Check The `RecommendationProvided` / `LowConfidence` Conditions Before Acting On Any Number — A Recommendation Minutes Old Carries Very Different Weight Than One With Days Of History Behind It
A Recommendation That Looks Wrong Next To One That Looks Fine	Check Both. Memory And CPU Are Computed By Different Maths (Peak Percentile Versus A More Relaxed CPU Path), But That Doesn’t Mean The Reasonable-Looking Number Has Been Verified — It Means It Hasn’t Been Questioned

Reference:

kubernetes/autoscaler — Vertical Pod Autoscaler
VPA Recommender Logic (source)
Real Setup: Home RKE2 Cluster (6 Nodes), 2026-08-06 — Part 2 Pending Longer Observation Window

[AWS]AWS Weekly Digest Report

ChungWei Wei — Thu, 03 Sep 2026 17:45:22 +0000

Situation

Had Been Watching A LinkedIn Account Post AWS Updates Every Week For A While — Always Just Updates, And Kept Wondering If There Was A More Human Way To Deliver Them. So, Riding The AI Wave, Built A Lambda + Bedrock + SES + S3 Pipeline: Every Friday It Pulls The AWS What’s New And Blog RSS Feeds, Hands Them To Amazon Nova Pro To Write A Categorized Digest, Emails It Out, Archives A Copy To S3 — And Also Has Claude Opus Look At This Account’s Actual Resource Usage And Suggest Optimizations.

sam build Passed. Template Validated. Everything Looked Done On Paper. First Real Deploy Still Found Two Bugs That Zero Amount Of Re-Reading The Code Would Have Surfaced, Because Both Only Show Up Once Real AWS Resources Are Talking To Each Other. The Account-Advice Feature Had A Longer Road: Two Prompt Versions That Produced Nothing Useful, A Model Swap That Revealed A Genuine Quality Gap Once The Prompt Was Actually Good, And Two More Bugs That Only Surfaced When Prepping To Open-Source The Project And Asking Whether Any Of It Would Hold Up On A Bigger Account.

Result First:


Setup	Lambda (Python 3.12, arm64) + Amazon Bedrock (Nova Pro For The Digest Body, Claude Opus 5 For Account Advice, Both Via Cross-Region Inference) + SES + S3, Deployed Via SAM
First Deploy Blocker	`CAPABILITY_IAM` Wasn’t Enough — Template Declares An Explicit IAM `RoleName`, Which Bumps The Requirement To `CAPABILITY_NAMED_IAM`
Bug 1	IAM Policy For `bedrock:InvokeModel` Was Scoped To The Wrong ARN — Substituted The Cross-Region Inference Profile ID Straight Into The `foundation-model` Resource, Which Isn’t How The Underlying Model ARN Is Actually Shaped
Bug 2	RSS Parser Silently Returned Zero Blog Posts Every Single Run — A Namespace-Stripping Regex Only Half Did Its Job
Two Failed Advice Prompts	Service names alone produced five lines of pure filler; adding usage-type granularity gave real signal, but the model still missed the two most valuable findings
Model Comparison	Same already-good prompt, same data, only the model swapped — Nova Pro forced a “savings” framing on 4 of 5 findings despite an explicit instruction not to; Claude Opus 5 cross-referenced multiple signals into real findings and correctly stayed quiet where there was nothing to save
Bug 3	`GetCostAndUsage` caps at 1,000 groups per response with no pagination handling — a bigger account would silently lose entries with zero error
Bug 4	The pattern-flagging step scanned only the already-truncated top-4-per-service display list — the smallest, most-signal-bearing usage types were exactly the ones most likely to fall outside it
How I Found Them	The first two by actually invoking the deployed function and reading CloudWatch Logs line by line, not just checking `StatusCode: 200`. The last two by asking “would this hold up on a bigger account?” before shipping the code to anyone else’s AWS account — not by watching anything break
Outcome	End-to-end run confirmed: 54 What’s New items + 3 blog posts parsed, Nova Pro digest generated, Opus 5 account advice generated (flag count on the same account went 7 → 14), saved to S3, emailed via SES

Neither Of The First Two Bugs Broke sam build. Neither Broke sam validate. Both Only Existed The Moment Real IAM Policies Met Real Bedrock APIs And Real RSS Bytes. The Last Two Were Quieter Still — This Account Is Structurally Too Small To Ever Trigger AWS’s Own Pagination Cap, So Both Were Found By Asking, Not By Watching Something Fail.

How It Broke (Root Cause)

Bug 1: The Cross-Region Inference Profile ARN Trap

Nova Pro On Bedrock Is Set Up To Use Cross-Region Inference — You Call It With An Inference Profile ID Like us.amazon.nova-pro-v1:0 Instead Of A Plain Model ID, And Bedrock Load-Balances The Actual Request Across A Handful Of Regions Behind The Scenes. The IAM Policy I’d Written Assumed The Profile ID Was Also The Model’s Identity For Permission Purposes, So It Built The Resource ARN Like This:

Resource:
  - !Sub 'arn:aws:bedrock:${BedrockRegion}::foundation-model/${BedrockModelId}'
  # BedrockModelId = us.amazon.nova-pro-v1:0

First Real Invoke After Deploy:

AccessDeniedException: User: arn:aws:sts::...:assumed-role/aws-weekly-digest-role.../aws-weekly-digest
is not authorized to perform: bedrock:InvokeModel on resource:
arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-pro-v1:0

Look Closely At That Resource — No us. Prefix. When You Invoke Through A Cross-Region Inference Profile, IAM Doesn’t Just Check Permission On The Profile Itself. It Also Checks Permission On The Underlying Foundation Model That The Profile Actually Routes The Request To — And That Underlying Model’s ARN Never Carries The us./eu./apac. Prefix. That Prefix Only Exists On The Profile ID, Not On The Real Model.

Didn’t Want To Guess At Which Regions It Actually Routes To, So Asked The API Directly Instead Of Assuming:

import boto3
c = boto3.client('bedrock', region_name='us-east-1')
r = c.get_inference_profile(inferenceProfileIdentifier='us.amazon.nova-pro-v1:0')
for m in r['models']:
    print(m['modelArn'])


arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-pro-v1:0
arn:aws:bedrock:us-west-2::foundation-model/amazon.nova-pro-v1:0
arn:aws:bedrock:us-east-2::foundation-model/amazon.nova-pro-v1:0

Three Destination Regions, Not One. The Original Policy Only Granted us-east-1, So Even Fixing The Model ID Without Fixing The Region Coverage Would Have Left An Intermittent Failure Waiting To Happen Depending On Which Region Bedrock Decided To Route To That Day. Fixed It By Explicitly Granting The Real Foundation-Model ARNs With A Region Wildcard, Plus The Inference-Profile ARN Separately:

Resource:
  - 'arn:aws:bedrock:*::foundation-model/amazon.nova-pro-v1:0'
  - 'arn:aws:bedrock:*::foundation-model/amazon.nova-lite-v1:0'
  - 'arn:aws:bedrock:*::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0'
  - 'arn:aws:bedrock:*::foundation-model/anthropic.claude-3-haiku-20240307-v1:0'
  - !Sub 'arn:aws:bedrock:${BedrockRegion}:${AWS::AccountId}:inference-profile/*'

Redeployed, Reinvoked — Clean Pass Through Bedrock On The Second Try.

Bug 2: The RSS Parser That Silently Gave Up On Every Blog Post

The AWS Blog RSS Feed Declares A Pile Of XML Namespaces On Its Root <rss> Tag — dc:, content:, slash:, atom:, And So On. To Dodge Dealing With Namespace-Aware ElementTree Parsing, The Original Code Just Regex-Stripped The xmlns Declarations Off The Root Element Before Parsing:

clean = re.sub(r'\s+xmlns(?::[a-z]+)?="[^"]*"', '', raw[start:])
root = ET.fromstring(clean)

Problem Is, Stripping The Declarations Doesn’t Strip The Prefixed Tags Still Sitting Inside Every <item> — Things Like <dc:creator>Jeff Barr</dc:creator> Are Still There. Once The xmlns:dc="..." Declaration That Would Have Told The Parser What dc: Means Is Gone, ElementTree Hits The First <dc:...> Tag And Throws:

unbound prefix: line 5, column 1

The Function Already Had A Try/Except Around The Whole Fetch, By Design — If The Blog Feed Is Down Or Malformed, The Digest Should Still Ship With Just The What’s New Section Instead Of Failing The Whole Run:

except Exception as e:
    print(f'[Blog] 跳過：{e}')
    return []

Which Is Exactly What It Did. Every Single Invoke, Silently, From The First Line Of Code. statusCode: 200, Email Sent, S3 File Written — Just With A Permanently Empty Blog Section, And No Symptom Visible Anywhere Except A CloudWatch Log Line You’d Only See If You Went Looking.

Fix Wasn’t Complicated Once Found — Strip The Prefix Off The Tags Too, Not Just The Declarations:

clean = re.sub(r'\s+xmlns(?::[a-z]+)?="[^"]*"', '', raw[start:])
clean = re.sub(r'</?[a-zA-Z0-9]+:', lambda m: m.group(0).replace(':', ''), clean)
root = ET.fromstring(clean)

<dc:creator> Becomes <creator>, </dc:creator> Becomes </creator>, And Since The Code Only Ever Reads title / link / description / pubDate Off Each Item Anyway, Renaming Tags It Never Touches Costs Nothing. Redeployed, Reinvoked — Blog Count Went From 0 篇 To 3 篇 In The Log Line.

Two Failed Prompts Before Account Advice Was Worth Reading

Once The Digest Itself Was Stable, Added An Opt-In Section: Feed This Account’s Actual Cost Explorer Usage To Bedrock, Append Optimization Advice To The Email. Version 1 Fed The Model Nothing But Service Names And Got Back Five Lines Of Pure Filler:

[AWS WAF]：現況推測：可能沒有設定 WAF 規則來保護應用程式。建議動作：設定 WAF 規則⋯⋯

“Possibly Not Configured, Recommend Configuring” — True Of Any Account, Because The Model Had Nothing But A Name To Guess From. Adding A Second GroupBy Dimension On USAGE_TYPE Changed The Input Entirely:

Amazon Virtual Private Cloud
    APS1-PublicIPv4:IdleAddress = 0.0892 ← the name says Idle
Amazon Elastic Container Service for Kubernetes
    APE2-AmazonEKS-Hours:extendedSupport = 0.479 ← paying extra to run an EOL version

Real Signal Now — But The Model Still Missed The Two Most Valuable Findings And Spent A Slot On A Low-Value CloudFront Check That Wasn’t Backed By Anything In The Data. Fix Moved Detection Out Of The Prompt Entirely: String-Match Known-Waste Usage-Type Patterns (IdleAddress, extendedSupport, NatGateway-Hours) In Code, Flag Them Before The Model Ever Sees Them, Let The Model Only Explain What’s Already Flagged. Third Version Got All Five Right.

Nova Pro vs Claude Opus 5: Same Data, Two Different Levels Of Reasoning

With The Prompt Already At Its Best Version, Swapping Only BEDROCK_MODEL_ID From Nova Pro To Claude Opus 5 (us.anthropic.claude-opus-5) Produced Two Different Kinds Of Output From The Same Data.

Nova Pro : Four Of Five Findings Led With “Expected Effect: Cost Savings” — Directly Against What The Prompt Said — On An Account That Spent $0.0014 Over Ninety Days. Each Finding Explained In Isolation, No Cross-Referencing.

Claude Opus 5 : 2,160 = 24 × 90 → Two Regions Each Running A DynamoDB Table Continuously For The Full Window. extendedSupport Hours Exactly Equal To Total Runtime → The Cluster Launched On An Outdated Version From Day One, Not Aged Into One. Zero BoxUsage In The Regions Holding The Flagged Snapshots → The Instances Are Long Gone, Only Backups Are Billing. And Correctly Stayed Quiet Once IPv4 Usage Matched The Load Balancer Count.

Cost, Measured, Not Estimated :

Model	Per Run	Per Year (52 Runs)
Nova Pro	$0.0035	$0.18
Claude Opus 5	$0.164 (in 4,296 / out 5,702, thinking included)	$8.53

47x More Expensive, $8.35 More A Year In Absolute Terms — Against The Cost Explorer API Call Itself Running $0.01/Week ($0.52/Year). At This Job’s Scale, Model Choice Was Never A Cost Problem, Only A Quality One. Production Switched To Opus 5 After This Comparison.

Switching Wasn’t Free, Though. Two Things Broke On The First Real Call: The Existing Claude Branch Sent A temperature Parameter On Every Call, And Opus 5 Has Dropped Sampling Parameters Entirely — A Flat 400. The Bare Model ID Also 400’d (“On-demand throughput isn’t supported”) Until Switched To The Same Inference-Profile Form Nova Already Used (us.anthropic.claude-opus-5). Adaptive Thinking Pushed A Single Advice Call To 82.9 Seconds, Which — Stacked On Top Of The Digest’s Own ~118 Seconds — Meant The Lambda Timeout Had To Double From 300s To 600s.

Two Bugs A $0.0014 Account Was Never Going To Trigger

Prepping To Open-Source The Project Meant Asking A Question The Account Running It Couldn’t Answer On Its Own: What Happens At Actual Scale?

Bug 3 — No Pagination On Cost Explorer Results. GetCostAndUsage Caps At 1,000 Groups Per Response, The Original Code Only Ever Read The First Page. The Failure Is The Dangerous Kind: The List Comes Back Shorter, With Nothing Indicating Anything Was Cut. Fixed With A Straightforward Pagination Loop.

Bug 4 — Flagging Ran Against The Already-Truncated Display List. The Display Step Trims Each Service To Its Top 4 Usage Types To Keep The Prompt Readable — But The Flagging Function Was Iterating That Same Trimmed List, Meaning The Smallest, Most Valuable Signals Were Exactly The Ones Most Likely To Be Cut First (An Idle IP At Usage 0.089 Sits Nowhere Near A Top-4-By-Cost Cutoff). Fixed By Scanning The Full List For Flags While Still Only Displaying The Top 4 Per Service. Flag Count On The Same Test Account Went From 7 To 14.

Notes

A gracefully-degrading except block is also a great place for a bug to live forever, invisibly. The whole point of catching that exception was so one broken feed wouldn’t take down the whole digest — which worked exactly as designed. But “worked as designed” and “worked correctly” turned out to be two different things, and only reading the actual log line told them apart. StatusCode: 200 was never going to say that.
IAM permission errors for Bedrock cross-region inference profiles don’t show up until the first real InvokeModel call. sam validate doesn’t know what ARN a profile ID actually resolves to at runtime — that information only exists behind the GetInferenceProfile API, so there’s no way to catch this at template-authoring time short of actually calling it and checking.
CAPABILITY_IAM vs CAPABILITY_NAMED_IAM is a one-line fix but a fully opaque error the first time you hit it. The moment a SAM template gives an AWS::IAM::Role an explicit RoleName (instead of letting CloudFormation generate one), the required capability silently changes.
A model producing generic advice is a signal problem before it’s a model problem. A cheap model fed good signal beats an expensive model fed a list of names.
Following an explicit instruction turned out to separate the two models more than any factual error did. The prompt said not to force a savings framing on this account. Nova ignored that on four of five findings; Opus followed it, and additionally did cross-referencing that the flagging logic alone never asked either model to do.
Both scale-dependent bugs look identical to correct behavior until they don’t. Same root cause as the RSS bug: success and silent-partial-failure produce identical-looking output. And neither was found by watching something break — both came from deliberately asking “would this hold up bigger?” before handing the code to someone else’s AWS account.

How To Prevent It

Scenario	What To Do
Writing an IAM policy for a Bedrock Cross-Region Inference Profile ID	Don’t string-substitute the profile ID into a `foundation-model` ARN. Call `GetInferenceProfile` to see the real destination model ARNs and grant those explicitly (or wildcard the region on a specific model ID)
Regex-stripping XML namespaces to avoid namespace-aware parsing	Strip the prefix off every tag occurrence too, not just the `xmlns` declarations on the root — otherwise the declarations disappear but the prefixed tags they were describing don’t
SAM template declares an explicit `RoleName` on an `AWS::IAM::Role`	Use `CAPABILITY_NAMED_IAM`, not `CAPABILITY_IAM`
Any code path with a silent `except: return []` for graceful degradation	After a “successful” deploy, still read the actual logs of a real invoke — don’t just trust `StatusCode: 200`
Handing an LLM an account’s resource list and expecting useful advice	Names alone produce advice that fits every account equally. Add usage-type or config-level granularity or the output degenerates to “consider configuring X” for anything on the list
Calling any AWS list/describe API expected to scale with account size	Check the documented page-size cap and implement pagination even if the test account never reaches it — a truncated-but-valid response produces no error
Building a “top N for display” step on top of a detection step	Run detection against the full, unfiltered list; truncate only at display time
Switching to a newer Claude generation on Bedrock	Confirm it still accepts `temperature`, use an inference-profile model ID, and re-measure latency — adaptive thinking can multiply a single call’s runtime

Reference:

[K8S] The Datasource Fix That Didn't Take

ChungWei Wei — Sat, 22 Aug 2026 12:21:55 +0000

Situation

Needed To Wire All Of Akamai’s Data Into An Existing Grafana On GKE. Split Into Two Dashboards: CDN Data Rode On A Plugin That Was Already There, And WAF/Bot Logs Went Through A Small Proxy I Wrote Myself, Because The Only Grafana Plugin That Speaks Akamai’s SIEM API Is A Paid Third-Party One. Didn’t Want To Pay For It. (Cheapskate.)

Both pieces hit the same kind of bug: something that looks fixed, deploys clean, and still doesn’t work — with nothing useful in the logs to explain why.

Result First:


Setup	Self-hosted Grafana on GKE. Akamai CDN traffic via the official plugin (unsigned, but already running), Akamai SIEM security events via a custom FastAPI proxy + the generic `yesoreyeram-infinity-datasource` plugin
Trigger	Plugin dashboard showed “No Data” even after the plugin loaded correctly. SIEM dashboard showed nothing at all, no error either
Real Cause (Trap One)	The plugin’s frontend called a legacy ID-based API route this Grafana version no longer serves — five separate call sites, not one
Real Cause (Trap Two + Three)	The SIEM datasource’s `url` was nested one level too deep, so requests went out with no host in them — and Grafana’s log had nothing to say about it, because the request never actually left. Fixing that in the YAML and redeploying changed nothing, because Grafana’s file-based datasource provisioner only applies YAML to a datasource that doesn’t exist in its database yet. Change the file for one that’s already there, and it’s silently ignored
Fix	Patch all five call sites, not just the obvious one; move `url` to the top level; and for an already-provisioned datasource, call the API directly to force the update — editing the YAML alone doesn’t do it

Every one of these failed the same way: no crash, no error, just nothing happening, and the fix looking correct right up until it silently wasn’t.

Trap One: An Unsigned Plugin’s Frontend Was Still Calling A Route That Doesn’t Exist Anymore

Akamai’s own plugin isn’t in Grafana’s signed catalog, so it has to load with GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS set. That part’s expected. What wasn’t expected: the plugin loaded fine, registered fine, and every panel just said “No Data.”

The frontend (module.js) was still building requests against Grafana’s old datasource-ID resource route:

/api/datasources/{id}/resources/...

This Grafana version only serves the uid-based route:

/api/datasources/uid/{uid}/resources/...

Patched the obvious one, the datasource’s getBackendDataSourceUrl helper, and redeployed. Still “No Data.” Turns out that helper wasn’t the only caller. The panel that actually fetches chart data calls the same pattern directly, twice, inline, in its own query() method, completely separate from the helper function. Four more call sites total, found only by grepping the whole bundle for the route pattern instead of trusting that one fix would propagate.

The part that made this worse than a normal bug hunt : fixing it and reloading the page didn’t show the fix either, at first. Grafana caches a plugin’s frontend assets keyed off the unchanged version string in plugin.json. Same version number, same cached JS, no matter how many times the container gets rebuilt underneath it. Had to stamp a fresh version string on every build just to be sure the browser wasn’t lying to me about what was actually running.

Trap Two: A `url` One Level Too Deep, And A Log That Had Nothing To Say About It

The SIEM side is a small FastAPI proxy that signs requests with Akamai’s EdgeGrid auth and hands Grafana back plain JSON, read by the generic (and actually signed) Infinity datasource plugin. Wired the datasource’s provisioning YAML, deployed, opened the dashboard: empty panels, no error banner, nothing in Grafana’s logs.

The mistake was one level of YAML nesting:

# wrong — url nested inside jsonData
jsonData:
  url: http://akamai-siem-proxy.monitoring.svc/siem

# right — url belongs at the datasource's own top level
url: http://akamai-siem-proxy.monitoring.svc/siem
jsonData:
  ...

With url in the wrong place, the top-level field Grafana actually reads stayed empty, and Infinity built its request as https:///siem?... — a URL with no host at all. Grafana’s own logs had nothing about this because the request never left the pod. Only way to actually see the failure was to bypass the dashboard and call POST /api/ds/query directly, simulating the exact panel query by hand, which finally returned the real error: http: no Host in request URL.

Trap Three: The Fix That Didn’t Take

Moved url to the top level, committed, let the GitOps pipeline sync, restarted Grafana. Opened the dashboard again. Still empty.

Checked the datasource’s actual stored config via the API — url was still blank, exactly as before the fix. Compared the restart’s logs against a Grafana pod that had just been created fresh: the fresh one logs a logger=provisioning.datasources line while it applies the YAML. The restarted one didn’t log that line at all. It wasn’t re-reading the file. It wasn’t touching that datasource.

Grafana’s file-based provisioner only writes a datasource’s config into its database the first time it sees that datasource. After that, the database row is authoritative, and the provisioner leaves it alone on every subsequent restart — even if the underlying YAML changed. There’s no flag for “please re-apply this,” and the docs don’t call this out anywhere obvious. The only way to actually change an already-provisioned datasource is to hit its API directly:

curl -u "admin:$ADMIN_PW" -X PUT "http://localhost:3000/api/datasources/uid/<uid>" \
  -H "Content-Type: application/json" \
  -d '{"name":"...","type":"...","access":"proxy","uid":"...","url":"...","jsonData":{...}}'

That call updates the database row directly, and it survives the next restart. Editing the YAML and redeploying is now step one of a two-step process for this cluster, not the whole fix — and it’ll stay that way for any datasource that already exists.

Notes

A url field that’s technically valid YAML in the wrong nesting level fails with no error, only silence. Grafana had no reason to complain — from its side, the field it expected was simply absent, and the datasource plugin just built a broken request quietly. Simulating the exact panel query via POST /api/ds/query surfaced the real error instantly; trusting the dashboard UI or the pod logs alone would not have.
A provisioner that only applies on first-create is a trap disguised as a convenience feature. It’s designed to avoid clobbering changes made through the UI, which is reasonable — but it means “I fixed the config file” and “the running system reflects that fix” are two different claims for anything already provisioned, and nothing tells you they’ve diverged.
An unsigned plugin’s frontend deserves the same scrutiny as its backend. The obvious fix (one helper function) looked complete. Grepping the whole bundle for every occurrence of the broken pattern found four more places doing the exact same thing, undocumented and un-abstracted.
Plugin asset caching keyed off a version string you control is easy to forget you control. If a rebuilt container doesn’t seem to have picked up a frontend change, check whether the cache key (plugin.json’s version) actually changed before assuming the build failed.
Once both pieces were actually working, a follow-on problem showed up that’s really the same theme again: a 12-hour SIEM query window can return 10,000+ events, and neither an unbounded table panel nor a naive “just get the count” query survive that volume without a real answer for how much data crosses the wire. Capping the table panel and adding a count-only mode that never materializes the full event list fixed it, and made the eventual fix look, in hindsight, exactly as inevitable as the query volume problem was invisible beforehand.

How To Prevent It

Scenario	What To Do
An unsigned/custom Grafana plugin shows “No Data” after loading correctly	Grep the entire frontend bundle for every API route pattern the backend actually serves, not just the one helper function that looks like the obvious entry point
A rebuilt plugin doesn’t seem to reflect a frontend change	Check whether `plugin.json`’s `version` actually changed — Grafana caches plugin assets keyed off that string, independent of what’s really on disk
A provisioned datasource/panel returns empty with nothing in any log	Bypass the dashboard and call `POST /api/ds/query` directly with the exact query the panel sends — it surfaces backend errors the UI silently swallows
Changing a datasource’s provisioning YAML for an already-existing datasource	Don’t trust a restart to apply it. Confirm via `logger=provisioning.datasources` in the pod’s boot log, or just call the datasource API directly (`PUT /api/datasources/uid/<uid>`) to be sure
A dashboard panel might pull a large, unbounded result set	Decide up front what each panel actually needs — a count doesn’t need the full payload — and enforce that at the data source, not just with a Grafana-side limit

Reference:

Grafana — Provisioning Data Sources
Grafana — Plugin Signature Verification
Real Incident: Self-hosted Grafana on GKE, Akamai CDN + SIEM integration, 2026-08

[K8S] The OIDC Fallback That Worked By Luck

ChungWei Wei — Mon, 17 Aug 2026 18:55:36 +0000

Situation

After I got Keycloak wired up, logging into the dashboard started getting stuck. No error, no crash log. Just one of those situations where you feel like you’re missing something obvious and can’t see it. Today I tried again, and this time there was a clear error in the log: Invalid parameter: redirect_uri

Result First:


Setup	Kubernetes dashboard (Headlamp) doing OIDC login against Keycloak, OIDC client config wired through a Helm chart’s “external secret” mode
First Symptom (4 days earlier)	Login just stalled after redirecting to the identity provider. No error, nothing actionable in the logs
Second Symptom (this time)	Identity provider rejected the callback outright: `Invalid parameter: redirect_uri`
Real Cause	The chart only wires `client-id`/`client-secret`/`issuer-url` unconditionally in external-secret mode. The callback-URL flag needs a separate, non-secret values field that I’d never set. Without it, the backend falls back to building the callback URL from the incoming request’s own scheme. Correct most of the time, wrong exactly when someone reaches the app over plain HTTP
Fix	Set the callback URL explicitly as a plain (non-secret) values field instead of relying on the Secret alone, the same string is already public in the identity provider’s client config anyway

Both symptoms trace back to the same root cause.

Trap One: A Chart’s OIDC Options Aren’t All Wired The Same Way

The chart supports pointing OIDC config at an existing Secret instead of writing credentials into values.yaml directly. Sane for anything that shouldn’t sit in git. Point it at a Secret with OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_ISSUER_URL, and OIDC_CALLBACK_URL, and the assumption is the app reads all four the same way.

It doesn’t. Reading the chart’s actual deployment template (not just its values.yaml comments) turned up the real logic: in external-secret mode, client-id, client-secret, and issuer-url get wired into CLI flags unconditionally. The callback URL only gets wired if a separate values field (not anything in the Secret) is non-empty:

{{- if or (ne $oidc.callbackURL "") (ne $callbackURL "") }}
- "-oidc-callback-url=$(OIDC_CALLBACK_URL)"
{{- end }}

$oidc.callbackURL here is config.oidc.callbackURL in values, a plain string I’d never set. Because the Secret already had the value and that seemed like the obvious place for it. The Secret key existed. The chart just never looked at it.

The fix that should be step one, not step three : for any chart that supports “point me at your own Secret” mode, read the actual template logic for which fields are wired unconditionally and which need an extra trigger. Don’t assume parallel-looking config fields all follow the same code path just because they’re documented next to each other.

Trap Two: A Silent Fallback Is Worse Than A Hard Failure

With that flag missing, the backend doesn’t error out. It falls back to constructing the callback URL from the request that’s currently hitting it:

func getOidcCallbackURL(r *http.Request, config *HeadlampConfig) string {
    if config.OidcCallbackURL != "" {
        return config.OidcCallbackURL
    }
    // ...falls back to r.Host + a scheme guessed from
    // X-Forwarded-Proto, or TLS state, or a default
}

This “usually correct” fallback is exactly why the first failure, four days earlier, produced no error: most requests arrived over HTTPS, the guessed scheme happened to match what the identity provider expected, and login silently worked by accident almost every time. The one thing that actually broke it was the app being reachable over plain HTTP with no forced redirect to HTTPS, and that wasn’t something I’d have thought to check. Nothing about “OIDC login” pointed at “check whether this app forces HTTPS.”

Reproducing it made the mechanism obvious once I tried both entry points on purpose:

# over HTTPS: the callback URL happens to match what the IdP expects
curl -sk "https://headlamp.example.internal/oidc?cluster=main" \
  | grep -oE 'redirect_uri=[^&"]+'
# redirect_uri=https%3A%2F%2Fheadlamp.example.internal%2Foidc-callback

# over plain HTTP: no forced redirect, so this request actually goes through as-is
curl -s "http://headlamp.example.internal/oidc?cluster=main" \
  | grep -oE 'redirect_uri=[^&"]+'
# redirect_uri=http%3A%2F%2Fheadlamp.example.internal%2Foidc-callback ← doesn't match what's registered

Same code path, same app. The result depends on which door you walked through: it passes testing whenever the tester happens to use the “normal” path, and only breaks for whoever doesn’t.

Notes

A fallback that’s “usually right” is a worse bug than one that’s always wrong. Always-wrong gets caught on the first test. Usually-right survives testing and shows up for someone else, later, as something that looks unrelated.
Config fields that live in the same block of a values file don’t necessarily follow the same code path. If a chart supports pulling secrets from an external Secret, check the template for which keys are actually read unconditionally versus which need an extra explicit trigger.
When a bug is intermittent, ask what’s different about the environment between “works” and “doesn’t,” not just what’s different in the app’s own logs. In this case the answer wasn’t in Headlamp or Keycloak at all. It was in whether the entry point enforced HTTPS.
A vague stuck-screen bug and a specific rejected-parameter error can be the same root cause, just surfaced through two different paths. Don’t assume a clearer error message later means a new bug. Check whether it’s the old one finally showing its face.

How To Prevent It

Scenario	What To Do
Any app config that supports “point me at your own Secret” for OIDC/SAML/etc.	Read the actual template/source for which fields are wired unconditionally versus which need a separate non-secret trigger; don’t assume documented-together means wired-together
A fallback/default value that depends on request context (scheme, host, headers)	Treat it as a source of intermittent bugs by default; pin the value explicitly wherever it’s cheap to do so, especially values that aren’t actually secret
A bug that “used to just not work” and now fails with a clear error	Check whether an upstream/downstream config changed in a way that makes the same underlying defect surface differently; don’t treat every clearer symptom as a brand-new investigation
Any service reachable over both HTTP and HTTPS with no forced redirect	Test scheme-dependent behavior (OIDC callbacks, cookie flags, CORS) over both entry points on purpose, not just the one you normally use

Reference:

Kubernetes — Authenticating with OpenID Connect Tokens
Keycloak Server Administration Guide
Real Incident: Home RKE2 cluster, Headlamp OIDC login via Keycloak

[GCP] GKE Internal Gatewa With Google Managed Cert Trap

ChungWei Wei — Sun, 09 Aug 2026 13:32:54 +0000

Situation

A Recent Task Was To Convert An Existing External Environment On GCP Into An Internal One. The Whole Setup Runs On GKE With Gateway API Handling Load Balancing. The Assumption Going In Was That A Google-Managed Certificate Would Attach To The Gateway API Internal LB The Same Way It Already Worked On The Gateway API External LB. That Assumption Ran Into Trouble Immediately.

Result First:


Expected	Attach An Existing Google-Managed Certificate Manager Cert To An Internal Regional Gateway, Same As Already Works On The External One
Actual	GKE’s Gateway Controller Silently Refuses — Treats The Certificate Reference As If It Doesn’t Exist
Root Cause	`gke-l7-rilb` Does Not Support Google-Managed Certificates At All. Verified Against The Official GatewayClass Capabilities Table, Then Confirmed Live By Actually Trying It
The Twist	The Underlying GCP Product — A Regional Internal Application Load Balancer — Has No Such Restriction. The Gap Is Specific To GKE’s Gateway API Controller, Not The Infrastructure It’s Built On
Fix	Bypassed The Controller Entirely — Built The Regional Internal ALB Directly Via Raw Compute Engine Resources, Reusing The Existing Backend Service

Turns Out “It Already Works On The External One” Was Never Good Evidence For Internal. Two GatewayClasses, Same API Surface, Same Cluster, Completely Different Certificate Support — And Nothing About The YAML You’d Write Tells You That Up Front.

Finding Out The Hard Way

First Move Was The Obvious One — Add The Certificate Manager Annotation To A Gateway And See What Happens:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  annotations:
    networking.gke.io/certificate-manager-certs: my-managed-cert
spec:
  gatewayClassName: gke-l7-rilb
  listeners:
    - name: https
      protocol: HTTPS
      port: 443

Applied Cleanly. Accepted: True. Then Programmed Flipped To False:

Error GWCER106: Gateway "..." is invalid, err: no certificates specified for protocol "HTTPS"

Read That Twice. The Annotation Is Right There. The Certificate Exists, Is Active, Has A Valid Expiry Date Months Out. GKE’s Own Error Message Is Reporting It As Though The Field Were Empty — Not “Certificate Invalid,” Not “Certificate Not Found,” Just Not Specified. That Phrasing Is The Tell: The Controller Isn’t Rejecting The Certificate. It’s Not Recognizing The Annotation As Applicable To This GatewayClass At All, And Falling Back To Its Generic “You Didn’t Give Me A Cert” Path.

Went To The Actual GatewayClass Capabilities Documentation Rather Than Assume From The Error Text Alone. The Table Is Explicit For Internal Regional (gke-l7-rilb) — Three Supported Certificate Mechanisms Listed:

Kubernetes Secret-Based Certificates
Self-Managed Compute Engine SSL Certificates
Self-Managed Certificate Manager Certificates

Google-Managed Certificates — Whether The Classic Compute Engine Kind Or The Certificate Manager Kind — Aren’t On That List For This GatewayClass. They Are For The External GatewayClasses (Both Regional And Global). That Asymmetry Is The Whole Bug: It’s Not That Internal Load Balancers Can’t Do Google-Managed Certs As A Category. It’s That This Specific GatewayClass’s Controller Implementation Never Wired Up That Path.

Notes

A Working External Example Told Me Nothing Useful About Internal. Same Cluster, Same Kind Of Certificate, Same General Idea Of “Attach A Cert To A Gateway” — And Zero Transferable Confidence, Because The Actual Support Matrix Is Keyed Per GatewayClass, Not Per Feature Category.
“No Certificates Specified” From A Config That Clearly Specifies One Is A Strong Signal The Field Is Being Ignored, Not Rejected. Worth Distinguishing Early — It Changes Where You Go Looking For Answers (Capability Docs, Not Certificate Troubleshooting).

The Twist: The Underlying Load Balancer Doesn’t Have This Problem

Before Accepting “Can’t Be Done,” Checked Whether The Restriction Was Really About The Load Balancer Product Or Just The Kubernetes-Level Abstraction Sitting On Top Of It. Google’s Own Certificate Manager Documentation Answered That Directly:

To deploy the regional Google-managed certificate to a regional external Application Load Balancer or regional internal Application Load Balancer , attach it directly to the target proxy.

gcloud compute target-https-proxies update PROXY_NAME --region=LOCATION \
  --certificate-manager-certificates=CERTIFICATE_NAME

No Caveat About Backend Type, No Mention Of GKE At All. The Regional Internal Application Load Balancer — As A GCP Product — Fully Supports Google-Managed Certificates On Its target-https-proxy. GKE’s Gateway API Just Never Exposed That Capability Through Its Own CRD For This GatewayClass. Two Completely Separate Layers: The Infrastructure Underneath Is Capable; The Kubernetes-Native Interface On Top Of It Has A Gap.

Confirmed It’s Not A Doc-Reading Error By Actually Building One — A Throwaway Regional Internal Backend Service, URL Map, Target HTTPS Proxy With That Exact Certificate Attached Via --certificate-manager-certificates, Forwarding Rule. It Came Up Fine. TLS Terminated Correctly, Traffic Reached The Backend.

Notes

A Kubernetes Abstraction’s Limitation Is Not Automatically The Underlying Cloud Product’s Limitation. GKE Gateway, Terraform’s Google Provider, gcloud, The Web Console — Each Is A Different Surface Over The Same Compute Engine Resources, And Each Can Lag The Others In Feature Coverage. When Something “Isn’t Supported,” Worth Asking Which Layer Is Actually Making That Claim.
The Fix Path That Worked Was Prefigured In The Error Message’s Vagueness. “No Certificates Specified” Rather Than A Certificate-Specific Rejection Was The Clue That The Real Gap Was In What The Controller Understood, Not In The Certificate Itself.

Working Around A Controller Gap, Not A Product Limitation

Given The Existing Gateway’s IP Was Already Depended On By Another Internal Service With The Address Hardcoded — Not Something To Touch Casually, And Recreating The Gateway Object Offered No Guarantee Of Reclaiming The Same IP Afterward — Chose Not To Rebuild It At All. Instead, Added A Second, Independent Entry Point Built From Raw Compute Resources, Pointed At The Exact Same Backend Service The Existing Gateway Already Used, With A Brand New Reserved Internal Address:

gcloud compute addresses create new-internal-addr --region=REGION --subnet=SUBNET \
  --purpose=SHARED_LOADBALANCER_VIP

gcloud compute url-maps create new-internal-urlmap --region=REGION \
  --default-service=EXISTING_BACKEND_SERVICE

gcloud compute target-https-proxies create new-internal-proxy --region=REGION \
  --url-map=new-internal-urlmap \
  --certificate-manager-certificates=CERTIFICATE_NAME

gcloud compute forwarding-rules create new-internal-fr --region=REGION \
  --load-balancing-scheme=INTERNAL_MANAGED \
  --network=NETWORK --subnet=SUBNET --address=new-internal-addr \
  --ports=443 --target-https-proxy=new-internal-proxy --target-https-proxy-region=REGION

Zero Existing Resources Touched. The Original HTTP-Only Gateway Kept Serving Its Dependent Exactly As Before. The New HTTPS Entry Point Came Up Pointed At The Same Backend, TLS Terminating Correctly, Verified From Inside The Cluster.

Notes

Reusing An Existing Backend Service Across Both The Old Gateway-Managed Path And The New Hand-Built One Meant Zero New NEGs, Zero New Health Checks — Genuinely Additive, Not A Parallel Copy Of The Whole Stack. A GCP Backend Service Can Be Referenced By Multiple URL Maps Simultaneously; No Need To Duplicate It Per Frontend.
“Rebuild The Load Balancer” And “Recreate The Same Kubernetes Object” Are Not The Same Sentence, Even Though They Sound Like It. The Actual Fork In The Road Was Never About Rebuilding Infrastructure — It Was About Which Control Plane Manages It Going Forward, And What Certificate Type That Choice Locks You Into.

Which One To Use

Scenario	What To Do
Attaching Any Certificate To A GKE Gateway API Object	Check The Official GatewayClass Capabilities Table For That Specific Class First — Don’t Infer From A Working Example On A Different GatewayClass In The Same Cluster
Getting `GWCER106` Or Any “No Certificate Specified” Error On A Gateway That Clearly References One	Treat It As A Sign The Field Isn’t Recognized For This GatewayClass, Not As A Certificate Problem — Go Check Capability Docs Before Re-Checking The Certificate
A Kubernetes Abstraction Claims A Cloud Feature Isn’t Supported	Verify Against The Underlying Cloud Product’s Own Documentation Before Accepting It As A Hard Limit — The Gap May Be In The K8s-Level Controller’s Feature Coverage, Not The Infrastructure
Need Google-Managed Cert Support On A Regional Internal ALB Backing A GKE Service, And The GatewayClass Won’t Cooperate	Build The Load Balancer’s Frontend (URL Map + Target HTTPS Proxy + Forwarding Rule) Directly Via Terraform/`gcloud`, Pointed At The Existing GKE-Managed Backend Service — No Need To Duplicate Backends Or NEGs
An Existing Internal IP Has Downstream Consumers With It Hardcoded	Don’t Recreate That Load Balancer Object To Add A Feature — Reserving The Exact Same IP Afterward Isn’t Guaranteed. Add A Second, Independent Entry Point Instead

Reference:

[K8S] The Time My Critical Security Bug Was Just kubectl Lying To Me

ChungWei Wei — Sun, 02 Aug 2026 17:53:58 +0000

Situation

My Home Lab Originally Only Used LLDAP For User Authentication, But Since The Systems I Wanted To Integrate Later Only Spoke OIDC, I Put Keycloak In Front Of It As A Bridge, With Everything Else Talking To Keycloak Instead. Wiring That Up And Testing It Turned Up Something Interesting: Got The LDAP Federation Wired Up, Got The OIDC Client Configured, Got Kubernetes Itself Trusting The Issuer. Then, Testing Whether Any Of It Actually Enforced Anything, I Ran One kubectl Command With A Made-Up Token String, And Got Back A Full Pod List. For About Thirty Seconds I Was Convinced I’d Found A Cluster-Wide Auth Bypass.

I Hadn’t. The Bug Was In My Test, Not My Cluster.

Result First:


Setup	Keycloak bridging LDAP (LLDAP) to OIDC, Kubernetes API server configured to trust the OIDC issuer
First Symptom	`kubectl --token="garbage"` returned a full, valid pod list — looked like anonymous cluster access
Real Cause	`sudo kubectl` silently picked up root’s existing admin kubeconfig; client-cert auth (TLS layer) authenticated the request before the bogus Bearer token was ever checked
Also Hit	LDAP Federation schema mismatch — tutorial’s default `objectClass` values don’t match what LLDAP actually uses, sync silently imports zero users, no error
Fix	Test with raw `curl` + explicit Bearer header, bypassing kubectl entirely; fix schema by querying the real directory with `ldapsearch` instead of copying tutorial defaults

Both traps share the same root lesson: when a config is supposed to match another system’s exact shape (a schema, an auth chain), verify against the real thing, don’t trust the tutorial default, and don’t trust a test tool that might be quietly injecting its own credentials.

Trap One: LDAP Federation’s Schema Match Is Exact, And Fails Silently

Keycloak’s LDAP User Federation builds its user-search filter from a field called userObjectClasses, roughly (&(objectClass=X)(objectClass=Y)(uid=%s)). Nearly every tutorial, including Keycloak’s own docs, defaults this to inetOrgPerson, organizationalPerson, because that’s the common AD/OpenLDAP combination.

My LDAP server is LLDAP (a lightweight LDAP implementation), and its actual user entries look like this:

ldapsearch -x -H ldap://ldap.internal:3890 \
  -D "uid=svc-bind,ou=people,dc=example,dc=internal" -w '<password>' \
  -b "ou=people,dc=example,dc=internal" "(uid=someuser)"


objectclass: inetOrgPerson
objectclass: posixAccount
objectclass: mailAccount
objectclass: person

No organizationalPerson. Copy the tutorial default in, and Keycloak’s search filter requires a class that literally doesn’t exist on any LLDAP user. The search always returns empty. The sync runs, reports added: 0, and throws no error at all. It just quietly does nothing. Switching to inetOrgPerson, person (both actually present) fixed it on the next sync.

The fix that should be step one, not step three : run ldapsearch against the real directory before configuring anything that has to match its schema. Don’t assume the tutorial’s defaults apply — especially once you’re off mainstream AD/OpenLDAP and onto something like LLDAP, FreeIPA, or 389-ds.

Trap Two: OIDC Login Working Doesn’t Mean Kubernetes Trusts It

Getting Headlamp to redirect to Keycloak, authenticate against LDAP, and come back with a valid ID token only proves the app recognizes you. Headlamp uses that ID token directly as a Bearer token against the Kubernetes API (it’s not proxying through its own ServiceAccount). There’s a separate, K8s-native trust relationship that has nothing to do with Keycloak or Headlamp:

# RKE2: /etc/rancher/rke2/config.yaml
kube-apiserver-arg:
  - "oidc-issuer-url=https://keycloak.example.internal/realms/homelab"
  - "oidc-client-id=headlamp"
  - "oidc-username-claim=email"
  - "oidc-groups-claim=groups"
  - "oidc-ca-file=/path/to/internal-ca.pem" # required if the issuer uses a self-signed cert

Two details that are easy to miss:

oidc-ca-file isn’t optional if your issuer runs on a self-signed cert. The API server fetches the OIDC discovery document and JWKS over its own system trust chain — it won’t inherit trust from anywhere else, and skipping this is a guaranteed x509 failure.
Successful OIDC authentication is not authorization. oidc-username-claim only answers “who is this,” RBAC still needs an explicit ClusterRoleBinding for that identity. A user with zero bindings logs into Keycloak and Headlamp just fine, then gets 403 Forbidden on every actual API call.

Trap Three: My Security Test Was Testing The Wrong Thing

Verifying whether OIDC actually gated the Kubernetes API, I ran this first:

sudo kubectl get pods -A --token="garbage" \
  --server=https://127.0.0.1:6443 --insecure-skip-tls-verify=true
# → 200 OK, full pod list

A made-up string as a token, and it returned everything. For a moment that reads as “this cluster has no auth at all.”

What actually happened: under sudo, kubectl defaults to reading root’s own kubeconfig, which already contains a valid admin client certificate. --token and --server are override flags that get merged into the same AuthInfo object. They don’t clear out a client-cert field that’s already sitting there. And TLS client-cert authentication happens at the handshake, before the request even carries an HTTP-level Bearer header. The request was authenticated as admin via mTLS the entire time. The garbage token was never actually evaluated.

The fix was to stop trusting a tool that might be silently carrying other credentials, and test the protocol directly:

# bogus token — should be 401
curl -sk -H "Authorization: Bearer garbage" \
  https://127.0.0.1:6443/api/v1/namespaces/default/pods
# → 401 ✓

# real OIDC token, no ClusterRoleBinding — should be 403
curl -sk -H "Authorization: Bearer <real_id_token>" \
  https://127.0.0.1:6443/api/v1/namespaces/default/pods
# → 403, with a message naming the identity and the denied verb ✓

401 and 403 that clearly trace back to the token’s content are trustworthy evidence. A 200 for an identity that shouldn’t have access is a signal to question the test setup first. It’s not a reason to immediately write up a vulnerability report.

Notes

LDAP schema mismatches fail silently, not loudly. added: 0 with zero errors looks like “nothing to sync,” not “your filter is wrong.” Query the real directory before configuring anything schema-dependent.
“The app logged me in” and “Kubernetes trusts this identity” are two separate trust relationships , configured in two separate places, and neither implies the other.
Authentication success is not authorization. Always confirm there’s an RBAC binding before declaring an OIDC integration “working.”
sudo + kubectl silently inherits root’s existing kubeconfig credentials. Any override flag you pass (--token, --server) gets merged into that AuthInfo, not substituted for it — client-cert auth wins regardless of what you put in --token.
When a security test returns a result that looks like “no protection at all,” the first suspect should be the test tool, not the system. High-level CLIs (kubectl, most cloud CLIs) auto-load credentials from multiple sources; test with the rawest possible client (curl, explicit headers) before trusting a surprising result.

How To Prevent It

Scenario	What To Do
Any LDAP/AD integration (SSO bridges, monitoring system directory auth, CI directory lookups)	Run `ldapsearch -x` (or your language’s LDAP client) against the real directory before configuring schema-matching fields like `userObjectClasses`/`memberOf` — especially off mainstream AD/OpenLDAP
Adding an OIDC identity provider in front of Kubernetes	Remember “app trusts OIDC” and “kube-apiserver trusts OIDC” are configured separately; confirm `oidc-username-claim` maps to whatever field you’ll actually bind RBAC to, and don’t forget `oidc-ca-file` for self-signed issuers
Manually verifying whether a token/credential check actually works	Test with the tool closest to the raw protocol (`curl` + manual headers) before trusting a high-level CLI — unless you’ve confirmed it isn’t silently carrying credentials you forgot about
Personal projects / CTF / homelab wanting hands-on enterprise IAM experience	Keycloak’s LDAP Federation + OIDC Client + `kube-apiserver --oidc-*` is one of the cheapest ways to actually operate a full centralized-identity flow end to end, and every step is independently testable

Reference:

Keycloak Server Administration Guide
Kubernetes — Authenticating with OpenID Connect Tokens
Real Incident: Home RKE2 cluster, Keycloak LDAP→OIDC bridge deployment

[K8S] Migrating From Envoy Gateway To Istio Ambient

ChungWei Wei — Sun, 26 Jul 2026 12:19:12 +0000

Situation

My Home RKE2 Cluster Had Envoy Gateway Handling North-South Traffic, Routed To Whatever Service Needed It. It Worked Fine. What It Didn’t Have Was Any Answer For East-West: The Handful Of Services Calling Each Other Internally Were Doing It In Plaintext, No Identity, No mTLS. I’d Been Meaning To Add A Service Mesh For A While, Mainly Because This Is Exactly The Kind Of Thing A Real Company Would Actually Do, And “Migrate A Live Ingress From One Implementation To Another, Zero Downtime” Is About As Real-Company As It Gets. Decided To Fully Replace Envoy Gateway With Istio’s Own Gateway In Ambient Mode, Rather Than Just Bolting Istio’s Mesh On The Side, And Did It The Same Day.

Result First:


Before	Envoy Gateway, north-south only, no mesh
Decision	Fully replace with Istio Gateway (not “add mesh, leave ingress alone”) — same mTLS either way, this route also removes a system and doubles as ingress-migration practice
Services Migrated	4 — Gitea (plain manifest), Bifrost/Headlamp/Keycloak (GitOps-managed)
Downtime	Zero
Biggest Surprise	The existing wildcard TLS cert needed zero changes — `certificateRefs` is a Gateway API field, not an Envoy or Istio proprietary mechanism
Actual Risk	Not the TLS cert (that was fine) — it was almost editing GitOps-managed routes by hand and getting silently reverted

The Two Routes Look Different In Effort, Not In What You Get

Without Comparing Service Mesh Options Against An Existing Gateway API Setup, It’s Easy To Assume Ripping Out The Current Ingress Controller Is What “Unlocks” Mesh Features Like mTLS. Gateway API (Who Handles Ingress) And Service Mesh (Who Handles Service-To-Service) Are Separate Concerns. You Can Install Istio’s Data Plane Next To An Untouched Envoy Gateway And Get Full mTLS Between Meshed Pods, No Ingress Changes Required. The Only Reason To Additionally Swap The Gateway Itself Is Consolidation (One Control Plane Instead Of Two) And, In This Case, The Migration Practice. If Someone’s Evaluating This For A Real Team And The Answer They’re Looking For Is “Which Gives Us More Mesh Capability” — Neither, They’re The Same. The Question Is Really “Do We Also Want To Consolidate Ingress.”

The Data Plane, Meanwhile, Went With Ambient Mode Over Sidecars. A Node-Level ztunnel Handles mTLS/L4 Instead Of A Proxy Container In Every Pod. My Nodes Aren’t Great, And Ambient’s Whole Pitch Is Lower Overhead At That Scale.

Trap One (That Wasn’t): The Wildcard Cert Just Worked

I Expected To Re-Issue Or At Least Re-Mount The Existing *.internal Wildcard TLS Cert When Switching Gateway Implementations. It Turns Out certificateRefs — The Field That Points A Gateway’s TLS Listener At A Kubernetes Secret — Is Part Of The Gateway API Spec Itself, Not Something Envoy Gateway Or Istio Each Bolt On Differently. Same Secret, Same Field Name, Different Controller Reading It:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: istio-gateway
  namespace: istio-system
spec:
  gatewayClassName: istio
  listeners:
  - name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        name: wildcard-tls # same Secret the old Envoy Gateway used
        namespace: envoy-gateway-system

The Only Thing Needed Was A ReferenceGrant In The Secret’s Namespace, Because The New Gateway Object Lives In A Different Namespace (istio-system) Than Where The Cert Secret Happened To Be Parked:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-istio-system-read-tls
  namespace: envoy-gateway-system
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: Gateway
    namespace: istio-system
  to:
  - group: ""
    kind: Secret
    name: wildcard-tls

Worth Calling Out: I’d Assumed Before Starting That A ReferenceGrant Was Already In Use Somewhere In The Existing Setup. It Wasn’t. The Old Gateway And The Secret Had Always Lived In The Same Namespace, So Cross-Namespace Referencing Had Simply Never Come Up Before. Good Reminder That “This Pattern Is Probably Already Handled Somewhere” Is A Guess, Not A Fact, Until You Actually Grep For It.

Trap Two: GitOps-Managed Routes Will Silently Revert Your Manual Edit

Three Of My Four Services (An AI Gateway, A K8s Dashboard, And An Identity Provider, I’ll Call Them Bifrost, Headlamp, And Keycloak, Their Real Names) Had Their HTTPRoute Managed By Flux — Which I Hadn’t Front-Of-Mind When I Started Testing. The Fourth, A Self-Hosted Gitea, Was A Plain Manual Manifest, So I Just kubectl patch-Ed It Directly, No Problem. For The Other Three, Editing The Live Object By Hand Would Have Worked For Exactly As Long As It Took Flux’s Next Reconcile Loop To Notice The Drift And Put It Back:

# This gets reverted on the next reconcile if the Kustomization has prune: true
# and the source repo still has the old parentRef
spec:
  parentRefs:
  - name: main-gateway # <- edit this live, Flux puts it right back
    namespace: envoy-gateway-system

The Actual Fix Was Editing The Source Repo, Not The Cluster:

git clone <gitops-repo>
# change parentRefs from main-gateway to istio-gateway in the HTTPRoute manifest
git commit -m "Migrate HTTPRoute to Istio Gateway"
git push
kubectl annotate kustomization <name> -n flux-system \
  reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite

None Of This Is Istio-Specific. It’s The Standard “Where’s The Actual Source Of Truth” Question A GitOps Setup Forces On Every Infra Change. But It’s Easy To Forget In The Moment When You’re Focused On Testing A New Gateway Config And Just Want To See If It Works.

Validating Without Touching DNS: Run Both Gateways In Parallel

The Zero-Downtime Part Came From Never Touching Production Routing Until The New Path Was Already Proven. MetalLB Handed The New Istio Gateway Its Own IP, Separate From The Old One, So Both Gateways Ran Side By Side. Testing The New Path Didn’t Require Touching DNS At All — curl --resolve Fakes The DNS Answer For A Single Request:

# hits the NEW gateway directly, real hostname for correct SNI/cert matching
curl --cacert ca.pem --resolve myservice.internal:443:<new-gateway-ip> \
  https://myservice.internal/
# 200

# side-by-side against the OLD gateway, still serving real production traffic
curl --cacert ca.pem --resolve myservice.internal:443:<old-gateway-ip> \
  https://myservice.internal/
# 200

Only After Every Service Came Back 200 On The New Path Did I Touch The Authoritative DNS Records (Two Servers, Both Had To Be Updated And Both Individually Verified — More On Why Below) And Point Them At The New Gateway’s IP. Repeated Per Service: Confirm On The New Path → Flip DNS For That One Service → Confirm Again On The Real Hostname → Move To The Next. Only After All Four Were Confirmed Live On The New Gateway Did I Uninstall The Old One.

Notes

certificateRefs Is A Gateway API Spec Field, Not An Envoy Or Istio Proprietary Mechanism. Switching Gateway Implementations Doesn’t Require Touching TLS Certs At All, As Long As The New Controller Also Implements Gateway API (Most Do). Confirm This Before Assuming A Cert Migration Task Even Exists.
A ReferenceGrant You’ve Never Needed Before Doesn’t Mean It’s Already Handled Elsewhere. It Might Just Mean The Situation Requiring It Never Came Up. Check, Don’t Assume.
GitOps-Managed Resources Revert Manual Edits On The Next Reconcile. This Applies To Any Ingress/Route Migration In A Flux Or ArgoCD-Managed Cluster, Not Just This One — Identify Which Resources Are GitOps-Owned Before Touching Anything Live.
curl --resolve Lets You Fully Validate A New Ingress Path — TLS, Routing, Backend — Without Touching DNS Or Risking Live Traffic. Flip DNS Only After Every Service Independently Passes On The New Path.
Actual Resource Overhead Was Smaller Than Expected. My Cluster’s Nodes Are Tight On Memory (One Physical Host Running All Five VMs, Already Low On Free RAM). I Was Ready To Abort Or Scale Back If Ambient’s Per-Node ztunnel Made That Worse. Real Numbers: 1-4 Percentage Points Of Memory Increase Per Node. Worth Measuring Before Assuming A Resource-Constrained Environment Can’t Handle It.

Real-World Application

Scenario	What To Do
Evaluating Whether To Add A Service Mesh On Top Of An Existing Gateway API Setup	Remember Gateway API And Service Mesh Are Orthogonal — You Can Add Mesh Capability (mTLS, Tracing) Without Touching Ingress At All. Only Consolidate The Two If You Specifically Want One Fewer System To Run
Migrating Ingress In Any Cluster Using Flux Or ArgoCD	Identify Which Routes Are GitOps-Managed Before Touching Anything. Edit The Source Repo, Not The Live Object — Manual Edits To Managed Resources Get Silently Reverted On The Next Reconcile
Any Zero-Downtime Ingress/LB Migration (Not Just Gateway API)	Run Old And New In Parallel On Separate IPs, Validate The New Path With `curl --resolve` Or Equivalent Per-Request DNS Override, And Only Flip Real DNS After Every Route Independently Passes
Resource-Constrained Clusters Considering Ambient Mode	Don’t Assume The Overhead Is Prohibitive Without Measuring — Take A `kubectl top nodes` Baseline Before Installing, Compare After. The Actual Number May Be Smaller Than The Sidecar-Model Reputation Suggests

Reference:

Istio — Ambient Mode
Istio — Gateway API Support
Real Incident: Home Lab RKE2 Cluster, Envoy Gateway → Istio Ambient Migration

[GCP] Three Traps Migrating GKE Gateway API To A Global LB

ChungWei Wei — Sat, 11 Jul 2026 16:58:38 +0000

Situation

Needed The PRD Domain’s Certificate To Auto-Renew, Which Meant Switching GatewayClass From Regional (gke-l7-regional-external-managed) To Global (gke-l7-global-external-managed). Certificate Manager’s Google-Managed Auto-Renewing Certs Only Support Global. That Night I Hit Three Traps Back To Back: Certificate Manager’s Annotation Syntax Is Completely Different, GCPBackendPolicy Can Only Bind One Service And Two Of Mine Fought Over It, And Global Security Policy’s Default Rule Is Allow, Not Deny. Five Days Later, Adding An Internal Gateway, The Second Trap Bit Me Again, Verbatim. That’s When I Knew It Wasn’t Bad Luck, It Was A Rule.

Result First:


Trigger	PRD Needed Certificate Manager Auto-Renewing Certs, Only Supported On Global GatewayClass
Traps Hit	① Certificate Manager Annotation Syntax (`GWCER105`) ② GCPBackendPolicy Conflict (Silent Failure, No Error At Apply Time) ③ Global Security Policy Defaults To Allow
Repeat Confirmation	5 Days Later, Adding An Internal Gateway, Trap ② Fired Again, Identical Symptom
Is This A GCP Design Flaw	Not Really. Regional And Global GatewayClass Map To Fundamentally Different Underlying Infrastructure, The Docs Just Don’t Spell Out The Cost Of Switching
Outcome	Global LB Migration Succeeded, PRD Cert Auto-Renewal Working

Gateway API’s Three-Layer Model, Quick Primer

If You Haven’t Touched Gateway API Yet: It Splits Ingress’s “One Resource Rules Everything” Mess Into Three Layers: GatewayClass (Vendor Declares Who Implements It, Here That’s GKE), Gateway (Infra Team Owns The Entry Point, Protocol, TLS), HTTPRoute (Developers Own The Routing Rules). Every Trap In This Post Happened At The GatewayClass Layer. Same GKE Gateway Controller, But Regional And Global GatewayClass Map To Completely Different Underlying GCP Resources (Regional External LB vs. Global External LB). Syntax, Limits, Behavior. None Of It Carries Over. Switching Between Them Isn’t Tweaking A Parameter, It’s Swapping Out The Whole Underlying Mechanism.

Trap One: Certificate Manager Mounts Differently On Regional vs Global

Copied The Regional Syntax First, Got This Immediately:

Error GWCER105: invalid TLS option key "networking.gke.io/certificate-map"

The Two GatewayClasses Don’t Share A Certificate-Mounting Syntax At All:

GatewayClass	Correct Syntax
`gke-l7-regional-external-managed`	`listeners[].tls.options["networking.gke.io/pre-shared-certs"]` (Regional Manages Its Own SSL Certificates, Doesn’t Support Certificate Manager)
`gke-l7-global-external-managed`	Not Under `listeners.tls` At All. Goes On The Gateway Object’s `metadata.annotations["networking.gke.io/certmap"]`, And That Listener Must Have No `tls` Block Whatsoever

It’s Not A Typo’d Key. It’s Two Entirely Different Mounting Mechanisms. One Lives At The Listener Level Under tls.options, The Other Lives At The Gateway Metadata Level As An Annotation. Different Locations Entirely. Copy The Regional Syntax Onto Global And It’s Guaranteed To Blow Up.

Trap Two: GCPBackendPolicy Only Binds One Service, And Fails Silently

This Is The Nastiest Of The Three Because It Doesn’t Error Out At kubectl apply Or helm upgrade Time :

kubectl get gcpbackendpolicy <name> -n <namespace> -o jsonpath='{.status}'
# conflicted with GCPBackendPolicy "xxx" of higher precedence, hence not applied

The Reason: A K8s Service (Regardless Of Whether It Backs A Regional Or Global Backend Service) Can Only Be Bound By One GCPBackendPolicy. If Both A Regional And A Global Gateway’s HTTPRoute Point At The Same Service (My Case: Each Side Needed A Different Cloud Armor Policy Attached), Whichever Got Created First Wins, And The Second One Just Silently Fails. No Error At Apply Time, You Have To Go Check status.conditions To Even See It. My Fix At The Time Was Going With “Only One Gateway Serves This Service At A Time,” Fully Tearing Down The Regional Gateway’s Resources So Global Could Claim The GCPBackendPolicy Binding. The Other Option Is Giving Each Gateway Its Own Dedicated Service (Same Pod Selector, Two ClusterIP Services).

Five Days Later, The Repeat : Adding An Internal Gateway (For Private Connectivity), I Hit The Exact Same Trap. PRD’s Main Service Had A Global Policy Attached, Which Couldn’t Bind To The Regional Backend Service The Internal Gateway Generated, Same SetSecurityPolicy ... does not exist Error. This Time I Just Applied What I’d Already Learned: Give The Internal Gateway Its Own Dedicated Service, Don’t Reuse The Main One. Actually Getting Bit A Second Time Was A Good Thing. It Proved “One Service, One GCPBackendPolicy” Wasn’t A One-Off Quirk Of This Particular Migration. It’s A Checklist Item For Every New Gateway You Ever Add.

Trap Three: Global Security Policy Defaults To Allow, Not Deny

gcloud compute security-policies describe <policy-name> --format=yaml
# priority 2147483647 "default rule" defaults to action: allow, your allowlist does nothing

Regional Policy (google_compute_region_security_policy) Doesn’t Have This Problem, But The Moment You Create A Global Policy (google_compute_security_policy), GCP Auto-Generates A Priority 2147483647 allow Default Rule. That Priority Collides With Whatever Deny-All Rule You Meant To Create By Hand (Cannot have rules with the same priorities). The Right Fix Isn’t Deleting And Recreating. It’s terraform import-ing The Auto-Generated Rule Into State, Then Using Terraform To Flip The Action To Deny:

terraform import google_compute_security_policy_rule.<resource_name> \
  "projects/<project>/global/securityPolicies/<policy-name>/priority/2147483647"
terraform plan # Should Show An In-Place Update: action: allow -> deny(403)
terraform apply

Notes

GWCER105 Isn’t A Typo’d Key, It’s Two Entirely Different Cert-Mounting Mechanisms. Regional Uses Listener-Level tls.options, Global Uses A Gateway-Metadata-Level Annotation. Different Locations Entirely. Copy One Onto The Other, It Breaks.
GCPBackendPolicy Conflicts Are The Nastiest Of The Three Because They Don’t Error At Apply Time. You Only See “Conflicted With … Of Higher Precedence, Hence Not Applied” By Checking status.conditions. First Created Wins, Second One Fails Silently.
Import The Global Security Policy’s Auto-Generated Allow Rule Into Terraform State, Don’t Delete And Recreate It. The Priority Is A Fixed Value (2147483647) That Collides With Any Hand-Built Deny-All Rule.
Global LB Provisioning Takes Noticeably Longer Than Regional. Regional Usually Programs In 1-3 Minutes, Google’s Docs Say Global Can Take 10-15 (Config Has To Propagate To Global GFE Nodes), And Even After Programmed: True, TLS Taking Full Effect At The Edge Can Take A Few More Minutes. If curl Throws SSL_ERROR_SYSCALL But openssl s_client Tests Fine, It’s Usually Just Still Propagating. Wait It Out.
This Same Migration Also Hit A Helm Field Manager Conflict , Which I Pushed Through With --server-side --force-conflicts And Ended Up Wiping A Namespace As A Result. That’s A Completely Different Mechanism (Helm 3’s Release Tracking/Delete Logic), Covered In A Separate Post, Not Repeated Here.

The Rule: Check Service Sharing Before Adding Any New Gateway

The Real Value Of This Post Isn’t The Three Traps Individually. It’s That Trap Two Got Confirmed Twice. Hit It Once, You’d Chalk It Up To “Just This Migration’s Bad Luck.” Hit It Again Five Days Later Adding An Internal Gateway, And Now It’s Confirmed: This Is A Checklist Item For Every New Gateway You Ever Add , Not A One-Time Fluke. External Or Internal, Regional Or Global. Before Adding A New Gateway, Check Whether It’ll Share A Service With An Existing One. If It Does, Pick One Of Trap Two’s Two Fixes (“Only One Gateway Serves This Service At A Time” Or “Give Each Gateway Its Own Service”) Upfront, Don’t Wait For The Silent Failure To Send You Digging.

Real-World Application

Scenario	What To Do
Any Environment Running GKE Gateway API (Not Just Regional/Global Mixed)	Before Adding Any New Gateway (External/Internal, Regional/Global), Check Whether It’ll Share A Service With An Existing Gateway. If So, Pick One Of Trap Two’s Two Fixes Upfront. Don’t Find Out Via `status.conditions` After The Fact
Teams Setting Up Auto-Renewing Certificates	Confirm Upfront That Certificate Manager Only Supports Global GatewayClass. Regional Is Stuck With Manually-Managed Pre-Shared Certs. This Constraint Decides Whether You’re Migrating At All, It’s Not A Choice Of Syntax
Environments Managing Cloud Armor / Security Policy Via Terraform	When Creating A Global Policy, Always Check Whether The Auto-Generated Allow Default Rule Got Imported Into State. Don’t Assume It Behaves Like A Regional Policy
Any GCP Resource Migration (Not Just Gateway API)	The Difference Between “I Just Happened To Hit This” And “This Is A Rule” Usually Only Becomes Clear On A Second, Independent Occurrence. When Something Looks Like A Coincidence, Write It Down Instead Of Closing The Ticket, So You Recognize It Instantly The Next Time

Reference:

Secure a Gateway | GKE
GatewayClass capabilities | GKE
GCPBackendPolicy | GKE Gateway API
Real Incident: Internal Service, PRD Regional→Global Migration; Internal Gateway Repeat Confirmation

[GCP] Deploy vLLM On Cloud Run (GPU & CPU)

ChungWei Wei — Sat, 11 Jul 2026 04:02:10 +0000

Situation

Want To Deploy vLLM On Cloud Run To Serve An LLM. Two Paths: GPU Mode (For Real Service) Or CPU Mode (For Tiny Model / Architecture Test Only). Both Fully Tested End To End.

Result First:

	GPU Mode	CPU Mode
Support Level	GA	Experimental
Tested Model	Gemma-4-E2B-it (L4)	Qwen2.5-0.5B-Instruct
Response Time	6.1s	114s
Build Effort	Low, Use Official Image	High, ~56 Min Multi-Stage Build
Use Case	Real Service, Demo	Architecture Test Only

GPU Is About 19x Faster Than CPU. CPU Mode Works, But Don’t Expect Speed.

How To (GPU Mode)

Requirement:

	L4	RTX PRO 6000 Blackwell
Min CPU	4 (8 Recommended)	20
Min Memory	16 GiB (32 GiB Recommended)	80 GiB
Region	asia-southeast1 / us-central1 / europe-west1 / europe-west4 / us-east4	asia-southeast1 / us-central1 / europe-west4 / asia-south2
Billing	Instance-Based Billing Required	Same

First-Time GPU Deploy Get A Default Quota (3 GPU / Project / Region).

Deploy With This Command:

gcloud run deploy vllm-gpu-service \
  --image=us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:gemma4 \
  --region=asia-southeast1 \
  --cpu=8 \
  --memory=32Gi \
  --gpu=1 \
  --gpu-type=nvidia-l4 \
  --no-gpu-zonal-redundancy \
  --no-cpu-throttling \
  --execution-environment=gen2 \
  --concurrency=4 \
  --max-instances=1 \
  --timeout=600 \
  --port=8080 \
  --startup-probe=tcpSocket.port=8080,initialDelaySeconds=60,timeoutSeconds=240,failureThreshold=2,periodSeconds=240 \
  --no-allow-unauthenticated \
  --args="vllm,serve,google/gemma-4-E2B-it,--enforce-eager,--enable-chunked-prefill,--enable-prefix-caching,--generation-config=auto,--enable-auto-tool-choice,--tool-call-parser=gemma4,--reasoning-parser=gemma4,--dtype=bfloat16,--max-num-seqs=64,--gpu-memory-utilization=0.9,--tensor-parallel-size=1,--port=8080,--host=0.0.0.0"

Test It:

$ curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  https://vllm-gpu-service-xxx.asia-southeast1.run.app/v1/chat/completions \
  -d '{"model":"google/gemma-4-E2B-it","messages":[{"role":"user","content":"用一句話介紹你自己"}],"max_tokens":100}'


{"choices":[{"message":{"content":"我是一個由 Google DeepMind 開發的開放權重大型語言模型，名為 Gemma 4。"}}],
 "usage":{"prompt_tokens":14,"completion_tokens":23,"total_tokens":37}}

Total Round Trip: 6.1s.

Notes

Don’t Use The Generic docker.io/vllm/vllm-openai:latest Image. It’s Built Against A Newer CUDA Than Cloud Run L4’s Driver (535.x / CUDA 12.2) Supports, Fails With RuntimeError: The NVIDIA driver on your system is too old (found version 12020). Use Google’s Own Rebuilt Model Garden Image Above Instead — This Is Also Why The Official Codelab Never Uses The Generic Image.
This Image’s Entrypoint Doesn’t Auto-Prepend vllm. It’s A Launcher Script (gcs_download_launcher.sh) That exec Whatever You Give It. Args Must Start With vllm,serve,..., Not Just serve,..., Or You Get serve: command not found.
--enforce-eager Is Required, Not Optional, Unless You Extend The Startup Probe A Lot. Without It, vLLM Does torch.compile + CUDA Graph Capture Across Batch Sizes 1 To 128 On First Boot, Which Takes Over 5 Minutes — Longer Than Most Startup Probe Configs, So The Deploy Just Rolls Back. With It, Cold Start Is Under A Minute, At The Cost Of Slightly Slower Steady-State Throughput.
Port Is 8080 Here, Not vLLM’s Usual 8000. Match --port To Whatever The Image Actually Listens On.
--max-instances Does Not Prevent Cold Start , It’s Only A Concurrency Ceiling. Use --min-instances=1 If Cold Start Is Unacceptable — That Means Paying For A GPU Instance 24/7. Without It, The Service Scales To Zero When Idle, No Charge.

How To (CPU Mode)

Only Makes Sense For Very Small Models. pip install vllm Assumes CUDA, Won’t Run Inference On A CPU-Only Host. Official Path Is Building From docker/Dockerfile.cpu — Don’t Hand-Roll A Simplified Version, Pull The Real File:

git clone --depth 1 https://github.com/vllm-project/vllm.git
cd vllm

Build With This cloudbuild.yaml:

steps:
  - name: 'gcr.io/cloud-builders/docker'
    env:
      - 'DOCKER_BUILDKIT=1'
    args:
      - 'build'
      - '-f'
      - 'vllm/docker/Dockerfile.cpu'
      - '--target'
      - 'vllm-openai'
      - '--build-arg'
      - 'max_jobs=8'
      - '-t'
      - 'asia-southeast1-docker.pkg.dev/$PROJECT_ID/YOUR_REPO/vllm-cpu:latest'
      - 'vllm'

images:
  - 'asia-southeast1-docker.pkg.dev/$PROJECT_ID/YOUR_REPO/vllm-cpu:latest'

options:
  machineType: 'E2_HIGHCPU_32'

timeout: 7200s


gcloud builds submit --config=cloudbuild.yaml --region=asia-southeast1 .

Deploy With This Command:

gcloud run deploy vllm-cpu-service \
  --image=asia-southeast1-docker.pkg.dev/YOUR_PROJECT/YOUR_REPO/vllm-cpu:latest \
  --region=asia-southeast1 \
  --memory=10Gi \
  --cpu=4 \
  --timeout=600 \
  --port=8000 \
  --set-env-vars=VLLM_CPU_KVCACHE_SPACE=4,VLLM_ENABLE_V1_MULTIPROCESSING=0,VLLM_HOST_IP=127.0.0.1,GLOO_SOCKET_IFNAME=lo \
  --args="Qwen/Qwen2.5-0.5B-Instruct,--port=8000,--host=0.0.0.0" \
  --startup-probe=tcpSocket.port=8000,initialDelaySeconds=60,timeoutSeconds=240,failureThreshold=1,periodSeconds=240 \
  --no-allow-unauthenticated

Test It:

$ curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  https://vllm-cpu-service-xxx.asia-southeast1.run.app/v1/chat/completions \
  -d '{"model":"Qwen/Qwen2.5-0.5B-Instruct","messages":[{"role":"user","content":"用一句話介紹你自己"}],"max_tokens":100}'


{"choices":[{"message":{"content":"我是由阿里云开发的超大规模语言模型，我叫通义千问。我的目标是帮助用户理解和创造。"}}],
 "usage":{"prompt_tokens":34,"completion_tokens":27,"total_tokens":61}}

Total: 114s, 34 Prompt Tokens + 27 Completion Tokens, 0.5B Model.

Notes

Cloud Build’s Default Docker Builder Doesn’t Have BuildKit Enabled. The Official Dockerfile Uses RUN --mount=type=cache,... (BuildKit-Only), Fails With the --mount option requires BuildKit. Add env: ['DOCKER_BUILDKIT=1'] To The Build Step.
Cloud Build Machine Types Are All High-CPU-Low-Memory (e2-highcpu-32 = 32 vCPU / 32GB RAM, No High-Mem Option Exists). Default MAX_JOBS=32 OOM-Kills The Build Machine Mid-Compile (status "INTERNAL_ERROR", Not A Normal Build Failure) During The Triton-CPU Stage. Lower It With --build-arg max_jobs=8. Full Build Then Takes About 56 Min.
Keep .git In The Build Context. vLLM Uses setuptools-scm To Derive Its Version From Git History — Strip .git To Save Upload Size And The Build Fails Hard With setuptools-scm was unable to detect version. A --depth 1 Shallow Clone Is Enough, Costs About 40MB Extra.
vllm serve Takes The Model Name As A Positional Argument , Not A --model= Flag.
Model Loads Fine But The Port Never Opens (shm_broadcast.py: No available shared memory broadcast block found in 60 seconds) — vLLM’s Multi-Process Architecture Needs /dev/shm For IPC, And Cloud Run’s Default Is Too Small. Mounting A Bigger In-Memory Volume At /dev/shm Does Not Work, Cloud Run Rejects It Outright (Mount paths [/dev/shm] are in one of disallowed mount points [/dev, /proc, /sys]). The Actual Fix Is Avoiding The Dependency, Not Enlarging It: --set-env-vars=VLLM_ENABLE_V1_MULTIPROCESSING=0,VLLM_HOST_IP=127.0.0.1,GLOO_SOCKET_IFNAME=lo. First Var Keeps The Engine Mostly In One Process; The Other Two Force torch.distributed’s TCP Rendezvous Onto Loopback, Because It Otherwise Times Out Retrying Inside Cloud Run’s gVisor Sandboxed Network ([c10d] The server socket ... has timed out, will retry).

Which One To Use

GPU Mode For Anything Real — Just Remember To Use Google’s Model Garden Image, Not The Generic vllm/vllm-openai. CPU Mode Hits Deeper Issues (shm Architecture Limits, gVisor Sandbox Network Compatibility) And Is Really Only Good For Validating The Architecture Or A Very-Low-Traffic Demo, Not Production.

Cloud Run Now Support GPU (NVIDIA L4 / RTX PRO 6000 Blackwell) Directly, No Need To Go Through GKE Anymore. GPU Mode Just Means Adding --gpu And --gpu-type To A Normal gcloud run deploy, Cloud Run Handles The Driver And Scheduling — That’s What Makes The GPU Path Above So Much Simpler Than It Used To Be.

Reference:

DEV Community: ChungWei Wei

[AI] LM Studio Couldn't Run This Model, So I Got It Running On My Own Mac

Situation

How It Broke (Root Cause)

The Runtime Is Frozen, The Model Isn’t

Wanting To Centralize It Ran Into A Harder Wall

The Fix

Skip The Runtime, Use The Library Directly

Wrap It So Nothing Downstream Has To Change

Make It Actually Stay Up

One Serialization Detail That Matters

A Week Later: Staying Up Isn’t The Same As Staying Fast

Notes

How To Prevent It

[GCP]GCP Weekly Digest Report

Situation

How It Broke (Root Cause)

Surprise 1: The SDK The Migration Plan Assumed No Longer Exists

Surprise 2: The Model From The Old Version Retires In Under Two Months

Surprise 3: The Blog RSS Feed The Old Code Points At Just… Isn’t RSS Anymore

Near-Miss: Testing Gemini Auth With The Wrong Google Account’s Cached Credentials

Building The Advice Half: Detection Belongs To Google, Not To Me

Advice Bug 1: The Location Wildcard That Looks Documented But Isn’t

Advice Bug 2: 42 Silent 403s That Reported Themselves As Success

Advice Bug 3: The Location List That Silently Excluded The Only Resources That Existed

Mail Blocker 1: SendGrid Banned The Account, Then The Org Wouldn’t Let Anyone Download A Key

Mail Blocker 2: 403 After Every Other Piece Was Already Right

Notes

How To Prevent It

[K8s] I Watched A VPA Recommendation For A Month — It Never Moved, Because It Was Never Really A Recommendation

Situation

What A Month Of History Actually Changed

The Answer To Part 1’s Open Question

Notes

The Number I Waved Through In Part 1

Notes

Real-World Application

[K8s]Vertical Pod Autoscaler Isn't A Cloud Feature — I Put It On A Bare RKE2 Cluster And Its First Answer Was 10x Reality

Situation

Confirming VPA Is Cloud-Agnostic

Two Install Paths, Picked The Less Manual One

Notes

The Part I Didn’t Expect: The First Real Recommendation

Why Memory Recommendations Are Built To Run High

Notes

What Happens Next

Real-World Application

[AWS]AWS Weekly Digest Report

Situation

How It Broke (Root Cause)

Bug 1: The Cross-Region Inference Profile ARN Trap

Bug 2: The RSS Parser That Silently Gave Up On Every Blog Post

Two Failed Prompts Before Account Advice Was Worth Reading

Nova Pro vs Claude Opus 5: Same Data, Two Different Levels Of Reasoning

Two Bugs A $0.0014 Account Was Never Going To Trigger

Notes

How To Prevent It

[K8S] The Datasource Fix That Didn't Take

Situation

Trap One: An Unsigned Plugin’s Frontend Was Still Calling A Route That Doesn’t Exist Anymore

Trap Two: A url One Level Too Deep, And A Log That Had Nothing To Say About It

Trap Three: The Fix That Didn’t Take

Notes

How To Prevent It

[K8S] The OIDC Fallback That Worked By Luck

Situation

Trap One: A Chart’s OIDC Options Aren’t All Wired The Same Way

Trap Two: A Silent Fallback Is Worse Than A Hard Failure

Notes

How To Prevent It

[GCP] GKE Internal Gatewa With Google Managed Cert Trap

Situation

Finding Out The Hard Way

Notes

The Twist: The Underlying Load Balancer Doesn’t Have This Problem

Notes

Working Around A Controller Gap, Not A Product Limitation

Notes

Which One To Use

[K8S] The Time My Critical Security Bug Was Just kubectl Lying To Me

Trap Two: A `url` One Level Too Deep, And A Log That Had Nothing To Say About It