DEV Community: Paulo Victor Leite Lima Gomes

shadow ai is now part of the ci/cd threat model

Paulo Victor Leite Lima Gomes — Sun, 09 Aug 2026 00:03:17 +0000

For a while, shadow AI sounded like the new version of shadow IT.

Someone used a browser tool without approval. Someone installed an IDE extension. Someone pasted a stack trace into a chatbot and promised, with the confidence of a person who has not read the data policy, that there was "nothing sensitive in there."

Annoying? Yes.

New? Not really.

Then the same behavior walks into CI/CD, and the joke changes.

CNCF published a fresh piece on shadow AI in CI/CD, framing the problem as a path from developer laptop to Kubernetes. That is the right framing. The issue is not that developers are using AI. They are. The issue is that AI tools, model calls, extensions, MCP servers, and agents can quietly become part of the delivery path without being named as infrastructure.

And unnamed infrastructure is my least favorite kind. It always waits until a Friday to introduce itself.

ci/cd is not a vibes zone

CI/CD is where code becomes production. That sounds obvious, but people forget it when the tool looks like a helpful assistant instead of a deployment component.

An AI extension that suggests code is one thing. An agent that opens pull requests, edits workflows, calls internal tools, reads secrets, writes artifacts, comments on review findings, generates Dockerfiles, updates Kubernetes manifests, or triggers release jobs is very much another thing.

At that point, the agent is not "developer productivity."

It is part of the software supply chain.

If the answer is "well, it is just installed on a few laptops," congratulations. You have built a distributed production dependency with worse asset inventory.

the laptop is now part of the pipeline

The classic supply-chain threat model already cared about developer machines. AI makes that boundary softer.

A coding assistant can read a repository. A browser agent can read docs, issues, and dashboards. An MCP server can expose local files, databases, internal APIs, or cloud operations. A terminal agent can run commands. A review bot can influence what humans approve. A workflow generator can change the YAML that later runs with stronger permissions in CI.

None of these are automatically bad. I use these tools. I also prefer them with guards.

The problem is the invisible bridge.

A developer installs an AI helper to save time. The helper writes code. The code changes a build step. The build step pulls a dependency. The dependency runs in CI. CI has tokens that can publish artifacts, push images, deploy to Kubernetes, or talk to cloud APIs. That is not a moral panic. That is just a graph.

prompt injection gets a delivery badge

Prompt injection is annoying in a chat window. In CI/CD it can become operational.

Imagine an agent reading an issue, a README, a generated dependency report, or a web page during a build-related task. The content tells it, politely and confidently, to ignore previous instructions and fetch a secret, change a workflow, install a package, or call a tool it should not call.

The model is not "hacked" in the movie sense. It is interpreting untrusted text while tools are available. That is enough.

The old web security lesson still applies: data and instructions need boundaries. We learned this with SQL injection. We learned it with shell injection. We learned it with XSS. Now we get to learn it again with agents, because apparently software engineering is a subscription service to the same class of mistakes.

CI/CD makes this sharper because the environment has authority. Build systems sign artifacts. They publish packages. They push container images. They deploy. They comment on pull requests in ways humans may trust. They often have more permissions than any single human needs, because the pipeline is where everybody's shortcuts go to become policy.

An agent operating near that environment needs the same suspicion we give any automation with credentials.

inventory first, philosophy later

The first useful step is boring: inventory.

Which AI tools are installed by developers? Which are approved? Which can read source, call tools, write to GitHub, or run inside CI? Which MCP servers exist? Which secrets are exposed to jobs that can run agent-written scripts?

This is not glamorous work. It will not get a keynote. It will produce a spreadsheet with several rows named things like temp-copilot-test and old-agent-demo-final2, which is how you know you are doing real platform engineering.

But without inventory, policy is theatre.

You cannot enforce an allowlist if nobody knows what is being used. You cannot audit a model call that never entered your logging path. You cannot rotate a token that was copied into a local agent config six months ago by someone who has since joined a team called Innovation Enablement, which is where accountability goes to wear a blazer.

policy belongs in the control plane

The second step is moving from advice to controls.

"Please do not paste secrets into random tools" is a good sentence. It is not a security architecture.

This is why the enterprise features around AI tools matter even when they sound dull. MCP allowlists, org-level defaults, session limits, review labels, audit trails, and policy-managed tool access are not corporate checkbox dust. They are the beginning of treating agents like production actors.

The policy should not say "no AI." That battle is mostly over, and banning useful tools usually creates a worse shadow version.

The policy should say: these agents can run here, with these tools, under these identities, with these logs, with these spending limits, against these repositories, and with these review requirements.

That is a sentence platform teams can implement.

ci permissions should be smaller than your ambition

Most CI systems are too trusting by default because they grew up around scripts written by people inside the company. That was already optimistic. Agent-generated changes make it funnier.

If an agent can modify a workflow file, and that workflow later runs with broad tokens, you have a privilege-escalation shape. If pull request builds can access secrets too early, you have a data-exfiltration shape. If a generated script can download and execute arbitrary dependencies, you have a supply-chain shape. If a bot can approve its own changes through a friendly review loop, you have invented management consulting for malware.

The answer is not exotic.

Use least privilege. Separate read and write tokens. Treat workflow changes as sensitive. Require human review for delivery definitions. Limit egress from build jobs. Pin actions where it matters. Keep production deploy credentials away from anything that runs untrusted code.

The agent angle does not replace normal CI/CD hardening.

It removes excuses for postponing it.

review the output, but also review the path

Humans are still in the loop, but the loop needs to include more than the final diff.

If an agent opened a pull request, I want to know what it touched, what commands it ran, which tools it called, which external context it used, and whether it changed the delivery path. A clean-looking diff can still have a dirty path behind it.

This does not mean every agent run needs a courtroom transcript. It means risky actions need traceability. For CI workflows, infrastructure-as-code, Kubernetes manifests, secrets plumbing, package publishing, auth code, and payment paths, the bar should be higher.

shadow ai is a naming failure

I do not think shadow AI in CI/CD is mainly a developer behavior problem. Developers use tools that help them ship. This is not surprising. Water is wet. YAML is still a crime scene.

The real problem is letting useful tools enter the delivery path without turning them into named, governed infrastructure.

Once an AI system can influence code, builds, artifacts, credentials, deployments, or Kubernetes state, it needs an owner. It needs inventory. It needs logs. It needs permissions. It needs limits. It needs a way to be disabled without asking six teams who installed what.

This is boring platform work.

Good.

Boring platform work is how we survive exciting tooling.

Shadow AI stops being a cultural debate the moment it touches CI/CD. From there, it is supply-chain infrastructure without a nameplate.

Put the nameplate on it.

Then give it the smallest key that lets it do the job.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

prometheus collectors are cloud control plane now

Paulo Victor Leite Lima Gomes — Sat, 08 Aug 2026 00:08:29 +0000

Prometheus used to have a nice emotional contract: you ran the thing, scraped the thing, and cursed at the thing when a target disappeared because somebody changed a label during a deployment and now the dashboard looks like a crime scene.

It was not pleasant, but at least ownership was obvious.

Then AWS announced CloudWatch managed Prometheus collectors, and the shape changed. CloudWatch can now run fully managed collectors that discover Prometheus-compatible targets in AWS infrastructure, scrape them through VPC connectivity, and deliver the metrics into CloudWatch. EKS, EC2, ECS, MSK, OpenSearch. The usual suspects.

This is genuinely useful.

It also means the scrape layer is not just a daemonset, sidecar, or collector you forgot to upgrade. It is becoming another cloud control plane.

That is worth paying attention to.

nobody enjoys operating scrapers

Let us be honest. Maintaining metrics collection infrastructure is not the part of observability that makes people feel alive. Somebody has to run collectors, patch them, scale them, give them network access, keep configs synced, notice when service discovery breaks, and stop the high-cardinality endpoint from eating the month.

Prometheus is excellent. Prometheus also makes you own the scraping path.

AWS taking more of that burden makes sense. The new collectors are agentless from the customer's point of view. You provide scrape configuration and a connection to the resources. CloudWatch provisions and scales the machinery. The docs say the collector creates ENIs in selected subnets, scrapes through them using OTLP, and delivers metrics to a CloudWatch dataset through a VPC endpoint.

That removes operational work and moves it into the platform contract.

the scrape config is production code

Every observability system eventually teaches the same annoying lesson:

The dashboard is not the product.

The pipeline is.

If the scraper misses a target, the dashboard lies politely. If relabeling is wrong, the alert routes to the wrong team. If a metric selector is too wide, the bill grows. If it is too narrow, the incident starts with everyone saying "that is weird, the graph is flat."

With managed collectors, scrape configuration is production code. It decides what gets seen, how it is named, where it lands, and how much it costs. That means it needs the same discipline as other production configuration:

code review
owners
environment separation
rollback
change history
label conventions
cost expectations
tests or at least validation queries

Yes, tests for observability config sound boring. So does checking a parachute.

managed does not mean ownerless

The trap with managed services is the little mental shortcut where "AWS runs it" becomes "we do not have to think about it." That shortcut is expensive.

Managed collectors can remove the need to operate collector hosts, in-cluster scrapers, or a pile of OpenTelemetry collector deployments. Great. Please remove toil. Tired humans deserve mercy.

But the collector still needs network permissions. It still needs scrape targets. It still needs security groups. It still needs subnets. It still needs metric selection. It still emits billable data. It still becomes part of the answer when the incident commander asks, "can we trust this graph?"

Observability is not decoration. Autoscalers, rollback decisions, SLOs, anomaly detection, agent workflows, and very tired humans at 03:17 all depend on it. Once the metrics pipeline becomes managed infrastructure, platform teams need to treat it as a shared dependency.

labels are where good intentions go to get weird

The hard part of metrics collection is rarely "can I scrape /metrics?"

It is usually:

what is this service called?
which team owns it?
is this production?
which customer, region, tenant, cluster, namespace, queue, model, or job does this represent?
is this cardinality useful or did we just create a tiny money printer pointed at ourselves?

Labels are the observability version of naming things and blaming DNS. They look simple until your query returns twelve answers and none of them are the one you need.

Managed collectors do not remove that problem. They make the boundary cleaner.

For ECS, AWS describes DNS-based discovery through Cloud Map so the collector follows changing task IPs. For EKS, discovery can follow Kubernetes targets. For EC2, direct instance scraping still needs ports and sane target definitions.

But if every team brings its own labels, the centralized collector becomes a centralized junk drawer.

The right platform move is to publish a small label contract and enforce it where possible:

service
team
environment
region
workload_type
slo_name when it exists

Then be suspicious of labels with unbounded values. User IDs, request IDs, raw URLs, pod hashes, prompt IDs, session IDs, and "temporary" fields are how observability turns into a billing incident wearing a Grafana badge.

cost is part of the signal now

The CloudWatch docs are clear that managed collectors are charged by the hour and CloudWatch OpenTelemetry metric ingestion pricing applies. They also mention VPC data transfer can show up when scrapers pull metrics from targets, and suggest compressing /metrics responses to reduce transfer volume.

That is architecture.

When metrics were collected by something your team operated, the cost was often hidden inside nodes, daemonsets, storage, and people pretending a giant Prometheus was "fine for now." With managed collectors, the bill can become more explicit. This is good if you measure it. It is surprising if you do not.

Metric volume needs a budget. Scrape interval needs a budget. Cardinality needs a budget. Retention and query patterns need a budget. Before anyone says "but it is just metrics", remember that "just logs" already bought many companies a very educational invoice.

The useful question is not "is managed collection cheaper?"

What is the cost per service of the metrics we actually use to operate the system?

observability has a supply chain too

We talk a lot about software supply chains now. Images, dependencies, SBOMs, signatures, provenance, all that healthy paranoia.

Metrics have a supply chain too.

Exporter. Endpoint. Network path. Scraper. Relabeling. Transport. Storage. Query. Alert. Dashboard. Human decision. If one link is wrong, the final graph may still look professional.

Managed collectors make some links stronger. Less self-managed collector drift. Better scaling. Fewer forgotten agents. More unified querying with CloudWatch and PromQL. That is valuable.

They also concentrate trust. If CloudWatch becomes the place where more Prometheus data lands, collector configuration becomes a first-class platform artifact. It should live near the IaC, have owners, and have a rollback path that does not involve three people clicking in a console while the incident channel develops a personality.

what i would do first

If I were rolling this out, I would avoid the grand observability transformation slide deck and start smaller.

First, pick one environment and one workload type. ECS services through Cloud Map, for example. Prove that discovery, security groups, scrape intervals, labels, dashboards, and alarms make sense.

Second, put the collector config in version control. No mystery config. No "we changed it in the console because the graph looked lonely."

Third, define a label contract before onboarding the second team. That is where standards become folklore if nobody writes them down.

Fourth, set cost alarms early. Not after the first strange invoice. Early. Scrape volume is part of the rollout.

Fifth, document the trust path for incident responders. If an alert fires from a managed collector, people should know where the target is, which config scraped it, and who owns the metric.

That is not bureaucracy.

That is how you make "managed" mean less work instead of less visibility.

the punchline

CloudWatch managed Prometheus collectors are a good feature because they remove a chunk of observability plumbing nobody wants to babysit.

But plumbing is not meaningless just because a cloud provider operates part of it.

The scrape layer decides what your company can see. It shapes alerts, dashboards, SLOs, autoscaling, rollback confidence, and sometimes the emotional weather of an incident call.

So yes, let AWS run the collectors if that reduces toil. Let CloudWatch pull Prometheus-compatible metrics from EKS, ECS, EC2, MSK, and OpenSearch. Use PromQL where it helps. Delete some self-managed collector snowflakes.

Just do not confuse managed infrastructure with absent ownership.

The metrics pipeline is becoming a cloud control plane.

Treat the scrape config like production code, the labels like an API, and the bill like an alert signal.

Otherwise the graph will look calm right up until reality files a ticket.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

docker content trust retiring is the migration teams forgot

Paulo Victor Leite Lima Gomes — Fri, 07 Aug 2026 00:03:48 +0000

Docker Content Trust is not disappearing in a dramatic way.

There is no breach headline. No emergency patch window. No breathless thread explaining why every container is suddenly unsafe.

Instead, Docker is doing something much more useful and much easier to ignore: it is retiring an old trust mechanism on a schedule.

Docker announced that Docker Content Trust and the Notary v1 service at notary.docker.io are being fully retired. Write brownouts already happened in July. Read brownouts are scheduled for August 10 and August 12, 2026. Full shutdown is scheduled for December 8, 2026.

Most teams will not notice, because ordinary docker pull and docker push do not use DCT unless it was explicitly enabled.

That is exactly why this is an interesting migration.

The danger is not that every container workflow breaks. The danger is that a few important workflows break because nobody remembers they were relying on a trust feature from a different era of the container ecosystem.

the old checkbox was doing real work

Docker Content Trust came from a useful idea: make image trust feel close to the Docker workflow.

Set DOCKER_CONTENT_TRUST=1. Use docker trust commands. Let the client verify signed metadata before accepting an image tag. For many teams, that was the first practical taste of container image signing.

It was not perfect, but it was a real control.

The uncomfortable part is what happens when an early control becomes organizational memory. A security exception gets written around it. A CI pipeline inherits an environment variable. A Kubernetes admission policy checks for it. A base-image publishing process signs with it because that was the approved pattern three platform teams ago.

Then years pass.

The people who installed the mechanism move on. The repo keeps building. The cluster keeps deploying. The control still exists, but nobody can clearly say whether it is essential, obsolete, redundant, or accidentally load-bearing.

That is the migration most teams forgot.

Not the command migration.

The ownership migration.

trust moved from docker-native to supply-chain architecture

Docker's guidance is blunt in the best way: Notary v1 is no longer maintained, and the ecosystem has moved toward OCI-native signing tools such as Sigstore Cosign and Notation.

That is the right direction.

It is also a change in responsibility.

DCT made trust feel like a Docker client feature. Modern image trust is more distributed. Signatures live as OCI artifacts. Identity may come from OIDC. Verification may happen in CI, in a registry policy, in an admission controller, or in a deployment platform. Provenance and SBOMs may become part of the same release evidence.

That is more powerful.

It is also less magical.

You do not migrate by replacing one command with another command and calling the work done. You migrate by deciding what must be true before an image can move from source code to production.

Who is allowed to sign?

Which identity signs release images?

Where are signatures stored?

Which registries are trusted?

What happens when verification fails?

Can a developer bypass the policy for an incident?

Does the cluster enforce the same thing CI claims to enforce?

Those questions are not Docker trivia. They are release architecture.

disabling dct is not a strategy

For some teams, the right immediate move really is simple.

Search for DOCKER_CONTENT_TRUST=1. Search for docker trust sign, docker trust inspect, and docker trust revoke. Search CI configs, shell profiles, Dockerfiles, Compose files, deployment scripts, and old platform templates. If DCT is set only by accident, remove it before the read brownouts find it for you.

That is fine.

But "unset the variable" should be treated as a triage step, not a supply-chain strategy.

If DCT was the only thing standing between your production release path and arbitrary mutable tags, removing it may keep builds green while reducing the evidence behind those builds.

This is where teams get tricked by operational success.

The pipeline passes.

The incident is avoided.

The migration ticket is closed.

And the organization quietly moves from imperfect verification to no verification at all.

That is not modernization. That is control deletion with a better changelog.

digest pinning is necessary but incomplete

Docker's migration post correctly separates repeatability from publisher identity.

Pulling by digest is a major improvement over trusting a tag. A digest says, "give me this exact content." That matters because tags can move, registry defaults can change, and rebuilds can produce new artifacts under familiar names.

If production depends on some-image:latest, you do not have a precise release input. You have a suggestion.

So yes, pin digests where production repeatability matters.

But digest pinning is not the same as trust.

A digest can prove that you got the content you requested. It does not, by itself, prove that the content came from the publisher, pipeline, or review process you intended to trust.

This distinction matters more now because AI agents are making artifact production cheaper. More code can be generated. More images can be built. More dependency updates can be proposed. More "small safe changes" can enter the queue.

The bottleneck is no longer only producing software.

The bottleneck is proving what happened to the software before it shipped.

admission control is where trust becomes real

Signing an image is only half the story.

The other half is enforcement.

If the cluster accepts unsigned images, signature policy is documentation. If the deployment system accepts images from any registry, registry policy is a slide. If CI checks one path but emergency deploys use another, your supply-chain story has a side door.

This is why tools like Kyverno, Ratify, Gatekeeper, Cosign, and Notation matter less as product names and more as places to make release claims enforceable.

The platform team should be able to say:

release images are signed by this identity
production deploys require a valid signature
allowed registries are explicit
base images come from approved sources
exceptions expire
verification failures are visible
the policy is tested before the brownout, not during it

That list is boring.

Good.

Boring is what mature trust looks like.

this is really about old controls aging out

DCT retirement is a container story, but the pattern is bigger than Docker.

Engineering organizations accumulate controls the same way they accumulate dependencies. Some are still valuable. Some have better replacements. Some are dead but harmless. Some are dead but important. The dangerous ones are the controls that are both forgotten and load-bearing.

AI makes this worse, not because AI changes image signing, but because it increases the amount of automation running through old paths.

An agent can update a Dockerfile.

An agent can regenerate a build pipeline.

An agent can migrate a service to a new base image.

An agent can open ten pull requests that all look plausible.

But if nobody knows what release evidence is required, the agent will optimize for passing checks, not preserving trust.

That is why the useful question this week is not "are we affected by Docker Content Trust retirement?"

The better question is: "do we know which supply-chain controls are actually protecting production?"

If the answer is no, the DCT timeline is a gift. It gives teams a dated reason to inventory old assumptions before December turns them into production surprises.

what i would do before august 10

I would start with discovery, not a tool debate.

Search for DOCKER_CONTENT_TRUST, docker trust, Notary v1 endpoints, and admission policies that mention DCT. Check CI variables, repo secrets, build images, deployment templates, and platform docs. Ask registry publishers whether they are moving to Cosign, Notation, or something else.

Then split the findings into three buckets.

First: accidental DCT usage that can be removed safely.

Second: intentional DCT usage that needs a migration path to modern signing.

Third: places where DCT was pretending to be a broader supply-chain strategy.

That third bucket is the important one.

It should produce platform work: digest pinning guidance, signing identity, registry policy, admission enforcement, exception handling, release evidence, and ownership.

The old trust checkbox is going away.

What replaces it is not a feature.

It is a release discipline.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

your gpu bill is not a model problem

Paulo Victor Leite Lima Gomes — Thu, 06 Aug 2026 00:04:39 +0000

The cloud bill has a very annoying habit.

It waits until everyone is talking about the exciting abstraction, then it shows up with the boring invoice.

For AI infrastructure, the exciting abstraction is the model.

Which model is smarter? Which one has better reasoning? Which one is cheaper per million tokens? Which one can explain a failing test with the confidence of someone who absolutely did not run it?

Useful questions.

Incomplete questions.

OpenCost 1.121.0 added Kubernetes inference cost tracking with llm-d and vLLM metrics. The interesting part is not only that it can calculate model and token cost. The interesting part is the distinction it forces platform teams to make.

What did the model's work cost?

And what did keeping the model ready cost?

Those are not the same question.

That difference is where many self-hosted AI spreadsheets go to die.

the model is not the workload

When teams talk about AI cost, they often jump straight to price-per-token.

That makes sense for SaaS APIs. You send tokens. You receive tokens. Somewhere in the middle, a billing system does billing-system things and your dashboard gets another little number to make you sad.

Self-hosting changes the shape.

Now you are not only buying inference.

You are running infrastructure.

There are GPUs. There is memory. There is a gateway. There are pods. There are model weights sitting in VRAM. There is routing. There is cache behavior. There are namespaces, teams, tenants, and probably one deployment nobody wants to touch because it was "temporary" in March.

This is why the OpenCost angle matters. It connects Kubernetes cost allocation to inference metrics from vLLM and llm-d. It is trying to answer cost questions in the place where the workload actually runs, not in a slide where all GPUs are magically 100% utilized and nobody ever pays for idle capacity.

That slide is beautiful.

It is also fiction.

usage cost can lie to you politely

The useful distinction is usage-based cost versus allocation-based cost.

Usage-based cost is about active work. The model processed tokens. GPU compute was consumed. Input and output tokens had real processing cost. KV cache hits changed that cost. Great.

Allocation-based cost is about availability. The model is loaded. GPU memory is reserved. The serving stack is alive. Common infrastructure is running. The system is ready to respond even if the request traffic is having a nap.

Both numbers are true.

Only one answers the build-versus-buy question.

If a self-hosted model costs one dollar per million tokens while it is actively working, and a hosted API costs two dollars per million tokens, the spreadsheet starts smiling.

"Look, we are cheaper."

Maybe.

If the GPU is busy enough.

If the model is mostly idle, the real cost per token includes the quiet hours too. A GPU with a model loaded is not free because the request queue is empty. It is just expensive in a less photogenic way.

This is the trap. Usage-based cost can make self-hosting look cheap because it ignores the carrying cost of readiness. Allocation-based cost includes the part production teams actually pay for at the end of the month.

The invoice does not care that your benchmark was efficient for seven minutes.

warm capacity is a product decision

There is a reason teams keep models warm.

Latency matters. Cold starts are bad. Users do not enjoy waiting while your platform gently remembers how to be useful. Some workloads need a model ready now, not after a scheduling adventure and a loading ceremony.

So keeping capacity warm can be the correct choice.

But it has to be a choice.

Not an accident.

This is where AI FinOps becomes platform engineering. The useful question is not "is this model expensive?" That is too vague. Everything is expensive if you squint with procurement nearby.

The better questions are more practical.

Which teams need this model warm?

Which traffic can tolerate queueing or scale-up?

Which low-volume workloads should use an external API instead?

Which models can share capacity?

Which routes should consolidate traffic to improve utilization?

Which experiments are quietly reserving production-grade hardware for demo-grade usage?

This is less glamorous than model selection. It is also where the money is.

cost labels become architecture

One detail I like in the OpenCost work is the label shape. The new metrics include things like model name, model version, namespace, workload type, and whether the cost basis is usage or allocation.

That sounds boring.

Good.

Boring labels are how the bill becomes debuggable.

Without them, AI cost turns into fog. Everyone knows the platform is expensive. Nobody knows if the expensive part is one team's chatbot, another team's batch summarizer, a forgotten staging deployment, or a model nobody retired because the migration ticket got renamed into a roadmap initiative and then disappeared.

Cost attribution is not just finance hygiene. It changes engineering behavior.

When teams can see the cost of keeping a model warm for their namespace, they can have a real conversation about latency, volume, and tradeoffs. When they only see an aggregate GPU bill, the conversation becomes vibes with a spreadsheet attachment.

We have seen this movie with Kubernetes before.

CPU requests looked like small YAML details until over-requested workloads became real money. Storage classes looked like plumbing until retained volumes became invoices. Logs looked harmless until everyone discovered that debugging output also has a billing department.

Inference is getting the same treatment.

The YAML is now connected to the token.

Wonderful and terrible, as usual.

the platform owns the gap

The gap between usage-based and allocation-based cost is the part I would watch.

It is the cost of readiness.

Sometimes that gap is healthy. A fraud system, incident assistant, or customer-facing workflow may justify warm capacity because response time matters. Fine. Put that in the architecture decision. Own it.

Sometimes the gap is waste. A model is deployed for a pilot with twelve users. A team copied production values into staging. A bigger model is serving requests that a smaller model could handle. A routing layer spreads traffic so thin that every model looks underutilized.

That is not a model problem.

That is a platform problem.

The platform owns the routing, isolation, quotas, schedules, autoscaling policy, cache behavior, and cost visibility. The model is only one component inside that system.

AI infrastructure is becoming normal infrastructure faster than people want to admit. It needs the same dull tools we already learned to respect: allocation reports, budgets, ownership labels, deployment review, capacity planning, and cleanup jobs with teeth.

Not because engineers love governance.

We do not.

Because otherwise the bill becomes the observability system.

And the bill has terrible UX.

self-hosting needs utilization, not optimism

I like self-hosting. I like owning the runtime. I like being able to tune the serving stack, inspect metrics, and avoid pretending that an API call is a complete architecture.

But self-hosting is not a personality trait.

It needs utilization.

If the allocation-based cost per million tokens beats the external API price, great. If it does not, the answer is not to argue with the spreadsheet until morale improves. The answer is to change the architecture or stop self-hosting that workload.

Route more traffic to fewer warm models.

Use smaller models where they are enough.

Separate latency-sensitive traffic from batch traffic.

Scale down experiments.

Expose cost per namespace and model version.

Make teams see the fixed cost they are asking the platform to carry.

The useful version of AI cost tracking does not end with "this model is expensive."

It ends with a decision.

Keep it warm because latency matters.

Move it to an API because utilization is bad.

Consolidate traffic because too many models are doing too little work.

Change routing because the current setup is optimized for a demo, not production.

That is the real lesson.

Your GPU bill is not a model problem.

It is a utilization story with Kubernetes labels.

Which is a very platform-engineering way for AI to become boring.

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

durable execution is something you install now

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 12:05:45 +0000

A process starts a multi-step job.

Step 1 validates the customer. Step 2 creates an internal record. Step 3 moves money, changes a subscription, sends an irreversible request to a partner, or does whatever your business considers serious.

Then the process dies.

Not politely.

The container gets killed. The deploy rolls. The host disappears. The worker restarts halfway through the thing it was doing.

Now the question is horrible:

What happens when it runs again?

If step 3 already happened, restarting from step 1 may duplicate something expensive. If you skip too much, you may leave the customer in a half-activated state. If you retry blindly, the external provider may accept the same request twice.

This is the shape of a lot of fintech work. Account opening, subscription activation, credit disbursement, repayment settlement, KYC refreshes, scheduled sweeps. They are not one database transaction. They are small business stories pretending to be function calls.

For a long time, the answer was not a product. The answer was discipline.

the old answer was owning the machinery

At a fintech on the American continent, the public architecture story was event-driven: Kafka, services, async processing, consistency through events, and careful thinking around idempotency. That style makes sense in a bank-sized environment.

But the boring part of the work was never just "publish an event."

The real work was deciding what each step meant, which message was the source of truth, how to identify duplicates, how to make consumers safe to replay, and how to compensate for something that had already crossed a boundary.

You ended up with sagas, state machines, idempotency keys, outbox tables, retry topics, reconciliation jobs, audit trails, alerts, and runbooks.

That is not a criticism. In serious financial systems, this discipline is part of the job. You cannot hand-wave around money by saying "the queue is eventually consistent."

But there is an operational cost to owning the mechanism.

Every team has to remember the same failure modes. Every new workflow has to rebuild the same scaffolding. Every incident asks: did the step finish, did the event publish, did the retry duplicate it, and is the customer state now real?

The painful thing is that most of this machinery is not product-specific.

The compensation logic is product-specific. The step boundaries are product-specific. Retry and reversal rules are product-specific.

But "remember completed step 3 after the worker crashes" is plumbing.

And plumbing eventually becomes a product.

DBOS and Temporal change the default

This is why frameworks like DBOS and Temporal are worth paying attention to, even if you do not adopt either tomorrow.

Temporal's model is built around workflows and activities. The Temporal service stores a durable event history for each workflow execution. When a worker crashes or restarts, workflow code can be replayed from that history, and completed activity results are not forgotten because a process died. The service has a record of what happened.

DBOS takes a different shape. DBOS Transact is positioned as an open-source durable execution library that runs inside your application. The current DBOS material still describes the core design as Postgres-backed: workflow and step state are checkpointed into Postgres, and application servers coordinate through the database instead of a separate central orchestrator.

That difference matters.

Temporal asks you to run, or pay for, a workflow service. Temporal Cloud is priced as a managed platform, with usage and storage dimensions around things like actions and workflow history storage.

DBOS is making a more Postgres-native bet. The open-source Transact library can run anywhere, and the paid DBOS offering is around Conductor, support, management tooling, and hosting. Their current public pricing lists Pro and Teams plans around managed apps, seats, and checkpoint volume.

I do not think the interesting point is which one is "better." That depends too much on the system, team, language, and operational model.

The interesting point is that both package the same formerly internal capability:

This multi-step job survives process death.

That sentence used to imply bespoke platform work. Now it can start with a framework.

the hard part did not disappear

There is a trap here, and it is exactly the trap that appears every time infrastructure gets better.

People confuse "the framework remembers what happened" with "the system now knows what should happen."

It does not.

If a workflow charges a customer, activates a subscription, provisions access, sends an email, and updates analytics, the framework can help make that workflow resumable. It can record step completion, retry activities, show where execution stopped, and make the history visible instead of buried inside logs.

But it cannot tell you whether charging the customer should happen before activation or after activation. It cannot tell you whether a failed activation should trigger a refund, a retry, a manual review, or a grace period. It cannot tell you whether a partner API is actually idempotent just because it accepts an idempotency key.

That is still design work.

At a fintech in the Gulf I work with now, this comes up in normal product workflows. A scheduled sweep is not interesting because a timer fired. It is interesting because the system needs to know which accounts were considered, which ones were skipped, and what should happen when the process wakes up after a deploy.

Durable execution helps a lot there.

A completed step recorded in Postgres is much better than a log line and a prayer. A workflow console is better than SSH-ing into a worker for clues.

But the useful conversation shifts upward.

Instead of asking, "how do we build the retry table?", the team asks, "where is the boundary where retry becomes compensation?"

Instead of asking, "how do we resume after a crash?", the team asks, "what does resumed mean for this customer?"

That is a better use of engineering time.

idempotency still lives at the edges

One thing I would be careful about in any durable execution pitch is idempotency.

Inside the framework boundary, you may get very strong guarantees. Completed workflow steps can be recorded. Activity results can be remembered. Retries can be controlled. Execution can resume after a crash.

At the external boundary, reality is messy.

Payment providers, banking partners, email systems, identity vendors, and card processors all have their own ideas about duplicate requests. Some support idempotency well. Some support it in documentation. Some support it until a timeout happens.

Boundary calls still need stable request identifiers, provider references, and reconciliation. You still need to know whether a timeout means "nothing happened", "something happened but you did not see it", or "check later."

Durable execution reduces the number of places where that logic leaks.

It does not repeal distributed systems.

durability as a purchasable property

The reason I like this category is that it fits a pattern that keeps repeating in software.

At first, a reliability property is craft. A strong team builds it internally because they have no choice. Then enough teams hit the same pain, and the property becomes a product.

Observability did this. Feature flags did this. Secrets management did this. CI/CD did this. Policy-as-code did this. Now durable execution is doing it.

The boring control-plane property becomes something you can buy, install, or outsource.

That is good. I do not miss every team building its own mini workflow engine by accident.

But the job does not vanish. It moves.

The value is no longer proving that your worker can survive a restart. The value is deciding which workflows deserve durability, which steps are atomic from the business point of view, which failures should retry, which failures should compensate, and who owns the customer outcome.

That is where durable execution gets interesting.

Not because DBOS or Temporal make failure disappear.

Because they make the failure visible enough that we can stop pretending the hard part was the retry loop.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

DBOS and the shrinking cost of durable workflows

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 10:37:23 +0000

The annoying failure is not the one where everything dies before anything important happens. That one is clean. You retry the job, or the user clicks again, or the scheduler runs next minute. Fine.

The real failure is the one where the process dies after step 3.

Step 1 created the customer record. Step 2 reserved something. Step 3 moved money, activated a subscription, sent a document, or called an external provider that will not forget what you asked it to do. Then the worker disappears. Kubernetes evicts the pod. The VM restarts. A deployment kills the process. Someone discovers that "stateless service" was more aspiration than architecture.

Now what? If you restart the job, you might charge twice. If you skip it, the customer is stuck halfway through activation. If you manually repair it, congratulations, you invented an operations queue with anxiety as the UI.

This is the problem durable execution is trying to make boring.

Not easy. Boring.

we used to build this ourselves

In fintech, this shows up everywhere because workflows are longer than a request and more fragile than the happy path diagram.

At a fintech in the Americas, the answer was the one many serious backend teams reached: sagas, Kafka, idempotency keys, state tables, retries, reconciliation jobs, and discipline.

That architecture was not dumb. It was right for its time.

You split a process into steps. Each step publishes an event or writes state. Commands carry idempotency keys. External calls have deduplication at the boundary when possible. Reconciliation jobs scan for weird states later because you know the system will find one. Everyone learns that "at least once" is not a delivery guarantee, it is a personality test.

The hard part was never only writing the saga. The hard part was owning it.

You had to decide what state meant, which failures should pause, which should compensate, and which should page a human. You had to keep idempotency keys flowing across service boundaries. You had to make sure a step that looked atomic in code was atomic in the business.

And because the mechanism was hand-rolled, every team ended up owning some version of the same machinery.

One team had an orchestration table. Another had choreography over Kafka topics. Another had a cron that repaired half-completed rows. All of this worked, until it did not, and then the organizational memory lived in the heads of the people who had been burned before.

This is why DBOS and Temporal are interesting to me. They did not invent the idea that a workflow should survive a crash. They productize the control-plane part we used to rebuild.

what DBOS changes

I first paid attention to DBOS while looking at subscription activation workflows and scheduled sweeps at a bank I work with now in the Gulf. The storage model caught my eye.

DBOS Transact makes workflows durable by checkpointing completed steps in Postgres. If the process crashes, the restarted app can resume from the last completed step.

It also changes the adoption shape. DBOS is not asking every team to operate a separate workflow cluster before they get durability. The open-source Transact libraries are positioned as application-adjacent durable execution built on Postgres-compatible storage. Their material still says the state can live in your application database. DBOS also sells the control-plane layer, with paid plans priced around users, applications, and checkpoint volume.

That matters because the first objection to workflow engines is often operational weight. DBOS is aiming at the space where you want crash recovery, durable queues, scheduled work, and observability without turning every service into research.

The tradeoff is clear. If durable execution metadata is in Postgres, Postgres becomes part of the runtime. That can be a feature or a coupling point. You get simpler ownership and queryability, but you still need to understand retention, migrations, backups, and the blast radius of putting workflow history near application data.

No free lunch. Fewer moving parts.

what Temporal gives you

Temporal has a different shape.

Temporal gives you a durable execution platform where workflows are recorded as event histories. A workflow can run for seconds, days, or years, and the service persists the events needed to recover and continue. Activities run outside deterministic workflow logic. The SDK replays workflow history to rebuild state.

That model is powerful when the workflow is long-lived, crosses services, and needs signals, timers, retries, versioning, and a strong operational surface.

Temporal Cloud pricing also makes the product shape explicit. It is a managed platform priced around Actions, active storage, retained storage, and support. You are buying a durable execution control plane.

That is a reasonable deal for many teams. It is also a real platform decision. Once Temporal is in the middle of a process, someone owns namespaces, workers, task queues, retry policies, versioning, security, and cost.

the hard part stayed yours

Here is the part that people sometimes skip when durable execution gets exciting.

A framework can remember that step 3 completed. It cannot tell you whether step 3 was the right boundary.

If step 3 called a payment provider, did the provider accept an idempotency key? If the network timed out, did the payment happen or not? If the workflow resumes tomorrow, should it retry, query the provider, compensate the customer, or stop for manual review?

Durable execution makes these questions visible. It does not answer them for you.

That visibility is useful. In hand-rolled systems, step boundaries often leak across topic names, retry loops, database rows, and tribal knowledge. In DBOS or Temporal, you are forced to make the workflow shape explicit. You name the step. You decide what gets checkpointed, what is safe to replay, and what must never be replayed.

That is the real design work.

Not "how do I keep a process alive forever?" The question is, "what does it mean for this business action to have happened?"

That is why fintech engineers get twitchy about retries. Retry is not a technical verb when money is involved. It can mean "try the HTTP call again." It can also mean "charge twice."

The boundary matters.

So does ownership. A durable workflow that crosses payments, subscriptions, notifications, and compliance is not owned by the framework. It is owned by a team, or by nobody.

durability became a purchasable property

The wider pattern is the interesting one.

A lot of backend engineering is the history of boring properties becoming products. We used to build deployment scripts. Then deployment became a platform property. We used to hand-roll metrics pipelines. Then observability became a product category. Some teams still build these things, for good reasons. But the default shifted.

Durable execution is going through the same move.

"This multi-step job survives a crash" used to be a badge of engineering maturity buried inside every serious backend system. Now it is increasingly an infrastructure property. You can install a library, run a platform, or pay for a managed service.

That is good. Fewer teams need to rediscover failure modes from first principles. More workflows can have explicit state instead of vibes. Scheduled sweeps, onboarding flows, subscription activation, refunds, identity checks, and cleanup jobs can stop depending on the fantasy that processes die only at convenient times.

But it also raises the bar for engineering judgment.

If durability is easy to add, teams will add it to things that should have stayed simple. If every background job becomes a workflow, you can create a system that is reliable but hard to reason about.

The question is not "should we use DBOS or Temporal for everything?"

The question is "which business processes deserve durable execution, and who owns the semantics when they resume?"

That is a better question than "how many cron retries should we add?"

And it is where the work belongs.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

durable execution is something you install

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 10:33:55 +0000

A process starts a multi-step job. Step 1 validates the customer. Step 2 creates an internal record. Step 3 moves money, changes a subscription, sends an irreversible request to a partner, or does whatever your business considers serious. Then the process dies.

Not politely.

The container gets killed. The deploy rolls. The host disappears. The worker restarts halfway through the thing it was doing.

Now the question is simple and horrible:

What happens when it runs again?

This is the shape of a lot of fintech work. Account opening, subscription activation, credit disbursement, repayment settlement, KYC refreshes, scheduled sweeps, ledger posting. They are not one database transaction. They are small business stories pretending to be function calls.

the old answer was discipline

But the boring part of the work was never just "publish an event." The real work was deciding what each step meant, which message was the source of truth, how to identify duplicates, how to make consumers safe to replay, and how to compensate for something that had already crossed a boundary.

You ended up with sagas, state machines, idempotency keys, outbox tables, retry topics, reconciliation jobs, audit trails, dashboards, alerts, and runbooks.

That is not a criticism. In serious financial systems, this discipline is part of the job. You cannot hand-wave around money by saying "the queue is eventually consistent" and then go for lunch.

But there is an operational cost to owning the mechanism. Every team has to remember the same failure modes. Every new workflow has to rebuild the same scaffolding. Every incident asks: did the step finish, did the event publish, did the retry duplicate it, and is the customer state now real?

The painful thing is that most of this machinery is not product-specific.

The compensation logic is product-specific. The step boundaries are product-specific. The decision that a transfer can be retried but a partner call needs an idempotency key is product-specific. But "remember completed step 3 after the worker crashes" is plumbing.

And plumbing eventually becomes a product.

dbos and temporal change the default

This is why frameworks like DBOS and Temporal are worth paying attention to, even if you do not adopt either tomorrow.

Temporal's model is built around workflows and activities. The Temporal service stores a durable event history for each workflow execution. When a worker crashes or restarts, workflow code can be replayed from history, and completed activities are not simply repeated. The service has a record of what happened.

DBOS takes a different product shape. DBOS Transact is positioned as an open-source library that runs inside your application. The current DBOS docs still describe a Postgres system database that stores workflow checkpoints, step outputs, and schedule and queue state. The DBOS pricing page now puts the paid model around Conductor, tooling, support, and hosting options, with Pro and Teams plans priced around users, managed apps, and checkpoint usage.

That difference matters.

Temporal asks you to run, or pay for, a workflow service. Temporal Cloud pricing is consumption-based, mostly around actions, storage, and support. The minimum Cloud plan is currently listed at $100/month, with included actions and storage before usage-based charges kick in.

DBOS is making a more Postgres-native bet. The open-source library gives you durable workflows in application code, while Conductor adds operational tooling like monitoring, recovery, versioning, alerts, and support.

The interesting bit is that both package the same formerly internal capability:

This multi-step job survives process death.

That sentence used to imply bespoke platform work. Now it can start with a framework.

the hard part did not disappear

There is a trap here, and it is exactly the trap that appears every time infrastructure gets better.

People confuse "the framework remembers what happened" with "the system now knows what should happen."

It does not.

But it cannot tell you whether charging the customer should happen before activation or after activation.

It cannot tell you whether a failed activation should trigger a refund, a retry, a manual review, or a grace period. It cannot tell you whether a partner API is actually idempotent just because it accepts an idempotency key. It cannot tell you whether your "step" is one business action or three business actions wearing a coat.

That is still design work.

At a bank I work with now in the Gulf, this comes up in normal product workflows. A scheduled sweep is not interesting because a timer fired. It is interesting because the system needs to know which accounts were considered, which ones were skipped, which external calls succeeded, and what should happen when the process wakes up after a deploy.

Durable execution helps a lot there. A completed step recorded in Postgres is much better than a log line and a prayer. A workflow console is better than SSH-ing into a worker for clues. But the useful conversation shifts upward.

Instead of asking, "how do we build the retry table?", the team asks, "where is the boundary where retry becomes compensation?" Instead of asking, "how do we resume after a crash?", the team asks, "what does resumed mean for this customer?"

That is a better use of engineering time.

idempotency still lives at the edges

One thing I would be careful about in any durable execution pitch is idempotency.

At the external boundary, reality is messier.

Durable execution reduces the number of places where that logic leaks.

It does not repeal distributed systems.

the wider pattern

The reason I like this category is that it fits a pattern that keeps repeating in software. At first, a reliability property is craft. A strong team builds it internally because they have no choice. Then enough teams hit the same pain, and the property becomes a product.

Observability did this. Feature flags did this. Secrets management did this. CI/CD did this. Policy-as-code did this. Now durable execution is doing it.

The boring control-plane property becomes something you can buy, install, or outsource.

That is good. I do not miss every team building its own mini workflow engine by accident.

But the job does not vanish. It moves.

That is where durable execution gets interesting.

Not because DBOS or Temporal make failure disappear.

Because they make the failure visible enough that we can stop pretending the hard part was the retry loop.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

DBOS and the shrinking cost of durable workflows

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 10:07:34 +0000

The annoying failure is not the one where everything dies before anything important happens. That one is clean. You retry the job, or the user clicks again, or the scheduler runs next minute. Fine.

The real failure is the one where the process dies after step 3.

This is the problem durable execution is trying to make boring.

Not easy. Boring.

we used to build this ourselves

In fintech, this shows up everywhere because workflows are longer than a request and more fragile than the happy path diagram.

At a fintech in the Americas, the answer was the one many serious backend teams reached: sagas, Kafka, idempotency keys, state tables, retries, reconciliation jobs, and discipline.

That architecture was not dumb. It was right for its time.

The hard part was never only writing the saga. The hard part was owning it.

And because the mechanism was hand-rolled, every team ended up owning some version of the same machinery.

This is why DBOS and Temporal are interesting to me. They did not invent the idea that a workflow should survive a crash. They productize the control-plane part we used to rebuild.

what DBOS changes

I first paid attention to DBOS while looking at subscription activation workflows and scheduled sweeps at a bank I work with now in the Gulf. The storage model caught my eye.

DBOS Transact makes workflows durable by checkpointing completed steps in Postgres. If the process crashes, the restarted app can resume from the last completed step.

No free lunch. Fewer moving parts.

what Temporal gives you

Temporal has a different shape.

That model is powerful when the workflow is long-lived, crosses services, and needs signals, timers, retries, versioning, and a strong operational surface.

the hard part stayed yours

Here is the part that people sometimes skip when durable execution gets exciting.

A framework can remember that step 3 completed. It cannot tell you whether step 3 was the right boundary.

Durable execution makes these questions visible. It does not answer them for you.

That is the real design work.

Not "how do I keep a process alive forever?" The question is, "what does it mean for this business action to have happened?"

That is why fintech engineers get twitchy about retries. Retry is not a technical verb when money is involved. It can mean "try the HTTP call again." It can also mean "charge twice."

The boundary matters.

So does ownership. A durable workflow that crosses payments, subscriptions, notifications, and compliance is not owned by the framework. It is owned by a team, or by nobody.

durability became a purchasable property

The wider pattern is the interesting one.

Durable execution is going through the same move.

But it also raises the bar for engineering judgment.

The question is not "should we use DBOS or Temporal for everything?"

The question is "which business processes deserve durable execution, and who owns the semantics when they resume?"

That is a better question than "how many cron retries should we add?"

And it is where the work belongs.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

DBOS and the shrinking cost of durable workflows

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 09:09:42 +0000

The annoying failure is not the one where everything dies before anything important happens. That one is clean. You retry the job, or the user clicks again, or the scheduler runs next minute. Fine.

The real failure is the one where the process dies after step 3.

This is the problem durable execution is trying to make boring.

Not easy. Boring.

we used to build this ourselves

In fintech, this shows up everywhere because workflows are longer than a request and more fragile than the happy path diagram.

At a fintech in the Americas, the answer was the one many serious backend teams reached: sagas, Kafka, idempotency keys, state tables, retries, reconciliation jobs, and discipline.

That architecture was not dumb. It was right for its time.

The hard part was never only writing the saga. The hard part was owning it.

And because the mechanism was hand-rolled, every team ended up owning some version of the same machinery.

This is why DBOS and Temporal are interesting to me. They did not invent the idea that a workflow should survive a crash. They productize the control-plane part we used to rebuild.

what DBOS changes

I first paid attention to DBOS while looking at subscription activation workflows and scheduled sweeps at a bank I work with now in the Gulf. The storage model caught my eye.

DBOS Transact makes workflows durable by checkpointing completed steps in Postgres. If the process crashes, the restarted app can resume from the last completed step.

No free lunch. Fewer moving parts.

what Temporal gives you

Temporal has a different shape.

That model is powerful when the workflow is long-lived, crosses services, and needs signals, timers, retries, versioning, and a strong operational surface.

the hard part stayed yours

Here is the part that people sometimes skip when durable execution gets exciting.

A framework can remember that step 3 completed. It cannot tell you whether step 3 was the right boundary.

Durable execution makes these questions visible. It does not answer them for you.

That is the real design work.

Not "how do I keep a process alive forever?" The question is, "what does it mean for this business action to have happened?"

That is why fintech engineers get twitchy about retries. Retry is not a technical verb when money is involved. It can mean "try the HTTP call again." It can also mean "charge twice."

The boundary matters.

So does ownership. A durable workflow that crosses payments, subscriptions, notifications, and compliance is not owned by the framework. It is owned by a team, or by nobody.

durability became a purchasable property

The wider pattern is the interesting one.

Durable execution is going through the same move.

But it also raises the bar for engineering judgment.

The question is not "should we use DBOS or Temporal for everything?"

The question is "which business processes deserve durable execution, and who owns the semantics when they resume?"

That is a better question than "how many cron retries should we add?"

And it is where the work belongs.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

Gateway API is the Kubernetes network contract now

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 00:03:50 +0000

Kubernetes networking has spent years doing something very Kubernetes: taking a simple idea, wrapping it in twelve implementation details, and then asking everyone to be calm because there is an annotation for that.

Ingress was supposed to be the friendly front door. In practice, it became a shared fiction with different controllers, different annotations, different load balancer behaviors, and different levels of "works on my cluster" energy.

This is why Gateway API v1.6 caught my attention.

Not because TCPRoute and UDPRoute graduating to Standard is the kind of news that makes normal people cancel dinner plans.

It is interesting because Kubernetes networking is finally admitting the real problem:

Networking is not one object owned by one team.

It is a contract between platform engineers, network operators, security people, application teams, and the poor person holding the incident phone.

Gateway API v1.6 makes TCPRoute and UDPRoute Standard in the v1 API. The release also moves experimental APIs into a clearer separate group, gateway.networking.x-k8s.io, so teams can see the line between stable contract and "please do not bet payroll on this yet."

That sounds like API housekeeping.

It is actually platform governance wearing a YAML hat.

ingress was too small for the job

Ingress was useful. I am not here to pretend we all imagined it. For a simple HTTP service, host, path, TLS secret, backend service, done. Great.

Then production arrived.

Someone needed redirects. Someone needed header matching. Someone needed gRPC. Someone needed traffic splitting. Someone needed cross-namespace routing. Someone needed TCP because not everything in life is an HTTP endpoint with a tidy JSON body. Someone needed UDP because DNS, games, voice, and other inconvenient realities exist.

The answer became annotations.

Annotations are fine when they are metadata. They are less fine when they become a shadow API where every controller has its own dialect and every migration requires reading the controller docs like ancient cave paintings.

You can build serious systems with annotations. Many teams did. But the contract is weak. The Kubernetes API says one thing, the controller-specific behavior says another thing, and the real platform behavior lives in tribal knowledge, Helm values, and a wiki page last updated by someone who now works somewhere else.

Gateway API exists because service networking needs a richer model than "here is an Ingress, good luck."

roles matter more than objects

The best part of Gateway API is not a specific route kind. It is the role model.

A platform team can own GatewayClass, deciding which implementation exists and how it is configured.

An infrastructure or network team can own Gateway, deciding where traffic enters and what listeners exist.

Application teams can own HTTPRoute, GRPCRoute, TCPRoute, or UDPRoute, describing how their traffic should attach to that entry point.

Security can care about which namespaces are allowed to bind to which listeners and what cross-namespace references are allowed.

This is not just prettier YAML. This is the shape of the organization finally showing up in the API.

That matters because most Kubernetes networking incidents are not caused by nobody knowing how packets work. They are caused by ownership being blurry.

Who is allowed to expose a service?

Who approves a public hostname?

Who owns TLS policy?

Who can route traffic across namespaces?

Who knows whether the load balancer behavior differs between dev and prod?

If the answer is "whoever found the right annotation first," congratulations, you have invented networking roulette.

Gateway API is useful because it makes those boundaries easier to express as Kubernetes resources instead of hallway negotiations with kubectl.

tcp and udp graduating is a big deal

The v1.6 headline is TCPRoute and UDPRoute graduating to Standard.

That is not glamorous. It is better than glamorous. It is boring in the way infrastructure should aspire to be.

HTTP gets the attention because it is where most web teams live. But platform teams know the cluster is full of protocols that do not fit cleanly into HTTP routing:

database proxies
Redis or Memcached
MQTT
DNS
custom TCP protocols
UDP services
legacy systems that survived because nobody was brave enough to ask why

Before Gateway API matures across these protocols, teams often end up with side paths. Some traffic uses Ingress. Some uses Service LoadBalancer. Some uses controller-specific CRDs. Some uses a cloud load balancer configured outside the cluster. Some uses Terraform, prayer, and a diagram with three arrows labeled "network."

That fragmentation is expensive.

Not always in cloud bill terms, although give it time. It is expensive in reasoning. During an incident, nobody wants to ask five systems where traffic might be going. They want one control plane, or at least one contract that explains the different routes.

TCPRoute and UDPRoute reaching Standard says layer 4 routing is no longer a side quest. It belongs in the Kubernetes networking contract too.

standard and experimental should be visibly different

I also like the experimental API group separation more than I expected.

Experimental features are not bad. They are how platforms grow. But the problem is when experiments look almost exactly like stable APIs until they break under you.

The new gateway.networking.x-k8s.io group makes the risk more visible. That is healthy.

Engineers are adults. Mostly. Let them use experimental features when the trade-off is worth it. But make the choice explicit. Put it in code review. Put it in the platform policy. Put it in the upgrade checklist.

If a team wants an experimental route feature for an internal tool, fine.

If a team wants it for card authorization traffic on Black Friday, maybe open a second tab and reconsider your life choices.

The important thing is not banning experiments. The important thing is keeping experimental contracts from silently becoming production foundations.

portability is still not magic

Gateway API is often described as portable. That is good, but we should not turn portability into another comforting lie.

Portable does not mean every implementation behaves identically in every environment. Load balancers still have personalities. Cloud networks still have limits. Controllers still differ in supported features. TLS, health checks, source IP behavior, cross-zone traffic, and observability can still surprise you.

The contract gets better. The work does not disappear.

A serious platform team should still ask:

Which Gateway API resources are supported by our implementation?
Which features are Core, Extended, or implementation-specific?
How do we test routes before production?
Who owns upgrades of the Gateway API CRDs?
What metrics prove traffic is going where we think it is going?
What is the rollback path if a route change behaves badly?

There it is again. Evidence. Ownership. Rollback. The usual boring friends.

Gateway API gives teams a cleaner place to put the network contract. It does not make the contract self-enforcing by magic.

the platform lesson

The bigger story is that Kubernetes keeps moving from "here are primitives" toward "here are shared contracts."

That is what mature platform work looks like.

A cluster is not just a place where pods run. It is a place where many teams negotiate identity, traffic, policy, cost, observability, upgrades, and failure boundaries.

The primitive-only version says:

"Create an Ingress and add these annotations."

The platform-contract version says:

"This Gateway is the approved public entry point. These listeners are owned by the platform team. These namespaces can attach routes. These route kinds are stable. These features are experimental. These metrics and alerts prove the path is healthy. These changes require review."

Less exciting. More useful.

This is why Gateway API v1.6 matters. TCPRoute and UDPRoute becoming Standard is not just about supporting more protocols. It is about reducing the number of weird escape hatches teams need when reality does not fit HTTP.

And the clearer experimental boundary is not just cleanup. It helps teams avoid accidentally treating tomorrow's idea as today's production contract.

what I would do now

If I owned a Kubernetes platform today, I would not rewrite every ingress next week because a blog post happened. That is how platform teams create migrations that nobody asked for and then act surprised when application teams hide.

I would do something smaller and more useful.

First, inventory current ingress and load balancer patterns. Find the annotation zoo. Find the controller-specific features. Find the services bypassing the main path.

Second, pick one Gateway API implementation already aligned with the platform direction. Do not make this a beauty contest with twelve vendors and a scoring spreadsheet from the land of eternal meetings.

Third, define ownership. Who creates GatewayClasses? Who owns Gateways? Who approves Routes? Who can attach across namespaces? Who reviews experimental features?

Fourth, migrate the next new service first. New contracts are easier when you are not also moving the old mess under pressure.

Fifth, measure it like production. Route status, listener health, backend failures, certificate issues, rejected attachments, and controller errors should be visible without someone spelunking through events at 2 a.m.

Gateway API is not "Ingress 2.0" in the lazy sense.

It is the Kubernetes network contract growing up.

That is not as catchy as a new dashboard screenshot, but it is much more valuable.

Because the real win is not that we get another API object.

The real win is that fewer critical networking decisions have to live in annotations, controller folklore, and one senior engineer's memory.

That engineer deserves a vacation.

The cluster deserves a contract.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

durable execution is something you install

Paulo Victor Leite Lima Gomes — Tue, 04 Aug 2026 15:02:50 +0000

Not politely.

The container gets killed. The deploy rolls. The host disappears. The worker restarts halfway through the thing it was doing.

Now the question is simple and horrible:

What happens when it runs again?

the old answer was discipline

You ended up with sagas, state machines, idempotency keys, outbox tables, retry topics, reconciliation jobs, audit trails, dashboards, alerts, and runbooks.

That is not a criticism. In serious financial systems, this discipline is part of the job. You cannot hand-wave around money by saying "the queue is eventually consistent" and then go for lunch.

The painful thing is that most of this machinery is not product-specific.

And plumbing eventually becomes a product.

dbos and temporal change the default

This is why frameworks like DBOS and Temporal are worth paying attention to, even if you do not adopt either tomorrow.

Temporal's model is built around workflows and activities. The Temporal service stores a durable event history for each workflow execution. When a worker crashes or restarts, workflow code can be replayed from history, and completed activities are not simply repeated. The service has a record of what happened.

DBOS takes a different product shape. DBOS Transact is positioned as an open-source library that runs inside your application. The current DBOS site still says Transact uses your existing Postgres database to store and recover workflow state and execution history. Their paid model is around DBOS Conductor, tooling, support, and hosting options, with Pro and Teams plans priced around checkpoint usage.

That difference matters.

Temporal asks you to run, or pay for, a workflow service. Temporal Cloud pricing is consumption-based, mostly around actions and storage. The minimum Cloud plan is currently listed at $100/month.

The interesting bit is that both package the same formerly internal capability:

This multi-step job survives process death.

That sentence used to imply bespoke platform work. Now it can start with a framework.

the hard part did not disappear

There is a trap here, and it is exactly the trap that appears every time infrastructure gets better.

People confuse "the framework remembers what happened" with "the system now knows what should happen."

It does not.

But it cannot tell you whether charging the customer should happen before activation or after activation.

That is still design work.

At a fintech in the Gulf I work with now, this comes up in normal product workflows. A scheduled sweep is not interesting because a timer fired. It is interesting because the system needs to know which accounts were considered, which ones were skipped, which external calls succeeded, and what should happen when the process wakes up after a deploy.

That is a better use of engineering time.

idempotency still lives at the edges

One thing I would be careful about in any durable execution pitch is idempotency.

At the external boundary, reality is messier.

Durable execution reduces the number of places where that logic leaks.

It does not repeal distributed systems.

the wider pattern

Observability did this. Feature flags did this. Secrets management did this. CI/CD did this. Policy-as-code did this. Now durable execution is doing it.

The boring control-plane property becomes something you can buy, install, or outsource.

That is good. I do not miss every team building its own mini workflow engine by accident.

But the job does not vanish. It moves.

That is where durable execution gets interesting.

Not because DBOS or Temporal make failure disappear.

Because they make the failure visible enough that we can stop pretending the hard part was the retry loop.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

durable execution is something you install

Paulo Victor Leite Lima Gomes — Tue, 04 Aug 2026 14:02:20 +0000

Not politely.

The container gets killed. The deploy rolls. The host disappears. The worker restarts halfway through the thing it was doing.

Now the question is simple and horrible:

What happens when it runs again?

the old answer was discipline

You ended up with sagas, state machines, idempotency keys, outbox tables, retry topics, reconciliation jobs, audit trails, dashboards, alerts, and runbooks.

That is not a criticism. In serious financial systems, this discipline is part of the job. You cannot hand-wave around money by saying "the queue is eventually consistent" and then go for lunch.

The painful thing is that most of this machinery is not product-specific.

And plumbing eventually becomes a product.

dbos and temporal change the default

This is why frameworks like DBOS and Temporal are worth paying attention to, even if you do not adopt either tomorrow.

Temporal's model is built around workflows and activities. The Temporal service stores a durable event history for each workflow execution. When a worker crashes or restarts, workflow code can be replayed from history, and completed activities are not simply repeated. The service has a record of what happened.

That difference matters.

Temporal asks you to run, or pay for, a workflow service. Temporal Cloud pricing is consumption-based, mostly around actions and storage. The minimum Cloud plan is currently listed at $100/month.

The interesting bit is that both package the same formerly internal capability:

This multi-step job survives process death.

That sentence used to imply bespoke platform work. Now it can start with a framework.

the hard part did not disappear

There is a trap here, and it is exactly the trap that appears every time infrastructure gets better.

People confuse "the framework remembers what happened" with "the system now knows what should happen."

It does not.

But it cannot tell you whether charging the customer should happen before activation or after activation.

That is still design work.

At a fintech in the Gulf I work with now, this comes up in normal product workflows. A scheduled sweep is not interesting because a timer fired. It is interesting because the system needs to know which accounts were considered, which ones were skipped, which external calls succeeded, and what should happen when the process wakes up after a deploy.

That is a better use of engineering time.

idempotency still lives at the edges

One thing I would be careful about in any durable execution pitch is idempotency.

At the external boundary, reality is messier.

Durable execution reduces the number of places where that logic leaks.

It does not repeal distributed systems.

the wider pattern

Observability did this. Feature flags did this. Secrets management did this. CI/CD did this. Policy-as-code did this. Now durable execution is doing it.

The boring control-plane property becomes something you can buy, install, or outsource.

That is good. I do not miss every team building its own mini workflow engine by accident.

But the job does not vanish. It moves.

That is where durable execution gets interesting.

Not because DBOS or Temporal make failure disappear.

Because they make the failure visible enough that we can stop pretending the hard part was the retry loop.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.