DEV Community: Paulo Victor Leite Lima Gomes

Finally, gitHub just turned Copilot into something much bigger, a cursor-ish

Paulo Victor Leite Lima Gomes — Tue, 25 Aug 2026 16:29:12 +0000

thats it!!! gitHub Copilot now is playing the game...

The new GitHub Copilot app is a desktop workspace where developers can:

Run multiple coding agents in parallel
Assign work directly from GitHub issues
Choose between different AI models
Review changes and manage pull requests
Use canvases to plan and explore ideas
Extend agents through plugins and MCP tools
Follow the work from task to merge in one place

So, is it GitHub’s answer to Cursor?

Yes, but with some steroids.

Cursor is still primarily an AI-first code editor. The Copilot app is moving toward something broader: a control center for AI agents working across your entire software delivery workflow.

GitHub already owns the repositories, issues, pull requests, Actions, reviews, permissions and organizational policies. Connecting agents directly to all of that is its biggest advantage.

The interesting competition is no longer:

VS Code vs. Cursor

It is becoming:

Which platform will control the fleet of AI agents building, reviewing and shipping our software?

Cursor proved the AI-first development experience. GitHub is now trying to absorb that experience into the platform where most software teams already operate.

Things escalated quickly. 🤖

Explore the GitHub Copilot app
Official getting-started announcement

GitHub #GitHubCopilot #Cursor #AI #SoftwareEngineering #AgenticAI

why your ai agent fails: it is a harness problem, not a prompt problem

Paulo Victor Leite Lima Gomes — Tue, 25 Aug 2026 09:02:43 +0000

The most common mistake in agent development is believing the prompt is the product.

An agent gets stuck in a loop. The team edits the system prompt.

The agent clicks the wrong thing. The team adds another sentence about being careful.

The agent says the task is done, but nothing actually changed. The team writes, "Before saying you are finished, verify that you are finished."

Then it fails again, just with a more polite explanation.

I get why this happens. Prompts are visible. Prompts are easy to change. Prompts feel like the part of the system where intelligence lives. If the agent behaves badly, it is tempting to assume the instruction was not clear enough or the model was not powerful enough.

But after you build with agents for a while, a different pattern shows up.

The expensive failures are usually not language failures. They are runtime failures.

The model did not know the browser had been redirected to login. The tool returned a partial success and nobody checked the side effect. The loop had no stop condition. The context window filled with junk. The agent retried the same broken action until it ran out of budget. The final answer sounded confident because the model was doing what models do: producing a plausible continuation.

That is not a prompt problem.

That is a harness problem.

the harness is the product

An AI harness is the deterministic software around the model.

It is the loop that decides when the model runs. It is the tool registry. It is the state store. It is the retry policy. It is the permission boundary. It is the verifier that checks whether the intended action really happened. It is the boring code that turns a stochastic reasoning engine into something you can put near production.

If the LLM is the brain, the harness is everything that stops the brain from having to be the operating system, database, browser driver, security policy, and QA team at the same time.

This distinction matters because LLMs are good at interpretation, planning, and fuzzy judgment. They are bad places to put deterministic guarantees.

The model can decide that "upvote this Hacker News post" probably means it should navigate to Hacker News, find the post, and click the upvote arrow. Fine.

The model should not be the only thing responsible for knowing whether the user is logged in, whether the click actually registered, whether the site returned a validation error, whether the same step has already failed five times, and whether the final state matches the original goal.

That is harness work.

Tejas Kumar makes this point nicely in his AI Engineer talk, "Harnesses in AI: A Deep Dive." The interesting part of the framing is not that agents need better tools. Everyone knows agents need tools. The interesting part is that the tools are not enough either. You need the runtime that governs how the tools are selected, executed, observed, and verified.

Without that runtime, even a strong model becomes a very expensive intern with browser access and no checklist.

the demo failure is painfully familiar

The Hacker News example is a good one because it is small enough to understand and real enough to hurt.

The task is simple: log into Hacker News and upvote a specific post.

An unharnessed agent can look competent for the first few steps. It opens the page. It sees the post. It clicks upvote. Then Hacker News redirects it to a login page.

This is where the illusion breaks.

To a human, the state change is obvious. We are not on the post anymore. We are on an authentication screen. The task is now blocked on session state.

To a weak agent loop, this may just look like another page. The model tries to continue. It may click around. It may navigate back. It may retry the upvote. It may burn tokens describing what it thinks happened. Eventually, if the loop has no hard verifier, it may claim success because it has reached the narrative shape of success.

"I have successfully upvoted the post."

No, you have not.

You clicked a thing, got redirected, lost the state, and narrated the happy path.

This is the exact failure mode I see in coding agents too. The agent says it fixed the test, but the test was never run. It says it updated the issue, but the API call failed. It says it opened a PR, but the branch was not pushed. It says it deployed, but the deployment job is red. The English output has no authority unless the system can prove the side effect.

Natural language completion is not completion.

prompts express intent; harnesses enforce reality

A prompt can tell the model what you want.

A harness decides what is allowed, what happened, and whether the job is done.

That is the line I would draw for any production agent.

Use prompts for judgment-heavy work:

interpret the user's request
choose between reasonable next actions
summarize a failure
decide which file or page looks relevant
explain tradeoffs

Use harness code for anything that must be true:

maximum number of steps
tool permissions
schema validation
authentication handling
retries and backoff
idempotency
budget limits
state persistence
success verification
audit logs

The more important the guarantee, the less I want it living only in prose.

This is not anti-prompt. A good prompt still matters. Bad instructions create bad work. But prompts should not carry the weight of infrastructure.

"Do not loop forever" is not a loop control strategy.

"Be careful with credentials" is not a secret management strategy.

"Verify before finishing" is not a verifier.

Those are intentions. Useful intentions, but still intentions.

what a real harness should own

The first thing a harness needs is a controlled loop.

Every agent loop should have a budget: maximum steps, maximum tokens, maximum wall-clock time, and ideally maximum repeated failures for the same action. If the agent tries the same thing three times and gets the same result, the harness should force a state change: inspect, escalate, try a different path, or stop with evidence.

The second thing is external state.

Do not rely on the context window as your source of truth. The context window is a working set, not a database. Keep durable state in a place the harness controls: task status, tool calls, browser URL, auth state, IDs created, files changed, commands run, errors observed, and verification results.

The third thing is tool discipline.

Tools should have typed inputs, typed outputs, clear error states, and narrow permissions. A browser tool should not pretend a click succeeded if the page navigated to an error. A GitHub tool should return the PR URL only after the API confirms creation. A file-editing tool should expose the diff, not just a cheerful message.

The fourth thing is interception.

Some states are not model problems. They are runtime events.

If the browser hits a login page, the harness should recognize that. If an API returns 401, the harness should classify it as auth, not ask the model to philosophize about why access failed. If a rate limit appears, the harness should apply a policy. If a destructive action is requested, the harness should require the right approval path.

The fifth thing is verification.

This is the big one.

The harness should never accept "done" because the model says "done." It should verify against the thing that matters.

For a coding agent, that might mean tests passed, typecheck passed, the diff only touches expected files, and the application still starts.

For a browser agent, that might mean a DOM state changed, a network request returned success, or a record appeared in the backend.

For a messaging agent, that might mean the message was actually sent by the platform API and the returned ID was recorded.

For a deployment agent, that might mean the new version is live and the health endpoint returns the expected response.

If you cannot verify it, the correct status is not success. It is unverified.

smaller models get much better when the harness is good

One of the more uncomfortable truths for AI product teams is that some "model quality" problems are really "we built a sloppy runtime" problems.

A stronger model can mask bad harness design for a while. It may recover from more weird states. It may infer missing context. It may be less likely to repeat itself. It may write a better apology when things go wrong.

But that is an expensive way to compensate for missing engineering.

If the harness handles auth, state, retries, tool schemas, and verification, the model's job gets smaller and cleaner. Smaller models become viable for more workflows. Latency drops. Cost drops. Behavior becomes easier to reason about because the deterministic parts are actually deterministic.

This is the part many teams miss. They jump from "the agent failed" to "we need a frontier model" when the real fix is often:

stop relying on context as memory
stop treating tool output as trustworthy without validation
stop letting the loop run until vibes improve
stop allowing natural language to define completion
stop giving the model responsibility for infrastructure state

Use the expensive model where judgment matters. Use code where guarantees matter.

That is not a philosophical preference. It is just cheaper and less fragile.

the architecture I would start with

If I were building a production agent from scratch, I would start with a small harness before adding cleverness.

At minimum:

a task object with explicit goal, constraints, status, and verifier
a step loop with hard limits
a tool registry with schemas and permission scopes
an execution ledger that records every tool call and result
state detectors for auth, navigation, rate limits, and common failure modes
a verifier that checks real side effects
a finalizer that can only report success after verification passes

The implementation does not need to be fancy. In many systems, this can be boring TypeScript or Python. The important thing is that the model is no longer floating in a chat transcript pretending to be a runtime.

A minimal mental model looks like this:

task -> plan -> tool call -> observe -> update state -> verify -> continue or stop

The model participates in the plan and maybe in choosing the next tool call.

The harness owns observation, state, limits, and truth.

That division of labor is what makes the system debuggable. When something fails, you can inspect the ledger. You can see the last known state. You can see whether verification ran. You can see whether the model misunderstood the page or the tool lied about the result.

Without that, you are debugging a transcript.

Debugging transcripts is miserable.

the punchline

Prompts are where agent design starts. They are not where agent reliability ends.

If your agent loops forever, misses auth walls, loses state, or claims work it did not do, adding another sentence to the system prompt may make you feel productive. It probably will not fix the system.

The fix is to build the harness.

Give the agent a controlled loop. Give it typed tools. Give it external state. Give it policies for auth, retries, budgets, and permissions. Give it verification that does not care how confident the model sounds.

Then the model can do the part it is actually good at: interpreting messy goals and choosing useful next actions.

Most agent failures are not proof that models are too dumb.

They are proof that we keep asking models to compensate for missing software architecture.

And software architecture, annoyingly enough, is still our job.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

agent handoffs need contracts, not vibes

Paulo Victor Leite Lima Gomes — Sat, 22 Aug 2026 14:02:24 +0000

A2A joining the Agentic AI Foundation is one of those announcements that can look boring if you only read the protocol names.

Another foundation. Another standard. Another diagram with agents talking to agents, tools talking to agents, agents discovering other agents, and everyone promising an open ecosystem.

Fine.

The interesting part is not the branding.

The interesting part is that agent-to-agent interoperability is leaving the demo layer.

A2A is now being positioned as part of an open agentic stack: AGENTS.md for instructions, MCP for agent-to-tool access, agentgateway for traffic mediation, and A2A for agent-to-agent discovery, delegation, and collaboration. At the same time, MCP's 2026-07-28 specification work is making the tool side more operational: stateless requests, explicit headers, tasks, apps, authorization hardening, and deprecation policy.

That combination matters.

Not because "agents can talk now."

They already could, in the same way two services can technically communicate if you glue enough JSON and optimism between them.

It matters because the industry is trying to turn agent collaboration into infrastructure.

And once something becomes infrastructure, the question changes.

The question is no longer "can this agent hand work to that agent?"

The question is "what contract governs the handoff when it fails at 2:13 in the morning and the customer-visible thing is still broken?"

mcp is tools, a2a is coworkers

The simple mental model is this:

MCP helps an agent connect to tools and data.

A2A helps an agent connect to other agents.

That distinction sounds neat, and mostly it is. An MCP server exposes capabilities: search this document store, file this ticket, query this database, call this deployment tool, list these resources, run this workflow. A2A exposes another agent as a collaborator: here is what I can do, here is how to reach me, here is the kind of task I can accept, here is how I return work.

One is agent-to-capability.

The other is agent-to-worker.

Both reduce integration friction. That is the good part.

It is also the part that should make platform teams sit up.

Lower integration friction never just removes work. It changes the shape of the work. When REST APIs became easy to expose, companies did not end up with five carefully governed APIs. They ended up with hundreds of APIs, some owned, some half-owned, some haunted by a team name from three reorganizations ago.

When queues became easy, everything became asynchronous until nobody knew where the retry storm started.

When Kubernetes made workloads portable, every team discovered they were a platform team, usually against their will.

Agent interoperability will do the same thing.

It will create more handoffs.

handoffs are where production systems lie to you

Most demo videos are about the happy path.

Agent A receives a request. It discovers Agent B. Agent B knows procurement, or payroll, or shipping, or incident response, or SAP, because apparently the enterprise future is just a group project with better latency. Agent B does its part. Agent A summarizes the result. Everyone claps politely.

That is not the version I care about.

I care about the version where Agent B accepts the task but cannot finish it.

Or starts it twice.

Or finishes it after Agent A has already timed out and delegated the same work elsewhere.

Or needs human approval in the middle.

Or returns a partial answer that looks final.

Or uses a tool Agent A was not allowed to use directly, which means delegation became permission laundering with a nicer diagram.

This is where "agent collaboration" stops being cute.

A handoff is not a vibe. It is a distributed systems boundary.

And distributed systems boundaries need contracts.

the contract starts with identity

If one agent delegates work to another, the first boring question is: who is actually acting?

Not "what is the agent name in the UI?"

Real identity.

Which user initiated the work? Which agent accepted it? Which organization owns that agent? Which runtime executed it? Which credentials were available? Which policy context traveled with the request? Which downstream actions happened as the user, as the originating agent, as the receiving agent, or as some service account nobody wants to admit exists?

This gets uncomfortable fast because agent delegation blurs authority.

If I ask an engineering agent to investigate an outage, and it asks a database agent for help, and the database agent calls a query tool, what identity should the database see?

The human?

The first agent?

The second agent?

The workflow?

The answer probably depends on the action. Read-only troubleshooting is different from changing a feature flag. Querying aggregate metrics is different from exporting customer rows. Asking a finance agent for an explanation is different from authorizing a refund.

That means the handoff contract needs to carry identity and scope explicitly.

Otherwise, the company will eventually discover that "agent-to-agent interoperability" became a convenient way for work to cross permission boundaries without anyone noticing.

delegation needs scope, not charisma

An agent card or capability document is useful. A receiving agent should be able to describe what it can do, how to call it, what formats it accepts, which modalities it supports, and what kind of task state it returns.

But production teams need more than a capability brochure.

They need a delegation contract.

What tasks may this agent accept? What tasks must it reject? Can it call other agents? Can it call paid tools? Can it access private data? Can it take irreversible actions? Can it ask a human for approval? Can it continue after the caller disconnects? Can it keep state? For how long? Under whose retention policy?

This is the part that separates interoperability from operability.

Interoperability says Agent A can reach Agent B.

Operability says Agent A knows what it is allowed to ask, what Agent B promised to do, what happens if Agent B does not finish, and who owns the mess.

That sounds boring because it is.

Good.

Boring is the right emotional temperature for production contracts.

retries are where agent workflows get expensive

Every handoff needs a failure model.

This is not optional. It is the difference between a multi-agent workflow and a loosely supervised pile of attempts.

Can a delegated task be retried safely? Is the task idempotent? Does the caller provide an idempotency key? Does the callee deduplicate? Can the caller cancel the task? What does cancellation mean if the callee already started downstream work? Can the task return partial progress? Can it be resumed? How long should the caller wait before escalating?

We already learned this with APIs, queues, background jobs, workflow engines, and incident automation.

"Try again" is not a strategy.

It is a strategy only after you know what retry means.

Agents make this worse because the handoff may not be a single request. It may be a long-running investigation, a plan, a code change, a data enrichment chain, or a customer support workflow that waits for human input.

MCP's newer task and explicit-handle work is interesting here because it pushes state into visible handles instead of hiding everything in a transport session. That is the right direction. A long-running operation should have a durable identity. A caller should be able to poll it, cancel it, resume it, and explain it later.

A2A-style agent work needs the same discipline.

If an agent gives another agent a task, the task should not evaporate into "the model is thinking."

It should become a thing.

A thing with an owner, state, deadline, budget, inputs, outputs, trace, and failure mode.

provenance is not a nice-to-have

The other thing every handoff needs is provenance.

When the final answer comes back, a reviewer should be able to see more than the polished summary. Which agent did which part? Which tools did each agent call? Which data sources contributed? Which intermediate results were accepted? Which were discarded? Which approvals happened? Which policies blocked actions? Which assumptions crossed the boundary?

Without that, multi-agent systems will create a new kind of production ambiguity.

The work "got done", but nobody can explain how.

That is not autonomy. That is an audit problem with a chat interface.

Provenance matters for security, but it also matters for quality. If an incident-response agent delegates log analysis to another agent, and that agent silently ignores one region because the query timed out, the final report may look confident and still be wrong. If a research agent asks a finance agent for market context, the final answer needs to show whether the finance agent used fresh data, cached data, or a fallback model making sad little guesses.

The point is not to drown humans in traces.

The point is to make the trace available when the answer matters.

the platform checklist

If I were responsible for agent interoperability inside a company, I would not start by arguing about which protocol wins.

I would start with inventory.

Which agents exist? Who owns them? What can they do? Which tools can they reach? Which data can they see? Which other agents can call them? Which environments run them? Which version is production?

Then I would require explicit capability documents.

Not marketing descriptions. Operational ones. Accepted task schemas, rejected scopes, auth requirements, latency expectations, retry semantics, cancellation behavior, data classification, budget rules, approval boundaries, and support contacts.

Then I would version the handoff contracts.

If Agent A depends on Agent B for a workflow, that dependency should be visible somewhere better than a prompt. Contract changes should have migration notes. Breaking changes should have dates. Deprecated behavior should have a removal policy. The boring API-management muscles still apply.

Then I would trace cross-agent work end to end.

One workflow ID across the caller, callee, tools, approvals, and final output. Not because tracing is fashionable, but because debugging a distributed agent workflow without correlation IDs sounds like a punishment for something I did in a previous sprint.

Then I would test failure paths.

Agent unavailable. Agent slow. Agent returns partial data. Agent asks for approval. Agent exceeds budget. Agent rejects scope. Agent starts work and then receives cancellation. Agent delegates to a forbidden downstream agent. Agent returns output that violates the caller's schema.

If those cases are not tested, the production behavior is mostly a rumor.

standards are good, but they do not remove ownership

I like that A2A is moving under neutral governance. I like that MCP is getting more operationally serious. I like that agent interoperability is becoming less vendor-specific and less glued together by bespoke adapters.

That is progress.

But standards do not remove ownership.

They make ownership more important.

Once protocols make handoffs easier, more teams will create them. Once more teams create them, the platform needs a way to govern identity, scope, retries, task state, budgets, approvals, provenance, audit, and human-readable failure.

The practical lesson is not "agents can talk."

The practical lesson is that agent handoffs are production contracts now.

Treat them like APIs. Treat them like queues. Treat them like jobs. Treat them like incident runbooks. Give them names, versions, owners, deadlines, traces, and failure modes.

Interoperability without operability is not a platform.

It is just a larger surface area for ambiguity.

And ambiguity is where production systems go to make humans look tired.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

shadow ai is now part of the ci/cd threat model

Paulo Victor Leite Lima Gomes — Sun, 09 Aug 2026 00:03:17 +0000

For a while, shadow AI sounded like the new version of shadow IT.

Someone used a browser tool without approval. Someone installed an IDE extension. Someone pasted a stack trace into a chatbot and promised, with the confidence of a person who has not read the data policy, that there was "nothing sensitive in there."

Annoying? Yes.

New? Not really.

Then the same behavior walks into CI/CD, and the joke changes.

CNCF published a fresh piece on shadow AI in CI/CD, framing the problem as a path from developer laptop to Kubernetes. That is the right framing. The issue is not that developers are using AI. They are. The issue is that AI tools, model calls, extensions, MCP servers, and agents can quietly become part of the delivery path without being named as infrastructure.

And unnamed infrastructure is my least favorite kind. It always waits until a Friday to introduce itself.

ci/cd is not a vibes zone

CI/CD is where code becomes production. That sounds obvious, but people forget it when the tool looks like a helpful assistant instead of a deployment component.

An AI extension that suggests code is one thing. An agent that opens pull requests, edits workflows, calls internal tools, reads secrets, writes artifacts, comments on review findings, generates Dockerfiles, updates Kubernetes manifests, or triggers release jobs is very much another thing.

At that point, the agent is not "developer productivity."

It is part of the software supply chain.

If the answer is "well, it is just installed on a few laptops," congratulations. You have built a distributed production dependency with worse asset inventory.

the laptop is now part of the pipeline

The classic supply-chain threat model already cared about developer machines. AI makes that boundary softer.

A coding assistant can read a repository. A browser agent can read docs, issues, and dashboards. An MCP server can expose local files, databases, internal APIs, or cloud operations. A terminal agent can run commands. A review bot can influence what humans approve. A workflow generator can change the YAML that later runs with stronger permissions in CI.

None of these are automatically bad. I use these tools. I also prefer them with guards.

The problem is the invisible bridge.

A developer installs an AI helper to save time. The helper writes code. The code changes a build step. The build step pulls a dependency. The dependency runs in CI. CI has tokens that can publish artifacts, push images, deploy to Kubernetes, or talk to cloud APIs. That is not a moral panic. That is just a graph.

prompt injection gets a delivery badge

Prompt injection is annoying in a chat window. In CI/CD it can become operational.

Imagine an agent reading an issue, a README, a generated dependency report, or a web page during a build-related task. The content tells it, politely and confidently, to ignore previous instructions and fetch a secret, change a workflow, install a package, or call a tool it should not call.

The model is not "hacked" in the movie sense. It is interpreting untrusted text while tools are available. That is enough.

The old web security lesson still applies: data and instructions need boundaries. We learned this with SQL injection. We learned it with shell injection. We learned it with XSS. Now we get to learn it again with agents, because apparently software engineering is a subscription service to the same class of mistakes.

CI/CD makes this sharper because the environment has authority. Build systems sign artifacts. They publish packages. They push container images. They deploy. They comment on pull requests in ways humans may trust. They often have more permissions than any single human needs, because the pipeline is where everybody's shortcuts go to become policy.

An agent operating near that environment needs the same suspicion we give any automation with credentials.

inventory first, philosophy later

The first useful step is boring: inventory.

Which AI tools are installed by developers? Which are approved? Which can read source, call tools, write to GitHub, or run inside CI? Which MCP servers exist? Which secrets are exposed to jobs that can run agent-written scripts?

This is not glamorous work. It will not get a keynote. It will produce a spreadsheet with several rows named things like temp-copilot-test and old-agent-demo-final2, which is how you know you are doing real platform engineering.

But without inventory, policy is theatre.

You cannot enforce an allowlist if nobody knows what is being used. You cannot audit a model call that never entered your logging path. You cannot rotate a token that was copied into a local agent config six months ago by someone who has since joined a team called Innovation Enablement, which is where accountability goes to wear a blazer.

policy belongs in the control plane

The second step is moving from advice to controls.

"Please do not paste secrets into random tools" is a good sentence. It is not a security architecture.

This is why the enterprise features around AI tools matter even when they sound dull. MCP allowlists, org-level defaults, session limits, review labels, audit trails, and policy-managed tool access are not corporate checkbox dust. They are the beginning of treating agents like production actors.

The policy should not say "no AI." That battle is mostly over, and banning useful tools usually creates a worse shadow version.

The policy should say: these agents can run here, with these tools, under these identities, with these logs, with these spending limits, against these repositories, and with these review requirements.

That is a sentence platform teams can implement.

ci permissions should be smaller than your ambition

Most CI systems are too trusting by default because they grew up around scripts written by people inside the company. That was already optimistic. Agent-generated changes make it funnier.

If an agent can modify a workflow file, and that workflow later runs with broad tokens, you have a privilege-escalation shape. If pull request builds can access secrets too early, you have a data-exfiltration shape. If a generated script can download and execute arbitrary dependencies, you have a supply-chain shape. If a bot can approve its own changes through a friendly review loop, you have invented management consulting for malware.

The answer is not exotic.

Use least privilege. Separate read and write tokens. Treat workflow changes as sensitive. Require human review for delivery definitions. Limit egress from build jobs. Pin actions where it matters. Keep production deploy credentials away from anything that runs untrusted code.

The agent angle does not replace normal CI/CD hardening.

It removes excuses for postponing it.

review the output, but also review the path

Humans are still in the loop, but the loop needs to include more than the final diff.

If an agent opened a pull request, I want to know what it touched, what commands it ran, which tools it called, which external context it used, and whether it changed the delivery path. A clean-looking diff can still have a dirty path behind it.

This does not mean every agent run needs a courtroom transcript. It means risky actions need traceability. For CI workflows, infrastructure-as-code, Kubernetes manifests, secrets plumbing, package publishing, auth code, and payment paths, the bar should be higher.

shadow ai is a naming failure

I do not think shadow AI in CI/CD is mainly a developer behavior problem. Developers use tools that help them ship. This is not surprising. Water is wet. YAML is still a crime scene.

The real problem is letting useful tools enter the delivery path without turning them into named, governed infrastructure.

Once an AI system can influence code, builds, artifacts, credentials, deployments, or Kubernetes state, it needs an owner. It needs inventory. It needs logs. It needs permissions. It needs limits. It needs a way to be disabled without asking six teams who installed what.

This is boring platform work.

Good.

Boring platform work is how we survive exciting tooling.

Shadow AI stops being a cultural debate the moment it touches CI/CD. From there, it is supply-chain infrastructure without a nameplate.

Put the nameplate on it.

Then give it the smallest key that lets it do the job.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

prometheus collectors are cloud control plane now

Paulo Victor Leite Lima Gomes — Sat, 08 Aug 2026 00:08:29 +0000

Prometheus used to have a nice emotional contract: you ran the thing, scraped the thing, and cursed at the thing when a target disappeared because somebody changed a label during a deployment and now the dashboard looks like a crime scene.

It was not pleasant, but at least ownership was obvious.

Then AWS announced CloudWatch managed Prometheus collectors, and the shape changed. CloudWatch can now run fully managed collectors that discover Prometheus-compatible targets in AWS infrastructure, scrape them through VPC connectivity, and deliver the metrics into CloudWatch. EKS, EC2, ECS, MSK, OpenSearch. The usual suspects.

This is genuinely useful.

It also means the scrape layer is not just a daemonset, sidecar, or collector you forgot to upgrade. It is becoming another cloud control plane.

That is worth paying attention to.

nobody enjoys operating scrapers

Let us be honest. Maintaining metrics collection infrastructure is not the part of observability that makes people feel alive. Somebody has to run collectors, patch them, scale them, give them network access, keep configs synced, notice when service discovery breaks, and stop the high-cardinality endpoint from eating the month.

Prometheus is excellent. Prometheus also makes you own the scraping path.

AWS taking more of that burden makes sense. The new collectors are agentless from the customer's point of view. You provide scrape configuration and a connection to the resources. CloudWatch provisions and scales the machinery. The docs say the collector creates ENIs in selected subnets, scrapes through them using OTLP, and delivers metrics to a CloudWatch dataset through a VPC endpoint.

That removes operational work and moves it into the platform contract.

the scrape config is production code

Every observability system eventually teaches the same annoying lesson:

The dashboard is not the product.

The pipeline is.

If the scraper misses a target, the dashboard lies politely. If relabeling is wrong, the alert routes to the wrong team. If a metric selector is too wide, the bill grows. If it is too narrow, the incident starts with everyone saying "that is weird, the graph is flat."

With managed collectors, scrape configuration is production code. It decides what gets seen, how it is named, where it lands, and how much it costs. That means it needs the same discipline as other production configuration:

code review
owners
environment separation
rollback
change history
label conventions
cost expectations
tests or at least validation queries

Yes, tests for observability config sound boring. So does checking a parachute.

managed does not mean ownerless

The trap with managed services is the little mental shortcut where "AWS runs it" becomes "we do not have to think about it." That shortcut is expensive.

Managed collectors can remove the need to operate collector hosts, in-cluster scrapers, or a pile of OpenTelemetry collector deployments. Great. Please remove toil. Tired humans deserve mercy.

But the collector still needs network permissions. It still needs scrape targets. It still needs security groups. It still needs subnets. It still needs metric selection. It still emits billable data. It still becomes part of the answer when the incident commander asks, "can we trust this graph?"

Observability is not decoration. Autoscalers, rollback decisions, SLOs, anomaly detection, agent workflows, and very tired humans at 03:17 all depend on it. Once the metrics pipeline becomes managed infrastructure, platform teams need to treat it as a shared dependency.

labels are where good intentions go to get weird

The hard part of metrics collection is rarely "can I scrape /metrics?"

It is usually:

what is this service called?
which team owns it?
is this production?
which customer, region, tenant, cluster, namespace, queue, model, or job does this represent?
is this cardinality useful or did we just create a tiny money printer pointed at ourselves?

Labels are the observability version of naming things and blaming DNS. They look simple until your query returns twelve answers and none of them are the one you need.

Managed collectors do not remove that problem. They make the boundary cleaner.

For ECS, AWS describes DNS-based discovery through Cloud Map so the collector follows changing task IPs. For EKS, discovery can follow Kubernetes targets. For EC2, direct instance scraping still needs ports and sane target definitions.

But if every team brings its own labels, the centralized collector becomes a centralized junk drawer.

The right platform move is to publish a small label contract and enforce it where possible:

service
team
environment
region
workload_type
slo_name when it exists

Then be suspicious of labels with unbounded values. User IDs, request IDs, raw URLs, pod hashes, prompt IDs, session IDs, and "temporary" fields are how observability turns into a billing incident wearing a Grafana badge.

cost is part of the signal now

The CloudWatch docs are clear that managed collectors are charged by the hour and CloudWatch OpenTelemetry metric ingestion pricing applies. They also mention VPC data transfer can show up when scrapers pull metrics from targets, and suggest compressing /metrics responses to reduce transfer volume.

That is architecture.

When metrics were collected by something your team operated, the cost was often hidden inside nodes, daemonsets, storage, and people pretending a giant Prometheus was "fine for now." With managed collectors, the bill can become more explicit. This is good if you measure it. It is surprising if you do not.

Metric volume needs a budget. Scrape interval needs a budget. Cardinality needs a budget. Retention and query patterns need a budget. Before anyone says "but it is just metrics", remember that "just logs" already bought many companies a very educational invoice.

The useful question is not "is managed collection cheaper?"

What is the cost per service of the metrics we actually use to operate the system?

observability has a supply chain too

We talk a lot about software supply chains now. Images, dependencies, SBOMs, signatures, provenance, all that healthy paranoia.

Metrics have a supply chain too.

Exporter. Endpoint. Network path. Scraper. Relabeling. Transport. Storage. Query. Alert. Dashboard. Human decision. If one link is wrong, the final graph may still look professional.

Managed collectors make some links stronger. Less self-managed collector drift. Better scaling. Fewer forgotten agents. More unified querying with CloudWatch and PromQL. That is valuable.

They also concentrate trust. If CloudWatch becomes the place where more Prometheus data lands, collector configuration becomes a first-class platform artifact. It should live near the IaC, have owners, and have a rollback path that does not involve three people clicking in a console while the incident channel develops a personality.

what i would do first

If I were rolling this out, I would avoid the grand observability transformation slide deck and start smaller.

First, pick one environment and one workload type. ECS services through Cloud Map, for example. Prove that discovery, security groups, scrape intervals, labels, dashboards, and alarms make sense.

Second, put the collector config in version control. No mystery config. No "we changed it in the console because the graph looked lonely."

Third, define a label contract before onboarding the second team. That is where standards become folklore if nobody writes them down.

Fourth, set cost alarms early. Not after the first strange invoice. Early. Scrape volume is part of the rollout.

Fifth, document the trust path for incident responders. If an alert fires from a managed collector, people should know where the target is, which config scraped it, and who owns the metric.

That is not bureaucracy.

That is how you make "managed" mean less work instead of less visibility.

the punchline

CloudWatch managed Prometheus collectors are a good feature because they remove a chunk of observability plumbing nobody wants to babysit.

But plumbing is not meaningless just because a cloud provider operates part of it.

The scrape layer decides what your company can see. It shapes alerts, dashboards, SLOs, autoscaling, rollback confidence, and sometimes the emotional weather of an incident call.

So yes, let AWS run the collectors if that reduces toil. Let CloudWatch pull Prometheus-compatible metrics from EKS, ECS, EC2, MSK, and OpenSearch. Use PromQL where it helps. Delete some self-managed collector snowflakes.

Just do not confuse managed infrastructure with absent ownership.

The metrics pipeline is becoming a cloud control plane.

Treat the scrape config like production code, the labels like an API, and the bill like an alert signal.

Otherwise the graph will look calm right up until reality files a ticket.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

docker content trust retiring is the migration teams forgot

Paulo Victor Leite Lima Gomes — Fri, 07 Aug 2026 00:03:48 +0000

Docker Content Trust is not disappearing in a dramatic way.

There is no breach headline. No emergency patch window. No breathless thread explaining why every container is suddenly unsafe.

Instead, Docker is doing something much more useful and much easier to ignore: it is retiring an old trust mechanism on a schedule.

Docker announced that Docker Content Trust and the Notary v1 service at notary.docker.io are being fully retired. Write brownouts already happened in July. Read brownouts are scheduled for August 10 and August 12, 2026. Full shutdown is scheduled for December 8, 2026.

Most teams will not notice, because ordinary docker pull and docker push do not use DCT unless it was explicitly enabled.

That is exactly why this is an interesting migration.

The danger is not that every container workflow breaks. The danger is that a few important workflows break because nobody remembers they were relying on a trust feature from a different era of the container ecosystem.

the old checkbox was doing real work

Docker Content Trust came from a useful idea: make image trust feel close to the Docker workflow.

Set DOCKER_CONTENT_TRUST=1. Use docker trust commands. Let the client verify signed metadata before accepting an image tag. For many teams, that was the first practical taste of container image signing.

It was not perfect, but it was a real control.

The uncomfortable part is what happens when an early control becomes organizational memory. A security exception gets written around it. A CI pipeline inherits an environment variable. A Kubernetes admission policy checks for it. A base-image publishing process signs with it because that was the approved pattern three platform teams ago.

Then years pass.

The people who installed the mechanism move on. The repo keeps building. The cluster keeps deploying. The control still exists, but nobody can clearly say whether it is essential, obsolete, redundant, or accidentally load-bearing.

That is the migration most teams forgot.

Not the command migration.

The ownership migration.

trust moved from docker-native to supply-chain architecture

Docker's guidance is blunt in the best way: Notary v1 is no longer maintained, and the ecosystem has moved toward OCI-native signing tools such as Sigstore Cosign and Notation.

That is the right direction.

It is also a change in responsibility.

DCT made trust feel like a Docker client feature. Modern image trust is more distributed. Signatures live as OCI artifacts. Identity may come from OIDC. Verification may happen in CI, in a registry policy, in an admission controller, or in a deployment platform. Provenance and SBOMs may become part of the same release evidence.

That is more powerful.

It is also less magical.

You do not migrate by replacing one command with another command and calling the work done. You migrate by deciding what must be true before an image can move from source code to production.

Who is allowed to sign?

Which identity signs release images?

Where are signatures stored?

Which registries are trusted?

What happens when verification fails?

Can a developer bypass the policy for an incident?

Does the cluster enforce the same thing CI claims to enforce?

Those questions are not Docker trivia. They are release architecture.

disabling dct is not a strategy

For some teams, the right immediate move really is simple.

Search for DOCKER_CONTENT_TRUST=1. Search for docker trust sign, docker trust inspect, and docker trust revoke. Search CI configs, shell profiles, Dockerfiles, Compose files, deployment scripts, and old platform templates. If DCT is set only by accident, remove it before the read brownouts find it for you.

That is fine.

But "unset the variable" should be treated as a triage step, not a supply-chain strategy.

If DCT was the only thing standing between your production release path and arbitrary mutable tags, removing it may keep builds green while reducing the evidence behind those builds.

This is where teams get tricked by operational success.

The pipeline passes.

The incident is avoided.

The migration ticket is closed.

And the organization quietly moves from imperfect verification to no verification at all.

That is not modernization. That is control deletion with a better changelog.

digest pinning is necessary but incomplete

Docker's migration post correctly separates repeatability from publisher identity.

Pulling by digest is a major improvement over trusting a tag. A digest says, "give me this exact content." That matters because tags can move, registry defaults can change, and rebuilds can produce new artifacts under familiar names.

If production depends on some-image:latest, you do not have a precise release input. You have a suggestion.

So yes, pin digests where production repeatability matters.

But digest pinning is not the same as trust.

A digest can prove that you got the content you requested. It does not, by itself, prove that the content came from the publisher, pipeline, or review process you intended to trust.

This distinction matters more now because AI agents are making artifact production cheaper. More code can be generated. More images can be built. More dependency updates can be proposed. More "small safe changes" can enter the queue.

The bottleneck is no longer only producing software.

The bottleneck is proving what happened to the software before it shipped.

admission control is where trust becomes real

Signing an image is only half the story.

The other half is enforcement.

If the cluster accepts unsigned images, signature policy is documentation. If the deployment system accepts images from any registry, registry policy is a slide. If CI checks one path but emergency deploys use another, your supply-chain story has a side door.

This is why tools like Kyverno, Ratify, Gatekeeper, Cosign, and Notation matter less as product names and more as places to make release claims enforceable.

The platform team should be able to say:

release images are signed by this identity
production deploys require a valid signature
allowed registries are explicit
base images come from approved sources
exceptions expire
verification failures are visible
the policy is tested before the brownout, not during it

That list is boring.

Good.

Boring is what mature trust looks like.

this is really about old controls aging out

DCT retirement is a container story, but the pattern is bigger than Docker.

Engineering organizations accumulate controls the same way they accumulate dependencies. Some are still valuable. Some have better replacements. Some are dead but harmless. Some are dead but important. The dangerous ones are the controls that are both forgotten and load-bearing.

AI makes this worse, not because AI changes image signing, but because it increases the amount of automation running through old paths.

An agent can update a Dockerfile.

An agent can regenerate a build pipeline.

An agent can migrate a service to a new base image.

An agent can open ten pull requests that all look plausible.

But if nobody knows what release evidence is required, the agent will optimize for passing checks, not preserving trust.

That is why the useful question this week is not "are we affected by Docker Content Trust retirement?"

The better question is: "do we know which supply-chain controls are actually protecting production?"

If the answer is no, the DCT timeline is a gift. It gives teams a dated reason to inventory old assumptions before December turns them into production surprises.

what i would do before august 10

I would start with discovery, not a tool debate.

Search for DOCKER_CONTENT_TRUST, docker trust, Notary v1 endpoints, and admission policies that mention DCT. Check CI variables, repo secrets, build images, deployment templates, and platform docs. Ask registry publishers whether they are moving to Cosign, Notation, or something else.

Then split the findings into three buckets.

First: accidental DCT usage that can be removed safely.

Second: intentional DCT usage that needs a migration path to modern signing.

Third: places where DCT was pretending to be a broader supply-chain strategy.

That third bucket is the important one.

It should produce platform work: digest pinning guidance, signing identity, registry policy, admission enforcement, exception handling, release evidence, and ownership.

The old trust checkbox is going away.

What replaces it is not a feature.

It is a release discipline.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

your gpu bill is not a model problem

Paulo Victor Leite Lima Gomes — Thu, 06 Aug 2026 00:04:39 +0000

The cloud bill has a very annoying habit.

It waits until everyone is talking about the exciting abstraction, then it shows up with the boring invoice.

For AI infrastructure, the exciting abstraction is the model.

Which model is smarter? Which one has better reasoning? Which one is cheaper per million tokens? Which one can explain a failing test with the confidence of someone who absolutely did not run it?

Useful questions.

Incomplete questions.

OpenCost 1.121.0 added Kubernetes inference cost tracking with llm-d and vLLM metrics. The interesting part is not only that it can calculate model and token cost. The interesting part is the distinction it forces platform teams to make.

What did the model's work cost?

And what did keeping the model ready cost?

Those are not the same question.

That difference is where many self-hosted AI spreadsheets go to die.

the model is not the workload

When teams talk about AI cost, they often jump straight to price-per-token.

That makes sense for SaaS APIs. You send tokens. You receive tokens. Somewhere in the middle, a billing system does billing-system things and your dashboard gets another little number to make you sad.

Self-hosting changes the shape.

Now you are not only buying inference.

You are running infrastructure.

There are GPUs. There is memory. There is a gateway. There are pods. There are model weights sitting in VRAM. There is routing. There is cache behavior. There are namespaces, teams, tenants, and probably one deployment nobody wants to touch because it was "temporary" in March.

This is why the OpenCost angle matters. It connects Kubernetes cost allocation to inference metrics from vLLM and llm-d. It is trying to answer cost questions in the place where the workload actually runs, not in a slide where all GPUs are magically 100% utilized and nobody ever pays for idle capacity.

That slide is beautiful.

It is also fiction.

usage cost can lie to you politely

The useful distinction is usage-based cost versus allocation-based cost.

Usage-based cost is about active work. The model processed tokens. GPU compute was consumed. Input and output tokens had real processing cost. KV cache hits changed that cost. Great.

Allocation-based cost is about availability. The model is loaded. GPU memory is reserved. The serving stack is alive. Common infrastructure is running. The system is ready to respond even if the request traffic is having a nap.

Both numbers are true.

Only one answers the build-versus-buy question.

If a self-hosted model costs one dollar per million tokens while it is actively working, and a hosted API costs two dollars per million tokens, the spreadsheet starts smiling.

"Look, we are cheaper."

Maybe.

If the GPU is busy enough.

If the model is mostly idle, the real cost per token includes the quiet hours too. A GPU with a model loaded is not free because the request queue is empty. It is just expensive in a less photogenic way.

This is the trap. Usage-based cost can make self-hosting look cheap because it ignores the carrying cost of readiness. Allocation-based cost includes the part production teams actually pay for at the end of the month.

The invoice does not care that your benchmark was efficient for seven minutes.

warm capacity is a product decision

There is a reason teams keep models warm.

Latency matters. Cold starts are bad. Users do not enjoy waiting while your platform gently remembers how to be useful. Some workloads need a model ready now, not after a scheduling adventure and a loading ceremony.

So keeping capacity warm can be the correct choice.

But it has to be a choice.

Not an accident.

This is where AI FinOps becomes platform engineering. The useful question is not "is this model expensive?" That is too vague. Everything is expensive if you squint with procurement nearby.

The better questions are more practical.

Which teams need this model warm?

Which traffic can tolerate queueing or scale-up?

Which low-volume workloads should use an external API instead?

Which models can share capacity?

Which routes should consolidate traffic to improve utilization?

Which experiments are quietly reserving production-grade hardware for demo-grade usage?

This is less glamorous than model selection. It is also where the money is.

cost labels become architecture

One detail I like in the OpenCost work is the label shape. The new metrics include things like model name, model version, namespace, workload type, and whether the cost basis is usage or allocation.

That sounds boring.

Good.

Boring labels are how the bill becomes debuggable.

Without them, AI cost turns into fog. Everyone knows the platform is expensive. Nobody knows if the expensive part is one team's chatbot, another team's batch summarizer, a forgotten staging deployment, or a model nobody retired because the migration ticket got renamed into a roadmap initiative and then disappeared.

Cost attribution is not just finance hygiene. It changes engineering behavior.

When teams can see the cost of keeping a model warm for their namespace, they can have a real conversation about latency, volume, and tradeoffs. When they only see an aggregate GPU bill, the conversation becomes vibes with a spreadsheet attachment.

We have seen this movie with Kubernetes before.

CPU requests looked like small YAML details until over-requested workloads became real money. Storage classes looked like plumbing until retained volumes became invoices. Logs looked harmless until everyone discovered that debugging output also has a billing department.

Inference is getting the same treatment.

The YAML is now connected to the token.

Wonderful and terrible, as usual.

the platform owns the gap

The gap between usage-based and allocation-based cost is the part I would watch.

It is the cost of readiness.

Sometimes that gap is healthy. A fraud system, incident assistant, or customer-facing workflow may justify warm capacity because response time matters. Fine. Put that in the architecture decision. Own it.

Sometimes the gap is waste. A model is deployed for a pilot with twelve users. A team copied production values into staging. A bigger model is serving requests that a smaller model could handle. A routing layer spreads traffic so thin that every model looks underutilized.

That is not a model problem.

That is a platform problem.

The platform owns the routing, isolation, quotas, schedules, autoscaling policy, cache behavior, and cost visibility. The model is only one component inside that system.

AI infrastructure is becoming normal infrastructure faster than people want to admit. It needs the same dull tools we already learned to respect: allocation reports, budgets, ownership labels, deployment review, capacity planning, and cleanup jobs with teeth.

Not because engineers love governance.

We do not.

Because otherwise the bill becomes the observability system.

And the bill has terrible UX.

self-hosting needs utilization, not optimism

I like self-hosting. I like owning the runtime. I like being able to tune the serving stack, inspect metrics, and avoid pretending that an API call is a complete architecture.

But self-hosting is not a personality trait.

It needs utilization.

If the allocation-based cost per million tokens beats the external API price, great. If it does not, the answer is not to argue with the spreadsheet until morale improves. The answer is to change the architecture or stop self-hosting that workload.

Route more traffic to fewer warm models.

Use smaller models where they are enough.

Separate latency-sensitive traffic from batch traffic.

Scale down experiments.

Expose cost per namespace and model version.

Make teams see the fixed cost they are asking the platform to carry.

The useful version of AI cost tracking does not end with "this model is expensive."

It ends with a decision.

Keep it warm because latency matters.

Move it to an API because utilization is bad.

Consolidate traffic because too many models are doing too little work.

Change routing because the current setup is optimized for a demo, not production.

That is the real lesson.

Your GPU bill is not a model problem.

It is a utilization story with Kubernetes labels.

Which is a very platform-engineering way for AI to become boring.

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

durable execution is something you install now

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 12:05:45 +0000

A process starts a multi-step job.

Step 1 validates the customer. Step 2 creates an internal record. Step 3 moves money, changes a subscription, sends an irreversible request to a partner, or does whatever your business considers serious.

Then the process dies.

Not politely.

The container gets killed. The deploy rolls. The host disappears. The worker restarts halfway through the thing it was doing.

Now the question is horrible:

What happens when it runs again?

If step 3 already happened, restarting from step 1 may duplicate something expensive. If you skip too much, you may leave the customer in a half-activated state. If you retry blindly, the external provider may accept the same request twice.

This is the shape of a lot of fintech work. Account opening, subscription activation, credit disbursement, repayment settlement, KYC refreshes, scheduled sweeps. They are not one database transaction. They are small business stories pretending to be function calls.

For a long time, the answer was not a product. The answer was discipline.

the old answer was owning the machinery

At a fintech on the American continent, the public architecture story was event-driven: Kafka, services, async processing, consistency through events, and careful thinking around idempotency. That style makes sense in a bank-sized environment.

But the boring part of the work was never just "publish an event."

The real work was deciding what each step meant, which message was the source of truth, how to identify duplicates, how to make consumers safe to replay, and how to compensate for something that had already crossed a boundary.

You ended up with sagas, state machines, idempotency keys, outbox tables, retry topics, reconciliation jobs, audit trails, alerts, and runbooks.

That is not a criticism. In serious financial systems, this discipline is part of the job. You cannot hand-wave around money by saying "the queue is eventually consistent."

But there is an operational cost to owning the mechanism.

Every team has to remember the same failure modes. Every new workflow has to rebuild the same scaffolding. Every incident asks: did the step finish, did the event publish, did the retry duplicate it, and is the customer state now real?

The painful thing is that most of this machinery is not product-specific.

The compensation logic is product-specific. The step boundaries are product-specific. Retry and reversal rules are product-specific.

But "remember completed step 3 after the worker crashes" is plumbing.

And plumbing eventually becomes a product.

DBOS and Temporal change the default

This is why frameworks like DBOS and Temporal are worth paying attention to, even if you do not adopt either tomorrow.

Temporal's model is built around workflows and activities. The Temporal service stores a durable event history for each workflow execution. When a worker crashes or restarts, workflow code can be replayed from that history, and completed activity results are not forgotten because a process died. The service has a record of what happened.

DBOS takes a different shape. DBOS Transact is positioned as an open-source durable execution library that runs inside your application. The current DBOS material still describes the core design as Postgres-backed: workflow and step state are checkpointed into Postgres, and application servers coordinate through the database instead of a separate central orchestrator.

That difference matters.

Temporal asks you to run, or pay for, a workflow service. Temporal Cloud is priced as a managed platform, with usage and storage dimensions around things like actions and workflow history storage.

DBOS is making a more Postgres-native bet. The open-source Transact library can run anywhere, and the paid DBOS offering is around Conductor, support, management tooling, and hosting. Their current public pricing lists Pro and Teams plans around managed apps, seats, and checkpoint volume.

I do not think the interesting point is which one is "better." That depends too much on the system, team, language, and operational model.

The interesting point is that both package the same formerly internal capability:

This multi-step job survives process death.

That sentence used to imply bespoke platform work. Now it can start with a framework.

the hard part did not disappear

There is a trap here, and it is exactly the trap that appears every time infrastructure gets better.

People confuse "the framework remembers what happened" with "the system now knows what should happen."

It does not.

If a workflow charges a customer, activates a subscription, provisions access, sends an email, and updates analytics, the framework can help make that workflow resumable. It can record step completion, retry activities, show where execution stopped, and make the history visible instead of buried inside logs.

But it cannot tell you whether charging the customer should happen before activation or after activation. It cannot tell you whether a failed activation should trigger a refund, a retry, a manual review, or a grace period. It cannot tell you whether a partner API is actually idempotent just because it accepts an idempotency key.

That is still design work.

At a fintech in the Gulf I work with now, this comes up in normal product workflows. A scheduled sweep is not interesting because a timer fired. It is interesting because the system needs to know which accounts were considered, which ones were skipped, and what should happen when the process wakes up after a deploy.

Durable execution helps a lot there.

A completed step recorded in Postgres is much better than a log line and a prayer. A workflow console is better than SSH-ing into a worker for clues.

But the useful conversation shifts upward.

Instead of asking, "how do we build the retry table?", the team asks, "where is the boundary where retry becomes compensation?"

Instead of asking, "how do we resume after a crash?", the team asks, "what does resumed mean for this customer?"

That is a better use of engineering time.

idempotency still lives at the edges

One thing I would be careful about in any durable execution pitch is idempotency.

Inside the framework boundary, you may get very strong guarantees. Completed workflow steps can be recorded. Activity results can be remembered. Retries can be controlled. Execution can resume after a crash.

At the external boundary, reality is messy.

Payment providers, banking partners, email systems, identity vendors, and card processors all have their own ideas about duplicate requests. Some support idempotency well. Some support it in documentation. Some support it until a timeout happens.

Boundary calls still need stable request identifiers, provider references, and reconciliation. You still need to know whether a timeout means "nothing happened", "something happened but you did not see it", or "check later."

Durable execution reduces the number of places where that logic leaks.

It does not repeal distributed systems.

durability as a purchasable property

The reason I like this category is that it fits a pattern that keeps repeating in software.

At first, a reliability property is craft. A strong team builds it internally because they have no choice. Then enough teams hit the same pain, and the property becomes a product.

Observability did this. Feature flags did this. Secrets management did this. CI/CD did this. Policy-as-code did this. Now durable execution is doing it.

The boring control-plane property becomes something you can buy, install, or outsource.

That is good. I do not miss every team building its own mini workflow engine by accident.

But the job does not vanish. It moves.

The value is no longer proving that your worker can survive a restart. The value is deciding which workflows deserve durability, which steps are atomic from the business point of view, which failures should retry, which failures should compensate, and who owns the customer outcome.

That is where durable execution gets interesting.

Not because DBOS or Temporal make failure disappear.

Because they make the failure visible enough that we can stop pretending the hard part was the retry loop.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

DBOS and the shrinking cost of durable workflows

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 10:37:23 +0000

The annoying failure is not the one where everything dies before anything important happens. That one is clean. You retry the job, or the user clicks again, or the scheduler runs next minute. Fine.

The real failure is the one where the process dies after step 3.

Step 1 created the customer record. Step 2 reserved something. Step 3 moved money, activated a subscription, sent a document, or called an external provider that will not forget what you asked it to do. Then the worker disappears. Kubernetes evicts the pod. The VM restarts. A deployment kills the process. Someone discovers that "stateless service" was more aspiration than architecture.

Now what? If you restart the job, you might charge twice. If you skip it, the customer is stuck halfway through activation. If you manually repair it, congratulations, you invented an operations queue with anxiety as the UI.

This is the problem durable execution is trying to make boring.

Not easy. Boring.

we used to build this ourselves

In fintech, this shows up everywhere because workflows are longer than a request and more fragile than the happy path diagram.

At a fintech in the Americas, the answer was the one many serious backend teams reached: sagas, Kafka, idempotency keys, state tables, retries, reconciliation jobs, and discipline.

That architecture was not dumb. It was right for its time.

You split a process into steps. Each step publishes an event or writes state. Commands carry idempotency keys. External calls have deduplication at the boundary when possible. Reconciliation jobs scan for weird states later because you know the system will find one. Everyone learns that "at least once" is not a delivery guarantee, it is a personality test.

The hard part was never only writing the saga. The hard part was owning it.

You had to decide what state meant, which failures should pause, which should compensate, and which should page a human. You had to keep idempotency keys flowing across service boundaries. You had to make sure a step that looked atomic in code was atomic in the business.

And because the mechanism was hand-rolled, every team ended up owning some version of the same machinery.

One team had an orchestration table. Another had choreography over Kafka topics. Another had a cron that repaired half-completed rows. All of this worked, until it did not, and then the organizational memory lived in the heads of the people who had been burned before.

This is why DBOS and Temporal are interesting to me. They did not invent the idea that a workflow should survive a crash. They productize the control-plane part we used to rebuild.

what DBOS changes

I first paid attention to DBOS while looking at subscription activation workflows and scheduled sweeps at a bank I work with now in the Gulf. The storage model caught my eye.

DBOS Transact makes workflows durable by checkpointing completed steps in Postgres. If the process crashes, the restarted app can resume from the last completed step.

It also changes the adoption shape. DBOS is not asking every team to operate a separate workflow cluster before they get durability. The open-source Transact libraries are positioned as application-adjacent durable execution built on Postgres-compatible storage. Their material still says the state can live in your application database. DBOS also sells the control-plane layer, with paid plans priced around users, applications, and checkpoint volume.

That matters because the first objection to workflow engines is often operational weight. DBOS is aiming at the space where you want crash recovery, durable queues, scheduled work, and observability without turning every service into research.

The tradeoff is clear. If durable execution metadata is in Postgres, Postgres becomes part of the runtime. That can be a feature or a coupling point. You get simpler ownership and queryability, but you still need to understand retention, migrations, backups, and the blast radius of putting workflow history near application data.

No free lunch. Fewer moving parts.

what Temporal gives you

Temporal has a different shape.

Temporal gives you a durable execution platform where workflows are recorded as event histories. A workflow can run for seconds, days, or years, and the service persists the events needed to recover and continue. Activities run outside deterministic workflow logic. The SDK replays workflow history to rebuild state.

That model is powerful when the workflow is long-lived, crosses services, and needs signals, timers, retries, versioning, and a strong operational surface.

Temporal Cloud pricing also makes the product shape explicit. It is a managed platform priced around Actions, active storage, retained storage, and support. You are buying a durable execution control plane.

That is a reasonable deal for many teams. It is also a real platform decision. Once Temporal is in the middle of a process, someone owns namespaces, workers, task queues, retry policies, versioning, security, and cost.

the hard part stayed yours

Here is the part that people sometimes skip when durable execution gets exciting.

A framework can remember that step 3 completed. It cannot tell you whether step 3 was the right boundary.

If step 3 called a payment provider, did the provider accept an idempotency key? If the network timed out, did the payment happen or not? If the workflow resumes tomorrow, should it retry, query the provider, compensate the customer, or stop for manual review?

Durable execution makes these questions visible. It does not answer them for you.

That visibility is useful. In hand-rolled systems, step boundaries often leak across topic names, retry loops, database rows, and tribal knowledge. In DBOS or Temporal, you are forced to make the workflow shape explicit. You name the step. You decide what gets checkpointed, what is safe to replay, and what must never be replayed.

That is the real design work.

Not "how do I keep a process alive forever?" The question is, "what does it mean for this business action to have happened?"

That is why fintech engineers get twitchy about retries. Retry is not a technical verb when money is involved. It can mean "try the HTTP call again." It can also mean "charge twice."

The boundary matters.

So does ownership. A durable workflow that crosses payments, subscriptions, notifications, and compliance is not owned by the framework. It is owned by a team, or by nobody.

durability became a purchasable property

The wider pattern is the interesting one.

A lot of backend engineering is the history of boring properties becoming products. We used to build deployment scripts. Then deployment became a platform property. We used to hand-roll metrics pipelines. Then observability became a product category. Some teams still build these things, for good reasons. But the default shifted.

Durable execution is going through the same move.

"This multi-step job survives a crash" used to be a badge of engineering maturity buried inside every serious backend system. Now it is increasingly an infrastructure property. You can install a library, run a platform, or pay for a managed service.

That is good. Fewer teams need to rediscover failure modes from first principles. More workflows can have explicit state instead of vibes. Scheduled sweeps, onboarding flows, subscription activation, refunds, identity checks, and cleanup jobs can stop depending on the fantasy that processes die only at convenient times.

But it also raises the bar for engineering judgment.

If durability is easy to add, teams will add it to things that should have stayed simple. If every background job becomes a workflow, you can create a system that is reliable but hard to reason about.

The question is not "should we use DBOS or Temporal for everything?"

The question is "which business processes deserve durable execution, and who owns the semantics when they resume?"

That is a better question than "how many cron retries should we add?"

And it is where the work belongs.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

durable execution is something you install

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 10:33:55 +0000

A process starts a multi-step job. Step 1 validates the customer. Step 2 creates an internal record. Step 3 moves money, changes a subscription, sends an irreversible request to a partner, or does whatever your business considers serious. Then the process dies.

Not politely.

The container gets killed. The deploy rolls. The host disappears. The worker restarts halfway through the thing it was doing.

Now the question is simple and horrible:

What happens when it runs again?

This is the shape of a lot of fintech work. Account opening, subscription activation, credit disbursement, repayment settlement, KYC refreshes, scheduled sweeps, ledger posting. They are not one database transaction. They are small business stories pretending to be function calls.

the old answer was discipline

But the boring part of the work was never just "publish an event." The real work was deciding what each step meant, which message was the source of truth, how to identify duplicates, how to make consumers safe to replay, and how to compensate for something that had already crossed a boundary.

You ended up with sagas, state machines, idempotency keys, outbox tables, retry topics, reconciliation jobs, audit trails, dashboards, alerts, and runbooks.

That is not a criticism. In serious financial systems, this discipline is part of the job. You cannot hand-wave around money by saying "the queue is eventually consistent" and then go for lunch.

But there is an operational cost to owning the mechanism. Every team has to remember the same failure modes. Every new workflow has to rebuild the same scaffolding. Every incident asks: did the step finish, did the event publish, did the retry duplicate it, and is the customer state now real?

The painful thing is that most of this machinery is not product-specific.

The compensation logic is product-specific. The step boundaries are product-specific. The decision that a transfer can be retried but a partner call needs an idempotency key is product-specific. But "remember completed step 3 after the worker crashes" is plumbing.

And plumbing eventually becomes a product.

dbos and temporal change the default

This is why frameworks like DBOS and Temporal are worth paying attention to, even if you do not adopt either tomorrow.

Temporal's model is built around workflows and activities. The Temporal service stores a durable event history for each workflow execution. When a worker crashes or restarts, workflow code can be replayed from history, and completed activities are not simply repeated. The service has a record of what happened.

DBOS takes a different product shape. DBOS Transact is positioned as an open-source library that runs inside your application. The current DBOS docs still describe a Postgres system database that stores workflow checkpoints, step outputs, and schedule and queue state. The DBOS pricing page now puts the paid model around Conductor, tooling, support, and hosting options, with Pro and Teams plans priced around users, managed apps, and checkpoint usage.

That difference matters.

Temporal asks you to run, or pay for, a workflow service. Temporal Cloud pricing is consumption-based, mostly around actions, storage, and support. The minimum Cloud plan is currently listed at $100/month, with included actions and storage before usage-based charges kick in.

DBOS is making a more Postgres-native bet. The open-source library gives you durable workflows in application code, while Conductor adds operational tooling like monitoring, recovery, versioning, alerts, and support.

The interesting bit is that both package the same formerly internal capability:

This multi-step job survives process death.

That sentence used to imply bespoke platform work. Now it can start with a framework.

the hard part did not disappear

There is a trap here, and it is exactly the trap that appears every time infrastructure gets better.

People confuse "the framework remembers what happened" with "the system now knows what should happen."

It does not.

But it cannot tell you whether charging the customer should happen before activation or after activation.

It cannot tell you whether a failed activation should trigger a refund, a retry, a manual review, or a grace period. It cannot tell you whether a partner API is actually idempotent just because it accepts an idempotency key. It cannot tell you whether your "step" is one business action or three business actions wearing a coat.

That is still design work.

At a bank I work with now in the Gulf, this comes up in normal product workflows. A scheduled sweep is not interesting because a timer fired. It is interesting because the system needs to know which accounts were considered, which ones were skipped, which external calls succeeded, and what should happen when the process wakes up after a deploy.

Durable execution helps a lot there. A completed step recorded in Postgres is much better than a log line and a prayer. A workflow console is better than SSH-ing into a worker for clues. But the useful conversation shifts upward.

Instead of asking, "how do we build the retry table?", the team asks, "where is the boundary where retry becomes compensation?" Instead of asking, "how do we resume after a crash?", the team asks, "what does resumed mean for this customer?"

That is a better use of engineering time.

idempotency still lives at the edges

One thing I would be careful about in any durable execution pitch is idempotency.

At the external boundary, reality is messier.

Durable execution reduces the number of places where that logic leaks.

It does not repeal distributed systems.

the wider pattern

The reason I like this category is that it fits a pattern that keeps repeating in software. At first, a reliability property is craft. A strong team builds it internally because they have no choice. Then enough teams hit the same pain, and the property becomes a product.

Observability did this. Feature flags did this. Secrets management did this. CI/CD did this. Policy-as-code did this. Now durable execution is doing it.

The boring control-plane property becomes something you can buy, install, or outsource.

That is good. I do not miss every team building its own mini workflow engine by accident.

But the job does not vanish. It moves.

That is where durable execution gets interesting.

Not because DBOS or Temporal make failure disappear.

Because they make the failure visible enough that we can stop pretending the hard part was the retry loop.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

DBOS and the shrinking cost of durable workflows

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 10:07:34 +0000

The annoying failure is not the one where everything dies before anything important happens. That one is clean. You retry the job, or the user clicks again, or the scheduler runs next minute. Fine.

The real failure is the one where the process dies after step 3.

This is the problem durable execution is trying to make boring.

Not easy. Boring.

we used to build this ourselves

In fintech, this shows up everywhere because workflows are longer than a request and more fragile than the happy path diagram.

At a fintech in the Americas, the answer was the one many serious backend teams reached: sagas, Kafka, idempotency keys, state tables, retries, reconciliation jobs, and discipline.

That architecture was not dumb. It was right for its time.

The hard part was never only writing the saga. The hard part was owning it.

And because the mechanism was hand-rolled, every team ended up owning some version of the same machinery.

This is why DBOS and Temporal are interesting to me. They did not invent the idea that a workflow should survive a crash. They productize the control-plane part we used to rebuild.

what DBOS changes

I first paid attention to DBOS while looking at subscription activation workflows and scheduled sweeps at a bank I work with now in the Gulf. The storage model caught my eye.

DBOS Transact makes workflows durable by checkpointing completed steps in Postgres. If the process crashes, the restarted app can resume from the last completed step.

No free lunch. Fewer moving parts.

what Temporal gives you

Temporal has a different shape.

That model is powerful when the workflow is long-lived, crosses services, and needs signals, timers, retries, versioning, and a strong operational surface.

the hard part stayed yours

Here is the part that people sometimes skip when durable execution gets exciting.

A framework can remember that step 3 completed. It cannot tell you whether step 3 was the right boundary.

Durable execution makes these questions visible. It does not answer them for you.

That is the real design work.

Not "how do I keep a process alive forever?" The question is, "what does it mean for this business action to have happened?"

That is why fintech engineers get twitchy about retries. Retry is not a technical verb when money is involved. It can mean "try the HTTP call again." It can also mean "charge twice."

The boundary matters.

So does ownership. A durable workflow that crosses payments, subscriptions, notifications, and compliance is not owned by the framework. It is owned by a team, or by nobody.

durability became a purchasable property

The wider pattern is the interesting one.

Durable execution is going through the same move.

But it also raises the bar for engineering judgment.

The question is not "should we use DBOS or Temporal for everything?"

The question is "which business processes deserve durable execution, and who owns the semantics when they resume?"

That is a better question than "how many cron retries should we add?"

And it is where the work belongs.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.

DBOS and the shrinking cost of durable workflows

Paulo Victor Leite Lima Gomes — Wed, 05 Aug 2026 09:09:42 +0000

The annoying failure is not the one where everything dies before anything important happens. That one is clean. You retry the job, or the user clicks again, or the scheduler runs next minute. Fine.

The real failure is the one where the process dies after step 3.

This is the problem durable execution is trying to make boring.

Not easy. Boring.

we used to build this ourselves

In fintech, this shows up everywhere because workflows are longer than a request and more fragile than the happy path diagram.

At a fintech in the Americas, the answer was the one many serious backend teams reached: sagas, Kafka, idempotency keys, state tables, retries, reconciliation jobs, and discipline.

That architecture was not dumb. It was right for its time.

The hard part was never only writing the saga. The hard part was owning it.

And because the mechanism was hand-rolled, every team ended up owning some version of the same machinery.

This is why DBOS and Temporal are interesting to me. They did not invent the idea that a workflow should survive a crash. They productize the control-plane part we used to rebuild.

what DBOS changes

I first paid attention to DBOS while looking at subscription activation workflows and scheduled sweeps at a bank I work with now in the Gulf. The storage model caught my eye.

DBOS Transact makes workflows durable by checkpointing completed steps in Postgres. If the process crashes, the restarted app can resume from the last completed step.

No free lunch. Fewer moving parts.

what Temporal gives you

Temporal has a different shape.

That model is powerful when the workflow is long-lived, crosses services, and needs signals, timers, retries, versioning, and a strong operational surface.

the hard part stayed yours

Here is the part that people sometimes skip when durable execution gets exciting.

A framework can remember that step 3 completed. It cannot tell you whether step 3 was the right boundary.

Durable execution makes these questions visible. It does not answer them for you.

That is the real design work.

Not "how do I keep a process alive forever?" The question is, "what does it mean for this business action to have happened?"

That is why fintech engineers get twitchy about retries. Retry is not a technical verb when money is involved. It can mean "try the HTTP call again." It can also mean "charge twice."

The boundary matters.

So does ownership. A durable workflow that crosses payments, subscriptions, notifications, and compliance is not owned by the framework. It is owned by a team, or by nobody.

durability became a purchasable property

The wider pattern is the interesting one.

Durable execution is going through the same move.

But it also raises the bar for engineering judgment.

The question is not "should we use DBOS or Temporal for everything?"

The question is "which business processes deserve durable execution, and who owns the semantics when they resume?"

That is a better question than "how many cron retries should we add?"

And it is where the work belongs.

references

To test my projects, I use Railway. If you want $20 USD to get started, use this link.