DEV Community: Improving

Observability-Driven Development

Improving — Thu, 09 Jul 2026 11:03:45 +0000

At almost every conversation around observability, I've noticed a pattern. Teams have metrics and logs in place but the moment tracing comes up, there's a pause. Either it's on the backlog, or it was set up partially and never completed, or nobody owns it.

Tracing is treated as the optional third pillar of observability, until an incident happens where it's the only metric that would have actually helped.

Lack of tracing isn't really the problem, instead it's a symptom. The deeper issue is that observability is still treated as something you wire up after the application is built. It's either an ops concern or a post-deployment checklist, but not a design decision.

There is a better way to approach this, and in this blog post, we will explore observability driven development, where observability is securely tied to the application from day 1.

Observability Paradox: More Tools, Worse Outcomes

The cost of hourly downtime exceeds $300,000 for 90% of firms, and 41% of enterprises say hourly downtime costs $1 million to over $5 million. As a result, observability is being discussed in every room and 70% engineers admitted that observability budgets have increased in the past year in a research done by Dynatrace in 2025. But yet there are downtimes and teams scramble through multiple dashboards to find the root cause. Mean time to resolution (MTTR) during production incidents has trended in the wrong direction for four consecutive years. In 2021, 47% of teams reported MTTR over an hour and by 2024, that number had climbed to 82%.

So, there's more investment happening, newer tools and AI being added, yet recovery is slower.

The cost of that slowdown is not just technical. High-impact outages carry huge losses, with median cost of $2 million per hour. Teams with full-stack observability reduces the downtime cost in half. The gap between those two outcomes is not about which tools you bought but it is about how you use them and when you bring them in development.

41% of leaders still learn about service interruptions through inefficient means including customer complaints, incident tickets, or manual checks, even as organizations continue investing in observability. Your dashboards are green, but your users are already affected. That is a visibility gap baked into how the system was built.

When Observability Is an Afterthought

Most teams start by building the service, writing the code, and shipping it to production. Then someone raises a ticket to setup monitoring dashboards and alerts. That's because monitoring is the last item on the launch checklist and not the first item on the design document.

Even if the team implements visibility from the start, things aren't helpful until an incident comes to light. For instance, a team building a new payments microservice does the right things at deployment time. Scraping Prometheus metrics, forwarding the logs to the aggregation stack and building dashboards, seems to cover the observability.

However, what a healthy request flow looks like end to end is not defined. Nobody mapped out which integration points needed tracing, and error budget was not set before the service went live.

It's only when the first real incident hits, the gaps show up immediately:

Metrics show something is wrong but not where
Logs require manual correlation across multiple services to build any picture
Third-party payment gateway sitting at the edge of the system has no instrumentation at all

You'll have a few engineers to spend hours finding the root cause to fix it. This happens because observability was never part of the initial design and architectural conversations. Observability was treated as something ops handles after the application is built, not something engineers designed for while building it.

Observability-Driven Development (ODD): Designing to Be Observed

Observability Driven Development (ODD) is a development practice that shifts observability left, treating it as a design concern from day one rather than an afterthought. Teams define their observability requirements the same way they define their functional requirements. To set up observability properly, a teams should note down answers to specific questions before any code is written:

What does a healthy request flow look like end to end?
Which integration points carry the most risk and need tracing?
What failure modes are possible, and how will they surface?
What does degraded performance look like versus a hard failure?

Test-driven development (TDD) changed how teams used to think about code quality, by making testing a design activity. ODD does the same for operational visibility. You are designing a system that is observable.

Key Aspects of ODD

Here's how ODD differs from standard observability practices:

Telemetry is a design requirement, not a deployment task: Logs, metrics, and traces are scoped during design the same way API contracts and data models are.

SLOs come before code: Error budgets and service level objectives are defined at the start, giving the team clear signals to build toward and clear thresholds to alert on.

OpenTelemetry as the instrumentation standard: OpenTelemetry is now the second largest CNCF project after Kubernetes. It has become the default choice for vendor-neutral, portable instrumentation.

Observability gaps are treated as bugs: If a critical path cannot be observed, that is a defect to fix, not a gap to live with.

Implementing ODD in Practice

A complete overhaul of your existing tooling is not required to implement ODD in your software development lifecycle (SDLC) processes. It only requires shifting a few things within the process.

Identify what needs to be observable during the design phase. Map critical paths, define instrumentation touchpoints, and set SLOs before writing code.

Instrument observability during your development phase as you build. Treat observability gaps the same way you treat failing tests - something to fix before the PR merges, not after the service ships.

Validate that telemetry actually works as a pre-launch item. Confirm traces are flowing, metrics are accurate, and alerts fire under the right conditions before go-live, not after the first incident.

Benefits of ODD That Show Up in the Business

Implementing ODD doesn't just improve your reliability metrics and product quality, but it directly affects your customer satisfaction and shows up in your financials.

Faster incident resolution: Teams that design for observability upfront cut MTTR by 40-50%. When traces are in place and instrumentation covers critical paths, engineers can find root causes in minutes. The difference between a 20-minute fix taking 20 minutes versus taking 3 hours is almost always a visibility gap, not a complexity gap.

Developers building instead of firefighting: Cisco research found developers spend more than 57% of their time pulled into war rooms for performance issues. ODD changes it as when a system is built to surface its own failures, lesser efforts are spent to diagnose them.

Business cost of downtime shrinks: Full-stack observability halves the median cost of a high-impact outage. Getting there requires designing for observability from the start, not layering tools on top after the fact.

Less noise, lower spend: Nearly 70% of observability data most teams collect is unnecessary. Intentional instrumentation at design time means teams collect the signals that matter, not everything available, which directly reduces observability spend and alert fatigue.

ODD in the Age of AI

AI has transformed software development. Codes are written by AI agents, reviewed by AI agents, and even validated by AI agents. AI also changes the observability problem in a fundamental way.

Traditional applications fail in predictable ways, like a service crashed, a query timed out or an API returned a 500. These were observable with standard tooling. But AI systems fail differently, like a model may return a confident but wrong answer, and a prompt can produce inconsistent outputs under similar conditions. Latency varies in ways that are difficult to attribute to any single component. These are subtle and silent failures and often challenging to catch reactively.

This is why ODD isn't optional for AI-powered applications. Designing for observability from day one means teams define upfront what they need to see - prompt and response logging, model latency tracking, token usage, output quality signals - and build the instrumentation to surface it.

Conclusion

The correct sequence to implement ODD is:

Bring observability into the design conversation
Define what the system needs to surface before building it
Treat instrumentation gaps the same way you treat bugs.

This sequencing refines the type and speed of outcomes. Less engineering time is lost to correlating logs and finding the root cause, and cost of downtime reduces due to leaner intentional telemetry.

As AI gets embedded deeper into application stacks, observability becomes complicated. The teams with a foundation built on ODD will be better positioned to handle that complexity than those trying to retrofit visibility after the fact.

Why AI Skills Are Now Part of the Product Owner’s Playbook

Improving — Thu, 09 Jul 2026 11:02:10 +0000

Product Owners have always been managers of information, i.e. translators. They sit between what the business needs and what the engineering team builds. They write specifications, refine backlogs, prioritize features.

AI changes the economics of work. Not by replacing the PO, but by compressing the parts of the role that used to consume weeks into hours. The result isn't a smaller role. It's a role that finally has time for the work that matters most: understanding the problem, validating whether the solution is right, and making sure the team is building towards impact rather than output.

I've started thinking of this as the shift from Information Technology to Impact Technology. To survive this shift, the Product Owners must master the Impact Technology cycle, transitioning from raw inputs to human value:

Data: Raw and unstructured building blocks.
Knowledge: Information processed, synthesized, and understood.
Insights: Identification of actionable patterns and connections.
Wisdom: Strategic application of those insights to navigate complex problems.
Positive Impact: Solving human needs and creating actual value.

We've spent years collecting data, organizing it, reporting on it. The industry is good at information. What it's not good at is turning that information into something that changes an outcome for a user. That's where the PO's value lives now: not in managing information, but in driving the cycle from data to knowledge to insight to impact.

Fifteen Years of Procrastination, Resolved in an Afternoon

I have a blog with over 600 posts spanning 20 years. For 15 of those years, the taxonomy was a mess. Overlapping categories, one-off tags, inconsistent naming. "Personal Growth" and "Personal Development" coexisted as separate categories. Posts sat uncategorized. The work to clean it up was always obvious and always deferred, because it was boring and manual and nobody wanted to spend weeks on it.

With AI, the cleanup took an afternoon.

I started by defining the rules: consolidate overlapping categories, remove one-off tags with fewer than three posts, flatten the hierarchy to under ten top-level themes. Then I ran frequency analysis across all posts to map where the overlaps lived.

For the execution, I used Python scripts through LM Studio, which let me run local models without sending data to a cloud API. I toggled between DeepSeek (7B parameters) for speed on straightforward categorization and a larger GPT OSS 20B model for posts that required more reasoning about where they belonged. Every script ran in dry-run mode first. For any post where the model's confidence was below 85%, it generated a CSV for manual review instead of making the change automatically.

Out of one batch of 210 posts, 182 were categorized at high confidence. The final taxonomy landed at nine core themes. The 15-year backlog was gone in an afternoon.

This is what AI does for a PO: it collapses the timeline on work that was always worth doing but never worth the manual cost. The PO's job in this workflow wasn't typing or categorizing. It was defining the rules, reviewing the edge cases, and making the judgment calls the model couldn't. That's the same skill set a PO uses when refining a backlog or validating acceptance criteria. The tool changed but the judgment didn't.

Describing the "What," Not the "How"

One of the shifts AI enables for Product Owners is moving from writing specifications to describing outcomes in natural language and watching them materialize.

This is where Behavior-Driven Development becomes a different kind of tool. The "In order to / As a / I want to" format paired with Gherkin scenarios (Given / When / Then) has been around for years. What's new is that an LLM can take those scenarios and produce working code from them directly. The PO writes the intent and the model generates the implementation. The PO reviews whether the output matches the intent.

I call the moment it clicks the "30-Second Wow," which means showing something concrete quickly and let the visuals make the point. We can take that prototype to stakeholders first. When stakeholders can see the thing and start saying "can we move this here?" or "could it also do that?", they stop evaluating and start contributing. That is where validated intent comes from. Once intent is confirmed, the prototype hands the Scrum Team something grounded to build against and improve their productivity much better than a 40-page specification document.

The PO's value in this loop is making sure outcome drives the work, not output. That requires understanding the user's problem deeply enough to recognize when the solution misses, even if it looks correct on the surface. AI handles the translation from intent to code. The PO handles the validation that the intent was correct in the first place.

Getting a Second Opinion

One discipline I've adopted: never trust a single AI session for anything that matters.

When I use an AI to help build a plan, a product strategy, a sprint scope, a prioritization framework, I spin up a separate instance and give it an adversarial prompt. "Find the holes in this plan. What am I not seeing? What assumptions haven't been tested?"

The reason for a separate instance is that the first session has context bias. It helped build the plan, so it's predisposed to defend it. A fresh instance approaches the work cold, the way a skeptical colleague would. The PO's role here isn't to accept either model's opinion as final. It's to hold both outputs and make the judgment call.

Game You're Playing

The resistance to AI in product roles usually comes from the same place it comes from everywhere: fear that the tool replaces the person. But the PO who thinks AI replaces them is confusing the output with the outcome.

AI can generate a specification. It can categorize 600 blog posts. It can produce a working prototype from a Gherkin scenario. It cannot tell you whether any of those outputs solve the right problem for the right user.

That judgment is the PO's job. AI makes product owners faster at everything around it. The Product Owner of the future is not a "human typewriter" transcribing requirements. You are a "Badass Maker" (in the spirit of Kathy Sierra). Your goal is not to master the tool, but to use the tool to make your users and your team awesome.

I keep coming back to a question: what is the prize of the game you're playing? Are you playing to produce output, or are you playing to create impact? If it's impact, then AI is the best tool a PO has ever been handed. Not because it does the work, but because it clears the path to the work that only a human can do.

As for me: I do not play to win. I play to learn. If you focus on learning and impact, the win takes care of itself.

Learn More

If your product team is exploring how to build AI into their workflow, Improving's AI training programs are designed for exactly this—practical, role-based, and focused on adoption that sticks.

For any comments or suggestions on this article, find Claudio on LinkedIn.

Mexico, Memories, and my Improving 100 Experience

Improving — Thu, 09 Jul 2026 11:00:53 +0000

"Mexico in June with +1 😄 Congrats!"

That was the message my manager sent me at midnight back in February after an All-Hands Meeting concluded. I remember I had a late-night client call that day and was planning to catch up on the AHM the next morning.

Well, I didn't know that I had missed something really important in the AHM. Turns out I was selected as one of the winners for the Improving 100 award under community impact category - first one from Improving India. And the reward was an all-expense paid trip to Cancun - I had to open Google Maps to find where it was!

While finding a +1 was the most challenging task (spoiler, I went solo), the travel blogger in me was already planning the itinerary to Teotihuacan and Chichen Itza!

Improving100 - What is it?

Improving100 is Improving's employee recognition program where they select and reward best performing individuals for the year across 22+ offices around the globe in the following categories:

Community Impact
Business Growth
Delivery Excellence
Technical Innovation

One of the highlights of this program is that employees nominate their peers - so anyone can nominate anyone - and the decision comes from leadership, who does the due diligence. That makes the recognition even more special.

India to Mexico - My 18th Country

I've been fortunate that my work has provided me opportunities to travel across the world - USA, France, Netherlands, UK and Sri Lanka - to name a few. Mexico was never in the cards, so, I was super excited for this trip.

There are no direct flights from India to Mexico, so you either do a stopover in the US or Europe. I chose to go via US and did a stopover in New York - where I facilitated a round table on Platform Engineering at Kubernetes Community Days New York - before heading to Mexico City.

That's where I was joined by my other teammates from Improving India - Bhavin Gandhi, Deeksha Sharma and Shehbaz Pathan - who had also won the Improving100 award.

We all spent a few days in Mexico City exploring the pyramids at Teotihuacan, tasting some Tequila, eating some authentic Tacos and Enchiladas and soaking in the 2026 Fifa world cup fever before heading to Cancun for the retreat.

Improving100 Gathering - The Best One Can Ask For

We work remotely here at Improving India, which means we see most of our colleagues on our 15inch displays on a Teams call. So, this opportunity to meet people in person was exciting. For most of us from Improving India, it was our first time in the country, and also the first-time meeting Improvers from across the globe.

We were put up in one the fanciest resorts that I had ever been to. A sea facing room with plush amenities, anyone would want to work hard to earn that luxury. There was a dedicated check-in only for Improvers as it was a massive group coming in. We were handed over our goody bags that included snacks, electrolytes, sunscreen and a cap - everything that one would need for Cancun.

Later in the evening, we had the welcome reception hosted by our US Managing Director Scott McMichael and CEO Curtis Hite. I was fortunate to have met Scott earlier in Atlanta, and he has an infectious energy! I also got to meet so many people during the welcome reception, some of them who I was meeting for the first time and a few who I had already interacted with virtually.

Excursions - Mexican Cooking Experience

One of the best things about the Improving100 outing was the excursion activities that each Improver had to choose from. These included deep sea fishing, sailing, golfing and Mexican cooking to name a few. For those of you who don't know, I love cooking and the moment I got the list to choose my activity, I opted for Mexican Cooking.

We were about 20 of us for this activity, some of them familiar faces while some new who I interacted with during the day. The activity started by Margarita making and me being a teetotaller, made a non-alcoholic one. That was followed by group salsa making activity which was super fun! The ingredients and utensils used were very Indian, so I'm going to make one at home soon! I also learned to make tortillas, which is very similar to the Indian Roti, but with a different dough.

Later, we hopped on a bus to go on a taco tour. We headed to one of the popular taco places in downtown. We started with a Cantarito making session followed by taco tasting. Being a vegetarian, I was served a cactus taco, and these were delicious! It was my first time have cactus, and it felt so much like Ivy Gourd, Tindora or Dondakaya that we have back in India. We completed the tour by playing a game of Loteria, which is the Mexican variant of Bingo.

Award Night - Sky Full of Stars

It was the last day and the most important one, as it was the award night. We had Improvers coming in their best outfits for the ultimate night of the outing. The stage was set in the beautiful resort with A Sky Full of Stars - I saw the setup and had this song playing on loop in my head!

Scott started the proceedings, followed by the recognitions. It was great to see the energy and genuine appreciation by everyone present there. We had group and individual photos taken post the award session, after which the dance floor was open.

The DJ started with some English music, and we were just stepping around the dance floor. And just then we heard a familiar beat, and our DJ decided to play a popular Hindi song. That's when us India Improvers all got in the groove and taught some Bollywood moves to other Improvers, some of whom were already fans of Shah Rukh and Aamir Khan!

It was surely one of the best evenings of the event.

More Than Just a Trip

Looking back, this Mexico experience wasn't just about the stunning landscapes, vibrant culture, or unforgettable food. It was about what it represented. It stood as a reminder that meaningful recognition goes beyond certificates or awards; it creates memories, builds connections, and leaves a lasting impact. A huge shout out to Emily Rosselle and the larger team for planning such a beautiful event!

Without a doubt, this has been one of the best employee recognition programs I've ever experienced.

From exploring new places to stepping outside my everyday routine, this trip gave me a chance to reflect, recharge, and truly appreciate the journey - both personal and professional - that led here.

If you'd like to experience Mexico through my lens, I've shared highlights and moments from the trip on my Instagram. Do check out the highlights - I've tried to capture not just the places, but the energy, the people, and the stories that made this trip unforgettable.

If you're interested in following what I'm working on next - across tech, travel, and everything in between - let's connect on LinkedIn. I'd love to stay in touch and share more of the journey.

Here's to more such experiences, more stories, and more reasons to celebrate the work we do. If you like this, join our team, as we are always looking for talented, fun folks to join us.

Originally published on Improving.com

Your Prompt Is a Cross-Border Data Transfer

Improving — Thu, 09 Jul 2026 10:59:13 +0000

Every time you hit Enter on an AI chatbot, you are signing off on a data transfer with your prompt. Most probably, you have no idea where that data went, which tools it touched, or whether it crossed a border on its way to receive the response.

In this blog post, I will walk you through this observability problem and solution in details, including why AI sovereignty is suddenly an architecture concern, why on-prem alone does not solve it, what OpenTelemetry's GenAI semantic conventions actually give you today, and how an "AI Receipt" demo I built turns the black box into something you can audit.

GenAI Stack: Four Capabilities, Growing Data Footprint

GenAI is not a chatbot anymore. What we deploy in production today is a stack of four capabilities, and each one widens the data footprint:

Chatbot: The model reasons for your prompt and answers from its weights alone.
RAG: The model retrieves context from your documents first, then answers grounded in what it found.
MCP: The model uses external tools - APIs, search engines, and databases, to do real work in external systems.
Agents: The model plans which of the above to invoke, chains them together, and acts without a human in the loop.

By the time you are running an agent, a single user prompt can fan out into a dozen calls across vector stores, internal APIs, and third-party services. That fan-out is where the sovereignty question gets uncomfortable.

The Data is Already Moving. Quietly.

Data movement without visibility is already happening in the data sensitive places: government services, banking chatbots, telecom log analysis, healthcare claims processing, and the customer support bots you talked to yesterday.

One user request fans out into multiple data transfers, and most of them leave the home region without the user (or often the operator) noticing.

According to LayerX's 2025 Enterprise AI and SaaS Data Security Report, based on real browser telemetry from enterprises:

45% of enterprise users are actively using AI tools.
77% of employees paste data into GenAI tools.
40% of files uploaded into GenAI tools contain PII or PCI data.

The report calls GenAI the single largest channel for corporate-to-personal data exfiltration, accounting for 32% of all such movement. 67% of that AI usage happens through unmanaged personal accounts, which means most enterprises cannot see it even when it is happening on their own networks.

This is the unmanaged surface. The managed surface, where you are deliberately building agents for production, is bigger and more invisible, because the data movement happens inside your own stack instead of on someone's laptop.

For companies shipping in AI regulated industries, the residency and visibility of the data become a central architecture question.

Pressing Enter feels local, but it is not. Your prompt may hit a model hosted in another region, get enriched by an MCP tool calling a SaaS API in a third region, and return through a logging pipeline in a fourth. Each of those hops is, technically and legally, a cross-border data transfer.

Why Sovereignty, Why Now?

Three things have collided in the last eighteen months: public trust in AI is shaky, regulators are no longer waiting, and enterprises are realizing that their AI roadmaps are tied to providers they cannot fully audit.

On the trust side, the 2025 KPMG and University of Melbourne global study found that:

54% of respondents are wary about trusting AI.
70% believe regulation is necessary.
Only 43% believe current laws are adequate.
76% expect international laws to govern AI.

The mandate for visible, accountable AI is not coming from a niche advocacy corner; it is the default public position.

On the regulatory side, the rules are arriving faster than most architecture reviews can absorb.

The EU AI Act's high-risk obligations begin to apply in August 2026.
The European Commission tabled the Cloud and AI Development Act on 3 June 2026.
In India, the DPDP Act and the DPDP Rules 2025 were notified in November 2025 and are now rolling out in phases.
China continues to tighten data export controls, and the US is moving in a different direction with deregulation pushes.

The net effect for any multinational is that there is no longer one global AI policy; there are several, and they are not aligned.

Real-World Cases

A handful of concrete cases make the pattern visible:

HPCL (India, oil and gas) moved AI workloads on-premise. The headline reason is security around National Critical Information Infrastructure, but the two reasons that surprised me were cost predictability (per-token cloud pricing made budgeting impossible at scale) and avoiding lock-in to any single hyperscaler's pricing curve.
NABARD (India, agricultural finance) went on-prem for what their CGM called "strategic independence," and to comply with the financial data localization requirements of the DPDP Act. They did not want their agri-finance data sitting in someone else's region, and they did not want their AI roadmap held hostage to a foreign provider's API decisions.
The Bundeswehr (Germany, defence) declined to use Palantir for its military cloud and AI project and is actively examining European alternatives - a decision driven less by capability than by the desire for non-US-controlled tooling for sensitive workloads.
EU public-sector procurement is being restructured around the four-level sovereignty framework, which means any vendor selling into critical European workloads will need an answer to "where exactly does the data go" that holds up to audit.

The Linux Foundation's 2025 Sovereign AI research report and Hugging Face's writing on sovereignty and open source reach the same conclusion from different angles: control over models, data, and infrastructure is now a first-class architectural concern, not a procurement footnote.

Four Layers of Sovereignty

There are four layers of AI sovereignty, and when people say, "sovereign AI", they usually mean one of these four things. Treating them as one bucket is what gets architectures into trouble.

The Four Layers

Data sovereignty: Does the data stay inside our borders? This is the question your local data protection law - whether that is GDPR, CCPA, LGPD, PIPL, or DPDP - is designed to answer.
Tech sovereignty: Do we own the IP, or are we renting it? If the vendor changes terms, can we keep operating?
Operational sovereignty: Can a foreign entity switch our tech off? If a major hosted-AI provider has a regional outage tomorrow, does your critical service degrade with it?
AI sovereignty: Does the model understand our context, our languages, our values? A model trained predominantly on one language and culture will not serve a government or enterprise operating in a different linguistic and regulatory context well, no matter where the weights are stored.

You can get data sovereignty right and still fail on operational sovereignty. You can host your own model and still leak data through a tool call. These layers need to be reasoned about separately.

Build vs Buy: The Practical Path

Once you take care of four layers of sovereignty, the next step could be to build everything from scratch: your own model, your own infrastructure, your own toolchain. It gives you complete control, but it also stalls every other engineering priority while you reinvent commodity infrastructure that hosted providers have already polished for years.

The opposite reaction is what most teams actually do. They pick the fastest hosted API, ship the feature, and treat sovereignty as a problem for next quarter. Velocity stays high, but it leaves you with no real answer when the auditor or security team asks where the data actually went on each request.

The practical path is to make different choices at different layers of the stack.

Bringing it In-house: BYO-GPU

The Bring-Your-Own-GPU pattern has matured over the last year for the layers where you want maximum control. Deploy the model on your own hardware, your own Kubernetes cluster, and route prompts it the same way you would route any other internal service.

The toolchain for this is now genuinely good:

Kubernetes for orchestration
vLLM for high-throughput inference serving
NVIDIA NIM for packaged, production-ready model microservices
NVIDIA Dynamo for multi-node, disaggregated inference at scale

The operating rule is simple: prompts and training data never leave the data center.

The Infrastructure Gap: We Have the Infrastructure. We Lack the Trace.

You can host your model locally, lock down your VPC, and still leak data on every request. Why? Because the moment you put an agent in front of that model, the agent starts calling MCP tools including a web search tool, weather API, third-party CRM, and SaaS knowledge base. Each of those calls can, and often does, leave your region.

The local LLM is not the leak. The agent's tool calls are. And without instrumentation at the request level, you have no idea it is happening.

What OpenTelemetry's GenAI Semantic Conventions Actually Give You

OpenTelemetry GenAI special interest group has been filling in the gaps for AI workloads specifically. As of writing, the umbrella Semantic Conventions release is at v1.41.0, but the GenAI conventions inside it is still marked Development status, with v1.36.0 acting as the stability baseline for existing instrumentations. That said, even in its current form it gives you enough vocabulary to instrument a real agent pipeline.

Five Signal Categories

The spec covers five signal categories for GenAI observability:

Model spans: A span for every call to an LLM. Operation name (chat, text_completion, embeddings, generate_content), model name, provider, token usage, parameters.
Agent spans: A span for the agent layer itself. Distinguishes create_agent and invoke_agent from a raw chat completion, so an orchestrator's reasoning step is visible separately from the underlying LLM call.
Events: Per-step lifecycle records captured as span events, useful when you want fine-grained timeline data without bloating span attributes.
Metrics: Token usage histograms, request duration, time to first token, all as proper OTel metrics you can chart in Prometheus or Grafana.
Exceptions: Standard semantic conventions for capturing GenAI errors (timeouts, model errors, tool errors) with a consistent error type vocabulary.

There are vendor-specific conventions on top of this for Anthropic, OpenAI, AWS Bedrock, and Azure AI Inference, so instrumentation for those providers can layer system-specific detail (request IDs, finish reasons, system fingerprints) without diverging from the core spec.

Attributes That Matter

In practice, a handful of attributes do most of the work. The ones I instrumented in the demo below, all directly from the spec, are:

gen_ai.request.model - The model name
gen_ai.request.model.id - Model version or full identifier
gen_ai.request.top_k, gen_ai.request.top_p - Sampling parameters
gen_ai.response.model.id - Response model identifier
gen_ai.usage.input_tokens - Tokens consumed
gen_ai.usage.output_tokens - Tokens generated
gen_ai.operation.name - chat, text_completion, embeddings, etc.

Tracing Tool Calls: MCP Semantic Conventions

The OpenTelemetry project has a dedicated semantic conventions spec for Model Context Protocol as a sub-area of the GenAI conventions, and this is the part that is most directly useful for the cross-border data transfer problem.

The MCP conventions define a Client span and a Server span for every MCP request. Both carry the same core attributes:

mcp.method.name - The JSON-RPC method (tools/call, initialize, prompts/get, resources/read, and so on)
mcp.session.id - So you can group every call from one MCP session together
mcp.protocol.version
gen_ai.tool.name and gen_ai.operation.name = execute_tool when the call is a tool invocation
mcp.resource.uri when a resource is being read

The detail that most teams miss is context propagation. MCP runs on top of JSON-RPC, and JSON-RPC has no native trace context mechanism. The OTel spec recommends injecting traceparent and tracestate (and baggage, if you use it) into the MCP request's params._meta field, so the receiving server can pick up the parent context and continue the trace, following the W3C Trace Context standard.

What You Still Have to Add Yourself

OpenTelemetry does not currently define a "where did this data go geographically" attribute set. That is the gap we're most interested in. The standard resource attributes (cloud.region, cloud.provider, service.name) get you part of the way for the services you own, but they say nothing about whether a span represents a cross-border movement.

For the demo below, I added a small custom namespace on top of the standard GenAI attributes:

data.sovereignty.source_region        # US-EAST
data.sovereignty.destination_region   # IN-MUMBAI
data.sovereignty.cross_border         # true
data.sovereignty.alert                # "Data crosses border: US-EAST -> IN-MUMBAI"
data.sovereignty.home_region          # US-EAST (resource attribute)
data.classification                   # internal | pii | confidential

Five attributes, derived from comparing the prefix of source and destination regions. Nothing fancy. But together with the standard gen_ai.* and mcp.* attributes, it is enough to flag every span where data left the home region, and to produce a per-request compliance verdict. This is the kind of layer that belongs to a community-driven standard eventually.

"AI Receipt" Demo

I built a working demo that puts all of the above together. The point of the project is to treat AI traces the way payment systems treat transactions. Every prompt gets a receipt that tells you exactly what happened on the data's journey.

Architecture

The demo runs as a small set of Docker Compose services, all instrumented with OTel:

You → AI Gateway (8000) → Agent Service (8004) → MCP Server (8002) ─┐
                       └→ LLM Router (8003) → Ollama (local) / Gemini (cloud)
                       └→ RAG Service (8001)                          │
All services → OTLP → OTel Collector → Jaeger ←──── AI Gateway parses
                                                     → AI Receipt JSON

The AI Gateway does intent detection and orchestration. The Agent Service is the multi-agent variant: it spawns three sub-agents (Policy Advisor, Document Analyst, Action Processor) that run in parallel and each call different MCP tools. The MCP server hosts nine functional tools (policy lookup, currency conversion, document search, PII scanning, GDPR compliance check, timezone conversion, port checking, log analysis, metrics calculation). Each tool is tagged with a deliberate region: most are US-EAST, but pii_scanner and compliance_checker sit in EU-FRANKFURT, and timezone_converter and port_checker sit in US regions, so the demo can actually demonstrate cross-border flows.

The LLM Router toggles between Ollama (local, US-EAST, runs on your host) and Gemini (cloud, US-EAST-1). Traces flow through a OpenTelemetry Collector configured with the standard OTLP gRPC receiver and a Jaeger exporter, plus a small attributes processor that stamps a demo version onto every span.

What Happens When You Send a Prompt

Each request walks through the same sequence:

You submit a prompt through the frontend chat panel.
AI Gateway receives the request, opens a root OTel span tagged with the home region (US-EAST-1), and runs a quick intent-detection step to decide whether the prompt needs the multi-agent path, the RAG path, or a direct LLM call.
Agent Service plans the work, spawns its sub-agents in parallel, and each sub-agent issues with MCP tool calls. Every MCP call gets its own child span with gen_ai.operation.name = execute_tool, gen_ai.tool.name, and sovereignty annotations describing the tool's region.
LLM Router picks the model. If the Local LLM toggle is on, the call goes to Ollama on the host (US-EAST-1). If it is off, the call goes to Gemini (for e.g, europe-west1), and the span is tagged data.sovereignty.cross_border = true with the destination region.
All spans are exported via OTLP to the OTel Collector, which adds the demo version attribute and forwards them to Jaeger.
AI Gateway closes the loop. Once the trace flushes, it queries Jaeger's HTTP API for the trace ID, walks the spans, and filters to the ones carrying sovereignty, GenAI, or MCP attributes. From those spans it computes the receipt.
Receipt is returned to the frontend, where the chat panel renders the natural-language answer and the receipt tab renders the structured compliance report.

What the Receipt Actually Shows

When you send a prompt with tracing enabled, the AI Gateway waits for the trace to flush, queries Jaeger's HTTP API for the trace ID, and walks the spans. It filters to spans that carry sovereignty annotations, GenAI attributes, or MCP attributes (so it ignores generic auto-instrumentation noise), then builds a structured report:

Total steps: Every meaningful span in the trace.
In-region count: How many stayed in US_EAST.
Cross-border count: How many had data.sovereignty.cross_border = true, with the destination region.
PII detected: Whether the prompt itself contained PII patterns (Aadhaar, PAN, email, phone, SSN).
Compliance verdict: FULL, PARTIAL, or NON-COMPLIANT against DPDPA, GDPR, and data localization rules.

Running It

Prerequisites:

Docker
Docker Compose
Ollama (runs on your host, not inside Docker)

Start it up by running the following commands:

ollama pull qwen2.5:1.5b
docker compose up --build

You can access the frontend at http://localhost:3000, Jaeger UI at http://localhost:16686.

The frontend has a chat panel, a Data Flow visualization that animates the agent's tool calls, and a receipt tab that decodes the trace into the sovereignty report. Flip DEMO_MODE=false in .env and add a GEMINI_API_KEY to see real cloud calls.

The demo repository is here: AI Sovereignty Demo

Beyond the Trace: Confidential Computing

OpenTelemetry tells you where the data went. It does not protect the data while it is in flight or in memory at the destination. For workloads where even a verified destination is not enough (the agent runs inside a trusted region but on a multi-tenant GPU node, for example), confidential computing is the layer that comes next. The Confidential Computing Consortium has a recent piece on protecting agentic AI workloads with confidential computing that pairs well with the observability story here. Trace what moves; encrypt what is in use.

Where This is Heading

AI Receipt is one proof-of-concept of a broader pattern. The pattern itself is what matters more than the specific implementation: sovereignty observability is becoming a first-class layer of the AI stack, alongside model serving, retrieval, and agent orchestration.

A few things look likely from here:

Cross-border attributes will become standard: OpenTelemetry already has resource attributes for cloud regions; what is missing is a community-driven attribute set that says, per span, whether the operation crossed a regulatory boundary and which one. That work will probably happen inside the OTel GenAI SIG over the next year, and the sooner enterprises start emitting their own attributes (whatever you call them), the easier the migration will be when the standard lands.
Receipts will move from afterthought to contract: Today, most AI systems generate a response and call it done. Regulated industries will start expecting a verifiable artefact per request: span-level, signed, and inspectable by the auditor without needing access to the underlying system. Payment networks did this thirty years ago; AI invocations are heading the same way.
Trace what moves, encrypt what is in use: Observability tells you where the data went. Confidential computing protects the data while it is being processed at the destination. The two layers are complements, not alternatives. Expect them to converge into a single sovereignty-and-attestation story over the next couple of years.
Audit layer will be open by default: No regulated enterprise wants its compliance posture to depend on a single vendor's proprietary dashboard. Open standards (OpenTelemetry), open instrumentation, and open audit logic are how this stays workable across jurisdictions and providers.

You do not have to choose between shipping AI and being sovereign. You do have to be willing to instrument what you ship, so that when the regulator, the auditor, or your own security team asks where the data went, you can answer with a receipt.

Don't just trust your AI. Trace it.

Keep the Conversation Going

If you found this useful, the demo repo is open, contributions and issues are welcome. You can connect with me on LinkedIn to discuss sovereign AI, GenAI observability, or where any of this falls apart in your environment.

Data sovereignty is a critical consideration as organizations begin integrating AI into their operations. Decisions around where data resides, how it is processed, and who has access to it can have significant implications for security, compliance, and governance. Our AI and data experts work with organizations navigating these challenges every day, helping them evaluate architectures and deployment models that align with their requirements.

References

Originally published at Improving

Stop Treating Legacy Systems as the Enemy

Improving — Thu, 09 Jul 2026 10:55:39 +0000

Many legacy software are built using "rapid application development" (RAD) tools like FoxPro or Microsoft Access. At that time, the senior engineers around, the ones who had spent decades with punch cards and mainframe batch jobs, looked at what FoxPro users were doing and called it cheating. FoxPro was too easy and high-level. It abstracted away the real work. A tool like that couldn't produce anything serious.

They were wrong as those FoxPro applications were able to run businesses and created the bedrock of successful enterprises. Some of them are still running today, decades later. The people who dismissed the tools could never dismiss the results.

I've watched this pattern repeat through every generation of technology I've worked in. Visual Basic was too simple for C++ developers. .NET was too abstracted for the Visual Basic crowd. And now, AI-assisted development gets the same skepticism from the previous generation. Every era's tools get treated as cheating by the people who built the last era.

The pattern isn't just about tools, though. It shows up in how we talk about the systems those tools produced. We call them "legacy" and mean it as an insult. That's a mistake.

What "Legacy" Actually Means

A system that has been running for fifteen years didn't survive by accident. It survived because it solved a real problem for real people, and it kept solving that problem through market shifts, leadership changes, and at least two or three technology cycles that were supposed to replace it. That's not a failure. That's evidence of success.

The problem isn't that legacy systems exist. The problem is that we approach them with contempt instead of curiosity. When the default attitude is "Rip and Replace," we lose something. Buried in that old code is business logic that nobody documented, edge cases that took years to discover, and decisions that were made for reasons the current team may not even know about.

Treating a legacy system like an enemy means treating your organization's history like a liability. I'd rather treat it like a foundation.

Refueling the Plane While It's Still in the Air

The urge to do a total rewrite is one of the most dangerous instincts in enterprise software. I have seen too many systems fail because they were 'Ground-and-Rewritten' instead of evolved. A total rewrite introduces massive risk and ignores the nuanced business rules embedded in the existing code.

I compare modernization to refueling a plane in mid-flight. The business cannot stop generating revenue because the technology underneath needs an upgrade. You can't land the plane to refuel the engines. You have to do it while it's still moving, and that demands a different discipline than a greenfield build.

The discipline is evolutionary maintenance: systematic, incremental, and protected by automated tests.

A Real-World Example

A recent example: migrating from Fluent Assertions to Shouldly across a large .NET codebase including over 3,000 tests. Deprecated APIs scattered through years of accumulated work. The process started with Context7, feeding the AI current library documentation that post-dates its training data. Without that step, the model works from stale knowledge and confidently introduces errors. With it, the AI reasons about actual API changes rather than guessing at them.

What surfaced next was something a purely manual process would likely have missed: Lamar and Marten shared a dependency. Updating one without the other would have broken the build in ways that wouldn't show up immediately. The AI found the link during the initial dependency scan.

Rather than forcing a full rewrite of existing test code, the AI generated custom extension methods to bridge the gap, preserving the developer experience on top while changing the underlying library beneath it. What would have been a multi-week manual effort compressed into two days with AI-assisted workflows.

The safety net is the non-negotiable part. Without automated tests, evolutionary maintenance is just a polite name for changing things and hoping nothing breaks. With them, you can refuel the plane with confidence, one engine at a time, the passengers never noticing.

The teams that get this right don't frame modernization as a one-time event. They treat it as ongoing technical hygiene, closer to changing the oil regularly than waiting for the engine to seize.

What You Find When You Look With Respect

Something happens when you approach a legacy codebase as archaeology rather than demolition. You find things! I've lost count of how many modernization projects have surfaced buried business logic that was still correct, still needed, and completely undocumented.

For examples:

An edge case handler that prevents a specific error nobody remembers anymore
A validation rule that catches a regulatory requirement the compliance team forgot was encoded in software
A workflow that routes exceptions to the right person because someone five years ago spent three weeks figuring out who that person should be

These are what I call 'little gems' in the overlooked. When you modernize with contempt, you demolish them along with everything else. When you modernize with respect, you discover them, and you carry them forward into the new system deliberately, rather than rediscovering the need for them six months after launch through a production incident.

This is the part of modernization that rarely makes the project plan. The plan says "migrate module X." It doesn't say "discover why module X has forty-seven edge cases that nobody can explain." But that discovery is where real value lives.

The teams who do it well are the ones who treat the legacy code as an artifact that encodes organizational knowledge, not just syntax waiting to be replaced.

A Different Starting Question

Most modernization conversations start with "How do we get rid of this old system?"

That's already the wrong question because it assumes the system is the problem. Majority of the time, the system isn't the problem, the system is the accumulated answer to problems the organization has faced over the years. The real question is: what has this system been holding together, and how do we keep that intact while we evolve everything around it?

The organizations that modernize well don't treat their legacy systems as enemies to defeat. They treat them as foundations to build on. The code changes. The infrastructure changes. But the business logic, the hard-won edge cases, and the organizational knowledge encoded in software carries forward.

If your team is facing a legacy modernization challenge, the first step isn't picking a new tech stack. It's understanding what you already have, and what it's been doing for you all along.

For more discussion, connect with Claudio on LinkedIn

Closing the AuthZ Gap in MCP: Policy-Driven Tool Invocation Control

Improving — Thu, 09 Jul 2026 10:54:26 +0000

Model Context Protocol (MCP) tools give AI agents direct access to production databases, internal APIs, and third-party platforms. But most teams deploying MCP today have no answer to a simple question: who authorized that tool call?

MCP Changed How Agents Talk to the World, But Not Who Controls It

With over 17,000 MCP servers available to use, according to Zuplo's report, momentum around MCP is quite real, and it is becoming the de-facto standard for how AI agents connect to external services. It functions like a USB cable for AI to eliminate the need for developers to write custom integrations for every single AI application.

The same report shows that 50% of professionals find MCP security or access-control a big challenge. MCP queries AI models with the help of agents for tool discovery, invocation and response handling. It lacks a native mechanism to control who invokes which tool, with what parameters and in whose context. Authorization at the invocation layer is still a problem.

What the Auth Model Covers Today and What it Doesn't

MCP's current authorization model covers connectivity and authentication. Servers can require OAuth 2.0 tokens, validate that an agent has valid identity and restrict which clients can connect. This becomes very useful since one cannot impersonate an agent and connect to your MCP server.

What it does not cover is authorization at the tool invocation layer. Once an agent is authenticated to the MCP server, there is no native spec primitive that controls which tools it can call, what parameters it can pass.

What happens at invocation reflects this gap directly. The agent resolves the tool from the server and makes a call with parameters. The MCP server executes downstream resources like a database, an internal API, etc.

The runtime does not ask: is this agent allowed to call this tool? Are these parameters acceptable? Does the tenant context and agent context match?

Where Security Breaks Down

In single tenant environments, teams rely on application-level checks or custom middleware that inspects calls before they reach the server. These approaches break down as the tool surface grows - checks drift out of sync with tool updates and need to be maintained per tool, per team. Neither scales across a shared platform serving multiple agent teams.

In multi-tenant environments, the stakes are even higher. The threat surface looks like this:

1. Cross-tenant tool calls

Agent A running in Tenant A's context invokes a tool that operates on Tenant B's data. There is no enforcement at the invocation layer to catch it before the call is executed.

2. Phantom agent identity

Your audit trail shows a service account, but the human who triggered the agent chain - the person who typed the prompt that started everything - is invisible by the time the tool call lands in your logs.

3. Unconstrained parameters

A tool calling a production database accepts query parameters. Nothing gets validated at the invocation layer.

4. Privilege escalation chains

Agent A calls Agent B, which invokes a tool. Each hop loses context and permission. By the time the tool is called, nobody can trace whose authority it is actually acting under.

What makes these more interesting is that the agent is not doing anything unauthorized in a traditional sense. It has permission and it uses it. The platform has no vocabulary for the fine-grained question: should this specific agent call this specific tool with these specific parameters right now or not?

Why Kubernetes RBAC Has No Answer Here

Most platform teams point to Kubernetes RBAC as their solution; it's already there and mature enough as well. But RBAC has a hard limit. It only controls access to Kubernetes API resources. You can control whether a "service account can list pods in this namespace or not". But you can't control "this agent can call the create_*record tool and not the delete_*record tool".

RBAC operates on resource-verb pairs at the Kubernetes API surface whereas MCP tool calls live at the application layer. They carry semantic context: which tool, which agent, which tenant, which parameters, what time. There is no medium between the two without something that understands the nature of tool invocation and can make policy-based decisions against runtime. It can be achieved by policy-as-code.

Policy-as-Code as the Enforcement Layer

The solution is to model your MCP tool invocations as Kubernetes custom resources and put a policy engine in the admission path. Enforce rules on every invocation before it is executed. The tool call becomes a first-class object that the platform can inspect, validate, mutate, and audit.

Kyverno is the strong option for this as it already sits in the Kubernetes admission path. It can inspect any arbitrary JSON structure in any custom resource. It supports both validating webhooks that can accept or block a call and mutating webhooks that can annotate a call before it proceeds.

This enforcement model has three layers, and each one solves a different problem related to your platform.

Three Patterns that Close the Gap

Pattern 1: Tool allowlisting per agent identity

A validating Policy that restricts which tools a given agent Service Account can invoke. It is denied by default and allows explicitly per agent-tool pair. If the agent's identity isn't in the allowlist for that tool, the call is blocked at admission before it ever reaches the MCP server.

spec:
  validationActions:
    - Deny
  matchConstraints:
    resourceRules:
      - apiGroups:   ["mcp.security.io"]
        apiVersions: ["v1alpha1"]
        operations:  ["CREATE"]
        resources:   ["mcptoolinvocations"]
  validations:
    - expression: >-
        has(object.spec.agentId) && object.spec.agentId != ""
      message: >-
        MCPToolInvocation must include a non-empty agentId in spec

    - expression: >-
        !(object.spec.agentId == "sre-agent") ||
        object.spec.toolName in [
          "pods_list",
          "pods_list_in_namespace",
          "pods_get",
          "pods_log",
          "pods_top",
          "events_list",
          "namespaces_list",
          "nodes_top",
          "configuration_view"
        ]
      messageExpression: >-
        "sre-agent is not permitted to invoke '" + object.spec.toolName + "'. " +
        "Allowed tools: pods_list, pods_list_in_namespace, pods_get, pods_log, " +
        "pods_top, events_list, namespaces_list, nodes_top, configuration_view"

Pattern 2: Multi-tenant isolation

A validating Policy that checks whether the tenant context in the invocation matches the agent's namespace or annotation. If they don't match, the call is denied. Cross-tenant invocations are blocked structurally, not by trusting the agent to behave correctly.

spec:
  validationActions:
    - Deny
  matchConstraints:
    resourceRules:
      - apiGroups:   ["mcp.security.io"]
        apiVersions: ["v1alpha1"]
        operations:  ["CREATE"]
        resources:   ["mcptoolinvocations"]
  validations:
    # Require tenantId to be present and non-empty
    - expression: >-
        has(object.spec.tenantId) && object.spec.tenantId != ""
      message: >-
        MCPToolInvocation must include a non-empty tenantId in spec.
        Invocations without tenant context are rejected.

Pattern 3: Human identity injection

A mutating Policy that annotates every tool invocation with the identity of the human who triggered the agent chain, the timestamp, and the agent chain ID. The audit trail always has a person attached to it, not just a service account.

spec:
  matchConstraints:
    resourceRules:
      - apiGroups:   ["mcp.security.io"]
        apiVersions: ["v1alpha1"]
        operations:  ["CREATE"]
        resources:   ["mcptoolinvocations"]
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >-
          Object{
            metadata: Object.metadata{
              annotations: {
                "mcp.security.io/triggered-by":
                  has(object.spec.triggeredBy) && object.spec.triggeredBy != ""
                    ? object.spec.triggeredBy
                    : request.userInfo.username,
                "mcp.security.io/triggered-at":
                  string(now()),
                "mcp.security.io/agent-id":
                  has(object.spec.agentId) ? object.spec.agentId : request.userInfo.username,
                "mcp.security.io/tenant-id":
                  has(object.spec.tenantId) ? object.spec.tenantId : object.metadata.namespace,
                "mcp.security.io/policy-version": "v1"
              }
            }
          }

Seeing the Policies in Action

The patterns described above are not theoretical. Here is what enforcement looks like against a real MCPToolInvocation custom resource, with kubernetes-mcp-server running in a Kubernetes cluster and Kyverno evaluating admission.

Before: No Mutating Policy

A remediation agent scales a deployment. The CR lands in etcd with no audit context.

# kubectl get mcptoolinvocation proxy-resources-scale-3a7a61 -n tenant-acme -o yaml
metadata:
  name: proxy-resources-scale-3a7a61
  namespace: tenant-acme
  # no annotations
spec:
  agentId: remediation-agent
  toolName: resources_scale
  triggeredBy: bob@acme.com   # agent-supplied - can be forged or omitted
  tenantId: tenant-acme

triggeredBy lives only in spec - it is what the agent chooses to send. Nothing confirms when this happened or which policy evaluated it.

After: mcp-inject-human-identity Mutating Policy Applied

Same call, same CR structure. Kyverno writes five annotations at admission time before the object reaches etcd.

metadata:
  name: proxy-resources-scale-9f2c84
  namespace: tenant-acme
  annotations:
    mcp.security.io/triggered-by:   "bob@acme.com"          # from spec.triggeredBy
    mcp.security.io/triggered-at:   "2026-06-10T14:32:01Z"   # Kyverno did it
    mcp.security.io/agent-id:       "remediation-agent"
    mcp.security.io/tenant-id:      "tenant-acme"
    mcp.security.io/policy-version: "v1"
spec:
  agentId: remediation-agent
  toolName: resources_scale
  triggeredBy: bob@acme.com
  tenantId: tenant-acme

These annotations are written by Kyverno - neither by the agent nor by proxy. The agent cannot set or alter them after admission. Even if the CR is later deleted, the Kubernetes audit log retains the creation record with the full annotation set.

How to Get Started

The right approach is to build incrementally, instrument first, enforce second, and tighten over time. Below is the practical sequence that works without breaking production agents.

1. Audit what your agents are actually calling

Before writing a single policy, spend a week in observation. Enable Kyverno in Audit mode and let it log every tool invocation. You will find tools being called that nobody knew about, parameters that nobody intended to allow, and service accounts with scope far wider than their function requires.

2. Model your MCPToolInvocation CRD deliberately

The shape of your custom resource determines what Kyverno can inspect. Put toolName, tenantId, and agentId in spec, not in labels or annotations. Policy JMESPath expressions are cleaner and easier to read in audit events when the fields live in spec.

3. Start with human identity propagation, not blocking

The mutating webhook that injects the triggering human identity has lower risk than a validating webhook that can deny invocations. Deploy the mutation pattern first. Get your audit trail working and verified. Compliance teams will often sign off on shared MCP infrastructure once they can answer "who triggered this" - give them that before you move to enforcement.

4. Flip one policy to Enforce at a time

Use the audit log data from step 1 to build your allowlists. Flip validationFailureAction to Enforce for one agent, one tool category at a time. Agent chains fail in surprising ways when a mid-chain call is denied - give yourself at least two weeks of audit coverage per policy before enforcing.

5. Propagate human context at the orchestration layer, not as an afterthought

The agent framework including LangGraph and AutoGen must stamp the triggering human identity onto the invocation before it reaches the admission webhook. This is an architectural decision that cannot be retrofitted cleanly. Make it a requirement before any agent gets write access to production tools.

Wrap Up

The AuthZ gap in MCP is a gap in the adoption curve. The spec is moving, the ecosystem is maturing, and native authorization primitives will come. But production agents are running right now, and the enforcement layer has to exist today.

In this blog post, we have described how MCP changed the way agents talk to the world, but not who controls it. The MCP's current authorization model ensures connectivity and authentication but not the authorization at the tool invocation layer.

Policy-as-code gives platform teams a concrete, uniform, auditable answer to the question: who authorized that tool call?

Kyverno's Policy model is one strong path to that answer, as it is forkable, framework-agnostic, and compliance-ready out of the box. However, using it in practice in production comes with its own set of challenges, which the Improving team is experienced in solving.

If you would like to discuss this article and have any suggestions, you can connect with Oshi Gupta and Sonali Srivastava on LinkedIn.

Originally published on Improving

Data Debt Is the Real Reason Your AI Predictions Don't Improve

Improving — Thu, 09 Jul 2026 10:49:49 +0000

The recommendation system was working perfectly six months ago. Now engagement is dropping, predictions feel inconsistent, and nobody can explain why. Accuracy improves slightly in staging once the ML team retrains the model, but production still looks unstable. Another experiment starts with larger model, better Embeddings, and more tuning and GPUs.

A week later, someone discovers the actual issue.

One upstream service changed how "active users" were calculated. A feature pipeline continued serving stale values for nearly a month. Half the training data was generated using the old definition, while inference was using the new one. The pipelines stayed green as nothing technically failed, and dashboards kept updating. The model simply learned from a distorted version of reality.

Many AI systems fail in production through slow reliability decay that hides inside data pipelines, feature stores, vector databases, APIs, and transformation layers. The invisible decay creates a false sense of optimization. Teams continue investing in experimentation while the actual bottleneck exists upstream in the data layer. Three patterns usually indicate this problem:

The model looks accurate during testing, but users still complain that predictions feel wrong in production.
Teams keep retraining the model, yet the same issues keep returning after every deployment.
Different dashboards show different numbers for the same metric because each system defines the data differently.

What Is Data Debt?

Before understanding data debt, it is important to understand the role data plays in AI systems. Data is the raw information collected from applications, users, transactions, APIs system interactions, etc. AI models learn patterns, make predictions, and generate outputs based entirely on this data.

In many ways, data acts as the foundation layer of an AI system. Just as a skyscraper depends on the strength of its foundation or a car depends on its engine, AI systems depend on the quality and reliability of the data flowing through them. If the underlying data is inconsistent, outdated, or poorly managed, the behavior of the entire system eventually becomes unreliable.

Data debt is the accumulated operational cost of unreliable, inconsistent, or poorly governed data systems. It emerges gradually through ignoring data quality, schema drift, weak lineage tracking, fragmented ownership, low-quality labeling processes, and brittle transformation pipelines.

In simple terms, data debt builds up when organizations continue scaling AI and data systems without maintaining the quality, consistency, and traceability of the underlying data. The systems may continue functioning, but the data slowly becomes harder to validate, debug, and trust over time.

Like technical debt, the problem compounds gradually. Small inconsistencies that seem manageable early on eventually spread across datasets, pipelines, feature stores, APIs, and AI workflows, making the entire system more difficult to maintain and optimize.

How Data Debt Affects AI Systems

Unlike infrastructure failures, data debt rarely causes immediate outages. AI pipelines continue functioning while reliability slowly degrades underneath. Models still train, dashboards still render, and pipelines still execute, but the outputs become increasingly difficult to trust.

Over time, data debt creates compounding downstream effects across AI systems:

Feature stores, the systems responsible for storing and serving ML features consistently across training and production environments, begin serving inconsistent feature distributions between inference and training workflows.
Pipeline dependencies become opaque, making debugging and root-cause analysis significantly slower.
Silent data quality failures propagate into production models without triggering infrastructure-level alerts.

The longer these issues remain unresolved, the harder they become to isolate because the dependency graph across datasets, services, and models keeps expanding.

Why Enterprises Accumulate Data Debt

SMBs and enterprise data systems evolve faster than governance models can keep up. New services, ingestion pipelines, analytics layers, and ML workflows are introduced continuously, often without standardized validation or ownership boundaries.

In distributed architectures, every system generates its own representation of business entities, events, and state transitions. Over time, those representations diverge. A "customer," "transaction," or "active user" may have entirely different definitions across operational systems, analytics pipelines, and ML feature stores.

The problem becomes significantly harder in AI workloads because models depend on historical consistency, not just raw availability.

Three structural factors accelerate data debt accumulation:

Large volumes of unstructured enterprise data such as logs, documents, support tickets, and chat records lack standardized schemas and validation layers.
Pipeline complexity increases exponentially as data moves across streaming systems, warehouses, feature stores, vector databases, and inference services.
Ownership fragmentation prevents consistent enforcement of quality controls, lineage tracking, and transformation standards.

Data debt is rarely caused by a single architectural mistake. It is usually the result of continuous optimization of trade-offs made under delivery pressure.

Where Data Debt Shows Up in AI Systems

Data debt shows up wherever AI systems depend on unreliable, inconsistent, or poorly governed data. It appears across training datasets, feature stores, retrieval pipelines, vector databases, APIs, and inference workflows. While the symptoms differ across architectures, the underlying issue is usually the same: the system is operating data that lacks consistency, traceability, or validation.

Forecasting models and traditional ML pipelines rely heavily on historical datasets and feature engineering, while Retrieval Augmented Generation (RAG) systems and autonomous agents continuously pull context from APIs, embeddings, or external systems during inference.

The failure patterns vary by architecture:

Forecasting and ML systems struggle with schema drift, missing historical records, and inconsistent feature generation across training and production environments.
RAG and LLM applications degrade because of stale embeddings, duplicated context, corrupted documents, or outdated vector stores serving irrelevant information.
Tool-using agents become unreliable when APIs expose conflicting definitions, incomplete state information, or inconsistent response formats across services.

In most cases, the visible issue appears at the model layer, but the actual failure originates much earlier in the data pipeline.

Anatomy of Data Debt

Data debt appears in predictable layers across modern AI systems. It usually starts upstream with inconsistent schemas, fragmented ownership, and weak validation controls, then spreads across pipelines, feature systems, and inference workflows. These problems are not limited to relational databases. They exist across feature stores, event streams, document systems, vector databases, and retrieval pipelines. Over time, inconsistencies become deeply embedded into the operational behavior of AI systems.

The most common forms of data debt include:

Data modeling issues: Different teams define the same entity differently. For example, marketing may define an "active_user" based on email engagement, while product defines it using session activity, resulting in conflicting churn predictions.
Data quality failures: Stale, incomplete, or noisy datasets silently affect model behavior. A delayed 'label refresh job' can introduce output drift for weeks before anyone notices.
Lineage and traceability gaps: Tracing predictions back to their original data source becomes increasingly difficult in complex AI systems. In RAG systems, a hallucinated response may ultimately come from a single corrupted PDF indexed into the vector store.
Pipeline fragility: Modern AI pipelines depend on multiple upstream systems. A schema change in a source table can silently break feature generation without triggering alerts or validation failures.

Individually, these issues seem manageable. Combined, they create systems where prediction reliability gradually deteriorates even when the models themselves remain unchanged.

Why Better Models Don't Fix Data Debt

Better AI models do not fix bad inputs because models cannot distinguish between a genuinely useful pattern and a flawed one unless the underlying data provides that context correctly. They learn statistical relationships from the data they receive. If the data contains noisy labels, stale features, inconsistent definitions, or biased signals, the model treats those patterns as valid during training.

Models with poor data create a dangerous illusion during evaluation. Offline metrics may improve because the model becomes highly optimized for flawed historical patterns, but production reliability continues to degrade under real-world conditions.

The result is overfitting at the system level:

Models learn operational noise as if it were a valid signal.
Evaluation pipelines reinforce flawed assumptions already present in the data.
Prediction instability increases as upstream inconsistencies propagate through training and inference workflows.

Paying Down Data Debt: What Works and Where It Fits

The bottleneck in modern AI systems has shifted from model capability to data reliability, consistency, and observability as model architectures become easier to access and standardize. As models become easier to access and deploy, long-term performance improvements increasingly depend on the quality and consistency of the underlying data systems.

This is the core idea behind data-centric AI, an approach popularized by Andrew Ng, where the focus moves from endlessly tuning models to improving datasets, validation, and data operations. In production environments, stable data systems usually lead to fewer regressions, faster debugging, and more reliable outputs than model experimentation alone.

Paying down data debt requires controls across multiple stages of the AI lifecycle:

Data contracts at producer-consumer boundaries prevent schema and definition mismatches before they propagate downstream.
Validation at ingestion catches stale records, malformed events, and missing fields before data reaches training or inference systems.
Observability and lineage make it possible to trace outputs back through features, embeddings, pipelines, and source systems during debugging.

These practices improve model accuracy and reduce operational instability across the entire AI stack:

Faster root-cause analysis during prediction failures
Fewer silent regressions caused by upstream pipeline changes
Easier auditing and reproducibility across ML workflows

Self-Check Questions for Data Maturity

Here are questions to ask for a simple self-check for data maturity:

Can production predictions be traced back to their original data source?
Are feature definitions and datasets versioned consistently?
Is there a clear source of truth for critical business entities?

If the answer to these questions is unclear, the system likely has accumulated data debt regardless of model sophistication. AI maturity is ultimately constrained by data maturity. Most organizations continue optimizing the visible model layer while the underlying data foundation remains unstable. The teams that build reliable AI systems are usually the ones investing most heavily in data discipline.

Conclusion

AI systems rarely fail because the models are weak. In most cases, the real issue exists much earlier in the pipeline. Inconsistent schemas, unreliable feature generation, stale datasets, missing lineage, and fragmented ownership quietly reduce the quality of predictions over time.

Many organizations are shifting toward more data-centric approaches to AI engineering by investing in validation, observability, lineage, and stronger data governance practices alongside model development. The goal is to ensure that models operate on reliable foundations.

The long-term challenge in AI is building systems with stronger data discipline. Teams that prioritize it tend to spend less time debugging unpredictable behavior and more time improving real-world outcomes.

Original article: https://www.improving.com/thoughts/data-debt-ai-predictions/

Author: Sarvani Yallapragada, Developer Advocate at Improving

Social: LinkedIn

Agent Memory Systems: Building Long-Term Context for AI

Improving — Thu, 28 May 2026 09:27:33 +0000

AI is almost everywhere now from agentic coding, autonomous workflows to day-to-day engineering tasks that you want to delegate to an agent. In many cases, an agent looks impressive in a single turn: you dump all the details of your pipeline or cluster issue into one prompt, and it responds with clean reasoning and a plausible plan.

The trouble starts when you try to work like a human does, iteratively over multiple turns.

Suddenly the agent forgets what you decided earlier, why you chose a certain approach, and sometimes even who is talking to it (SRE? platform engineer? backend dev?).

Agent memory fills this gap. Without a memory system, your agent behaves like a goldfish, it can only remember what fits inside a fixed context window, and once that window is saturated or a new session begins, the continuity breaks.

Memory is how you turn a one-shot chatbot into something that can maintain state, learn from outcomes, and stay aligned with your constraints over time.

Let’s start with understanding how context engineering solves this goldfish problem.

What is Context Engineering?

Context engineering refers to the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference, including all the other information that may land there outside of the prompts.

It includes techniques such as:

Prompt engineering
Structured outputs
State handling
RAG (Retrieval-Augmented Generation)
Memory (short-term + long-term)
Context packing / token budgeting

Why Agentic Memory Matters

Let’s understand from an example.

You’re building a DevOps pipeline, and you ask the agent to add one more step, and it replies with “I don’t know what you’re talking about.”

That does not mean the model you’re using is not good or capable. It’s because it has no context. It doesn’t know about your previous step in a single-turn conversation system.

If your system recalls previous conversation state, evidence, and decisions, the agent can understand exactly what pipeline you mean and what step should be added, even if you were referring in different sessions.

Without memory, the AI agent behaves like Dory from Finding Nemo. It might remember a few recent turns, and then it stops, especially if the context window is small, or you cross a certain token limit.

Once you restart a conversation or start a new session, it forgets nearly everything unless you build persistence.

Agentic memory is required for workflows that are iterative, multi-step, and long-running, helping achieve:

Continuity: Enables agents to remember previous turns in a conversation.
Learning and adaptation: Allows agents to learn from past successes and failures.
Advanced reasoning: Supports planning, personalization, and maintaining state.

Memory Architecture Patterns

Agent memory is not a single bucket where you dump everything from chat history.

In practice, it is layered because different information has different lifetimes, retrieval needs, and failure modes.

A useful mental model is:

Short-term memory: What the agent needs right now to finish the current task.
Long-term memory: What should persist across sessions.

Long-term memory typically splits into:

Episodic memory
Semantic memory
Procedural memory

Short-term Memory (Session / Working Memory)

Short-term memory is generally considered an agent session buffer. It holds recent conversation plus the immediate working state needed to complete the current task.

It prevents the agent from resetting mid-debug or mid-execution and is typically implemented as a sliding window of messages plus a state object containing plans, variables, tool outputs, and assumptions.

Once the task ends or the buffer grows too large, short-term memory is summarized, pruned, or selectively promoted into long-term memory.

Long-term Memory (Persistent Memory)

Long-term memory can be divided into three categories:

Episodic Memory

Stores past interactions as events with outcomes.

Examples:

What happened
What failed
What worked

Useful when revisiting the same system over time because it preserves continuity and prevents repeating dead ends.

Semantic Memory

Stores stable facts and constraints about the user, project, and environment.

Examples:

Deployment conventions
User preferences
Team policies
Architecture decisions

This keeps the agent consistent and personalized across sessions.

Procedural Memory

Stores repeatable how-to knowledge.

Examples:

Workflows
Runbooks
Checklists
Operational procedures

This allows the agent to execute proven processes instead of improvising each time.

How Agentic Memory Systems Work

Memory-enabled agents mimic the practical shape of human memory.

Humans have:

Sensory intake
Working memory
Long-term storage

Agents recreate this through an operational loop.

Practical Agent Loop

Imagine you ask an agent:

Deploy payments service to the Kubernetes staging cluster using Helm. Enable HPA and make sure rollout is healthy.

The agent will generally follow these steps:

Read
- Parse the goal, target, constraints, and current state.
Retrieve
- Pull relevant semantic, procedural, and episodic memories.
Assemble
- Build a token-budgeted context with only the relevant information.
Act
- Execute deployment steps and inspect failures.
Evaluate
- Verify rollout health, pod status, and deployment success.
Write-back
- Store durable learnings, fixes, and operational insights.

The write-back phase is what turns chat into learning.

Without it, you are only doing retrieval, not memory.

Agent Memory vs RAG (Retrieval-Augmented Generation)

RAG is about retrieving external knowledge such as:

Documentation
Tickets
Wikis
Runbooks

It is fundamentally a stateless retrieval workflow.

Memory, in contrast, is about persistent internal context:

User preferences
Constraints
Decisions
Outcomes
Historical interactions

What Should Become Memory?

Memory should contain information that improves future performance without introducing noise.

Examples include:

Explicit “remember this” instructions

Remember that all production deployments must go through manual approval in ArgoCD.

Stable preferences

Examples:

Uses Argo CD for GitOps
Prefers YAML over Helm templates
Follows strict naming conventions

Decisions and milestones

Examples:

Migrated from Jenkins to GitHub Actions
Standardized observability with Prometheus + Grafana

User corrections

Examples:

The API endpoint is /v2/orders, not /v1/orders
We’re running on EKS, not GKE

Outcomes and lessons learned

Examples:

Terraform state locking issues resolved by moving to S3 + DynamoDB backend

RAG and Memory Together

The most effective AI systems use both RAG and memory together.

RAG provides organizational knowledge.
Memory provides personalized context.

Together they create agents that are both knowledgeable and context-aware.

Vector Stores for Semantic Memory

Semantic memory stores stable facts, preferences, and knowledge.

As agents accumulate hundreds or thousands of facts, scalable retrieval becomes necessary.

This is where vector stores become useful.

Vector databases convert text into embeddings represented as numerical vectors.

When the agent needs information:

The current query is embedded.
Similar vectors are retrieved.
Relevant memories are injected into context.

Retrieval Strategies

Common strategies include:

Similarity search (top-k)
Re-ranking
Recency bias
Filtering by scope

Popular Vector Databases

Pinecone
Weaviate
Milvus
Qdrant
Chroma
pgvector

Context Window Management and Token Accounting

The context window is the model’s working memory.

Even though modern models support huge windows, effective context management is still difficult.

Adding too much information:

Degrades reasoning quality
Increases cost
Adds latency

Long-Running vs Short-Running Agents

Short-running agents may fit everything into a single context window.

Long-running agents operating across sessions accumulate far more information than any window can hold.

These agents require selective retrieval strategies.

The Context Stuffing Trap

A common mistake is including all available information without curation.

This introduces noise and buries critical information.

Helpful techniques include:

Semantic chunking
Memory buffering
Just-in-time retrieval
Hierarchical summarization
Progressive disclosure
Sliding windows

Packing Order Matters

Models pay more attention to the beginning and end of the context.

Recommended ordering:

System instructions and high-priority context
Immediate user query and relevant memories
Avoid burying critical information in the middle

Memory Management: Pruning and Compression

As memory grows, it requires active management.

Without management:

Retrieval slows down
Storage grows indefinitely
Old memories conflict with newer information

Pruning Strategies

Pruning selectively forgets irrelevant information.

Common strategies include:

TTL (time-to-live)
Least Recently Used (LRU)
Relevance scoring
User-requested deletion

Most production systems combine several of these techniques.

Memory Compression

Compression stores information in more compact forms.

Useful techniques include:

Rolling summaries
Hierarchical summarization
Topic clustering
Deduplication

Measuring Compression Quality

Compression should preserve critical information.

Quality checks include:

Ensuring important facts remain retrievable
Detecting contradictions
Avoiding over-compression

Final Thoughts

We are entering the era of personalized AI agents, where memory becomes foundational infrastructure.

Without memory, agents lose continuity across sessions and interactions.

With memory, they can:

Learn from outcomes
Maintain long-term context
Personalize interactions
Execute workflows more reliably

In this article, we explored:

Context engineering
Memory architecture patterns
Agent loops
RAG vs memory
Semantic retrieval systems
Context management
Pruning and compression

After working on 250+ projects and helping companies generate billions, one thing is clear and that is that most organizations don't fail at AI because of technology. They fail because they skip the trust-building stages like developing agentic memory systems that make AI safe to scale.

In the next part of this blog, we will be implementing memory patterns and learning how all these pieces come together to form a sophisticated agentic system. That system will make sure that agents not only talk in one session but also across sessions and remember all the past events.

Embedding AI Into Daily Development: What Software Engineers Actually Learn

Improving — Fri, 15 May 2026 06:06:04 +0000

The industry is currently obsessed with "Vibe Coding," but that framing misses the point for working engineers. As architects, we need to look past the trend and towards the outcome.

We have spent decades coupled to our tools, often mistaking the act of typing for the act of engineering. AI is the latest evolution of that tension. Professional engineers must shift from a code-first to an outcome-first mindset. The move is from Vibe Coding to Vibe Solving.

Vibe Coding focuses on the engine: the syntax, the generation, the satisfying feeling of code appearing on screen. Vibe Solving focuses on the destination: the business problem resolved, the user served, the system more maintainable than before. We are not syntax curators — we are problem solvers with better tools than we have ever had.

AI is no longer an experiment for software engineers. It is becoming part of daily development workflows, from writing specs to maintaining legacy systems. In this post I want to cover what that shift looks like in practice: how to kill the cold start, where AI delivers its highest return, how to keep the work auditable, and why transparency with clients is no longer optional.

Beyond the Blank Page

Thousands of hours disappear every year to the friction of sitting down to write a technical spec, a project plan, or even a well-scoped ticket — and having nothing come out.

The greatest tax on engineering productivity is the cold start.

To address this, I use a Voice-to-Insight workflow. The idea is simple: move into forward-only mode by capturing thoughts out loud, bypassing the internal critic that stalls written work.

The key unit in this workflow is the 30-second Wow. Instead of trying to convince people through abstract language, you quickly create something tangible — a rough sketch, a narrated idea, a snapshot — and use AI to turn it into a visible output like user stories or a working prototype. The idea moves from concept to something people can see and react to, which breaks resistance and builds shared understanding.

By using MacWhisper for speaker-diarized transcription and local models, you can process raw thoughts into a Voice Vault: a structured folder of transcripts and summaries. From there, tools like Windsurf (any LLM works here) help convert raw thinking into actionable backlogs or technical outlines — without a single minute of manual typing.

The raw material already exists in your own words. AI's job is to preserve it and reshape it into something a team can act on.

AI in Software Maintenance

Maintenance is the graveyard of most engineering backlogs. The work is pattern-heavy, rule-heavy, and boring in a way humans have always been bad at sustaining. That is also precisely why AI returns its highest ROI there.

A concrete example: I recently oversaw a migration of 3,000 tests from Fluent Assertions to Shouldly, alongside a major version update of the Marten and Lamar libraries. Historically, that is a multi-week job — one that gets deferred until it becomes a crisis.

With AI, it took two days.

The key technique was solving the LLM's training cutoff problem. The model did not know the latest breaking API changes in Marten. Ask it to migrate your tests and it will confidently produce syntax from two versions ago. To close that gap, I used Context7: an MCP Server that provides up-to-date documentation for libraries and frameworks. The AI is no longer working from stale training data — it is working from the version of reality I just handed it.

Two things are worth being honest about here.

First, the developer role changed, not disappeared. My value those two days was architectural review and verification. Second, the risk profile is not automatically lower. A model operating at scale can introduce a subtle error across thousands of tests just as easily as it introduces a correct pattern. Oversight is non-negotiable. Without it, AI-assisted maintenance just compresses the time to failure.

Investigation Markdown for AI Context

When AI is doing heavy lifting on a codebase, you cannot treat it as a black box. I require an Investigation Markdown file for every non-trivial task.

Investigation MD is a ledger that captures which decision paths were tried, which were abandoned, and why. It records the questions asked, the answers accepted, and the ones overridden. When an LLM's context window clears — and it will — that Markdown file is the only thing that tells the next human or the next AI instance where the team actually stands.

I call this managing the binary hands: the AI is fast and capable, but it does not remember yesterday, does not know why a prior approach was abandoned, and does not carry the context that exists only in the heads of your team. The Investigation Markdown makes that context explicit and persistent.

A few operating rules I hold teams to when AI is in the loop:

Plan before touching files. Use Plan Mode in Windsurf (or Claude Code) to audit the AI's intended changes before it touches the file system.
Get a second opinion. Spin up a separate model instance with an adversarial prompt. Ask it to find holes in the first model's plan.
Ground the citations. Prompt the LLM to trace claims back to source transcripts. If the AI cannot point to an origin, the claim does not belong in the strategy.

AI also starts paying back at the leadership level when you analyze Daily Scrum transcripts for blind spots: the technical concerns that surfaced and then got ignored, the risks buried under routine updates. A human lead cannot catch all of that across several minutes of conversation per day. A model can. The limitation is honest: this is only as good as the transcripts feeding it and the prompts guiding it. It does not replace an engineer, but it makes a sharp engineering manager sharper.

Professional Obligation

Using AI goes beyond improving productivity — it is a professional obligation. A consultant not using AI is showing up to a construction site with a manual screwdriver when power tools are available: slower, less accurate, and billing the client for the difference.

Transparency with clients follows from that. I do not hide that I use AI. I frame it around what they actually care about: that their budget is buying judgment rather than typing, that the oil change is happening while the car is still moving. That framing earns more trust than hiding the tool ever could.

What the Time Buys

We are drowning in data and starving for wisdom. We need to stop thinking of IT as Information Technology. The real shift is toward Impact Technology: using judgment, experience, and the best tools available to deliver outcomes that actually matter.

AI's real return is not the code it writes. It is the time it gives back. Time for the things AI cannot do: empathy with a client under pressure, design decisions that account for constraints not found in any repository, the hard conversation with a team about a risk they have been avoiding.

What else will you find if you stop putting off the cleanup and start using the tools?

For comments or suggestions on this article, find me on LinkedIn.

Why Most AI Training Fails

Improving — Mon, 11 May 2026 09:38:18 +0000

I have taken more online AI courses than I care to count. And I am going to be honest with you: most of them followed the exact same pattern. A long walk through the history of AI, a glossary of terminology, a bunch of model names and acronyms, maybe some screenshots of someone else using ChatGPT, and then a list of prompts to take home. I would finish a course and realize I could not remember half of what I had just watched. Not because the content was wrong. Because none of it connected to anything I actually do at work.

Sound familiar?

If it does, you are not alone. McKinsey's 2025 Global Survey found that 78% of organizations now use AI in at least one business function. But a WalkMe study from August 2025 reported that only 7.5% of employees have received any extensive AI training, and ManpowerGroup's 2026 Global Talent Barometer found that 56% of workers globally received no AI training of any kind. So the tools are everywhere, but the ability to use them well? That is a completely different story.

For managers and individual contributors, the friction shows up in the same places: meetings that produce unclear outcomes, writing that takes too long to start, and decisions that require pulling together information under time pressure. The courses that are supposed to help with this — don't. They teach information that is easy to absorb during the session and just as easy to forget by the next morning.

Why AI "Doesn't Work" for Most People

AI fails for most professionals not because the technology is broken, but because nobody showed them a different way to approach it.

Someone pastes a meeting transcript into an AI tool and asks for a summary. The output sounds confident, but it includes decisions that were never actually made. The immediate reaction? This thing cannot be trusted.

Or someone asks AI to draft a response to a client. The words are technically fine, but the tone is off. And then for the trivial stuff — someone asks if you are going to be at the meeting on Friday — the answer is just "Yep, I'll be there." You do not need AI for that. The challenge is knowing which messages are worth involving AI in and which ones are not.

These experiences pile up, and pretty soon you start wondering: everybody seems excited about AI, so why am I just continually frustrated with it? I hear that question a lot. And the answer is almost always the same: nobody taught you a different way to interact with the tool.

The Trust Problem No One Addresses Well

When people first sit down to work with AI in any kind of structured way, the number one concern is trust. Not whether AI is useful in theory, but a very practical worry: "I don't know if it's just going to lie to me." I hear some version of that in almost every conversation.

And the concern is grounded. Language models can fabricate. But here is what most training programs miss: they either ignore the trust problem entirely, or they spend an hour explaining the technical reasons behind hallucinations without ever showing you what to do about it.

The practical fix is not asking you to trust AI. It is teaching you how to challenge it. Ask where a claim came from. Ask for direct quotes from the source material. Ask what assumptions were made. Once you learn how to provide the right inputs and ask for the receipts, trust stops being a yes-or-no question and becomes conditional: trust it when you can verify it.

Why Treating AI Like a Search Engine Fails

Most people approach AI the way they approach Google: type something, get a result, move on. But AI works through interaction, and the first response is almost never the one you should keep. A lot of professionals figure this out the first time they push back on an AI response and watch it get better. The realization is uncomfortable, because the problem was not the tool. It was how they were using it.

A single vague prompt almost guarantees disappointment. The more context you give AI, the better the response it will give. Sometimes that context emerges through a back-and-forth conversation as your intent and goals surface through feedback on what you like and don't like. That one reframe changes everything that comes after it.

Why Learning AI Now Matters More Than Later

A lot of professionals are waiting, figuring they will pick up AI skills once the tools stabilize. I understand that instinct. But I think it is a mistake.

Imagine the 100-meter dash at the Summer Olympics. The starting pistol fires and every runner launches off the blocks. But one runner stands up, watches everyone else, studies their techniques, and decides to join once they understand the field. By the time they start running, the race is over.

AI adoption is following that same pattern. Gallup's Q3 2025 workforce survey found that 45% of U.S. employees now use AI at work, nearly doubling from 21% in 2023. And EY's Work Reimagined Survey found that companies are missing out on up to 40% of potential AI productivity gains because of gaps in talent strategy.

People who start earlier build intuition. They recognize when an output is fragile. They know how to recover without starting over. Waiting does not give you a better starting position. It just puts you further behind.

Think about what it would feel like to hire someone today who says, "So am I going to have to use this Internet thing?" Nobody asks that question anymore. But AI is heading in the same direction. Right now, learning AI is still seen as getting ahead. Soon enough, not knowing it will just be falling behind.

Why This Is a Training Problem, Not a Tool Problem

Most AI frustration has nothing to do with missing features. It comes from missing habits.

The numbers back this up. DataCamp's 2026 State of Data and AI Literacy Report found that 82% of enterprise leaders say they provide AI training, yet 59% still report an AI skills gap. The most common format is video-based courses, and 23% of leaders say video training does not translate to real-world application. Organizations are investing in training that is not changing how people work.

When I was designing our AI training, I made a very deliberate decision: I am not just going to tell you about a thing. I am going to have you do the thing. Because the difference between watching someone use AI and actually using it yourself is the difference between a forgettable session and a skill that sticks.

Knowing what a hallucination is does not help you when a meeting summary misrepresents a decision. What helps is learning how to provide context, how to demand evidence, and how to refine output without starting over. Prompt libraries promise shortcuts, but real work rarely fits templates. The durable skill is structured thinking: learning how to frame your requests with enough context and constraints that the system responds appropriately, regardless of which tool you are using.

Three Skills That Change Day-to-Day Work

When I was building the curriculum, I asked AI itself to research what professionals are most frequently asking for help with. The answer kept pointing to three areas.

1. Uncovering Insights from Messy Inputs

Meetings generate noise. Transcripts run long. Reports have more detail than anyone can process quickly. AI can help condense and organize all of that — but only if you stay accountable for verifying what comes back.

Asking for a summary is the easy part. The harder and more valuable skill is asking AI to show its sources. If it claims a decision was made, ask it to point to the exact passage. If a takeaway does not sound right, push back: "I don't remember that from the meeting. Show me where that is." That habit is the difference between speed and error.

2. Generating Ideas Without the Blank Page

The most paralyzing moment in any task is the beginning — staring at a blank page, trying to figure out where to start. AI solves this not by writing the final version but by giving you something to react to. Once you are reacting instead of creating from nothing, you are moving.

Here is a technique I share in every session: ask for multiple options rather than a single answer. Ask for five ideas. Review them. The first two might be terrible. The third might have something worth exploring. Tell AI to go deeper on that one and throw the rest away. That sets up an iteration cycle, and iteration is where the real value lives.

3. Drafting Communication with Accountability

AI can get you 90% of the way on a piece of writing that would have taken significant time to start from scratch. I have never had a situation where AI gave me something I did not have to tweak at all — it never gets it 100% right. But that remaining 10% — the nuance, the tone, the judgment about what to include — that is your job. AI handles the heavy lifting and you focus on the part that requires your expertise.

I draw a clear line here: AI can draft, but it does not send. The human owns tone, intent, and consequences. The discomfort people feel about AI handling communications entirely? That is well-placed. Having AI prepare a draft for your review is a fundamentally different thing from having it respond on your behalf.

Why Hands-On Practice Changes Outcomes

There is a difference between seeing AI used and using it yourself. Demos look clean, but real work does not. When you actually practice with AI, you see where your inputs were too vague, where constraints were missing, and where the first confident-sounding response was wrong.

And then something shifts. You structure a real interaction with clear context and constraints, and the output actually works. The realization is blunt: it did not fail randomly. It failed predictably. That is the moment you stop blaming the tool and start changing how you interact with it.

When you provide context, set boundaries, and iterate, AI produces drafts that hold together. Trust becomes conditional instead of binary, and the rework drops. Someone who spent 45 minutes writing a client update discovers that with clear context and two rounds of iteration, AI produces a usable draft in minutes. The remaining time goes to judgment: refining tone, checking accuracy, deciding what to leave out.

The professionals who make AI stick are the ones who apply it to their own problems early. Someone tackles a proposal outline they have been putting off. Someone else feeds in a meeting transcript and pulls action items. When the stakes feel real, the learning sticks faster.

Does Teaching AI Fundamentals Actually Change Anything?

This is a fair question. Professionals are busy, AI changes fast, and it is reasonable to ask why anyone should invest in learning the basics when the tool will be different in six months. The argument holds up if fundamentals training means memorizing features, watching demos, and leaving with a list of prompts. That kind of training does not change behavior.

But fundamentals defined as interaction discipline — how to structure context, how to iterate, how to verify — are not tied to any particular model or release cycle. They work the same way in ChatGPT as they do in Copilot, and they will work in whatever ships next year. The interface changes. The thinking does not.

The gap most professionals are stuck in is not between basic and advanced knowledge. It is between occasional use and reliable use. You have tried AI, gotten mixed results, and not changed your interaction patterns. That gap closes by practicing a different way of working, not by learning more theory.

Even experienced users pick up useful techniques in fundamentals-focused settings, because knowing a lot about AI and using it effectively are two different things. For the large population using AI occasionally and inconsistently, the bottleneck is almost always interaction habits — not technical depth.

Applying AI to Real Work

Training fails when it stays abstract. Real work has constraints: policies, customers, tone, and risk. Shorter, focused sessions tied to real tasks tend to produce more lasting change than marathon lectures.

A practical starting point? Look at what frustrates you. Tasks that are slow or mentally draining often contain parts AI can compress. Someone who spends two hours each week writing status updates can likely compress that to 20 minutes with the right interaction structure, freeing up time for work that actually requires their judgment.

And the applications go beyond text. AI can generate images, create visual aids for presentations, and produce supporting content. Once you see that you can describe a concept and have AI produce a working version, the range of tasks you consider using AI for expands.

Early use tends to focus on low-risk situations: notes, options, internal drafts. Over time, some uses stick and others disappear. The professionals who make AI part of how they work going forward are the ones who found two or three use cases where it reliably saved them time and built those into their routine.

Key Takeaways

AI does not underdeliver because the technology is broken. It underdelivers because most people were never taught how to interact with it. That is a training problem, and 56% of workers globally have not received any AI training at all.

Three skills make the biggest difference for professionals adopting AI:

Uncovering insights from messy inputs — and staying accountable for verifying what comes back
Generating ideas by pushing past the blank page — through iteration, not one-shot prompts
Drafting communication where AI does the heavy lifting — and you own the final 10%

The skills that actually stick — structured thinking, iteration, verification — are not tied to any specific tool or model. They work regardless of what platform you are using. But if your organization is waiting to build those skills, that wait has a price: EY's research found that companies are leaving up to 40% of their potential AI productivity gains on the table because of gaps in how they develop talent.

AI as a Professional Skill

None of this is complicated. But it does require a different approach than most professionals have been taught. And the longer teams wait to build these habits, the more time gets lost to rework, rechecking, and correcting mistakes that did not have to happen.

If any of this resonated — or if you have your own AI training stories (the good, the bad, and the frustrating) — I would genuinely enjoy hearing about them. You can find me on LinkedIn.

OWASP Top 10 for LLMs: A Practitioner’s Implementation Guide

Improving — Mon, 11 May 2026 09:35:40 +0000

Large Language Models (LLMs) are becoming a core part of modern applications — from copilots and chatbots to AI agents connected to tools and internal systems. As adoption grows, so do the security risks.

The OWASP Top 10 for LLM Applications (2025) highlights the most common security issues teams must address when building AI-powered systems. These risks go beyond traditional application security because LLMs interact with prompts, external data, tools, and autonomous workflows.

In this post, we'll cover a practical overview of each risk and how teams can detect, prevent, and test for them.

LLM01:2025 — Prompt Injection

Prompt injection is when an attacker slips malicious instructions into user input or content the model reads, tricking it into doing something it shouldn't.

Direct injection: A user directly tells the model to ignore its rules.
Indirect injection: The model reads an external document or web page that secretly contains instructions and follows them without realizing it.

Example: An LLM connected to internal tools retrieves a document containing hidden instructions telling it to export database credentials. The model follows the instruction and triggers a data leak.

How to Detect It

Watch for phrases like "ignore previous instructions" or "pretend you are" in user input
Compare inputs against known malicious prompt patterns
Alert on unusual tool calls — especially ones fetching or exporting data unexpectedly
Log all inputs and outputs so you can trace what happened after an incident

How to Prevent It

Make sure system-level rules can't be overridden by user messages
Sanitize and validate any external content before passing it to the model
Use clear separators between instructions and data in your prompts
Apply least-privilege access — the model should only be able to call what it needs
Add output filters to block unsafe responses before they reach users

How to Test It

Run red-team tests that simulate both direct and indirect injection attempts. Use automated prompt fuzzing to probe edge cases. After any prompt changes, run regression tests to confirm your safety rules still hold.

LLM02:2025 — Sensitive Information Disclosure

This happens when an LLM leaks personal data, API keys, credentials, or internal documents in its responses. It can occur through direct questions, indirect prompt injection, or a retrieval system that doesn't properly restrict access to sensitive documents.

Example: An internal HR assistant retrieves employee salary records during a broad query and includes them in its response — even though the user asking had no right to see them.

How to Detect It

Scan model outputs for PII (names, emails, ID numbers) and secrets (API keys, passwords)
Monitor what documents the retrieval system is fetching and whether they match the user's access level
Flag responses with unusual patterns like long random strings, which could be tokens or keys

How to Prevent It

Redact sensitive data before it gets indexed or fed into the model
Only retrieve documents the current user is actually allowed to see
Add an output filter that blocks responses containing classified data
Keep sensitive data stores separate from general knowledge sources

How to Test It

Try prompting the system to extract personal records or credentials through indirect queries. Verify that restricted data can't be retrieved through similarity-based tricks. Check that access controls on your retrieval system are actually working end-to-end.

LLM03:2025 — Supply Chain Vulnerabilities

LLM applications depend on many third-party components — base models, plugins, vector databases, MCP servers, and embedding providers. Any one of these can be a weak link. A malicious or compromised dependency can manipulate outputs, steal data, or take unexpected actions.

Example: An application uses a third-party MCP server for document processing. A malicious update modifies the server's tool responses to inject hidden instructions, causing the app to expose sensitive data.

How to Detect It

Keep a full inventory of every model, plugin, connector, and tool your application uses
Generate and maintain a Software Bill of Materials (SBOM) so you know what's inside
Watch for unexpected changes in model or tool behavior after updates
Correlate version upgrades with any new security anomalies

How to Prevent It

Vet vendors before integrating their tools — check their security practices and update history
Verify model weights and tool packages using checksums and cryptographic signing
Give third-party tools the minimum permissions they need, nothing more
Isolate external services in controlled network segments where possible

How to Test It

Regularly scan dependencies for known vulnerabilities. Test that third-party tools behave exactly as documented with no hidden inputs and no unexpected outputs. Before upgrading a dependency in production, simulate the upgrade in a test environment first.

LLM04:2025 — Data and Model Poisoning

Data poisoning happens when malicious data is introduced into training datasets or the retrieval corpus. In fine-tuning, poisoned samples can embed hidden behaviors that activate on specific triggers. In RAG systems, an attacker can insert crafted documents into the vector store so the model retrieves and trusts corrupted context.

Example: A RAG system indexes public documentation. An attacker adds a document with hidden instructions that changes how the model responds whenever a specific keyword is used.

How to Detect It

Track where every piece of data comes from before it enters your pipeline
Look for documents that appear in retrieval results far more often than you'd expect
Monitor for sudden shifts in model behavior after a dataset update
Check embeddings for outliers that don't fit the rest of your corpus

How to Prevent It

Control who can write to your vector store — don't allow open ingestion
Require human review for any high-impact data before it's added
Version your datasets so you can roll back if something goes wrong
Don't automatically ingest content from untrusted external sources

How to Test It

Use canary data — known triggers — to check whether the model has been altered. Compare model behavior before and after dataset updates. Periodically audit your retrieval corpus for documents that don't belong.

LLM05:2025 — Improper Output Handling

Output risk occurs when LLM responses are used directly — rendered as HTML, inserted into SQL queries, or passed to shell commands — without any validation. Because model output is probabilistic, it can contain unexpected characters or code-like content. Treating it as trusted input is the mistake.

How to Detect It

Scan model outputs for suspicious patterns: script tags, SQL special characters, shell operators
Watch downstream systems for unexpected queries or commands
Enable Content Security Policy (CSP) violation reporting to catch injected scripts

How to Prevent It

Always encode output before rendering it — treat it the same way you'd treat user-submitted content
Never pass model output directly to a shell command, SQL query, or code evaluator
Use parameterized queries instead of string concatenation
Validate outputs against a strict schema — for example, require JSON with defined fields

How to Test It

Deliberately include injection payloads in model responses during testing and verify they are neutralized before rendering. Review all code paths where LLM output flows into execution layers or sensitive APIs.

LLM06:2025 — Excessive Agency

When an LLM agent is given too much autonomy — access to APIs, databases, or infrastructure without proper guardrails — it can chain together actions that were never intended. This can cause real damage: deleted records, unexpected transactions, or service disruptions, often triggered by an ambiguous instruction or injected prompt.

How to Detect It

Log every action the agent takes, including its reasoning steps
Alert when an agent exceeds a set number of actions in a sequence
Track cross-system changes that could indicate the agent acted beyond its scope

How to Prevent It

Require human approval before the agent takes any high-risk or irreversible action
Limit how many steps an agent can chain together
Give agents time-limited credentials with the minimum permissions needed
Keep planning and execution separate — don't let the model decide and act in one step

How to Test It

Test agents against adversarial and ambiguous prompts to identify how they behave under pressure. Verify that kill switches actually stop an agent mid-task. Run stress tests to observe what happens when objectives conflict.

LLM07:2025 — System Prompt Leakage

The system prompt often contains safety rules, tool schemas, internal logic, and operational details that were never meant to be visible. If an attacker can get the model to reveal this content, they learn exactly how to bypass your controls.

Example: A user repeatedly asks the model to repeat its hidden instructions. After several attempts, the model partially reveals the safety rules embedded in its system message.

How to Detect It

Watch for responses that look like internal instructions or policy text
Flag repeated meta-questions like "what are your instructions" or "ignore your rules"
Use automated red-teaming tools to simulate extraction attempts

How to Prevent It

Don't store credentials, API endpoints, or secrets inside the system prompt
Use output filters that block responses referencing hidden instructions
Keep policy logic separate from natural language instructions
Structure prompts so system rules cannot be disclosed in response to user requests

How to Test It

Run structured extraction prompts specifically designed to coerce the model into revealing system content. After every prompt update, re-test to confirm that nothing new has leaked. Rotate system prompts if exposure is confirmed.

LLM08:2025 — Vector and Embedding Weaknesses

RAG systems rely on vector similarity to retrieve relevant documents. Attackers can craft documents with embeddings specifically designed to dominate retrieval results, hijacking the context the model receives. Poorly secured vector stores can also expose source content through embedding inversion — where attackers attempt to reconstruct original content from stored embeddings.

Example: A malicious document inserted into a public knowledge base is embedded to closely match frequent queries, causing it to be consistently retrieved and influence the model's output.

How to Detect It

Monitor for documents appearing far more often than expected across unrelated queries
Check for sudden shifts in the distribution of your embedding space
Audit who can write to your vector store and when changes were made

How to Prevent It

Restrict write access to the vector store — require authentication for all ingestion
Combine semantic similarity with keyword or rule-based filtering as a second check
Encrypt embeddings at rest and isolate vector infrastructure
Periodically re-index and validate your corpus to catch tampered documents

How to Test It

Simulate retrieval hijacking by inserting adversarial documents and checking whether they surface in results. Compare retrieval output from a clean corpus against your live one. Audit ingestion logs to see when and what was added.

LLM09:2025 — Misinformation

LLMs can confidently generate content that is factually wrong — fabricated statistics, non-existent citations, and outdated information. In applications used for decision-making, legal work, or reporting, this can cause serious real-world harm.

How to Detect It

Cross-check claims against trusted knowledge sources or retrieval results
Flag responses that make factual claims without citations in high-stakes domains
Monitor for contradictions across multi-turn conversations

How to Prevent It

Ground responses in retrieved, verifiable sources rather than relying on the model's memory
Require citations for any regulated or high-stakes use case
Add confidence indicators so users know when the model is less certain
Require human review before allowing the model to publish in high-impact contexts — do not permit autonomous publishing

How to Test It

Run benchmark evaluations using fact-sensitive datasets. Test with adversarial prompts designed to produce hallucinated references and measure how often they appear. Put corrections in place and notify affected parties if fabricated content has already been published.

LLM10:2025 — Unbounded Consumption

Without limits, LLM interactions can spiral into excessive token usage, recursive agent loops, or rapid API call chains. The result is infrastructure strain, massive cost overruns, or denial of service — sometimes triggered accidentally, sometimes by a malicious user probing for weaknesses.

How to Detect It

Track token usage per session and per user against expected baselines
Alert on recursive tool calls or unusually deep action chains
Use cost anomaly detection on your API and compute bills

How to Prevent It

Set hard token limits and cap response lengths
Apply rate limiting per user, per tenant, or per session
Limit how deep an agent can chain actions
Require confirmation before the model starts a high-cost operation

How to Test It

Simulate recursive prompts and measure whether your safeguards kick in. Test rate limiting and quota enforcement under high concurrency. After any incident, audit usage logs to understand the financial and operational impact.

Conclusion

LLM security is an engineering discipline, not an afterthought. The OWASP Top 10 for LLM Applications highlights that securing AI systems requires more than traditional application security practices. Teams must also address risks related to prompts, training data, external dependencies, and autonomous agents.

Building secure LLM systems requires layered protections, careful data management, strong observability, and continuous testing. The table below summarizes the key controls across all ten risk categories as a quick-reference checklist for teams designing, deploying, or operating LLM-enabled systems.

Risk	Detect	Prevent	Respond
Prompt Injection	Log inputs, pattern match	Sanitize inputs, least-privilege	Trace and remediate
Sensitive Disclosure	Scan outputs for PII/secrets	Redact data, enforce access controls	Block and audit
Supply Chain	SBOM, behavior monitoring	Vet vendors, verify checksums	Rollback, isolate
Data Poisoning	Track data provenance, monitor embeddings	Control ingestion, version datasets	Roll back corpus
Improper Output Handling	Scan for injection patterns	Encode outputs, parameterized queries	Review execution paths
Excessive Agency	Log agent actions, action limits	Human approval, least-privilege creds	Kill switch, audit
System Prompt Leakage	Watch for meta-questions	No secrets in prompts, output filters	Rotate prompts
Vector/Embedding Weaknesses	Monitor retrieval patterns	Restrict write access, encrypt embeddings	Re-index, audit logs
Misinformation	Cross-check claims, flag unsourced content	Ground in retrieval, require citations	Notify, correct
Unbounded Consumption	Track token usage, cost anomalies	Rate limits, hard token caps	Audit usage, throttle

Understanding these risks is the first step. For edge cases and complex deployments, consider working with security experts who specialise in AI systems.

If you found this post useful or have real-world experiences to share, feel free to connect on LinkedIn.

Everyone Talks About Golden Paths. Nobody Talks About Building Them.

Improving — Mon, 11 May 2026 09:32:33 +0000

Everyone's talking about Platform Engineering lately. Walk into any major technical conference like KubeCon, and you're bombarded with talks on "Golden Paths" and "IDPs." And it's great that the industry is finally focusing on developer experience instead of just more YAML.

But there's a massive gap between the conference talks and your terminal.

You leave these sessions feeling inspired, only to sit back down at your desk and stare at a mess of legacy deployment scripts. Most of the advice out there tells you why you need a platform, but almost nobody shows you how to actually build one without a massive team or a million-dollar budget.

That's exactly what I spoke about at the Platform Engineering Day co-located event at KubeCon Europe 2026. This post is a written version of that talk, with everything you need to go from zero to a working golden path — without needing a big platform team or expensive tooling.

What is a Golden Path?

A golden path is just the "opinionated" route your org sets up to get code into production.

It's about making the right way the easiest way.

I like to think of it as a product, not a mandate. You aren't forcing teams into a cage; you're giving them a well-lit highway with the guardrails already bolted on. If a team really needs to go off-road and hack together something custom, they can — but 99% of the time, they'll choose the highway because it's faster and safer.

A golden path isn't "done" unless it hits these four marks:

Opinionated: You've already made the boring decisions so the developer doesn't have to and can focus on shipping features.
Self-service: If a developer has to ping someone on Slack or open a Jira ticket, it's not a golden path. It's a hurdle.
Safe by default: Security and health checks aren't "extra steps" — they're just part of the plumbing.
Progressive: You don't build the whole highway at once. You start with a single paved mile.

When this works, the ROI is immediate. You stop seeing people copy-pasting crusty YAML from a repo last touched in 2022. New hires actually ship code on day one instead of day ten.

But here's where most organisations get stuck.

They understand what a golden path is. They've seen the talks, read the blog posts, maybe even drawn the diagram on a whiteboard. But when it's time to actually build one, the question is always the same: where do we start?

The problem is that a lot of teams assume they need a full platform in place before they can build a golden path — a proper IDP, a self-service portal, the whole thing. So they wait. And nothing gets built.

You don't need a full platform to start paving a golden path.

The Gap Nobody Talks About

Most teams aren't lacking a Golden Path because they're lazy or "don't get it." They're stuck because they can't find the starting line. Most advice from talks and blog posts assumes you're either part of a 50-person platform team or starting a greenfield project. In the real world? You have neither.

What you do have is a folder with a bunch of deployment scripts. Some are six months old; others were written three years ago by someone who hasn't worked at the company since 2023. Every team is doing their own thing — same task, but a dozen different, messy ways to get it done.

That isn't a platform problem; it's a fragmentation problem. And you don't need to buy a shiny new tool to fix it.

The other myth that kills progress is the "Big Bang" approach — sitting in a room, architecting the perfect platform, getting stakeholder approval, and buying three new SaaS tools before shipping a single thing. That's a recipe for a six-month roadmap that ends in a "deprioritized" project.

Building a Golden Path isn't a project with a deadline. It's an evolution. It matters less where you're starting and more that you're actually moving.

The good news? Your deployment scripts, messy as they are, are already your Phase 0. You are closer than you think.

A Maturity-First Approach to Building Golden Paths

You don't have to reinvent the wheel here. The CNCF Platform Engineering Maturity Model already gives us a roadmap.

This model breaks things down across five pillars — investment, adoption, interfaces, operations, and measurement — to help you figure out exactly where you're standing.

We take that model and map it directly to golden path construction. Instead of asking "how do we build a golden path?", you ask "what does the next maturity level look like for us?" That shift makes the whole thing much less overwhelming.

Each phase builds on the previous one. You can find the complete demo in the Golden Path Construction Demo Git Repo that maps out these phases with actual code — designed as a template you can fork and adapt to your own organisation.

Phase 0: The Chaos

This is where most teams are, even if they won't admit it.

Every team has their own deployment script. Same goal, different approach. One team uses inline kubectl commands; another has a YAML file that's been copied and modified so many times nobody knows what the original looked like. Images are pinned to latest, resource limits are missing, health checks are broken or absent.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: nginx:latest        # ❌ latest tag
        ports:
        - containerPort: 80
        # ❌ no resource limits
        # ❌ no health checks
        # ❌ no namespace
EOF

Nothing is wrong with any individual script. The problem is there are ten of them, and none of them talk to each other.

Phase 1: Standardize

This is the most important phase — and also the easiest one to ship.

You pick one script. One template. Every team uses it. That's it.

The template enforces the basics by default: resource limits, health checks, proper labels, a namespace. Nobody has to remember to add them. They're just there.

template:
  metadata:
    labels:
      app: ${APP_NAME}
      managed-by: platform-team
  spec:
    containers:
    - name: ${APP_NAME}
      image: ${IMAGE}
      ports:
      - containerPort: 80
      resources:
        requests:
          memory: "64Mi"
          cpu: "50m"
        limits:
          memory: "128Mi"
          cpu: "100m"
      livenessProbe:
        httpGet:
          path: /
          port: 80
        initialDelaySeconds: 10
      readinessProbe:
        httpGet:
          path: /
          port: 80
        initialDelaySeconds: 5

Developers just pass the necessary values; everything else is governed by the template.

What changes from Phase 0: instead of ten different scripts with ten different outcomes, you have one script with one consistent output.

What you gain: predictability. Every deployment looks the same. Debugging becomes faster. Onboarding becomes easier. And you've done this without any new tooling or platform investment.

This is also your first win to show leadership. You haven't built a platform. You've standardised how your teams deploy. That's already valuable.

Phase 2: Validate

Standardization tells teams what to do. Validation makes sure they actually do it.

In this phase, you move from a shell script to a config-driven approach. Teams fill in a YAML file with their app details. A validation layer checks the inputs before anything touches the cluster. Bad configs are rejected early, with a clear error message — not a cryptic Kubernetes failure three minutes later.

def validate(config):
    errors = []

    for field in ["name", "image", "team"]:
        if not config.get(field):
            errors.append(f"❌ '{field}' is required")

    name = config.get("name", "")
    if name and not re.match(r'^[a-z][a-z0-9-]*$', name):
        errors.append(f"❌ 'name' must be lowercase and DNS-compatible (got: '{name}')")

    image = config.get("image", "")
    if image and (":latest" in image or ":" not in image):
        errors.append(f"❌ 'image' must use a specific version tag, not ':latest' (got: '{image}')")

    env = config.get("environment", "dev")
    if env not in ENV_DEFAULTS:
        errors.append(f"❌ 'environment' must be one of {list(ENV_DEFAULTS.keys())} (got: '{env}')")

    return errors

What changes from Phase 1: the interface is now declarative, not imperative. Teams describe what they want, not how to do it.

What you gain: fewer misconfigurations, faster feedback loops, and a foundation that's ready to scale.

Phase 3: GitOps

This phase is a single change with a big impact.

Everything from Phase 2 stays exactly the same — the validation, the manifest generation, the standards. The only thing that changes is the last step. Instead of kubectl apply, you do a git push.

A CD tool like ArgoCD watches the repository and deploys automatically when it sees a change. Every deployment is now a commit. You get a full audit trail for free. Rollback is just a git revert.

What changes from Phase 2: humans are no longer directly touching the cluster.

What you gain: traceability, consistency across environments, and the groundwork for everything that comes next. Git becomes your single source of truth.

Phase 4: IDP and Self-Service

This is where the golden path becomes fully self-service.

Everything underneath is the same as Phase 3. The validation still runs. The manifest is still generated. ArgoCD still deploys. The developer just doesn't see any of it.

Instead, they open a portal, fill in a form with their app name, image tag, team, and environment, and hit deploy. No YAML. No terminal. No kubectl.

The platform carries all the knowledge so the developer doesn't have to.

What changes from Phase 3: the interface. That's it.

What you gain: any developer in your organisation can now deploy safely and correctly, regardless of their Kubernetes experience. The platform enforces everything. The developer just ships.

This is the line from the talk that captures the whole framework best:

The interface changes. The logic doesn't.

From Phase 1 to Phase 4, the core logic is the same. You're just wrapping it in a better interface at each step. That's what makes this approach so practical — you're not rebuilding from scratch at every phase. You're building on what you already have.

Why This Approach Works

Most platform initiatives fail because they try to boil the ocean. They treat a golden path like a destination you reach after eighteen months of development. By the time the platform is "ready," the requirements have changed, the budget is gone, and developers have already moved on to their own shadow IT solutions.

This maturity-first approach flips the script.

You get ROI on Day One

When you start at Phase 1 by just standardizing a single template, you aren't waiting for a portal to be built. You are solving the "copy-paste YAML" problem immediately.

Every hour a developer doesn't spend debugging a bad deployment script is an hour they spend shipping features.

You don't need a UI to prove that value to leadership.

It reduces cognitive load, not just tickets

A common mistake is thinking a golden path is just about automation. It's actually about psychology. When a developer knows there is a "safe" way to deploy, they stop worrying about breaking the cluster. That confidence leads to faster iterations. By building progressively, you lower the barrier to entry for new hires without overwhelming your existing team with a massive new toolset to learn.

It earns developer trust

Developers are naturally skeptical of "mandated" platforms. They've seen too many internal tools that make their lives harder. By evolving your existing scripts into a golden path, you are meeting them where they already live — fixing their current pain points instead of forcing them to adopt a whole new workflow overnight. Trust is built in increments, not in a grand reveal.

Leadership loves predictable growth

For a CTO or VP of Engineering, "we are building a platform" sounds like a high-risk, high-cost gamble. "We are maturing our deployment lifecycle from Phase 1 to Phase 2" sounds like a predictable, measurable improvement. This model gives you a language to speak to leadership that justifies the investment without making impossible promises.

Get Started

Building a golden path isn't about hitting a finish line. It's about building a culture where the right way to work is also the easiest way.

By following a maturity-based approach, you stop treating "The Platform" as a project and start treating it as a product that evolves with your team. You don't need to wait for a massive budget or a greenfield project. You just need to look at your current Phase 0 and decide which piece of logic is worth standardizing today.

The rewards are worth the effort: faster onboarding, fewer outages, and developers who actually enjoy their deployment process.

If you're currently staring at a mess of scripts and wondering how to map out your own Phase 1, I'd love to hear about it — reach out to me on LinkedIn to discuss your platform journey or share what's working for your team.