DEV Community: Kamesh Sampath

Teaching Agents When Not to Act: Agentic DevOps with Snowflake CoCo

Kamesh Sampath — Fri, 03 Jul 2026 06:17:16 +0000

Why the future of DevOps isn’t just about what AI can do, but what it’s authorized to do.

Agentic DevOps is not about teaching agents to act faster. It is about teaching them when not to act.

Like most engineers, I’ve had moments where I trusted automation a little more than I should have.

One of those moments came when I set up an auto-fix bot for a CI pipeline.

It found a race condition and “fixed” it by removing the lock. The deadlock that followed made its way into production and surfaced at 2am on a Friday.

The AI wasn’t really the problem. It did exactly what I asked it to do.

The problem was that I never defined what a safe fix looked like.

And the agent had no way to challenge that assumption.

AI-assisted DevOps is not Agentic DevOps

People use “AI-assisted DevOps” and “Agentic DevOps” interchangeably.

They are not the same thing.

AI-assisted DevOps is a text generator with a human execution layer.

Copilot suggests. You run.
Claude Code proposes. You accept.
The agent generates. The human ships.

Agentic DevOps is an execution model.

Agent scaffolds infrastructure.
Provisions identities.
Scans code, opens issues, proposes fixes.
Cleans up when the work is complete.

You state the outcome. The agent executes the workflow.

That is the pattern I have been exploring withSnowflake CoCo.

CoCo gives me an AI-native execution layer. The CI/CD workflow described here is an implementation pattern:

Use intent to define the desired outcome
Use policy to define the agent’s authority
Use human review when the agent is not confident enough to act.

The important part is not the repository.

The important part is the operating model.

What changed when I applied IDD to CI/CD

I have written before about Intent-Driven Development. The idea is simple: structure prompts around outcomes instead of procedural steps.

The four blocks are:

Goal
Requirements
Constraints
Output

The agent figures out how.

You define the what and the why.

When I started building this CI scaffold, the scan prompt initially said something like:

Find bugs and open issues

Simple. Direct. Not enough.

The agent found bugs.

It opened issues.

It also tried to auto-fix a 400-line authentication module with high confidence and wrong assumptions.

The problem was the missing constraint.

“Fix my bugs” carries an implicit constraint : “Do it safely.”

IDD forces that constraint into the prompt and the workflow.

Not as a comment.

Not as a README note.

As a policy that the agent reads before acting.

The Decision Tree

The constraint became a scoring system.

Before opening any issue or proposing a fix, the scan agent evaluates three dimensions.

SEVERITY : how dangerous the issue is if left unfixed.
COMPLEXITY : how much the fix touches.
CONFIDENCE : how certain the agent is that its fix is correct.

Then it routes the issue.

The governance loop: Mapping an agent’s confidence against its authorized ceiling of action to determine when a human must step in.

conservative (default)

auto-fix only if severity=low AND complexity=low AND confidence=high everything else -> needs-review

aggressive

auto-fix if confidence >= medium

off

never auto-fix, always needs-review

This is not a magic rules engine.

It is a simple judgment model with explicit inputs.

The race-condition fix from my Friday story would have scored:

severity: high
complexity: high
confidence: medium
route: needs-review

A human would have seen the issue, read the context, and decided whether to run the fix.

That is the difference I care about.

An agent that always acts is dangerous.

An agent that never acts is just autocomplete with a workflow file.

The useful middle is an agent that can decide when not to act.

Intent defines the goal. Policy defines the boundary.

Infrastructure-as-Intent does not mean the agent gets unlimited freedom.

It means the desired outcome is expressed clearly enough for the system to act.

But action still needs a boundary.

For this workflow, that boundary lives in configuration.

For example a policy document could be:

fix_mode:
  max_auto: conservative

The intent says what the agent should achieve.

The policy says how far the agent is allowed to go without human review.

Change it through a PR.

The commit history becomes the governance record.

When your security team asks who allowed the agent to auto-fix beyond low-risk changes, you do not need to reconstruct a Slack thread.

You show the PR that changed conservative to aggressive , the reviewer, and the merge date.

That is the record I want Agentic DevOps systems to preserve.

Not just what the agent did.

Who allowed it to act at that level.

For runtime experiments, there can still be a variable override.

Runtime overrides are necessary, but they introduce drift. Every temporary experiment requires a sunset tag and a hard expiration date. If the override outlives its sprint, the agent must revert to the version-controlled default and trigger a CI alert. This is how we ensure ‘temporary’ hacks don’t calcify into unreviewed permanent policy.

Maybe the team wants to try aggressive mode for a sprint. Set the variable. When the experiment ends, delete it. The policy falls back to the reviewed default.

Every scan run should log the active ceiling and its source.

Fix ceiling: conservative
Source: repository configuration

The point is not to slow the agent down.

To make authority visible, embed the active ceiling status directly into your PR templates. Don’t just show the outcome of the agent’s work; show the governance state that authorized it. A persistent ‘Governance Status’ block is the audit evidence your security team actually wants.

This is the engine of CoCo Automations. As agents move from reactive tasks to proactive operations — like daily drift detection — the ‘conservative’ vs. ‘aggressive’ dial becomes our primary safety check. Setting these modes at the repository level ensures that even when the agent operates autonomously, it is tethered to a pre-approved ceiling of authority.

Visibility alone, however, is not a substitute for intervention.

Human override should always exist

The needs-review route is not a dead end.

It is the place where human judgment enters the workflow.

A developer can review the issue, inspect the context, and decide whether the agent should proceed with the fix.

Even for a complex issue.

Even for a low-confidence issue.

The override exists by design.

Agent judgment is the default.

Human judgment is the final decision.

agent proposes -> human decides -> agent executes

A system where the agent can never be overridden is broken.

A system where the agent always acts is also broken.

The override is the acknowledgement that developers know their codebases better than any model.

They should always have the last word.

This is where Agentic DevOps becomes useful.

The agent is not replacing the engineer. It is doing the execution work after the engineer has set the boundary.

ICR did not drop. It got smarter.

The Intent Compression Ratio of this scaffold workflow is 48.

One conversation instruction.

48 distinct state-changing operations.

That number did not change when I added the decision tree.

What changed is that each operation now carries a risk assessment.

The agent that runs 48 operations conservatively is more useful than the agent that runs 48 operations blindly.

The metric I track is not only ICR.

I also track the agent-raised PR merge rate:

The percentage of agent-created PRs that engineers merge without modification.

That number measures trust.

A team at 60%+ merge rate has accepted the agent as part of the workflow.

A team at 10% has a noisy bot.

Same ICR. Different values.

Trust takes time.

The conservative ceiling builds it.

What I mean by Agentic DevOps with CoCo

Not : replace engineers with AI.

Yes : give engineers a CI pipeline that knows its limits.

The value is not faster code generation by itself.

Faster generation without better control creates faster review queues and faster mistakes.

The value is compressing the loop.

Where intent meets infrastructure: Integrating autonomous agents into traditional CI/CD loops.

That compression matters because the agent is calibrated.

It does not open a PR for every needs-review issue.

It does not silently change complex code because the prompt sounded confident.

It does not confuse autonomy with permission.

Agentic DevOps means the agent can say:

I am not sure enough to act on this one.

That is not a limitation.

That is the design.

Why this matters

Most AI engineering conversations still treat autonomy as the finish line.

Can the agent write the code?

Can it open the PR?

Can it deploy the change?

Those are useful questions, but they are not the first questions.

The first question should be:

What has to be true before the agent is allowed to act?

That is where intent becomes operational.

That is where policy becomes part of the development loop.

That is where CoCo, CI, and IDD start to become more useful than a smarter autocomplete.

Agentic DevOps is not about giving agents unlimited power.

It is about giving them:

explicit intent
bounded authority
visible decisions
human override

That is the CI agent I want.

One that knows how to act.

And one that knows when not to.

By making agentic authority visible and configurable, we are not just automating tasks; we are codifying intent. This is the final piece of theIntent-Driven Development puzzle: ensuring that the ‘evidence’ generated by our agents maps directly back to human-authorized intent.

Implementation References

The article describes the Agentic DevOps pattern rather than depending on a single repository. The following public references provide implementation context for the GitHub, GitLab, and Snowflake CoCo workflows discussed in this post.

CoCo skill

Snowflake-Labs/devops-snowflake-coco-agents

CoCo skill implementation for the DevOps agent workflow. This provides the core skill context behind scaffold, scan, issue routing, fix behavior, and the human-review loop.

GitHub template

Snowflake-Labs/github-coco-agent

Template for exploring CoCo-driven Agentic DevOps workflows with GitHub Actions. It shows how the pattern maps to GitHub-native CI/CD, repository configuration, issue routing, and PR-based governance.

GitLab template

snowflake-dev/gitlab-coco-agent

Template for exploring the same Agentic DevOps workflow in GitLab CI. It shows that the pattern is not tied only to GitHub Actions and can be adapted across CI/CD systems.

About the Author

Kamesh Sampath is a Lead Developer Advocate at Snowflake, author, and long-time open-source contributor with 25+ years in enterprise software. He works across data engineering and AI with developer communities, helping practitioners turn modern data platforms into systems that hold up in production.

Through talks, writing, and hands-on demos, Kamesh makes cloud, data, and AI topics easier to understand and apply — grounded in real-world constraints. His sessions mix deep technical detail with practical patterns that developers and data teams can apply right away.

Lately, he’s been speaking about Apache NiFi (Snowflake Openflow), AI (Snowflake Cortex), and PostgreSQL.

He believes technology becomes powerful when it is shared, taught, and built together.

GitHub |LinkedIn |Blog | X

From Intent to Evidence: Why AI Skills Need Tests

Kamesh Sampath — Mon, 22 Jun 2026 13:37:42 +0000

Moving beyond “it worked once” to build verifiable, trustworthy AI agents in real-world developer workflows.

Intent is the source. Evidence is the contract.

AI skills should not be trusted because they worked once. They should be trusted because they can reach the intended state repeatedly, with executable tests proving the outcome.

AI agents are easy to celebrate when they work once, but that is also the trap. The demo runs, the code looks right, the file appears in the right place, and everyone nods. Then you run it again. This time a file is missing, a parameter is read differently, or a command is skipped. The final result is close, but not correct. This is where AI-native development gets real: we do not only need agents that can generate; we need skills that can be trusted.

In my earlier posts on Infrastructure-as-Intent, Intent-Driven Development, Intent Compression Ratio, and token economics, I argued that developers are moving away from writing every step by hand and toward expressing outcomes. Not every command, not every click, and not every line of glue code — just the outcome. But there is a missing layer in that story. Intent needs evidence.

Without evidence, intent is only a better prompt

“It Worked Once” Is Not Engineering

Every developer knows this. We do not trust code because it compiled once, we do not trust infrastructure because one deployment succeeded, and we do not trust an API because one request returned 200 OK. We trust systems when we can verify them again and again.

AI skills should be held to the same standard. Today, many AI workflows still feel like demo-driven development: we run an agent, inspect the output, and if it looks good, we move on. That is fine for exploration, but it is not enough for reusable skills.

A reusable skill needs a stronger question. Not: did the agent produce something useful? But rather: did the system reach the intended state, and can it do that again?

The Gap Between Output and Outcome

There is a gap between a plausible result and a proven result. I call this the Verification Gap. On one hand, the agent gives an answer that looks right. On the other hand, the system is in the correct state. That gap matters because AI agents operate through probabilistic reasoning. The same instruction can lead to different paths, tools, or decisions depending on context and model behavior. Often those paths converge on the same outcome. Sometimes they do not.

The Verification Gap: The difference between an agent generating a plausible output and proving the system actually reached the correct state.

The mistake is not using AI; the mistake is assuming one successful run is proof. A better question is: how do we make non-deterministic behavior measurable enough for developers to work with? An answer that looks right is not the same as a system that is right.

CoCo as the Proving Ground

I encountered this problem firsthand while building skills for Snowflake Cortex Code, also known as CoCo. Because CoCo is an agentic coding assistant that combines reasoning, tool use, and code execution, it acts as a practical environment for testing AI skills in real-world workflows. I needed a way to prove that an agent’s answer wasn’t just plausible, but that the system was actually in the right state. That practical need led me to build inspect-coco.

A CoCo skill can package developer intent into a reusable capability. Instead of asking an agent to reason from scratch every time, we can give it a skill that captures a workflow, convention, tool pattern, or common engineering task. This is the shift I described in Infrastructure-as-Intent: we move from prescribing steps to expressing outcomes, turning a skill into a reusable unit of intent.

That is powerful, but it is also risky if we do not test it. A skill can hide many actions behind one instruction — it may create files, change configurations, call tools, or update a project. The user sees a simple request, and the skill performs the work. This is exactly what I mean by Intent Compression Ratio. A high-value skill compresses many steps into one intent, but high compression requires high confidence. If one intent represents ten steps, we need to know whether all ten steps completed correctly.

The same challenge exists whether you are building skills for CoCo, Claude Code, Codex, Gemini CLI, OpenHands, or any other agentic development environment. Once an agent can inspect repositories, modify files, execute commands, and automate parts of the software lifecycle, the question becomes the same: how do we know it did the right thing?

CoCo exposed the gap early. It became the first proving ground because it offered a practical environment where skills could interact with real projects, tools, and workflows. That made it easier to test a simple question: can intent be translated into repeatable, verifiable outcomes?

inspect-coco bridges the foundational evaluation capabilities of Inspect AI with the concrete execution environment of Snowflake CoCo.

While the pattern is not limited to CoCo, CoCo is the concrete runtime. Inspect AI is the evaluation foundation. inspect-coco connects the two.

What inspect-coco Does ?

inspect-coco is a developer-first test harness for the instructions and workflows that become AI skills. The flow is intentionally familiar:

It uses Inspect AI to run evaluations.
It uses CoCo to execute the skill.
It uses Docker to isolate the environment.
It uses a test script to verify the result.

The core evaluation loop: translating intent into execution, verification, and reliable measurement.

The key is the test. inspect-coco does not ask another model whether the answer looks good; it runs the skill and checks the system state. Did the file exist? Did the content match? Did the command succeed, and did the project end up in the expected shape?

If yes, the test passes; if not, it fails. That is a language developers already understand. If you prefer, the verification layer can also use familiar testing frameworks such as pytest, allowing teams to reuse existing assertions, fixtures, and testing practices instead of learning a new evaluation model. The important idea is not the framework itself, but that the outcome is verified by executable tests rather than judged by appearance.

Three Files

An evaluation can start with just three files. That is intentional; no heavy platform is required to understand the idea.

instruction.md (Markdown) describes the intent.
tests/test.sh (Shell script) verifies the outcome.
task.toml (TOML) tells the evaluation how to run.

A developer-first evaluation requires minimal overhead: intent, verification, and configuration.

The simplest example is the hello-world evaluation in the repository. The instruction asks the skill to create /workspace/hello.txt. The agent can choose its path, but the test objectively checks the result: does the file exist, and does the content match exactly? Exit 0 means pass; anything else means failure.

That is boring in the best possible way.

Unit Tests for Skill Instructions

One design choice in inspect-coco is worth calling out: it does not require starting with a fully packaged skill and running the whole lifecycle end to end. It can start with an instruction. That instruction may later become part of a CoCo skill, a Claude Code command, a Codex workflow, or another agentic environment. But before it becomes reusable automation, the instruction itself should be tested.

A skill is not only code; it is packaged intent. If the intent is vague, the skill will be vague. If the instruction is hard to verify, the skill will be hard to trust. If the constraints are weak, the agent has too much room to improvise. Because of this, inspect-coco treats the instruction as a testable artifact, making it closer to unit testing for instructions than demo testing for agents.

The question is not just whether the complete agent workflow can succeed. The earlier question is whether the instruction is clear, constrained, and verifiable enough to become a reliable skill in the first place. Instruction-level tests tell us if the unit of intent is well-formed before packaging it, while end-to-end tests tell us if the whole workflow actually worked. Both are useful, but they answer fundamentally different questions.

Intent Needs Structure

The instruction follows a simple structure, where each section has a specific job:

Goal: Says what we want.
Requirements: Say what must be true.
Constraints: Reduce unwanted choices.
Output: Tells us what success looks like.

This structure matters because vague intent creates drift. If the instruction is loose, the agent has more room to guess. It may still produce something useful, but useful is not the same as correct. A good instruction narrows the space. It does not remove all non-determinism, but it makes the outcome easier to test.

To enforce this, inspect-coco also includes an IDD-style check for instruction quality, catching weak instructions early. The framework does not only ask if the skill worked; it also asks if the intent was clear enough to test.

Repeated Runs Matter

One passing run is useful; it tells us the skill can work. But it does not tell us whether the skill is reliable. Because AI systems vary between runs, inspect-coco can execute the same task multiple times, measuring results across epochs and reporting pass rates such as pass@k.

Measuring reliability: Because AI systems are non-deterministic, true confidence comes from repeated execution and measuring stability over time.

A skill that passes once out of three is not the same as a skill that passes ten out of ten. Both may look good in a demo, but only one is ready to trust. The goal is not to pretend AI is deterministic; rather, it is to measure how stable the outcome is. This is where evaluation becomes more than a checklist — it becomes a feedback loop.

Why Not Just Use LLM-as-Judge?

LLM-as-judge has value when the output is subjective. A summary may need to be judged for clarity, a response checked for tone, or a support answer reviewed for usefulness.

But many developer tasks are not subjective. A file exists or it does not. A test passes or it does not. A command exits cleanly or it does not. A generated project runs or it does not. For those cases, we should not ask for an opinion; we should run a test.

That is the design choice behind inspect-coco: when the outcome can be verified by the system, the system should verify it.

Why Inspect AI ?

I built inspect-coco on Inspect AI because agent evaluation needs more than prompt scoring. Developer skills often act in a real environment: they create files, modify projects, run commands, and use tools. So, the evaluation framework needs to support real execution.

Inspect AI provides that foundation, giving us tasks, solvers, scorers, sandboxes, repeated runs, and logs. inspect-coco adds a CoCo-specific layer on top:

CoCo execution
CoCo skill scaffolding
IDD instruction checks
Docker-based environments
Snowflake authentication via OAuth, personal access tokens, and JWT
Script-based verification

That is the important split. Inspect AI is the general evaluation foundation, CoCo is the concrete runtime where I am proving the pattern, and inspect-coco connects the two for developers who want to build reliable CoCo skills today.

Developer-First by Design

I wanted this to feel like normal development. Not like a research project, not like a dashboard-first platform, and not like another abstract AI evaluation layer. A developer should be able to look at the files and understand what is happening:

Markdown for intent
Shell for verification
TOML for configuration
Docker for isolation
CLI for execution
Pass or fail for confidence

That is the whole point. AI changes how we build, but it should not remove the habits that made software engineering work.

Evaluation Should Be Close to the Skill

inspect-coco can scaffold evaluations from an existing CoCo plugin, meaning a skill can grow a test right beside it. This is important because testing should not feel like a separate ceremony. If writing evaluations is too far away from writing skills, developers will skip it.

The local evaluation loop: Bringing testing closer to the skill development process prevents evaluation from becoming a skipped ceremony.

The closer the test is to the skill, the more likely it becomes part of the workflow. That is how software teams learned to treat tests, and AI skill development should learn the same lesson.

From Prompt Engineering to Intent Engineering

Prompt engineering taught us how to talk to models. Intent engineering asks a harder question: can we express the outcome clearly enough for a system to act on it? But there is an even more important question after that: can we prove the system reached that outcome?

This is where AI-native development has to mature. The future will not belong to teams that generate the most code; it will belong to teams that can express intent clearly, compress complexity safely, and verify outcomes continuously. CoCo skills give us a way to package intent, and inspect-coco gives us a way to test it. That is the bridge from intent to evidence.

In Intent-Driven Development, the question is not only whether the agent produced something. The better question is whether the system reached the intended state, and whether it can do it again. That is what we should measure, and that is what we should improve. That is what turns AI skills from impressive demos into reliable developer tools.

Intent is the source. Evidence is the contract. Every serious AI skill needs tests.

Demo GIF in the article showing inspect-coco in action.

References and Further Reading

Project

Snowflake and CoCo

My IDD Series

Related Reading

About the Author

Lately, he’s been speaking about Apache NiFi (Snowflake Openflow), AI (Snowflake Cortex), and PostgreSQL.

He believes technology becomes powerful when it is shared, taught, and built together.

GitHub |LinkedIn |Blog | X

ICR and Token Economics

Kamesh Sampath — Wed, 20 May 2026 03:26:00 +0000

Why the hidden cost of low-intent AI systems is not just more tokens, but weaker architecture.

From prompt optimization to intent optimization: higher ICR means more useful work, fewer wasted tokens, and better token economics.

Introduction

We talk a lot about tokens.

Input tokens.

Output tokens.

Context windows.

Pricing.

Latency.

Caching.

But I think we are missing the deeper question.

Why are we using so many tokens in the first place?

Most conversations about token optimization start too late. They focus on trimming prompts, compressing context, caching responses, or choosing cheaper models.

Those things matter.

But they are symptoms.

The deeper issue is often not the model.

It is the interaction architecture.

A weak interaction architecture forces the user to explain too much, repeat too much, clarify too much, and carry too much system knowledge in every request.

That is not just a usability problem.

It is an economic problem.

Because in AI-native systems, tokens are no longer just text.

Tokens are infrastructure.

They affect cost.

They affect latency.

They affect context capacity.

They affect reliability.

They affect how much useful work an AI system can perform before the interaction collapses under its own verbosity.

This is where Intent Compression Ratio (ICR) becomes useful.

In my previous article, Intent Compression Ratio: Measuring the Power of Intent, I looked at ICR as a way to understand how much procedural complexity moves from the developer to the system.

In this post, I want to look at the same idea from another angle:

What happens to token economics when intent is poorly compressed?

The Token Problem Is Not Just Token Count

It is tempting to think token optimization means making prompts shorter.

That is only partially true.

A short prompt can be bad if it is ambiguous.

A long prompt can be necessary if the system has no context.

The real question is simpler:

How many tokens does the system need to faithfully understand and fulfill the user’s intent?

That is the economic unit that matters.

Not prompt length alone.

Not output length alone.

But the total token footprint required to move from intent to outcome.

That includes:

The first prompt.
The clarification turns.
The repeated context.
The hidden system instructions.
The tool call explanations.
The retries.
The corrections.
The familiar “no, that is not what I meant” loop.

When you add all of that up, a small human request can become a very large machine interaction.

I call this token amplification.

Token Amplification

Token amplification happens when compact human intent expands into a disproportionately large token footprint.

For example, a developer may say:

Deploy the payment service with autoscaling and observability.

That is a compact intent.

But in a low-ICR system, the developer may need to explain:

What service means.
Which environment.
Which runtime.
Which scaling rules.
Which observability stack.
Which naming conventions.
Which security constraints.
Which deployment policy.
Which rollback behavior.
Which team standards.

Then the system asks clarifying questions.

Then the user repeats context.

Then the system generates something close, but not quite right.

Then the user corrects it.

By the end, the original intent has become a long token trail.

The system did not use more tokens because it understood more.

It used more tokens because it understood less.

That distinction matters.

ICR as an Economic Signal

For this discussion, I use a deliberately simple version of ICR:

ICR = Intent Fulfilled / Total Tokens

Higher ICR means more useful work completed per token consumed.

Lower ICR means the system burns more tokens to achieve the same outcome.

This makes ICR more than a developer experience idea.

It is also a cost metric.

A high-ICR system does not merely feel better to use.

It is more economical.

It reduces repeated explanations.

It reduces clarification loops.

It reduces redundant context transfer.

It reduces orchestration overhead.

It reduces the probability of retries.

That means token optimization is not only a prompt engineering problem.

It is an intent design problem.

Prompt Verbosity Is Often a Tax on Missing Intent

We have all seen this pattern.

A prompt starts simple.

Then it grows.

“Act as a senior engineer…”

“Follow these conventions…”

“Use this format…”

“Do not forget…”

“Before answering…”

“Assume the following context…”

“Here are the constraints…”

“Here are examples…”

“Here are exceptions to the examples…”

Some of this is useful.

But often, prompt verbosity is compensating for missing system memory, missing context, missing abstractions, or missing intent contracts.

The user is not just expressing the goal.

The user is rebuilding the world around the goal.

Every time.

That is expensive.

Not only in dollars, but in cognition.

The developer becomes responsible for carrying the system model in their head and serializing it into text repeatedly.

That is not intelligence.

That is manual context transport.

Skills Are Where Intent Becomes Executable

Intent by itself is not enough.

If I say:

Deploy the payment service.

The system still needs to know what “deploy” means in my environment.

Does it mean Kubernetes? Snowpark Container Services? Terraform? A CI/CD pipeline? A staging rollout first? Which observability defaults? Which governance rules? Which rollback policy?

This is why skills are becoming such an important primitive in agentic systems.

Claude has Skills. Cortex Code has Skills. Gemini has Gems. Codex has Skills.

Different names, but they point in a similar direction.

A skill is not just a saved prompt.

At its best, a skill is compressed operational knowledge.

It can encode the steps, constraints, conventions, tools, expected output, safety boundaries, and domain-specific judgment required to complete a recurring task.

Without skills, the user has to serialize all of that context into the prompt.

With skills, the system can load the right operational pattern and execute the intent with less repeated explanation.

That is ICR in practice.

The human expression gets smaller, but the executable meaning gets richer.

The Economics of Repetition

The point is not to make every prompt shorter.

The point is to stop repeating what the system should already know.

A lot of token waste does not come from the user’s intent.

It comes from everything wrapped around the intent.

The same architectural context.

The same coding standards.

The same deployment rules.

The same governance constraints.

The same output format.

The same safety instructions.

Again and again.

In a single prompt, this feels harmless.

Across a workflow, it becomes expensive.

Across agents, tools, and retries, it becomes amplification.

This is the economics of repetition.

Low-ICR systems make humans restate context.

High-ICR systems turn repeated context into reusable capability.

That is why skills matter.

A skill is one way to stop paying the repetition tax every time. It packages recurring operational knowledge so the user can express the outcome instead of re-explaining the procedure.

The token savings are not just from fewer words.

They come from moving repeated meaning out of the prompt path and into the system.

But compression creates a second responsibility: transparency.

A Small Experiment: ICR Lab

To make this visible, I built a small open-source Streamlit app called ICR Lab.

The app is intentionally simple. It does not call a live LLM, benchmark models, or try to be an agent framework.

It simulates token behavior across different interaction styles so the economics are easier to see.

The current version compares five modes:

Verbose Prompting : large prompts with repeated context.
Clarification Heavy : multiple rounds of back-and-forth before the system can act.
Context-Aware : the system reuses some context and reduces repeated explanation.
Intent-Optimized : the user expresses compact semantic intent and the system assumes more structured responsibility.
Over-Compressed : the user strips so much context that the system misinterprets the intent, triggering correction rounds that amplify tokens beyond what a verbose prompt would have cost.

The line I keep coming back to is this:

Same intent. Different interaction architecture. Different token economics.

Same intent. Different interaction architecture. Different token economics. ICR Lab visualizes how token amplification changes across interaction modes.

That is the entire point of the demo.

The task does not change.

The model does not change.

The interaction pattern changes.

And the token footprint changes with it.

The fifth mode exists to prevent the wrong takeaway. Over-compression is not optimization. When the system lacks enough shared context to decompress a terse intent, it guesses wrong — and the correction cost exceeds the savings.

ICR is not “make prompts shorter”. It is “make meaning denser”.

The Wrong Lesson: Just Make Prompts Shorter

There is a dangerous interpretation here.

Someone might say:

So high ICR means shorter prompts?

Not exactly.

A short prompt with no shared context is just ambiguity.

A high-ICR interaction is not merely brief.

It is dense.

It carries more meaning per token because the surrounding system understands the domain, constraints, policies, tools, and desired state.

That understanding can come from many places:

Platform profiles.
Reusable skills.
Semantic memory.
Declarative manifests.
Typed tool interfaces.
Repository context.
Organizational conventions.

The point is not to delete words.

The point is to stop repeating what the system should already know.

Compression Without Visibility Is Opacity

There is another trap.

High compression can become dangerous when it hides too much.

If I say:

Deploy the connector.

And the system creates users, roles, policies, network rules, secrets, and tasks behind the scenes, I may get a great experience.

Until something breaks.

Then I need to know what was created, in what order, with what constraints, which step failed, what can be retried safely, and what state was left behind.

This is why ICR must be paired with transparency.

The system should absorb complexity, but not erase visibility.

High ICR without transparency is magic.

High ICR with transparency is engineering.

That distinction matters even more when money enters the picture.

Token-efficient systems that are impossible to debug only move the cost elsewhere.

They save tokens on the happy path and burn engineering hours during failure.

From Prompt Optimization to Intent Optimization

Prompt optimization asks:

How do I phrase this better?

Intent optimization asks:

Why does the user need to phrase this much at all?

That is the deeper shift.

The future of AI systems will not be defined only by larger context windows.

Larger context windows help, but they can also hide bad architecture.

If the system can accept more context, we may simply dump more context into it: more documents, more history, more instructions, more examples, more logs, more everything.

But stuffing the context window is not the same as understanding intent.

A bigger backpack does not make you organized.

It just lets you carry more mess.

The better direction is often not:

Give the model more tokens.

It is:

Require fewer tokens to preserve the same intent.

That is intent optimization.

What ICR Starts to Reveal

Once you look at AI systems through ICR, a few things become obvious.

A verbose prompt is not always rich intent.

Sometimes it is a workaround for missing abstraction.

A clarification loop is not always intelligence.

Sometimes it is the system externalizing its uncertainty back onto the user.

A large context window is not always better architecture.

Sometimes it is a bigger place to store repeated noise.

An agent workflow is not automatically efficient.

Sometimes agents multiply token usage by passing bloated context between themselves.

And a skill is not just automation.

It is a compressed unit of repeatable intent.

That last point matters.

In Intent Driven Development, I argued that developers are not becoming less important because AI can generate code.

They are becoming more important because the bottleneck has moved.

The bottleneck is no longer syntax.

It is clarity of intent.

Token economics reinforces that shift.

A vague developer with AI creates token sprawl.

A precise developer with intent-aware systems creates leverage.

After seeing these patterns, the next question is natural:

Why simulate this instead of using a live LLM?

Why ICR Lab Uses Simulation

ICR Lab v1 uses deterministic simulation instead of live model calls.

That is deliberate.

The goal is not to benchmark OpenAI, Anthropic, Snowflake Cortex, local models, or anything else.

The goal is to isolate the interaction pattern.

If the app used live LLM calls immediately, the conversation would drift into model quality.

Which model is cheaper?

Which model answers better?

Which model counts tokens differently?

Which model follows instructions more reliably?

Those are valid questions.

But they are not the question I wanted to explore.

The question is simpler:

How does interaction architecture change token economics?

Simulation keeps that visible.

Same input.

Same modes.

Same curves.

Same comparison.

No API keys.

No model variance.

No hidden backend.

Live model integrations can come later.

The concept needs to be clear before the instrumentation becomes real.

Closing Thought

The future may not belong to systems that consume the most tokens.

It may belong to systems that need the fewest tokens to understand us correctly.

That does not mean smaller models.

It does not mean shorter prompts.

It does not mean hiding complexity behind magical interfaces.

It means better intent architecture.

Systems that know the context.

Systems that preserve constraints.

Systems that compress repeated patterns.

Systems that expose what they did.

Systems that turn verbose human instruction into transparent machine execution.

That is where ICR and token economics meet.

Not in counting tokens for the sake of counting tokens.

But in asking a better question:

How much useful intent did this system fulfill per token consumed?

That question changes the optimization target.

And once you see it, token cost stops looking like a pricing problem alone.

It starts looking like a design problem.

Try the Demo

I built a small companion app for this post: ICR Lab.

It is open source and runs as a Streamlit app.

The goal is simple:

Enter a task.

Compare interaction modes.

Watch token amplification happen.

Then watch it collapse as intent gets better compressed.

It is not a benchmark.

It is a playground for building intuition.

Because sometimes the best way to understand a systems problem is to make the invisible curve visible.

About the Author

Lately, he’s been speaking about Apache NiFi (Snowflake Openflow), AI (Snowflake Cortex), and PostgreSQL.

He believes technology becomes powerful when it is shared, taught, and built together.

GitHub |LinkedIn |Blog | X

Intent Compression Ratio: Measuring the Power of Intent

Kamesh Sampath — Mon, 11 May 2026 15:40:46 +0000

A way to measure how much procedural complexity moves from developers to AI agents, skills, and intent-driven systems.

From many operations to one intent: visualizing Intent Compression Ratio.

We’ve spent years optimizing execution.

Better APIs.

Better CLIs.

Better automation.

But developers don’t struggle because execution is slow.

They struggle because execution is hidden.

Infrastructure doesn’t fail loudly.

It fails silently — when you get the order or the desired state wrong.

The Problem Isn’t Steps. It’s the Graph

Take a realistic ask:

“I need to deploy a connector with Snowflake Openflow.”

In practice, this often ends with generating a Programmatic Access Token (PAT) to be used by clients like Apache NiPyAPI.

But getting there is not one step.

It’s a graph.

Programmatic Access Token flow as an execution graph: multiple object operations compressed into one intent.

Each node is valid on its own.

But the ordering and desired state are not optional.

A network policy must exist before it’s attached.

An auth policy must reference the correct role.

The PAT must be generated after all constraints are satisfied.

Get this wrong?

No exception.

No obvious failure.

Just a token that doesn’t authenticate.

A connector that doesn’t connect.

And hours spent debugging across identity, network, and auth layers.

Infrastructure is not procedural.

It is relational.

And we’ve been asking developers to execute that relationship manually.

This is also why many AI agent workflows struggle today. We ask agents to operate on systems that still expose too much procedural complexity.

If an agent has to reason through 12 procedural steps to achieve one outcome, the probability of a hallucinated configuration, missed dependency, or ordering error rises quickly.

This is where Intent Compression Ratio(ICR) starts to matter.

High ICR becomes a prerequisite for agentic readiness.

kubectl apply Was a Hint

We’ve seen this pattern before.

kubectl apply didn’t simplify Kubernetes.

It changed the interface.

From:

Step-by-step execution

To intent:

“Here is the desired state.”

And the system:

Builds the dependency graph
Determines ordering
Reconciles toward the desired state

It doesn’t remove complexity.

It absorbs the graph.

But We Never Measured the Absorption

We say:

Intent-driven
AI agents
Skills

But we don’t ask:

How much complexity moved from the developer to the system?

That’s the gap.

Intent Compression Ratio

I’ve started calling this Intent Compression Ratio (ICR):

A way to measure how much execution complexity is collapsed into intent.

To quantify this, we need a way to measure the “work” the system is doing for us.

Ops : Are the total required operations every distinct state-changing API call e.g. IAM binding, network rule, object creation, policy attachment, or validation step.
Intent expressions are the semantic goals defined by the user.

A system with an ICR of 1 is just a wrapper.

A system with an ICR of 10 is an architectural partner.

At first glance, this looks like productivity.

It isn’t.

It measures who owns the system’s complexity.

What Actually Gets Compressed

Not just steps.

Cognitive load

The developer no longer tracks ordering or dependencies.

Dependency graph

The system constructs and executes the DAG.

Desired state

The system ensures correctness of what is being built, not just how.

Failure surface

Fewer invalid states. Fewer silent failures.

Higher ICR doesn’t just mean faster.

It means safer and more predictable systems.

ICR in Action

Let’s ground this with a few examples from the Snowflake platform.

Snowflake is a concrete example but Intent Compression Ratio is the general idea. I use Cortex Code as an Agentic Coding tool proof point.

The pattern is broader than any one platform: whenever developers have to coordinate object creation, dependency ordering, policy enforcement, and desired state, there is an opportunity to compress the graph into intent.

Programmatic Access Token

Intent:

“Deploy a Google connector with Openflow.”

Underlying reality:

Create service user
Create service role
Grant role to user
Create network rule
Create network policy
Attach network policy to user
Create authentication policy
Attach authentication policy to user
Configure PAT policy constraints
Generate PAT
Store PAT securely for client usage
Validate token exchange via Apache NiPyAPI

ICR ≈ 12 Ops / 1 intent = 12

Or expressed as intent compression:

1 intent : 12 Ops

This is not step reduction.

This is dependency graph compression.

Network Access

Intent:

“Allow access to googleapis.com.”

Underlying reality:

Rule definition
Policy creation
Attachment
Grants

ICR ≈ 4 Ops / 1 intent = 4

Or:

1 intent : 4 Ops

External Volume

Intent:

“Set up an Iceberg table with Snowflake.”

Underlying reality:

Storage integration
IAM configuration
External bindings
Volume configuration
Access validation
Table setup

ICR ≈ 6 Ops / 1 intent = 6

Or:

1 intent : 6 Ops

Where Skills and AI Agents Fit

This becomes critical in an AI-driven world.

AI agents are not good at:

Long procedural chains
Fragile ordering
Implicit dependencies

They are good at:

Intent
Bounded abstractions
Repeatable execution units

This is where skills come in.

A skill is not just automation.

It is a compressed execution graph with guarantees.

A skill compresses the execution graph behind a single expression of intent.

Whether implemented via Cortex Code or any other agentic system, a skill:

Encodes ordering
Enforces constraints
Ensures the correct desired state
Handles sensitive steps, such as token handling, safely

So when an agent executes:

“Deploy a connector with Openflow.”

It doesn’t reason through 12 Ops.

It invokes a single semantic unit.

ICR becomes a proxy for how usable a system is for agents.

The Tradeoff

Not all compression is good.

Too Much Compression: Opaque Systems

You run:

“Deploy my connector.”

It works — until it doesn’t.

Now the connector fails to authenticate.

You don’t know which role was created.

You don’t know which policy was applied.

You don’t know if the network rule exists.

You don’t know whether the PAT was generated under the right constraints.

You try to debug.

But there is no visibility into intermediate steps.

There is no way to fix just one part.

Everything is hidden behind the abstraction.

So you abandon it and go manual.

High ICR.

Zero observability.

That is not effective compression.

That is opacity.

Too Little Compression: Fully Manual Systems

Now take the opposite extreme.

You manually create everything:

User
Role
Grants
Network rule
Network policy
Auth policy
Token

Everything is visible.

But now you own everything.

You attach the wrong policy.

Or attach it in the wrong order.

Or bind the wrong role.

Or generate the token before the constraints are actually in place.

Nothing errors.

Your PAT is generated.

Your connector is deployed.

But it doesn’t work.

Debugging spans identity, network, and auth layers.

Full control.

Full responsibility.

That is not simplicity.

That is exposed complexity.

Glass Box Compression

The solution to opacity is not to go back to manual scripts.

It is observable intent.

I think of this as Glass Box Compression.

Unlike a black box, where the system performs magic and leaves you clueless when it breaks, a glass box system absorbs complexity while remaining transparent.

Now consider a well-designed skill:

“Deploy a connector with Openflow.”

It executes the full dependency graph.

It enforces ordering.

It validates desired state.

It exposes what it created.

And when something fails, it tells you:

What failed
Why it failed
What was expected
What can be retried safely

You can inspect the graph.

You can override specific parts.

You can re-run safely.

This is high ICR with visibility.

That is effective compression.

The Shift

We used to optimize for:

Performance
Scale
Reliability

Now we also optimize for:

How much complexity the system can safely absorb.

Because in an intent-driven world:

Developers express goals
AI agents invoke abstractions
Systems execute graphs

The interface moves closer to intent.

The execution moves deeper into the system.

And the value shifts from:

“How many steps can I automate?”

To:

“How much complexity can I safely stop thinking about?”

Closing Thought

The question is no longer:

“What can this system do?”

It is:

“How much of the system can you safely not think about?”

That is the real power of intent.

And that is what Intent Compression Ratio begins to measure.

This brings the series full circle.

In Infrastructure as Intent, I argued that infrastructure is moving from imperative execution toward declarative intent: developers should describe the outcome, not manually orchestrate every step.

In The Ghost in the Machine, I explored why intent still needs structure. We need the “Spirit of UML” — not necessarily UML diagrams themselves, but a way to model relationships, dependencies, and constraints before systems can safely automate them.

In Intent-Driven Development, I looked at the developer shift: from writing every instruction to shaping the intent that AI systems and platforms can execute.

Intent Compression Ratio connects those ideas.

If intent is the interface, and relationships are the hidden graph, then ICR asks:

How much of that graph can the system safely absorb?

That is the bridge from philosophy to measurement.

In the next part of this series, I’ll look at how to build high-ICR skills into custom AI agents.

About the Author

Kamesh Sampath is a Developer Advocate at Snowflake, an author, and a long-time open-source contributor with 25+ years in enterprise software. He works across data engineering and AI with developer communities, helping practitioners turn modern data platforms into systems that hold up in production.

Lately, he has been speaking about Apache NiFi, Snowflake Openflow, Snowflake Cortex, and PostgreSQL.

He believes technology becomes powerful when it is shared, taught, and built together.

GitHub | LinkedIn | Blog | X

Intent Driven Development: The Shift Developers Can’t Ignore

Kamesh Sampath — Fri, 17 Apr 2026 03:46:01 +0000

We spent decades writing code for machines. Now we need to express intent for them.

There’s a quiet anxiety in software development right now.

If AI can write code… where does that leave developers?

The answer is simpler and more uncomfortable than it seems.

Builders are not becoming obsolete.

They are becoming more important.

Because in a world where machines can generate code, the real bottleneck is no longer syntax.

It is clarity of intent.

AI is not replacing developers.

It is amplifying them.

A vague developer with AI produces noise.

A precise developer with AI produces systems.

The leverage has shifted but the responsibility has not.

Developers who can express intent are still in control.

But the interface has changed.

We are no longer just writing code.

We are shaping what systems should do with just pure natural language.

This shift doesn’t come out of nowhere.

My previous article on Infrastructure as Intent, I explored how systems are moving toward declared outcomes over explicit instructions.

In The Spirit of UML, I argued that abstraction is becoming essential again in the age of AI.

In this article, I propose that I ntent D riven D evelopment(IDD) is where these ideas meet in practice.

We’re Still Thinking Like Syntaxists

Even with AI tools everywhere, most developers still:

Write code line by line
Focus on syntax and structure
Debug implementation details

AI becomes:

Faster autocomplete
Better Stack Overflow

But that misses the point.

The real shift is not AI writing code.

It is:

Developers expressing intent clearly enough that machines can realize it.

From Syntax to Semantics

We were trained to be syntaxists.

We think in functions, APIs, and control flow.

Intent Driven Development requires semanticists:

Intent
Constraints
Context
Outcomes

Code doesn’t disappear.

But it stops being the starting point.

Intent becomes the source. Code becomes the artifact.

Writing Better Prompts is Writing Better Software

In this world, prompts are not casual inputs.

They are design artifacts.

Weak:

Build a dashboard

Intent-driven:

Build a Streamlit dashboard using Snowflake TPCH data. Show revenue trends over time, allow filtering by region and customer segment, and generate natural language insights. Keep the UI minimal and ensure queries are efficient.

The difference is precision.

A Real Example: From Intent to App (Cortex + Streamlit)

Development Loop

Start with intent that clearly defines the end outcomes:

Build a Streamlit dashboard using Snowflake TPCH data.

Show revenue trends over time
Allow filtering by region and customer segment
Use Cortex to generate natural language insights
Keep the UI minimal but interactive
Ensure queries are efficient and reusable

No boilerplate. No step-by-step instructions just clear intents.

Using tools like Cortex Code, this intent can translate into:

Snowflake queries over TPCH tables
Aggregations for revenue trends
A Streamlit UI with filters and charts
LLM-generated summaries like: “Revenue dipped in Q3 due to decline in APAC region”

You didn’t handwrite everything.

You guided the system with intent.

Where Developers Still Matter

The first version won’t be perfect.

Maybe the query is inefficient.

Maybe the chart is unclear.

Maybe the insights are generic.

You refine the intent:

Optimize queries for large datasets
Make insights specific to region trends
Add anomaly detection

Each iteration improves the system without dropping to low-level code immediately.

But… How Do You Know It’s Good?

You build something like this and think:

“Looks right… but can I trust it?”

This is where most AI-driven apps fail.

Not in building but in knowing whether they’re accurate and efficient.

Adding a Evaluation Layer with TruLens

Intent Driven Development needs evaluation.

Frameworks like TruLens make this practical.

They help measure whether your system aligns with your intent:

Context Relevance → Is the system finding the right data?
Groundedness → Are insights tied to actual data?
Answer Relevance → Is the system completing its goal?
Plan Quality → Is the agent forming a plan that is optimal for the user’s goal?
Tool Selection → Is the agent choosing the right tools for each task?
Plan Adherence → Does the agent follow its own plan, or drift mid-execution?
Tool Calling → Are tools being called correctly and with the right inputs?
Logical Consistency → Does the agent maintain coherent reasoning across steps?
Execution Efficiency → Is the agent solving the task with minimal steps, cost, and latency?

These aren’t just metrics.

They are checks on your intent and the system’s ability to complete your goals.

The New Loop

Intent Driven Development changes the workflow:

Define intent
Generate system
Measure results
Refine intent

You’re not just debugging code.

You’re debugging meaning.

Why This Matters Now

AI can act on intent
Systems are too complex to handcraft
The bottleneck is idea → implementation

IDD reduces that gap.

It lets developers operate closer to the problem than the plumbing.

The Developer Shift

The next generation of developers will:

Express constraints clearly
Structure intent precisely
Guide AI systems
Measure quality continuously

The best developers won’t write the most code.

They’ll express the clearest intent and validate it.

Closing

We taught machines to understand syntax.

Now we need to lead them with intent.

AI hasn’t taken control away from developers.

It has raised the bar for what control means.

The developers who thrive in this shift won’t be the ones who write the most code.

They’ll be the ones who can:

express intent clearly
define constraints precisely
And validate outcomes rigorously

Because in this new model, code is no longer the source of truth.

Intent is.

AI is powerful but it is still just an executor.

The developer remains the authority.

And like any good system, the quality of the output will always depend on the clarity of the input.

It starts with something simple.

The next intent you write.

Resources

If you want to explore Intent Driven Development hands-on, here are a few starting points.

Try the Intent Yourself

Use this prompt with Cortex Code (or a similar tool) and iterate on it:

Intent Prompt:

Build a Streamlit dashboard using Snowflake TPCH data.

Requirements:

Show revenue trends over time
Allow filtering by region and customer segment
Use semantic queries over Snowflake tables
Generate natural language insights for trends and anomalies
Ensure all insights are grounded in actual query results
Keep the UI minimal, fast, and interactive

Constraints:

Optimize queries for large datasets
Structure the code for reuse and clarity
Return both data visualizations and textual summaries

Output:

Streamlit app code
Snowflake queries used
Example insights generated from the data

Start with this, then refine the intent.

That’s the practice: improving the clarity of what you want the system to do.

👉 Check out the TruLens cheatsheet: https://github.com/Snowflake-Labs/sf-cheatsheets/blob/main/rag-evaluation-cheatsheet.md

Tools

The Ghost in the Machine: Why AI Needs the Spirit of UML

Kamesh Sampath — Sat, 28 Feb 2026 13:48:19 +0000

Introduction

AI can generate code at 10x speed. It still doesn’t understand the systems it’s building.

This story is about why we need to revive the spirit ofUnified Modelling Language( UML ) — not as diagrams, but as living infrastructure.

Back then, the hot trend wasModel-Driven Development (MDD) — generating code from models. UML was the flagship: Class Diagrams, Sequence Diagrams, the whole catalog. The belief was simple: if the model was correct, the system would be correct.

We were wrong.

UML didn’t fail because modeling was useless. It failed because of the Sync Tax.

You draw a class diagram. Rational Rose generates clean code. You ship it. QA finds bugs. Hotfixes go in. Production diverges. The model is now stale.

Keeping diagrams and code aligned becomes real work. Expensive work. The tax exceeds the value. So teams stop paying it.

We declared, “The code is the documentation,” and moved on.

Starting to sound familiar?

If you ask me, we are back in the MDD era. We’ve just rebranded it as AI-Driven Development.

The diagram is gone. The model remains. Only now, instead of Sync Tax , we pay the Hallucination Tax.

AI agents generate code at unprecedented speed. Ask, “Generate a Snowflake pipeline to ingest customer events,” and it produces SQL, tasks, maybe even a clean dbt model in seconds.

But the real question is: Does it understand the system it is operating in?

Specifically, when you give it that prompt, does it see the whole picture?

Your warehouse sizing and cost guardrails?
Your RBAC model and masking policies?
Which schema is production-certified versus experimental?
Which data is PII and which roles are allowed to see it?

Then there’s the Context Gap — the ritual of re-explaining your infrastructure to an agent with limited working memory. Every new chat resets shared understanding. Yes, there are hacks. But none of them feel native.

The spirit of modeling was never about diagrams. It was about shared context.

AI has the muscle. It lacks shared context.

So here’s the twist: UML and AI are two halves of a broken whole.

UML was too rigid for humans to maintain. AI is too fluid for systems to trust.

To solve theField Velocity problem, we don’t go back to 2005.

We make AI pay the Sync Tax for us.

And the only way to do that is to encode intent as an enforceable structure.

The Living Model: From “Pretty Picture” to “Policy”

Operationalizing Human Intent: Our conceptual model for combining an AI Agent with a central knowledge repository of Architectural Memory to enforce security, governance, and infrastructure best practices for automated and compliant code generation.”

In my earlier blog on I nfrastructure-as-Intent, I wrote about how human expertise and experience can be carved into agentic Skills (In Part 1, we defined Skills as the codified expertise and human experience that turn operators into mentors for AI agents.) ** ** , turning each of us from operators into mentors.

That shift forces us to stop authoring static models and start inferring living ones.

The truth of your system already exists — scattered across scripts, RBAC definitions, masking policies, and CI/CD pipelines.

So why does AI still hallucinate?

Because of the Context Gap — the absence of shared context.

Here is how we revive the UML spirit: by turning Skills into the rules of engagement for our AI agents.

The Guardrail Effect

When we practiced UML, we followed the structure. First system architecture. Then sequence diagrams. Then class diagrams. Step by step, we built a shared understanding.

In the AI era, shared understanding becomes shared constraints.

Instead of executing a prompt literally, the AI must query the Skills.

If a Skill declares, “All customer data requires masking primitive X,” that declaration becomes an enforceable structure. The agent generates within it, not around it. The constraint is applied at the point of creation.

That is the spirit of UML — reborn as an executable policy.

Policy is what machines can execute. Intent is what they can reason about.

Semantic Reasoning

Shared constraints unlock something UML never could: semantic reasoning.

UML provided syntactic structure. The reasoning lived in our heads.

AI can reason — but only if we give it structure.

Ask, “If I change this stream’s schema, who breaks?”

The AI uses the Skill graph to simulate impact and produce a blast-radius report before a single ALTER TABLE runs.

That is the difference between generation and understanding.

This is where intent stops being documentation and starts becoming infrastructure.

Closing the Context Gap

In my earlier post, I introduced the idea of a Shared Manifest — a persistent memory layer for AI agents. Not as a long prompt, but as structured Skills.

With Skills, the agent carries Architectural Memory.

It knows your certified schemas. Your cost guardrails. Your intent. Because it operates inside the same world model as you.

The Living Model is not documentation. It is infrastructure.

This is Infrastructure-as-Intent in action.

We use the rigidity of the model to keep AI honest.

We use the fluidity of AI to pay the Sync Tax — bringing the spirit of UML back as living infrastructure.

A Call to Builders

We don’t need a return to 2005-era static diagrams. We need an Architectural Memory for our systems.

Agentic AI tools have already defined the schema. Don’t panic about learning another DSL 🤣. It’s not one.

It’s a disciplined Markdown. Clear English. Proper structure. The same rigor you apply when writing a design doc.

Except now, that document is executable.

It’s not documentation for humans alone. It is codified wisdom for agents.

The era of blind velocity is over.

It’s time to give the muscle a brain.

Visuals note: The conceptual diagrams in this post were generated by Google Gemini.

About the Author

Lately, he’s been speaking about Apache NiFi (Snowflake Openflow), AI (Snowflake Cortex), and PostgreSQL.

He believes technology becomes powerful when it is shared, taught, and built together.

GitHub |LinkedIn |Blog | X

Infrastructure-as-Intent: The Field Velocity Blueprint

Kamesh Sampath — Mon, 16 Feb 2026 16:45:10 +0000

Infrastructure-as-Intent: Bridging human creativity with AI precision to accelerate data value.

We ’ve spent years building automation pipelines to deploy infrastructure. We call it Infrastructure-as-Code (IaC) — whether that’s Terraform, Ansible, or even simple CLI scripts. It saves us from nightmares when we need consistency between where we develop and where we deploy. Development == Production.

But when Snowflake launched Cortex Code — an AI coding agent designed for the Data Cloud — it sparked a new thought for me: why can’t we take IaC automation into its next phase? Instead of declaring how to build everything step by step, what if we simply declared the outcome we want?

This idea wasn’t born out of nowhere. It came from my own attempts to modernize my scripts — snow-utils — a collection of utilities I built to repeatedly set up Snowflake environments for demos. They started as a way to eliminate the rinse-and-repeat plumbing work.

The scripts solved the plumbing problem. But they introduced a different one: remembering the right combination of parameters every single time.

That frustration led me to a deeper pattern — one that could fundamentally change how we interact with cloud environments, and one that fits naturally into this new AI-assisted era.

The “Field Velocity” Problem

So we found the problem, built a solution, and then refined it further — something that could take us into the next phase. But before we get there, let’s pause and ask: why does this even matter?

The answer is simple and universal: speed.

We invented computers to help us do things faster. But as computer science evolved, the systems we built became increasingly complex. Somewhere along the way, speed started fighting complexity.

That’s what I call the Field Velocity problem.

What is it exactly?

It’s the time it takes to go from a customer asking:

Can we query Snowflake-managed Iceberg tables from DuckDB?

to a working, deployed proof of concept.

If you look closely at the history of Infrastructure-as-Code, you’ll notice something consistent. The bottleneck is rarely the idea. It’s the plumbing.

So our customer’s question will translate into following tasks(plumbing):

Configuring IAM roles – > Editing trust policies – > Setting up buckets. – > Creating network rules – > DDL Statements.

If I put it mathematically, we spend 90% of our time on plumbing and only 10% on the actual value.

So the fix? Invert the ratio.

That inversion is what I call Infrastructure-as-Intent.

Same wine. New bottle. 😉

Field Velocity: Flipping the ratio with Infrastructure as Intent

The Evolution: From Power CLI to Atomic Primitives

I come from a strong Linux and infrastructure background, and I’m a die-hard CLI person. If something is repetitive, my instinct is to script it.

That “Power CLI” reflex is what led me to write snow-utils.

Let me explain with a simple example.

Say I want to connect Snowflake to an AWS S3 bucket. Sounds straightforward. In reality, it looks like this:

Log into AWS Console → Create S3 bucket
Create an IAM policy
Create an IAM role and attach the policy
Log into Snowflake → Create the EXTERNAL VOLUME
Run DESCRIBE EXTERNAL VOLUME to hunt for AWS user ARN
Go back to AWS → Edit the Trust Relationship JSON and paste those values

And yes — to err is human. But one small typo in that trust policy JSON, or grabbing the wrong property from the Snowflake DESCRIBE output, can cost you hours of debugging before everything finally works.

That’s when I decided to wrap this entire chain of interdependent steps into a single command.

Instead of manually wiring everything together, I created a primitive that understood the sequence.

Now, for the same task, it’s simply:

snow-utils extvolume:up

Done.

One clean shot. Goal scored.

Over time, it became deterministic. It worked every time — no manual hunting, no fragile copy-paste, no mysterious “Access Denied” errors.

That’s when I realized something important: these weren’t just scripts anymore. They were atomic primitives.

Deconstruction: Moving from brittle, monolithic scripts to composable, reusable primitives.

The Shift: From Operator to Mentor

Once I had the primitives working reliably, I started asking a new question: why can’t this be more intelligent?

Around that time, Snowflake announced Cortex Code. What caught my attention wasn’t just that it was an AI coding agent, but the concept of Skills — giving the agent structured capabilities to orchestrate tools and perform tasks in a more contextual way.

That’s when I saw the transition from Infrastructure-as-Code (IaC) to Infrastructure-as-Intent (IaI) taking shape.

Instead of keeping a mental checklist of parameters, flags, expiration rules, and policy combinations, why couldn’t I just express the intent and let the system handle the wiring?

“Set up a Snowflake External Volume.”

“Create a Snowflake PAT.”

It suddenly felt possible.

But this wasn’t just about making things simpler. It was about changing roles.

With IaC, I was the operator — running commands, supplying parameters, ensuring everything aligned.

With IaI, I started becoming the mentor.

I wasn’t just teaching syntax. I was embedding experience.

Years of small production scars. Lessons from silent authentication failures. The instinct to validate IAM structures before applying them. The habit of forcing a dry run before destructive changes. The discipline of demanding explicit confirmation when something could break environments.

That’s what I call codifying wisdom — capturing operational judgment and passing it into the system so it behaves the way I would.

Let’s not fear AI. It doesn’t invent wisdom on its own — it executes what we mentor it to do.

Intelligence still comes from somewhere.

Let me make this concrete.

Suppose I want to create a Snowflake Programmatic Access Token (PAT). Traditionally, that involves multiple careful steps: creating a service user, applying network restrictions, attaching the correct policies, defining expiration windows, and ensuring everything aligns securely.

Instead of remembering every option and edge case, I wrote a simple markdown file called SKILLS and embedded all the logic behind the PAT primitive into it — including the commands and my opinions about how it should be done safely.

Now all I need to tell Cortex Code is:

“Set me up a PAT that’s valid for 7 days and expires 30 days from now.”

That’s it.

No more remembering long chains of parameters. No wrappers around wrappers. No cognitive overhead.

What emerged was a composable architecture — where the logic (snow-utils) is unbundled from the interface (Cortex Code).

The primitives remain solid.
The skills carry the experience.
The interface interprets intent.

That separation is what turns automation into intent.

The New Workflow: The Human defines the “Skill” (intent & guardrails), and the AI handles the execution.

The Pedagogical Pattern: Intentional Failure as a Feature

When I spoke about intent in the previous section, I didn’t just mean my intent to run a set of commands or automation scripts. I had another intent in mind as well.

Too many intents? I can already see an “Intent Overflow Exception” coming. 😉

Jokes apart — as a Power CLI user, clear intent helps me execute tasks efficiently. But a mentor does more than execute. A mentor creates situations where you apply your skills across related areas. A mentor shapes understanding.

As a Developer Advocate, I build demos, tutorials, and utilities like snow-utils. I usually share my GitHub repositories with a README explaining how to run the demo you just saw in my session.

That works.

But nothing matches personalized teaching — where the session leader’s experience, empathy, and small nuances surface naturally. The little warnings. The subtle explanations. The “watch this part carefully” moments.

That’s a classic pedagogical pattern.

Let me explain with the Horizontal Iceberg REST Catalog (HIRC) and DuckDB demo.

All you have to say now is:

“Deploy HIRC DuckDB demo.”

But the real power isn’t just in deploying the environment. It’s in how the Skill is structured.

I intentionally embedded a learning flow into it:

Say what you are going to do
Do it
Let it fail
Explain why it failed
Fix it
Show how it succeeds

That pattern is deliberate.

The goal isn’t just automation. It’s understanding.

That’s the human part — and that doesn’t emerge automatically. You have to mentor it into the system.

Now the Skill doesn’t just deploy infrastructure. It delivers a learning experience.

The “Recorder” Bonus: Intelligent & Interactive Reproduction

Context Adaptation: Transforming a manifest from “Works on my machine” (AWS) to “Works on your cloud” (Azure)

Since I’m documenting intent as a Skill, the natural next step was to share it.

That’s how we normally share knowledge — notes, blogs like this, repositories, configuration files.

So I asked myself: what if I wanted my product team to reproduce the exact same demo? Or reproduce a tricky bug?

In a classic IaC world, I would share a configuration file. The other person would clone the repo, pull it down, and then start adjusting dozens of environment settings — region values, parameter overrides, warehouse sizes, credentials — before they could even run it.

It works, but it assumes their environment matches mine.

Now think about how humans do this.

If I explain a setup to you, you don’t copy it blindly. You adapt it to your context. You change the region. You adjust the resource sizes. You map it to your naming conventions. You interpret the intent, not just the configuration.

That’s exactly what the IaI pattern enables with shared manifests.

When I share a manifest, I’m not just sharing a file. I’m sharing intent.

That intent can then be materialized in your environment — with adaptation.

Same architectural goal. Different context.

This isn’t blind automation like traditional IaC. It’s careful intent materialization.

And that difference is subtle — but important.

The Future: The Velocity Catalog

What we’ve discussed so far is not just another pattern or a personal experiment. It’s a mental shift we need to make in this AI era.

snow-utils is simply an example of how this shift took shape for me. The same idea can apply to any tool or script.

The real opportunity is this: we need to mentor AI to think and act the way we would — embedding our experience, judgment, and operational discipline into it.

Maybe that’s the next step for us: inject intent into DevOps.

Resources

The HIRC demo is just one example. Here is the 4-minute time-lapse demo that shows how this pattern works:

The same pattern applies across workloads. In the demo repository, I’ve also included the AI Smart Crowd Counter.

All you have to say is:

“Deploy AI Smart Crowd Counter.”

The underlying primitives remain the same. The intent changes.

One moment you’re setting up a data catalog. The next, you’re provisioning a Streamlit app and an image-processing pipeline.

Same intent model. Different outcome.

That’s the leverage.

If you’re experimenting with AI agents in your DevOps workflows, start by stabilizing your primitives. Once they’re solid, you can begin layering Skills on top of them.

And from there, you can build your own Velocity Catalog.

For reference, the underlying orchestration layer used here is Snowflake Cortex Code:

👉 Snowflake Cortex Code CLI Documentation

Visuals note: The conceptual diagrams in this post were generated by Google Gemini.

About the Author

Lately, he’s been speaking about Apache NiFi (Snowflake Openflow), AI (Snowflake Cortex), and PostgreSQL.

He believes technology becomes powerful when it is shared, taught, and built together.

GitHub |LinkedIn |Blog | X

[Boost]

Kamesh Sampath — Thu, 27 Feb 2025 18:27:32 +0000

Kamesh Sampath

Feb 25 '25

Let's Build Together: A Local Playground for Apache Polaris

#apachepolaris #snowflake #kubernetes #localstack

8 min read

Let's Build Together: A Local Playground for Apache Polaris

Kamesh Sampath — Tue, 25 Feb 2025 04:01:17 +0000

Why I Built a Developer-First Apache Polaris Starter Kit ?

As builders, we all know the pain of setting up a new development environment. Hours spent configuring dependencies, troubleshooting integration issues, and getting different services to play nicely together. When I started working with Apache Polaris, I faced these same challenges – and decided to do something about it.

The Challenge: Getting Started with Apache Polaris

Apache Polaris is a powerful open source Iceberg REST catalog implementation, originally contributed to the Apache Software Foundation by Snowflake. This donation to open source has made enterprise-grade data catalog capabilities accessible to the broader data community via simple REST APIs.

Setting up Polaris in a development environment can be challenging. You need:

A robust container orchestration platform
A working metastore (typically PostgreSQL)
S3-compatible storage
Various security configurations and credentials

Each of these components requires careful setup and configuration. For builders just getting started or wanting to experiment with Polaris, this overhead can be a significant barrier.

The Solution: A Complete Development Environment

This is why I created an open source starter kit that provides everything needed to get Polaris up and running in a local development environment. The project follows the true spirit of open source collaboration, building upon and integrating with other excellent open source tools in the ecosystem.

The kit automates the setup of:

A lightweight k3s Kubernetes cluster using k3d
LocalStack for AWS S3 emulation
PostgreSQL metastore with proper configurations
All necessary security credentials and configurations

A key aspect of this starter kit is its comprehensive automation using Ansible. The polaris-forge-setup directory houses Ansible playbooks that:

Automate the entire setup process
Verify if components are ready for use
Handle catalog setup and configuration
Provide cleanup capabilities for development iterations
Enable smooth transitions to higher environments

This automation-first approach serves two purposes:

Immediate Development: Developers can get started quickly with minimal manual intervention
Production Readiness: The Ansible scripts serve as a template for scaling to higher environments, making it easier to adapt the setup for staging or production use cases

By keeping everything open source and focusing on community-driven development, we ensure that builders can:

Learn from the implementation
Customize for their specific needs
Contribute improvements back to the community
Build upon a foundation of trusted open source tools

What is Snowflake OpenCatalog?

Snowflake OpenCatalog is an enterprise-grade implementation and managed service of upstream Polaris, making it incredibly easy to integrate with your existing data stack. By handling the operational complexities of running Polaris at scale, it allows teams to focus on their data applications:

Managed Infrastructure: Snowflake handles all operational aspects including:
- Polaris server management and scaling
- Security and access control
- High availability and reliability
- Regular updates and maintenance
Enterprise Integration: Seamless connectivity with:
- Snowflake's ecosystem of data services
- Popular query engines and tools
- Existing data governance frameworks
- Enterprise security systems
Production-Ready Features:
- Advanced access controls and auditing
- Cross-region and cross-cloud support
- Enterprise-grade SLAs
- Professional support

From Local Development to Enterprise Scale

This starter kit provides an ideal path for builders working with Apache Polaris and considering OpenCatalog for production deployment. By working with the upstream version in this development environment, you:

Gain hands-on experience with core concepts
Understand the underlying architecture
Can prototype and test implementations
Build expertise that transfers to OpenCatalog
Have a clear path to production scaling

When you're ready to move to production, the concepts and patterns you've learned here will help you make the most of OpenCatalog's enterprise capabilities while Snowflake handles the operational complexity.

Technical Design Decisions

Why Kubernetes with k3s and k3d?

While Docker Compose is often the go-to choice for local development environments, Apache Polaris's distributed nature benefits significantly from Kubernetes's capabilities. Here's why:

Advanced Networking: Kubernetes provides sophisticated networking between components:
- Automatic service discovery and DNS resolution
- Internal load balancing for scalable services
- Ingress management for external access
- Network policies for traffic control
Declarative Configuration: Using tools like Helm and Kustomize, we can:
- Maintain separate configurations for different environments
- Version control our infrastructure setup
- Apply consistent changes across deployments
- Manage complex dependencies between services
Reliable State Management:
- StatefulSets for databases and stateful services
- PersistentVolumes for durable storage
- Backup and restore capabilities
- Data replication when needed
Security and Configuration:
- Native secrets management
- Role-Based Access Control (RBAC)
- ConfigMaps for configuration management
- Service accounts for component authentication
Production Readiness:
- Same tools and patterns used in production
- Easy scaling of components
- Built-in monitoring and logging
- Consistent behavior across environments

I specifically chose k3s because it's lightweight and perfect for development environments. Using k3d allows us to run k3s in Docker containers, making it even more convenient for local development. It provides a full Kubernetes experience without the resource overhead of something like minikube.

LocalStack for S3 Integration

While we could have required developers to use actual AWS S3, LocalStack provides a perfect local alternative. It emulates AWS services locally, which means:

No cloud costs during development
No need for AWS credentials
Faster development cycles
Ability to work offline

PostgreSQL as the Metastore

PostgreSQL was a natural choice for the metastore. It's:

Well-documented and widely used
Easy to containerize
Highly reliable
Supported out of the box by Polaris

Kustomize for Deployment Management

Kustomize allows us to manage Kubernetes manifests in a clean, declarative way. It makes it easy to:

Maintain different configurations for different environments
Override settings without modifying base configurations
Keep configurations DRY and maintainable

Getting Started

Let me walk you through how to get up and running with this starter kit.

Ensure you have the prerequisites installed:

# Required tools and their version checks:

# Docker (Desktop or Engine)
docker --version
# Example output: Docker version 24.0.7

# Kubernetes CLI
kubectl version --client
# Example output: Client Version: v1.28.2

# k3d (>= 5.0.0)
k3d version
# Example output: k3d version v5.6.0

# Python (>= 3.11)
python --version
# Example output: Python 3.12.1

# uv (Python packaging tool)
uv --version
# Example output: uv 0.1.12

# Task
task --version
# Example output: Task version: v3.34.1

# LocalStack (>= 3.0.0)
localstack --version
# Example output: 3.0.0

Sign-up for Localstack

Initial Setup

Clone the repository and set up your environment:

git clone https://github.com/snowflake-labs/polaris-local-forge
cd polaris-local-forge

# Set up environment variables
export PROJECT_HOME="$PWD"
export KUBECONFIG="$PWD/.kube/config"
export K3D_CLUSTER_NAME=polaris-local-forge
export K3S_VERSION=v1.32.1-k3s1
export FEATURES_DIR="$PWD/k8s"

Python Environment Setup

# Install uv
pip install uv

# Set up Python environment
uv python pin 3.12
uv venv
source .venv/bin/activate  # On Unix-like systems
uv sync

Deploy the Environment

The setup process is automated through several scripts:

# Generate required sensitive files
$PROJECT_HOME/polaris-forge-setup/prepare.yml

# Create and set up the cluster
$PROJECT_HOME/bin/setup.sh

# Wait for deployments to be ready
$PROJECT_HOME/polaris-forge-setup/cluster_checks.yml --tags namespace,postgresql,localstack

Deploy Polaris

This is where things get interesting - deploying Polaris itself. You have two options for the container images:

Option 1: Use Pre-built Images

Apache Polaris doesn't currently publish official images, but you can use our pre-built images with PostgreSQL dependencies:

docker pull ghcr.io/snowflake-labs/polaris-local-forge/apache-polaris-server-pgsql
docker pull ghcr.io/snowflake-labs/polaris-local-forge/apache-polaris-admin-tool-pgsql

Option 2: Build Images Locally

Alternatively, you can build the images from source:

# Update IMAGE_REGISTRY in Taskfile.yml, then run:
task images

If you choose to build locally, remember to update the image references in:

k8s/polaris/deployment.yaml
k8s/polaris/bootstrap.yaml
k8s/polaris/purge.yaml

Deploy and Verify

Apply the Kubernetes manifests:

# Apply Polaris manifests
kubectl apply -k $PROJECT_HOME/k8s/polaris

# Verify deployments and jobs
$PROJECT_HOME/polaris-forge-setup/cluster_checks.yml --tags polaris

Setting Up Your First Catalog

Before creating your first catalog, configure your AWS environment variables:

export AWS_ENDPOINT_URL=http://localstack.localstack:14566
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_REGION=us-east-1

# Run the catalog setup
$PROJECT_HOME/polaris-forge-setup/catalog_setup.yml

Pro Tip: You can customize the default catalog settings by modifying values in polaris-forge-setup/defaults/main.yml. This file contains configurable parameters for your catalog, principal roles, and permissions.

Play with the Catalog

Once your catalog is set up, you can explore its functionality using the provided Jupyter notebook. The notebook notebooks/verify_setup.ipynb walks you through:

Creating a namespace
Defining a table
Inserting sample data
Verifying data storage in LocalStack

This hands-on exploration helps you understand how Polaris integrates with:

The PostgreSQL metastore for catalog management
LocalStack's S3 emulation for data storage
The overall Apache Iceberg table format structure

You can visually verify your setup by checking the LocalStack console at https://app.localstack.cloud/inst/default/resources/s3/polardb, where you'll see:

Catalog storage structure
Metadata files
Actual data files

Video Walkthrough

For a detailed visual guide of setting up and using this development environment, check out my walkthrough video:

This video demonstrates the entire process from initial setup to running your first queries.

Troubleshooting Tips

If you run into issues, here are some helpful commands for debugging:

# Check Polaris server logs
kubectl logs -f -n polaris deployment/polaris

# Check PostgreSQL logs
kubectl logs -f -n polaris statefulset/postgresql

# Check LocalStack logs
kubectl logs -f -n localstack deployment/localstack

# Check events in the polaris namespace
kubectl get events -n polaris --sort-by='.lastTimestamp'

The Impact: Streamlined Development Experience

With this starter kit, what used to take days of setup and configuration now takes minutes. Builders can focus on creating and experimenting with Polaris rather than wrestling with infrastructure setup.

The kit is open source and available on GitHub. I welcome contributions and feedback from the community. Together, we can make the development experience even better for everyone working with Apache Polaris.

Building should be about creating, not configuring. This starter kit aims to remove the friction from getting started with Apache Polaris, allowing builders to focus on what matters most – creating great applications.

Dont forget to check another project where I used this starter kit https://github.com/kameshsampath/balloon-popper-demo.

Related Projects and Tools

Core Components

Apache Polaris - Data Catalog and Governance Platform
PyIceberg - Python library for Apache Iceberg
LocalStack - AWS Cloud Service Emulator
k3d - k3s in Docker
k3s - Lightweight Kubernetes Distribution
Ansible - Automation Platform

Development Tools

Docker - Container Platform
Kubernetes - Container Orchestration
Helm - Kubernetes Package Manager
kubectl - Kubernetes CLI
uv - Python Packaging Tool

Elements of Event Driven Architecture(EDA)

Kamesh Sampath — Wed, 08 Nov 2023 08:06:04 +0000

Well, we encounter lots of data in our everyday life e.g. weather reports, flight timings, food deliveries etc., All these data are continuous and flowing from variety of sources. Such continuously flowing data is called a Data Stream.

A raw data(stream) is useless for any application unless it has some identifiable element to it.

Let me explain it with few examples, Joe joined Acme crop as a Developer on 25 October 2023, Mini placed an order for two pizzas at 12:00 PM. In these data examples if we take out the verbs(actions) "joined" and "ordered" along with the time "25 Oct" and "12:00 PM" , the whole data become useless.

For an application to effectively use the data stream, the data in the stream should be an event, a data that has time and action associated with it. A stream of data as events brings a great value to applications via analytics, triggers, chaining etc.,

With Mini placed an order for two pizzas at 12:00 PM event, we could extract the following analytical information,

Is Mini placing order for two pizzas everyday?
Is the order placed at 12:00PM everyday?
Is the order is delivered by the same food chain?
What pizzas were ordered?
Are food chains delivering the orders on time ?

The process of using the events and building such analytical information is called Data Processing. Data Processing could be done by a human, an application, an IoT device etc.,

With any data streaming scenario there is always two essential primitive entities,

Event Producer is someone or something that produces an event in our example above Mini is event producer who places the order for pizza.

Event Consumer is again someone or something that consumes or uses the event produced by the Event Producer. Taking the same Mini's order example there could be Pizza house that takes, processes and delivers the order.

As we understood the Event Producers produces events that are being consumed by Event Consumers, software architectures started to evolve around building applications that act as an Event Producer/Consumer and leverage these streams of events. An architectural style of building applications around events is called an Event Driven Architecture(EDA).

When building applications using EDA, enforces few basic requirements on the Platform and a software Framework that will be used to build such applications.

The platform that will be used to build the such applications need to be:

Scalable - as events are continuously flowing there could be sudden spikes to number of events that might come in, the platform need to be scalable or elastic to handle such spikes
Durable - as events can be consumed immediately or bit late in time, the platform should support a mechanism of delivering events durably at time of need
Resilient - The platform should be capable of handling failures and recovering from them without data loss
Data Retention - Retaining of data until a configurable amount time
Responding to Events - The platform should also be able to respond to events at bare minimum acknowledge the event on receive
Ordering - As events are associated with time, ordering of the events helps consumers who need to process them in specific order e.g within a time range or date range etc.,

A sole platform alone might not be enough to build an effective EDA styled application. There is greater need for an integration into the platform via plugins, API etc., In other words, a framework that is extensible and pluggable, and works on common semantics.

The framework should support:

Data Sources - the sources from where the events are generated i.e. Event Producers
Data Sinks - the destinations where the processed event is drained into i.e. Event Consumers
API - An interface to connect and work with the platform, data sources and data sinks.

Some great of Data Streaming platforms,

Apache Kafka - Developed at LinkedIn and Opensources to Apache Software Foundation
Redpanda - is a simple, powerful, and cost-efficient streaming data platform that is compatible with Kafka® APIs while eliminating Kafka complexity.

Apache Kafka also supports processing of streaming data using its ecosystem of Streaming API and ksqlDB. But for an effective architecture, it is always nice to have the core data streaming and data processing to be decoupled(Separation of Concerns). Such decoupling helps in processing data from heterogeneous sources e.g. Apache Kafka, Database, File System CSV files etc.,

Apache Flink is one such framework and distributed processing engine for stateful computations over unbounded(Apache Kafka) and bounded data streams(Database).

Just to summarise we learnt,

What is a Data Stream and an Event
What is a Data Producer and Data Consumer
An architecture style that is used to build application around Events
An effective EDA platform
Some great platform and frameworks that could be used to build EDA applications.

Trigger CI using Terraform Cloud

Kamesh Sampath — Mon, 10 Apr 2023 01:23:33 +0000

Continuous Integration(CI) pipelines needs a target infrastructure to which the CI artifacts are deployed. The deployments are handled by CI or we can leverage Continuous Deployment pipelines. Modern day architecture uses automation tools like terraform, ansible to provision the target infrastructure, this type of provisioning is called IaC.

Usually CI/CD and IaC don't run in tandem. Many times we want to trigger the CI pipeline only when the target infrastructure is ready to bootstrap with software components that are required by CI/CD pipelines.

As part of this DIY blog let us tackle the aforementioned problem with an use case.

Use Case

As CI/CD user I would like to provision a Kubernetes Cluster on Google Cloud Platform(GKE) using Terraform. The successful provision of the cluster should notify a CI pipeline to start bootstrapping ArgoCD on to GKE.

What you need ?

A Terraform Cloud Account. Create a workspace on the terraform cloud to be used for this exercise.
Google Cloud Account used to create the Google Kubernetes Engine(GKE) cluster.
Though we can use any CI platform, for this demo we will use Harness CIas our CI platform. You can do a free tier signup from here.

Demo Sources

The demo uses the following git repositories a sources,

IaC vanilla-gke: the terraform source repository that will be used with terraform cloud to provision GKE.
Kubernetes manifests bootstrap-argocd: the repository that holds kubernetes manifests to bootstrap argo CD on to the GKE cluster
Harness CI Pipeline tfc-notification-demo

Fork and Clone the Sources

To make fork and clone easier we will use gh CLI. Download the add gh to your $PATH.

Let us create a directory where we want to place all our demo sources,

mkdir -p "$HOME/tfc-notification-demo"
cd "$HOME/tfc-notification-demo"
export DEMO_HOME="$PWD"

IaC

Clone and fork vanilla-gke repo,

gh repo clone harness-apps/vanilla-gke
cd vanilla-gke
gh repo fork
export TFC_GKE_REPO="$PWD"

Bootstrap Argo CD Sources

Clone and fork bootstrap-argocd repo,

cd ..
gh repo clone harness-apps/bootstrap-argocd
cd bootstrap-argocd
gh repo fork
export ARGOCD_BOOTSTRAP_REPO="$PWD"

Harness CI Pipeline

Clone and fork tfc-notification-demo repo,

cd ..
gh repo clone harness-apps/tfc-notification-demo
cd tfc-notification-demo
gh repo fork
export TFC_DEMO_REPO="$PWD"

For rest of the blog we will reference the repositories vanilla-gke and bootstrap-argocd and tfc-notification-demo as $TFC_GKE_REPO, $ARGOCD_BOOTSTRAP_REPO and $TFC_DEMO_REPO.

Harness CI

In the following sections we will define and create the resources required to define a CI pipeline using Harness platform.

Create Harness Project

Create new harness project named terraform_integration_demos using Harness Web Console,

Update its details as shown,

Follow the wizard leaving rest to defaults and on the last screen choose Continuous Integration,

Click Go to Module to go to project home page.

Define New Pipeline

Click Pipelines to define a new pipeline,

For this demo will be doing manual clone, hence disable the clone,

Click on Pipelines and delete the default Build pipeline,

Add `harnessImage` Docker Registry Connector

As part of pipelines we will be pulling image from DockerHub. harnesImage Docker Registry Connector helps pulling the public Docker Hub images as an anonymous user.

Let us configure an harnesImage connector as described in docker registry connectors. The pipelines we create as part of the later section will use this connector.

Configure GitHub

GitHub Credentials

Create a GitHub PAT for the account where you have have forked the repositories $TFC_GKE_REPO and $ARGOCD_BOOTSTRAP_REPO. We will refer to the token as $GITHUB_PAT.

From the Project Setup click Secrets,

Update the encrypted text secret details as shown,

Click Save to save the secret,

Connector

As we need to clone the sources from GitHub, we need to define a GitHub Connector, from the Project Setup click Connectors,

From connector list select GitHub,

Enter the name as GitHub,

Click Continue to enter the connector details,

Click Continue and update the GitHub Connector credentials,

When selecting the Personal Access Token make sure you select the GitHub PAT that we defined in previous section,

Click Continue and use select Connect through Harness Platform,

Click Save and Continue to run the connection test, if all went well the connection should successful,

Google Cloud Service Account Secret

We need Google Service Account(GSA) credentials(JSON Key) to query the GKE cluster details and create resources on it.

Set environment

export GCP_PROJECT="the Google Cloud Project where Kubernetes Cluster is created"
export GSA_KEY_FILE="path where to store the key file"

Create SA

gcloud iam service-accounts create gke-user \
  --description "GKE User" \
  --display-name "gke-user"

IAM Binding

Add permissions to the user to be able to provision kubernetes resources,

gcloud projects add-iam-policy-binding $GCP_PROJECT \
  --member="serviceAccount:$GSA_NAME@$GCP_PROJECT.iam.gserviceaccount.com" \
  --role="roles/container.admin"

Download And Save GSA Key

IMPORTANT: Only security admins can create the JSON keys. Ensure the Google Cloud user you are using has Security Admin role.

gcloud iam service-accounts keys create "${GSA_KEY_FILE}" \
    --iam-account="gke-user@${GCP_PROJECT}.iam.gserviceaccount.com"

GSA Secret

Get back to the Project Setup click Secrets,

Add the GSA secret details as shown,

IMPORTANT: When you browse and select make sure you select the $GSA_KEY_FILE as the file for the secret.

Click Save to save the secret,

Terraform Workspace

On your terraform cloud account create a new workspace called vanilla-gke. Update the workspace settings to use Version Control and make it point to $TFC_GKE_REPO.

Configure the workspace with following variables,

For more details on available variables, check Terraform Inputs.

IMPORTANT: The GOOGLE_CREDENTIALS is Google Service Account JSON Key with permissions to create GKE cluster. Please check the https://github.com/harness-apps/vanilla-gke#pre-requisites for the required roles and permissions. This key will be used by Terraform to create the GKE cluster. When you add the key to terraform variables, you need to make it as base64 encoded e.g. cat YOUR_GOOGLE_CREDENTIALS_KEY_FILE | tr -d \\n

Going forward we will refer to the Terraform Workspace as $TF_WORKSPACE

Lookup your terraform cloud organizations

And set it's value to the variable $TF_CLOUD_ORGANIZATION.

Create need Terraform API Token that can be used to pull the outputs of terraform run(cloud). From your terraform user settings Create an API token,

And save the API token to the variable $TF_TOKEN_app_terraform_io. We will use this variable in CI pipeline.

Harness CI Pipeline

Getting back to Harness web console, navigate to your project terraform_integration_demos, click Pipelines and Create a Pipeline --> Import From Git,

Update the pipeline details as shown,

IMPORTANT: Make sure the Name of the pipeline is bootstrap argocd pipeline to make the import succeed with defaults.

Click the bootstrap argocd pipeline from the list to open the Pipeline Studio and click on the stage Bootstrap Argo CD to bring up the pipeline steps,

You can click on each step to see the details.

The Pipeline uses the following secrets,

google_application_credentials - the GSA credentials to manipulate GKE
terraform_cloud_api_token - the value of $TF_TOKEN_app_terraform_io
terraform_workspace - the value $TF_WORKSPACE
terraform_cloud_organization - the value $TF_CLOUD_ORGANIZATION

We already added google_application_credentials secret as part of the earlier section. Following the similar pattern let us add the terraform_cloud_api_token, terraform_workspace and terraform_cloud_organization as text secrets.

HINT:
From the Project Setup click Secrets,

TIP: You can also skip adding terraform_workspace and terraform_cloud_organization, we can extract the values from the webhook payload using the expressions <+trigger.payload.workspace_name> and <+trigger.payload.organization_name> respectively.

Notification Trigger

For the Harness CI pipelines to listen to Terraform Cloud Events we need to define a Trigger, navigate back to pipelines and select the bootstrap argocd pipeline --> Triggers,

Click Add New Trigger to add a new webhook trigger(Type: Custom),

On the Configuration page enter the name of the trigger to be tfc notification,

Leave rest of the fields to defaults and click Continue, leave the Conditions to defaults and click Continue.

On the Pipeline Input update the Pipeline Reference Branch to be set to main

NOTE: The Pipeline Reference Branch does not have any implication with this demo as we do manual clone of resources.

Click Create Trigger to create and save the trigger.

Copy Webhook URL

Let us refer to this value as $TRIGGER_WEBHOOK_URL.

Terraform Notification

On your terraform cloud console navigate to the workspace Settings --> Notifications,

Click Create Notification and select Webhook as the Destination,

Update the notification details as shown,

Since we need to bootstrap argo CD only on create events we set the triggers to happen only on Completed,

Click Create Notification to finish the creation of notification.

NOTE: The creation would have fired a notification, if the cluster is not ready yet the pipeline would have failed.

Congratulations!!. With this setup any new or updates thats done to the $TFC_GKE_REPO will trigger a plan and apply on Terraform Cloud. A Completed plan will trigger the bootstrap argocd pipline to run and apply the manifests from $BOOTSTRAP_ARGOCD_REPO on the GKE cluster.

An example of successful pipeline run

Summary

By using the terraform notifications feature we were able to make the CI pipelines listen to IaC events and run the CI pipelines as needed.

Using Workload Identity Continuous Integration(CI) Pipelines

Kamesh Sampath — Thu, 23 Mar 2023 05:22:53 +0000

Overview

In the first part of this series we understood what a Workload Identity is and in the second part how it can help in doing keyless Google API invocations by deploying a demo application. In this blog we learn how to use Workload Identity with SaaS Continuous Integration(CI) providers.

Many SaaS Continuous Integration(CI) providers e.g Harness CI uses what is called a Delegate

Harness Delegate is a service you run in your local laptop or on Cloud to connect your artifacts, infrastructure, collaboration, verification and other providers, with Harness Manager.

Here are some advantages of using Delegates,

Total control of source code, as they can be within organisations cloud infrastructure
Cloud Cost Optimisations as CI pipelines can leverage organisations existing cloud infrastructure
Leverage the native cloud services from CI pipelines

In this tutorial we will deploy a Harness Delegate on to GKE and understand how enabling Workload Identity on GKE can simplify the CI Pipeline.

CI Pipeline UseCase

Builds go application, ideally you can build any application but go is taken as an example here.
Package application build artifact as a container image
Push the image to Google Artifact Registry(GAR).
Cache the build artifacts, dependencies(go modules) on to Google Cloud Storage(GCS) to make build process faster and quicker.

Tutorial

Before we get to tutorial make sure you have signed up for free tier Harness CI account.

Pre-requisites

A Google Cloud Account with a Service Account with roles:
- Kubernetes Engine Admin - to create GKE cluster
- Service Account roles used to create/update/delete Service Account
  - iam.serviceAccounts.actAs
  - iam.serviceAccounts.get
  - iam.serviceAccounts.create
  - iam.serviceAccounts.delete
  - iam.serviceAccounts.update
  - iam.serviceAccounts.get
  - iam.serviceAccounts.getIamPolicy
  - iam.serviceAccounts.setIamPolicy
  - (OR) simply you can add Service Account Admin and Service Account User roles
- Compute Network Admin - to create the VPC networks
Google Cloud SDK
terraform
kubectl
helm
Taskfile

Download Sources

git clone https://github.com/harness-apps/workload-identity-gke-demo.git && cd "$(basename "$_" .git)"
export DEMO_HOME="$PWD"

Environment Setup

Variables

When working with Google Cloud the following environment variables helps in setting the right Google Cloud context like Service Account Key file, project etc., You can use direnv or set the following variables on your shell,

export GOOGLE_APPLICATION_CREDENTIALS="the google cloud service account key json file to use"
export CLOUDSDK_ACTIVE_CONFIG_NAME="the google cloud cli profile to use"
export GOOGLE_CLOUD_PROJECT="the google cloud project to use"
export KUBECONFIG="$DEMO_HOME/.kube/config"

You can find more information about gcloud cli configurations at https://cloud.google.com/sdk/docs/configurations.

As you may need to override few terraform variables that you don't want to check in to VCS, add them to a file called .local.tfvars and set the following environment variable to be picked up by terraform runs,

export TFVARS_FILE=.local.tfvars

Check the Inputs section for all possible terraform variables that are configurable.

An example .local.tfvars looks like,

project_id                 = "my-awesome-gcp-project"
region                     = "asia-south1"
cluster_name               = "wi-demos"
kubernetes_version         = "1.24."
harness_account_id         = "REPLACE WITH YOUR HARNESS ACCOUNT ID"
harness_delegate_token     = "REPLACE WITH YOUR HARNESS DELEGATE TOKEN"
harness_delegate_name      = "wi-demos-delegate"
harness_delegate_namespace = "harness-delegate-ng"
harness_manager_endpoint   = "https://app.harness.io/gratis"

Create Environment

We will use terraform to create a GKE cluster with WorkloadIdentity enabled for its nodes,

task init

Create GKE cluster

The terraform apply will creates a GKE Cluster,

task create_cluster

Deploy Harness Delegate

The following section deploys a Harness Delegate on to our GKE cluster. To be able to successfully deploy a Harness Delegate we need update the following values in our .local.tfvars file,

harness_account_id
harness_delegate_token
harness_delegate_namespace
harness_manager_endpoint
Use Account Id from Account Overview as the value for harness_account_id,

Use the Harness Cluster Hosting Account from the account details to find the matching endpoint URL. e.g for prod-2 it is https://app.harness.io/gratis and set that as value for harness_manager_endpoint.

TIP:
You can find the endpoint corresponding to your Harness Cluster Hosting Account from https://developer.harness.io/tutorials/platform/install-delegate/

Copy the default token from Projects --> Project Setup --> Delegates(Tokens) and set it as value for harness_delegate_token.

harness_delegate_name: defaults to harness-delegate
harness_delegate_namespace: defaults to harness-delegate-ng

With us having updated the .local.tfvars, run the following command to deploy the Harness Delegate,

task deploy_harness_delegate

NOTE: It will take some time for delegate to come up and get connected.

Wait for the delegate to be connected before proceeding to next steps.

You can view status of the delegate from the Project --> Project Setup --> Delegates page,

You can also check the running Harness delegate pods by using kubectl,

kubectl get pods -n harness-delegate-ng

The output should be something like the pod name may vary based on your harness_delegate_name value.

NAME                                 READY   STATUS    RESTARTS   AGE
harness-delegate-6bfd78d5cb-5h8x9   1/1     Running   0          2m23s

Build Application

Having deployed the Harness delegate, let us build a CI pipeline that will build and push the same go app to GAR.

Import Template

The sources already has build stage template that can be used to create the CI pipeline.

Navigate to your Harness Account, Account Overview --> Organizations and select default.

From the Organization overview page select Templates,

Click New Template and choose Import From Git option,

Fill the wizard with values as shown,

NOTE: If you want to use your fork of harness-apps/workload-identity-gke-demo then update Repository with your fork.

Create Pipeline

Navigate to Builds --> Pipelines, click Create Pipeline.

Click Add Stage and click Use template, choose ko_gar_build_push template that we imported earlier and click Use template to complete import.

Enter details about the stage,

Click Setup Stage to create the stage and fill other details i.e Template Inputs,

We use default namespace to run builder pods. The build pod runs with a Kubernetes Service Account(KSA) harness-builder.

NOTE:
The harness-builder KSA is mapped to Google IAM Service Account(GSA) harness-delegate to inherit the GCP roles using Workload Identity in this case to push the images to Google Artifact Registry(GAR).

Click Run to run the pipeline to see the image build and pushed to GAR,

As successful run would have pushed the image into GAR in this example its asia-south1-docker.pkg.dev/pratyakshika/demos/lingua-greeter:latest

Cleanup

To clean up all the Google Cloud resources that were created as part of this demo,

task destroy

Summary

By using Workload Identity Delegate we have simplified and secured our CI pipelines which can now use any Google API services by configuring the GSA with right roles/permissions. The CI SaaS platform no longer need to store/update the Google API credentials.

Having deployed Workload Identity Delegate you can also do keyless signing of your container images using Google Application Credentials. For more info check cosign.

For more tutorials and documentation please visit https://developer.harness.io.