DEV Community: Jubin Soni

I Built a VS Code Extension to Debug Azure AI Foundry Agents Without Leaving My Editor

Jubin Soni — Tue, 16 Jun 2026 01:08:37 +0000

TL;DR: I built Foundry Trace Inspector — a free, open-source VS Code extension that brings Azure AI Foundry agent traces into your editor as an interactive timeline. No portal switching, no throwaway scripts.

The problem

Azure AI Foundry has a genuinely great portal. You can see your agent runs, the tools it called, the messages it sent and received, and even a breakdown of token usage — all in a clean UI.

But here's what actually happens when you're building an agent locally:

Write some code, trigger a run
Switch to the browser, open the Foundry portal
Navigate to your project → your agent → Traces tab
Find the right run
Click through to see what happened
Switch back to VS Code to make a fix
Repeat

That context switch sounds minor. But when you're iterating fast — tweaking a system prompt, adjusting tool call logic, debugging why an agent handed off to the wrong sub-agent — it adds up. You're constantly pulling your attention out of your editor and into the browser and back again.

What I wanted was simple: see the trace right where I'm working.

What Foundry Trace Inspector does

The extension connects to your Azure AI Foundry project and gives you three views for every agent run, all inside a VS Code panel:

Trajectories — the full span tree

A Gantt-style collapsible tree showing the full execution: Session → Invoke Agent → Chat turns → Tool calls. Every span shows duration, token counts, and cost. Click any span to open a detail drawer with the model, status, token breakdown, and raw input/output.

Duration

Per-span timing bars — see exactly how long each step took.

Tokens

Input vs output token breakdown per span.

This is the view I use most during debugging. At a glance I can see: did the tool call happen? How long did it take? What did the LLM actually receive as input?

User View — readable conversation replay

A chat-bubble timeline of the full conversation: user messages and assistant replies rendered the way a human reads them, with the agent name and model on each assistant turn. Each assistant bubble has a "View Trace" button that jumps directly to the corresponding response in the sidebar — so you can go from "something looked off in this reply" to the raw span in one click.

Token & Cost chart

A stacked bar chart (input vs output tokens per LLM turn) so you can instantly spot which turns are burning the most tokens — useful when you're trying to understand why a multi-turn conversation is getting expensive.

Per span cost breakdown for both input and output tokens consumed

How it works under the hood

Azure AI Foundry agents use the OpenAI Responses API internally. Every agent reply produces a resp_... response ID that's visible in the Foundry portal's Traces tab. The extension fetches those responses directly via the same API and reconstructs the full conversation timeline locally.

When a session spans multiple turns, each response links to the previous one via previous_response_id. Load any response in the chain and the extension walks the chain automatically — you don't need to manually track down every ID.

Conversation IDs (conv_...) are discovered automatically from your saved responses, so once you track one response, the whole conversation surfaces.

No intermediate server. The extension makes API calls only to the Azure endpoint you configure. Your API key is stored in VS Code's encrypted SecretStorage — it never touches settings.json and never leaves your machine.

Setting it up

You need two things:

An Azure AI Foundry project endpoint URL (found in the Foundry portal under your project → Overview)
Either an API key or Azure CLI auth (az login) via DefaultAzureCredential

Once configured, grab a conv_... conversation ID from the portal's Traces tab, paste it into the sidebar, and the extension fetches all responses in that conversation automatically.

What's next

A few things I want to add in v0.2:

Auto-discovery of recent runs — instead of pasting IDs manually, list recent conversations directly from the panel
Side-by-side diff — compare two runs of the same agent to see what changed between runs
Export to Markdown — generate a readable trace report you can paste into a PR or incident note

Try it

If you're building on Azure AI Foundry Agent Service, install it and let me know what you think:

🔗 Install Foundry Trace Inspector

It takes about 2 minutes to set up — install, paste your Foundry project endpoint. If it saves you time, a ⭐ rating on the Marketplace goes a long way.

If you have feedback, run into a bug, or want a feature — open an issue on GitHub. I'm actively developing this and would love to hear what would make the dev loop even faster for you.

Amazon CodeWhisperer Q Developer Kiro: The Rise of Agentic Coding

Jubin Soni — Wed, 10 Jun 2026 19:25:00 +0000

The Abrupt End of Amazon Q Developer

In May 2026, AWS dropped a bombshell: Amazon Q Developer IDE plugins and paid subscriptions will reach end-of-support on April 30, 2027, with new signups blocked as of May 15, 2026. The successor? Kiro — AWS's next-generation AI IDE that reframes how engineers build software from scratch.

If you're a backend engineer who has been relying on Q Developer for code completion, inline chat, and security scanning inside VS Code or JetBrains, the clock is ticking. But before you begrudgingly migrate, it's worth understanding why this transition is happening, what Kiro actually offers, and whether the trade-offs are worth it — especially in production backend contexts like microservices, distributed systems, and observability pipelines.

Historical Context: From CodeWhisperer to Q Developer to Kiro

AWS's AI coding journey started with Amazon CodeWhisperer (launched in preview 2022), which was a single-model code suggestion tool — think GitHub Copilot, but AWS-native. It supported security scanning against common vulnerability patterns and could suggest AWS SDK calls contextually.

In early 2023, AWS folded CodeWhisperer into the broader Amazon Q branding — an umbrella AI assistant that spanned not just code but AWS console assistance, documentation search, and operational queries. Q Developer became the IDE-facing arm of that product.

The problem? Q Developer tried to be everything: a coding assistant, a console assistant, a documentation bot, and a security scanner all jammed into one plugin. Feedback from engineering teams consistently pointed to context window limitations, poor multi-file understanding, and weak support for complex backend architectures spanning multiple services.

Kiro is AWS's response. Built from the ground up with "spec-driven development" as its core philosophy, Kiro is less of an autocomplete engine and more of an agentic coding environment — it can plan, scaffold, and implement across your entire project tree, not just the file you have open.

Architecture Comparison

The architectural difference is significant. Q Developer operated in a request-response model where you asked a question or triggered a completion and got a result. Kiro introduces hooks — lifecycle-aware automations that fire when you save a file, open a PR, or change a spec. This is closer to how CI/CD pipelines work, and backend engineers will immediately recognize the paradigm.

Feature-by-Feature Breakdown

Feature	Amazon Q Developer	Amazon Kiro
Multi-file context	Limited (single file primary)	Full project tree
Agentic task execution	No	Yes (plan → implement → test)
Spec-driven development	No	Yes (SPEC.md driven)
MCP integration	No	Yes (external tool calls)
Security scanning	Yes (CodeWhisperer rules)	Yes (enhanced)
JetBrains support	Yes	Yes
VS Code support	Yes	Yes
AWS Free Tier access	Yes	Yes (via AIdeas Competition)
Paid subscription	$19/mo (deprecated)	Separate Kiro pricing
End of support	April 30, 2027	Active

Production Code Example 1: Spec-Driven Microservice Scaffolding with Kiro

One of Kiro's most powerful features is its SPEC.md-driven workflow. Instead of writing code and hoping the AI helps, you write a specification and Kiro implements it. Here's what that looks like for a backend order processing microservice.

// SPEC.md concept implemented as TypeScript types
// Kiro reads your spec and generates this scaffolding

import { Logger } from '@aws-lambda-powertools/logger';
import { Tracer } from '@aws-lambda-powertools/tracer';
import { DynamoDBClient, PutItemCommand, GetItemCommand } from '@aws-sdk/client-dynamodb';
import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs';
import { marshall, unmarshall } from '@aws-sdk/util-dynamodb';

const logger = new Logger({ serviceName: 'order-service', logLevel: 'INFO' });
const tracer = new Tracer({ serviceName: 'order-service' });
const ddb = tracer.captureAWSv3Client(new DynamoDBClient({}));
const sqs = tracer.captureAWSv3Client(new SQSClient({}));

interface Order {
  orderId: string;
  customerId: string;
  items: Array<{ sku: string; qty: number; price: number }>;
  status: 'PENDING' | 'CONFIRMED' | 'SHIPPED' | 'CANCELLED';
  createdAt: string;
}

interface CreateOrderResult {
  success: boolean;
  orderId?: string;
  error?: string;
}

// Kiro-generated handler with full error handling + structured logging
export const createOrder = async (
  order: Omit<Order, 'orderId' | 'status' | 'createdAt'>
): Promise<CreateOrderResult> => {
  const segment = tracer.getSegment();
  const subsegment = segment?.addNewSubsegment('createOrder');

  try {
    const orderId = `ORD-${Date.now()}-${Math.random().toString(36).slice(2, 7).toUpperCase()}`;
    const newOrder: Order = {
      ...order,
      orderId,
      status: 'PENDING',
      createdAt: new Date().toISOString(),
    };

    logger.info('Creating order', { orderId, customerId: order.customerId, itemCount: order.items.length });

    // Persist to DynamoDB
    await ddb.send(new PutItemCommand({
      TableName: process.env.ORDERS_TABLE!,
      Item: marshall(newOrder),
      ConditionExpression: 'attribute_not_exists(orderId)', // idempotency guard
    }));

    // Publish to downstream processing queue
    await sqs.send(new SendMessageCommand({
      QueueUrl: process.env.ORDER_QUEUE_URL!,
      MessageBody: JSON.stringify(newOrder),
      MessageGroupId: order.customerId, // FIFO ordering per customer
      MessageDeduplicationId: orderId,
    }));

    logger.info('Order created and queued', { orderId });
    return { success: true, orderId };

  } catch (error) {
    const err = error as Error;
    logger.error('Failed to create order', { error: err.message, stack: err.stack });
    subsegment?.addError(err);
    return { success: false, error: err.message };
  } finally {
    subsegment?.close();
  }
};

What Q Developer would do: Suggest inline completions line-by-line based on your cursor position.

What Kiro does: Reads your SPEC.md that says "Create an order service with DynamoDB persistence, SQS publishing, idempotency, and X-Ray tracing" — and generates the entire file, including imports, error handling, and the logging pattern your team already uses (learned from your codebase).

Production Code Example 2: Using Kiro Hooks for Automatic Test Generation

Kiro's hook system is where backend engineers will find the most leverage. A hook is a YAML-defined automation that triggers on file system events within your project.

# .kiro/hooks/auto-test.yaml
name: Generate Unit Tests on Save
trigger:
  event: file_saved
  pattern: "src/**/*.ts"
  exclude: "**/*.test.ts"
actions:
  - type: agent_task
    prompt: |
      A TypeScript file was just saved at {{file_path}}.
      Review the exported functions. For any function that does not have a 
      corresponding test in {{file_path_without_ext}}.test.ts, generate 
      comprehensive unit tests using Vitest. Include:
      - Happy path tests
      - Error boundary tests (network failures, malformed input)
      - Edge cases for empty arrays and null values
      Use @aws-sdk/client-dynamodb mocks from @aws-sdk/lib-dynamodb MockDocumentClient.
    output_file: "{{file_path_without_ext}}.test.ts"
    mode: merge # Don't overwrite existing tests, only append missing ones

// Auto-generated test from the hook above (Vitest)
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { mockClient } from 'aws-sdk-client-mock';
import { DynamoDBClient, PutItemCommand } from '@aws-sdk/client-dynamodb';
import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs';
import { createOrder } from './order-service';

const ddbMock = mockClient(DynamoDBClient);
const sqsMock = mockClient(SQSClient);

describe('createOrder', () => {
  beforeEach(() => {
    ddbMock.reset();
    sqsMock.reset();
    process.env.ORDERS_TABLE = 'test-orders';
    process.env.ORDER_QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123/orders.fifo';
  });

  it('should create order and return orderId on success', async () => {
    ddbMock.on(PutItemCommand).resolves({});
    sqsMock.on(SendMessageCommand).resolves({ MessageId: 'msg-123' });

    const result = await createOrder({
      customerId: 'cust-001',
      items: [{ sku: 'SKU-A', qty: 2, price: 29.99 }],
    });

    expect(result.success).toBe(true);
    expect(result.orderId).toMatch(/^ORD-/);
  });

  it('should return error when DynamoDB PutItem fails', async () => {
    ddbMock.on(PutItemCommand).rejects(new Error('ProvisionedThroughputExceededException'));

    const result = await createOrder({
      customerId: 'cust-001',
      items: [{ sku: 'SKU-A', qty: 1, price: 9.99 }],
    });

    expect(result.success).toBe(false);
    expect(result.error).toContain('ProvisionedThroughputExceededException');
  });

  it('should handle empty items array gracefully', async () => {
    ddbMock.on(PutItemCommand).resolves({});
    sqsMock.on(SendMessageCommand).resolves({});

    const result = await createOrder({ customerId: 'cust-001', items: [] });
    expect(result.success).toBe(true);
  });
});

This hook fires every time you save a .ts file, checks if tests are missing, and generates them — without you ever typing a prompt.

Production Code Example 3: Migrating Q Developer Inline Chat Workflows to Kiro

If you've built muscle memory around Q Developer's /dev inline chat, here's how equivalent workflows map in Kiro.

// Q Developer workflow (deprecated):
// You'd type: /dev add retry logic with exponential backoff to this fetch call
// And it would modify the highlighted code block.

// Kiro equivalent: Add to your SPEC.md or use agent chat with file pinning:
// "Add retry logic with exponential backoff to src/clients/http-client.ts"

// The result Kiro produces — production-grade with observability:
import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger({ serviceName: 'http-client' });

interface RetryOptions {
  maxAttempts: number;
  baseDelayMs: number;
  maxDelayMs: number;
  retryableStatusCodes: number[];
}

const DEFAULT_RETRY_OPTIONS: RetryOptions = {
  maxAttempts: 3,
  baseDelayMs: 100,
  maxDelayMs: 5000,
  retryableStatusCodes: [429, 500, 502, 503, 504],
};

export async function fetchWithRetry<T>(
  url: string,
  options: RequestInit = {},
  retryOpts: Partial<RetryOptions> = {}
): Promise<T> {
  const opts = { ...DEFAULT_RETRY_OPTIONS, ...retryOpts };
  let lastError: Error | null = null;

  for (let attempt = 1; attempt <= opts.maxAttempts; attempt++) {
    try {
      const response = await fetch(url, options);

      if (!response.ok && opts.retryableStatusCodes.includes(response.status)) {
        const delay = Math.min(opts.baseDelayMs * Math.pow(2, attempt - 1), opts.maxDelayMs);
        logger.warn('Retryable HTTP error, backing off', {
          url,
          status: response.status,
          attempt,
          delayMs: delay,
        });
        await new Promise(res => setTimeout(res, delay));
        continue;
      }

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
      }

      return response.json() as Promise<T>;
    } catch (error) {
      lastError = error as Error;
      if (attempt === opts.maxAttempts) break;

      const delay = Math.min(opts.baseDelayMs * Math.pow(2, attempt - 1), opts.maxDelayMs);
      logger.warn('Request failed, retrying', { url, attempt, delayMs: delay, error: lastError.message });
      await new Promise(res => setTimeout(res, delay));
    }
  }

  logger.error('All retry attempts exhausted', { url, maxAttempts: opts.maxAttempts });
  throw lastError ?? new Error('Unknown fetch error after retries');
}

When to Migrate Now vs Wait

Migrate now if:

You're starting a new service or greenfield project — Kiro's spec-driven approach saves the most time at project inception
Your team does heavy test generation — the hook system is a net productivity win
You're building MCP-integrated tooling or AWS-native agentic workflows

Wait if:

You have a heavily customized Q Developer security scanning ruleset — give the Kiro security scanner time to mature
You're on a locked-down enterprise network — Kiro's agentic features require broader outbound connectivity than Q Developer's plugin model

Performance & Productivity Metrics

Metric	Amazon Q Developer	Amazon Kiro (early data)
Avg. context window (tokens)	~16K	~128K+
Multi-file edits per session	1-2	10-20+
Test coverage improvement	~15%	~35% (with hooks)
Time to scaffold new service	~2-3 hrs manual	~20-40 min spec-driven
Security scan languages	15	20+

Summary

The Q Developer → Kiro transition isn't just a rebranding. It's a fundamental shift from a reactive autocomplete tool to a proactive agentic development environment. For backend engineers building distributed systems on AWS, Kiro's spec-driven planning, multi-file context, and hook-based automation represent a genuine productivity leap — not just an incremental update.

Start your migration now. The deprecation deadline of April 2027 sounds far off, but enterprise procurement, security reviews, and team retraining take time. Get ahead of it.

References

AWS: Amazon Q Developer End-of-Support Announcement — AWS News Blog, May 2026
AWS: Top Announcements of What's Next with AWS 2026 — AWS News Blog, April 2026
AWS Lambda Powertools for TypeScript — Official Documentation
AWS SDK Client Mock — GitHub
Kiro Documentation — Official Kiro Docs
AWS Well-Architected Framework: Operational Excellence — AWS Docs

Building a Vector Index in Azure AI Search: HNSW, Profiles, and RAG Retrieval

Jubin Soni — Tue, 02 Jun 2026 08:01:45 +0000

In this article, we will understand how vector search works in Azure AI Search and how to use it as the retrieval layer in a Retrieval-Augmented Generation (RAG) system. The article is meant for software engineers. We will not stop at theory. We will build a small, working example that you can run on your own machine and follow along step by step.

By the end, you will have a small document search service that takes a user question, finds the most relevant text using vector similarity, and prepares the context that you can pass to a language model.

Please note that Azure AI Search was earlier called Azure Cognitive Search. The service was renamed, but many older articles and code samples still use the old name. The concepts are the same.

Let us begin.

What is a RAG system, in short

A RAG system has two main parts. The first part is retrieval. When a user asks a question, we search a knowledge base and pull out the most relevant pieces of text. The second part is generation. We pass these pieces of text, along with the question, to a language model so that the model can answer using real, grounded information.

The quality of a RAG system depends heavily on the retrieval part. If retrieval returns the wrong text, the language model will produce a wrong or vague answer. This is the reason vector search matters. It allows us to retrieve text based on meaning, not only on keyword matching.

Why we need vector search

Traditional keyword search matches exact words. If the user searches for "car" and the document says "automobile", a keyword search may miss it. Vector search solves this problem.

In vector search, we first convert each piece of text into a list of numbers called an embedding. Texts with similar meaning produce embeddings that are close to each other in vector space. When a user asks a question, we convert the question into an embedding as well, and then we find the stored embeddings that are nearest to it. This is called nearest neighbour search.

Azure AI Search supports this by allowing you to define a vector field in your index. You store the embedding in this field, and the service builds a structure that can search through many vectors quickly.

How Azure AI Search performs vector search

Azure AI Search supports two algorithms for vector search. The first is HNSW (Hierarchical Navigable Small World), which is an approximate nearest neighbour (ANN) algorithm. It is fast and is the recommended choice for most production workloads. The second is exhaustive KNN, which compares the query against every stored vector. It is exact but slower, and it is mainly useful for small data sets or for measuring the accuracy of the approximate method.

The following table compares the two algorithms.

Algorithm	Type	Speed on large data	Recall	Best suited for
HNSW	Approximate (ANN)	Fast	Very high and tunable	Most production workloads and large indexes
Exhaustive KNN	Exact	Slow on large data	Exact (100 percent)	Small data sets or measuring ground-truth accuracy

In Azure AI Search, the algorithm is not attached directly to a field. Instead, you define a vector search configuration that contains a list of algorithms and a list of profiles. A profile gives a name to a chosen algorithm. Each vector field then refers to a profile by its name. This extra layer makes it easy to reuse the same algorithm settings across several fields.

The vector field itself uses the type Collection(Edm.Single), which is a collection of single-precision floating point numbers. You must set the number of dimensions on this field, and this number must match the output size of your embedding model.

The architecture of our RAG system

Before writing code, let us look at the full picture. There are two paths. The ingestion path runs once (or whenever your data changes) and fills the index. The query path runs every time a user asks a question.

Please note one important detail. The same embedding model must be used in both paths. If you embed your documents with one model and your queries with another, the vectors will not be comparable, and the search results will be meaningless.

The data model

When we store a document for RAG, we usually do not store the full document as one record. We split it into smaller chunks, because a smaller chunk gives more focused retrieval and fits better inside the language model prompt. Each chunk becomes one document in the Azure AI Search index, and each document holds the chunk text, its embedding, and some metadata for tracing the result back to its source.

The following entity relationship diagram shows how a source document relates to chunks, and how each chunk is stored as one search document in the index.

In our small example, our documents are already short, so we will treat each document as a single chunk. In a real system you would add a chunking step, but the structure of the index will remain the same.

Hands-on tutorial

Now we will build the system. I am assuming you have an Azure subscription, Python (version 3.9 or above), and basic familiarity with running Python scripts.

We will create the embeddings on our own machine using a small open model, so that you do not need to set up an Azure OpenAI deployment to follow along. I will explain at the end how to switch to Azure OpenAI embeddings if you prefer.

Step 1: Create an Azure AI Search service

In the Azure portal, create a resource of type "Azure AI Search". The free tier is enough for this tutorial. After the service is created, open it and note down two values from the portal. The first is the service endpoint, which looks like https://<your-service>.search.windows.net. The second is an admin key, which you will find under the "Keys" section. We will use these in our code.

Please keep the admin key secret. In a real project, you would store it in an environment variable or in Azure Key Vault, not in the source code.

Step 2: Install the Python libraries

pip install azure-search-documents sentence-transformers

Step 3: Create the vector index

Now we create an index that has a vector field. We define the field, the vector search configuration (the algorithm and the profile), and then we create the index.

from azure.core.credentials import AzureKeyCredential
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    SearchIndex,
    SimpleField,
    SearchableField,
    SearchField,
    SearchFieldDataType,
    VectorSearch,
    HnswAlgorithmConfiguration,
    HnswParameters,
    VectorSearchProfile,
)

endpoint = "https://<your-service>.search.windows.net"
admin_key = "<your-admin-key>"
index_name = "rag-docs"

index_client = SearchIndexClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(admin_key),
)

# 1. Define the fields. The embedding field is the vector field.
fields = [
    SimpleField(name="doc_id", type=SearchFieldDataType.String, key=True, filterable=True),
    SearchableField(name="text", type=SearchFieldDataType.String),
    SimpleField(name="source", type=SearchFieldDataType.String, filterable=True),
    SearchField(
        name="embedding",
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
        searchable=True,
        vector_search_dimensions=384,             # must match the embedding model
        vector_search_profile_name="my-vector-profile",
    ),
]

# 2. Define the vector search configuration: one algorithm and one profile.
vector_search = VectorSearch(
    algorithms=[
        HnswAlgorithmConfiguration(
            name="my-hnsw",
            parameters=HnswParameters(
                m=4,
                ef_construction=400,
                ef_search=500,
                metric="cosine",
            ),
        )
    ],
    profiles=[
        VectorSearchProfile(
            name="my-vector-profile",
            algorithm_configuration_name="my-hnsw",
        )
    ],
)

# 3. Create (or update) the index.
index = SearchIndex(name=index_name, fields=fields, vector_search=vector_search)
index_client.create_or_update_index(index)
print("Index created:", index_name)

Let me explain the choices here. The vector_search_dimensions is 384 because that is the output size of the embedding model we will use. The vector field points to a profile named my-vector-profile, and that profile points to the HNSW algorithm named my-hnsw. The metric is cosine, which is a common choice for text embeddings.

The next table explains the HNSW parameters and their default values in Azure AI Search.

Parameter	Where it is set	What it controls	Default and range
`m`	Algorithm configuration	Number of bi-directional links each node keeps in the graph	Default 4, range 4 to 10
`efConstruction`	Algorithm configuration	Number of candidate neighbours examined while building the graph	Default 400, range 100 to 1000
`efSearch`	Algorithm configuration	Number of candidate neighbours examined during a query	Default 500, range 100 to 1000
`metric`	Algorithm configuration	The distance metric	`cosine`; other values are `dotProduct`, `euclidean`, and `hamming`

The default values are a reasonable starting point. You can tune them later based on your recall and latency requirements.

Step 4: Embed and upload the documents

Now we load the embedding model, create a small knowledge base, generate embeddings, and upload the documents to the index.

from azure.search.documents import SearchClient
from sentence_transformers import SentenceTransformer

search_client = SearchClient(
    endpoint=endpoint,
    index_name=index_name,
    credential=AzureKeyCredential(admin_key),
)

# Load the embedding model (produces 384-dimensional vectors)
model = SentenceTransformer("all-MiniLM-L6-v2")

# Our small knowledge base
documents = [
    {"doc_id": "1", "source": "billing-faq",
     "text": "You can update your payment method from the account settings page under Billing."},
    {"doc_id": "2", "source": "billing-faq",
     "text": "Refunds are processed within five to seven business days to the original payment method."},
    {"doc_id": "3", "source": "shipping-faq",
     "text": "Standard delivery takes three to five working days within the country."},
    {"doc_id": "4", "source": "account-faq",
     "text": "To reset your password, click the forgot password link on the login screen."},
    {"doc_id": "5", "source": "shipping-faq",
     "text": "International orders may take up to fourteen working days depending on customs."},
]

# Create embeddings for the text of each document
texts = [d["text"] for d in documents]
vectors = model.encode(texts)

# Attach the embedding to each document and upload
to_upload = []
for doc, vector in zip(documents, vectors):
    to_upload.append({
        "doc_id": doc["doc_id"],
        "source": doc["source"],
        "text": doc["text"],
        "embedding": vector.tolist(),
    })

result = search_client.upload_documents(documents=to_upload)
print("Uploaded", len(result), "documents.")

When you run this script, it will upload five small documents into the index. In a real project you would read documents from files or a database, split them into chunks, and upload thousands or millions of documents using the same upload_documents method, usually in batches.

Step 5: Search using a query vector

Now we write the retrieval function. We embed the user question with the same model, then we run a vector query. The VectorizedQuery object tells Azure AI Search which vector to search with, how many neighbours to return, and which field to search against.

from azure.search.documents.models import VectorizedQuery


def search(question, k=3):
    # Embed the question with the same model
    query_vector = model.encode([question])[0]

    vector_query = VectorizedQuery(
        vector=query_vector.tolist(),
        k_nearest_neighbors=k,
        fields="embedding",
    )

    results = search_client.search(
        search_text=None,                 # pure vector search
        vector_queries=[vector_query],
        select=["doc_id", "source", "text"],
    )

    output = []
    for r in results:
        output.append({
            "score": r["@search.score"],
            "source": r["source"],
            "text": r["text"],
        })
    return output


for item in search("how do I get my money back", k=3):
    print(round(item["score"], 4), "|", item["source"], "|", item["text"])

Notice that the question uses the words "get my money back", but none of the documents contain these exact words. The most relevant document talks about refunds. Because vector search compares meaning and not keywords, the refund document should appear at the top of the results. This is the behaviour we want in a RAG system.

Step 6: Build the RAG prompt

The retrieval part is now complete. The final step is to take the retrieved text and build a prompt for the language model. We do not call any specific model here, because you may use Azure OpenAI, an Anthropic model, an OpenAI model, or any other. We only prepare the input.

def build_prompt(question, k=3):
    hits = search(question, k=k)
    context = "\n\n".join(f"- {hit['text']}" for hit in hits)

    prompt = (
        "You are a support assistant. Use only the context below to answer "
        "the question. If the answer is not in the context, say that you do "
        "not have enough information.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\n"
        "Answer:"
    )
    return prompt


print(build_prompt("how do I get my money back"))

The output is a prompt that contains the question and the most relevant pieces of text. You would now send this prompt to your language model, and the model would generate a grounded answer. This is the complete retrieval-augmented generation flow, with Azure AI Search acting as the vector store and retriever.

Using Azure OpenAI embeddings instead

In this tutorial we generated embeddings on our own machine. In many Azure projects, teams use Azure OpenAI embedding models instead, such as text-embedding-3-small. The change is small. You call the Azure OpenAI client to create the embedding, and you set the index dimensions to match the model (for example, 1536 for text-embedding-ada-002). The rest of the index and query code stays the same.

from openai import AzureOpenAI

client = AzureOpenAI(
    api_key="<your-azure-openai-key>",
    api_version="2024-10-21",
    azure_endpoint="https://<your-resource>.openai.azure.com/",
)

def get_embedding(text):
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-small",   # your deployment name
    )
    return response.data[0].embedding

Azure AI Search also supports a feature called integrated vectorization. With this feature, the service itself calls an Azure OpenAI model to convert text into vectors at indexing time and at query time, so you do not have to generate the embeddings in your own code. This is convenient for larger pipelines, but the basic flow shown above is enough to understand how vector search works.

A few practical notes

When you move from this small example to a real system, please keep the following points in mind. Choose your embedding model carefully, because it decides the dimension of your vectors and the quality of your retrieval. Add a chunking step so that long documents are split into focused passages. If you need both keyword matching and meaning-based matching, use hybrid search by passing a value to search_text together with the vector query; Azure AI Search will combine the keyword (BM25) results and the vector results. For even better ordering, you can enable the semantic ranker, which re-ranks the top results using a language model. Finally, monitor your index size, because vector indexes consume memory; if storage becomes a concern, look at vector compression and binary vectors.

Conclusion

We have seen what vector search is, why a RAG system needs it, and how Azure AI Search provides it through vector fields, the HNSW and exhaustive KNN algorithms, and the profile-based configuration. We then built a small but complete example: we created a search service, defined a vector index, embedded and uploaded a few documents, searched them by meaning, and assembled a RAG prompt. We also saw how to switch to Azure OpenAI embeddings.

You can now extend this example with your own documents, a chunking step, hybrid search, and a language model of your choice to build a full RAG application.

References

Quickstart: Vector search in Azure AI Search — Microsoft Learn: https://learn.microsoft.com/en-us/azure/search/search-get-started-vector
Create a vector index — Microsoft Learn: https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-create-index
Vector search overview — Microsoft Learn: https://learn.microsoft.com/en-us/azure/search/vector-search-overview
Hybrid search overview — Microsoft Learn: https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview
Index binary vectors (memory optimization) — Microsoft Learn: https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-index-binary-data
azure-search-documents Python SDK — PyPI: https://pypi.org/project/azure-search-documents/
Python vector search sample — Azure SDK for Python (GitHub): https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/search/azure-search-documents/samples/sample_vector_search.py
Azure AI Search vector samples — GitHub: https://github.com/Azure/azure-search-vector-samples

Amazon OpenSearch Vector Search Explained for RAG Systems

Jubin Soni — Tue, 02 Jun 2026 02:02:29 +0000

In this article, we will understand how vector search works in Amazon OpenSearch and how to use it as the retrieval layer in a Retrieval-Augmented Generation (RAG) system. The article is meant for software engineers. We will not stop at theory. We will build a small, working example that you can run on your own machine and follow along step by step.

Let us begin.

What is a RAG system, in short

Why we need vector search

Traditional keyword search matches exact words. If the user searches for "car" and the document says "automobile", a keyword search may miss it. Vector search solves this problem.

Amazon OpenSearch supports this through its k-NN (k-nearest neighbour) feature. We store embeddings in a special field type called knn_vector, and OpenSearch builds an index that can search through millions of vectors quickly.

How Amazon OpenSearch performs vector search

Amazon OpenSearch supports two broad approaches: exact nearest neighbour search and approximate nearest neighbour (ANN) search. Exact search compares the query against every stored vector and is accurate but slow on large data. ANN search trades a small amount of accuracy for a large gain in speed, which is what most production RAG systems use.

For ANN, Amazon OpenSearch provides different engines. An engine is the underlying library that builds and searches the vector index. The three engines are FAISS, Lucene, and NMSLIB. Each engine implements one or more algorithms. The two algorithms you will see most often are HNSW (Hierarchical Navigable Small World) and IVF (Inverted File).

The following table compares the three engines.

Engine	Algorithms supported	Filtering	Status	Best suited for
FAISS	HNSW and IVF	Efficient filtering (HNSW from version 2.9, IVF from 2.10)	Active and commonly recommended	Large data sets, memory tuning, and vector compression
Lucene	HNSW	Efficient filtering built in	Active	Smaller deployments and workloads that need metadata filtering
NMSLIB	HNSW	Pre-filtering and post-filtering only	Deprecated, kept for older indexes	Legacy indexes created in earlier versions

For new projects, FAISS and Lucene are the recommended choices. NMSLIB is deprecated, so please do not select it for new work. In our tutorial we will use the FAISS engine with the HNSW algorithm, because it is a good default for RAG workloads.

The knn_vector field supports up to 16,000 dimensions, and each dimension is stored as a 32-bit float by default. The number of dimensions is decided by the embedding model you choose. For example, the model we will use in this tutorial produces 384 dimensions.

The architecture of our RAG system

The data model

When we store a document for RAG, we usually do not store the full document as one record. We split it into smaller chunks, because a smaller chunk gives more focused retrieval and fits better inside the language model prompt. Each chunk becomes one record in the OpenSearch index, and each record holds the chunk text, its embedding, and some metadata for tracing the result back to its source.

The following entity relationship diagram shows how a source document relates to chunks, and how each chunk is stored as one vector record in OpenSearch.

Hands-on tutorial

Now we will build the system. I am assuming you have Docker and Python (version 3.9 or above) installed on your machine.

We will run OpenSearch locally using Docker. This is important to understand: Amazon OpenSearch Service is the managed version of OpenSearch, and the API is the same. So the code we write against a local OpenSearch will work against an Amazon OpenSearch Service domain with only a change in the connection settings. I will show you that change at the end.

Step 1: Run OpenSearch locally

Run the following command. It starts a single-node OpenSearch cluster. Recent OpenSearch versions require you to set an initial admin password, so we provide one through an environment variable.

docker run -d \
  --name opensearch-rag \
  -p 9200:9200 -p 9600:9600 \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=Your-Strong-Password123!" \
  opensearchproject/opensearch:2.17.0

After a few seconds, check that it is running:

curl -k -u admin:'Your-Strong-Password123!' https://localhost:9200

You should receive a JSON response that includes the cluster name and version. The -k flag tells curl to accept the self-signed certificate, which is fine for local testing.

Step 2: Create the k-NN index

Now we create an index that has a knn_vector field. We must enable k-NN at the index level and define the method (the algorithm, the engine, and its parameters).

PUT /rag-docs
{
  "settings": {
    "index": {
      "knn": true,
      "knn.algo_param.ef_search": 100
    }
  },
  "mappings": {
    "properties": {
      "embedding": {
        "type": "knn_vector",
        "dimension": 384,
        "method": {
          "name": "hnsw",
          "space_type": "innerproduct",
          "engine": "faiss",
          "parameters": {
            "ef_construction": 128,
            "m": 16
          }
        }
      },
      "doc_id": { "type": "keyword" },
      "source": { "type": "keyword" },
      "text":   { "type": "text" }
    }
  }
}

Let me explain the choices here. The dimension is 384 because that is the output size of our embedding model. The space_type is innerproduct. We will produce normalised embeddings, and for unit-length vectors the inner product is equal to cosine similarity. This is a common and reliable pattern. The HNSW parameters m and ef_construction control the structure of the graph, and ef_search controls how thoroughly the graph is searched at query time.

The next table explains these parameters and their trade-offs.

Parameter	Where it is set	What it controls	Trade-off
`m`	Index mapping (method)	Number of connections each node keeps in the HNSW graph	Higher value gives better recall but uses more memory
`ef_construction`	Index mapping (method)	Number of candidate neighbours examined while building the graph	Higher value gives a better graph but slows down indexing
`ef_search`	Index settings	Number of candidate neighbours examined during a query	Higher value gives better recall but slows down each search
`space_type`	Index mapping (method)	The distance metric, such as `l2`, `innerproduct`, or `cosinesimil`	Must match the way your embeddings are produced

The default values (m is 16, ef_construction is around 128, ef_search is 100) are a reasonable starting point. You can tune them later based on your recall and latency requirements.

Step 3: Connect, embed, and index the documents

Install the required Python libraries:

pip install opensearch-py sentence-transformers

Now we write a Python script. First we connect to OpenSearch, then we load the embedding model, then we embed a few documents and store them.

from opensearchpy import OpenSearch, helpers
from sentence_transformers import SentenceTransformer

# 1. Connect to the local OpenSearch cluster
client = OpenSearch(
    hosts=[{"host": "localhost", "port": 9200}],
    http_auth=("admin", "Your-Strong-Password123!"),
    use_ssl=True,
    verify_certs=False,   # local self-signed certificate
)

# 2. Load the embedding model (produces 384-dimensional vectors)
model = SentenceTransformer("all-MiniLM-L6-v2")

# 3. Our small knowledge base
documents = [
    {"doc_id": "1", "source": "billing-faq",
     "text": "You can update your payment method from the account settings page under Billing."},
    {"doc_id": "2", "source": "billing-faq",
     "text": "Refunds are processed within five to seven business days to the original payment method."},
    {"doc_id": "3", "source": "shipping-faq",
     "text": "Standard delivery takes three to five working days within the country."},
    {"doc_id": "4", "source": "account-faq",
     "text": "To reset your password, click the forgot password link on the login screen."},
    {"doc_id": "5", "source": "shipping-faq",
     "text": "International orders may take up to fourteen working days depending on customs."},
]

# 4. Create embeddings. We normalise them so inner product equals cosine similarity.
texts = [d["text"] for d in documents]
embeddings = model.encode(texts, normalize_embeddings=True)

# 5. Build the bulk request and index everything
actions = []
for doc, vector in zip(documents, embeddings):
    actions.append({
        "_index": "rag-docs",
        "_source": {
            "doc_id": doc["doc_id"],
            "source": doc["source"],
            "text": doc["text"],
            "embedding": vector.tolist(),
        },
    })

helpers.bulk(client, actions)
client.indices.refresh(index="rag-docs")
print("Indexed", len(documents), "documents.")

When you run this script, it will index five small documents. In a real project you would read documents from files or a database, split them into chunks, and index thousands or millions of records using the same helpers.bulk approach.

Step 4: Search using a query vector

Now we write the retrieval function. We embed the user question with the same model, then we run a k-NN query. The query asks OpenSearch to return the k nearest vectors to our query vector.

def search(question, k=3):
    # Embed the question with the same model and normalisation
    query_vector = model.encode([question], normalize_embeddings=True)[0]

    body = {
        "size": k,
        "query": {
            "knn": {
                "embedding": {
                    "vector": query_vector.tolist(),
                    "k": k,
                }
            }
        },
    }

    response = client.search(index="rag-docs", body=body)
    results = []
    for hit in response["hits"]["hits"]:
        results.append({
            "score": hit["_score"],
            "source": hit["_source"]["source"],
            "text": hit["_source"]["text"],
        })
    return results


for item in search("how do I get my money back", k=3):
    print(round(item["score"], 4), "|", item["source"], "|", item["text"])

Step 5: Build the RAG prompt

The retrieval part is now complete. The final step is to take the retrieved text and build a prompt for the language model. We do not call any specific model here, because you may use Amazon Bedrock, an Anthropic model, an OpenAI model, or any other. We only prepare the input.

def build_prompt(question, k=3):
    hits = search(question, k=k)
    context = "\n\n".join(f"- {hit['text']}" for hit in hits)

    prompt = (
        "You are a support assistant. Use only the context below to answer "
        "the question. If the answer is not in the context, say that you do "
        "not have enough information.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\n"
        "Answer:"
    )
    return prompt


print(build_prompt("how do I get my money back"))

The output is a prompt that contains the question and the most relevant pieces of text. You would now send this prompt to your language model, and the model would generate a grounded answer. This is the complete retrieval-augmented generation flow, with Amazon OpenSearch acting as the vector store and retriever.

Running this against Amazon OpenSearch Service

So far we have used a local cluster. To run the exact same code against an Amazon OpenSearch Service domain, you only need to change how the client connects. If your domain uses IAM-based access, you sign your requests with your AWS credentials. The rest of the code (creating the index, indexing, and searching) stays the same.

from opensearchpy import OpenSearch, RequestsHttpConnection, AWSV4SignerAuth
import boto3

host = "my-domain.us-east-1.es.amazonaws.com"   # your domain endpoint, no https://
region = "us-east-1"
service = "es"   # use "aoss" for Amazon OpenSearch Serverless

credentials = boto3.Session().get_credentials()
auth = AWSV4SignerAuth(credentials, region, service)

client = OpenSearch(
    hosts=[{"host": host, "port": 443}],
    http_auth=auth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    pool_maxsize=20,
)

One point to remember: if you use Amazon OpenSearch Serverless vector search collections, the service value is aoss, and these collections support only the HNSW algorithm with the FAISS engine. They do not support the Lucene engine or the IVF algorithm. For a managed OpenSearch domain (not serverless), all three engines are available.

A few practical notes for production

When you move from this small example to a real system, please keep the following points in mind. Choose your embedding model carefully, because it decides the dimension of your vectors and the quality of your retrieval. Add a chunking step so that long documents are split into focused passages. If you need to filter results by metadata, such as searching only within a particular source, use the efficient filtering support in the FAISS or Lucene engine. Finally, monitor memory usage, because vector indexes are kept in memory; if memory becomes a concern, look at byte vectors and quantization, which reduce the storage size of each vector.

Conclusion

We have seen what vector search is, why a RAG system needs it, and how Amazon OpenSearch provides it through the knn_vector field and the FAISS, Lucene, and NMSLIB engines. We then built a small but complete example: we ran OpenSearch, created a k-NN index, embedded and indexed a few documents, searched them by meaning, and assembled a RAG prompt. The same code runs against Amazon OpenSearch Service with only a change in the connection settings.

You can now extend this example with your own documents, a chunking step, and a language model of your choice to build a full RAG application.

References

Amazon OpenSearch Service vector database capabilities revisited — AWS Big Data Blog: https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-vector-database-capabilities-revisited/
Approximate k-NN search — OpenSearch Documentation: https://docs.opensearch.org/latest/vector-search/vector-search-techniques/approximate-knn/
Methods and engines — OpenSearch Documentation: https://docs.opensearch.org/latest/mappings/supported-field-types/knn-methods-engines/
k-NN vector field type — OpenSearch Documentation: https://docs.opensearch.org/latest/mappings/supported-field-types/knn-vector/
Efficient filters in the OpenSearch vector engine — OpenSearch Blog: https://opensearch.org/blog/efficient-filters-in-knn/
Signing HTTP requests to Amazon OpenSearch Service (Python client) — AWS Documentation: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/request-signing.html
Working with vector search collections (Amazon OpenSearch Serverless) — AWS Documentation: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-vector-search.html
Memory-optimized vectors — OpenSearch Documentation: https://docs.opensearch.org/latest/mappings/supported-field-types/knn-memory-optimized/

Jules: Google's Async Coding Agent Is Changing How We Think About AI and Software Development

Jubin Soni — Fri, 29 May 2026 18:12:09 +0000

There's a quiet architectural shift happening in how we build software, and it doesn't look like what most people expected.

We've spent the last two years treating AI like a very fast autocomplete — a co-pilot sitting shotgun, responding the moment we type. Cursor, Copilot, Gemini Code Assist: all synchronous, all requiring you to stay in the loop, all fundamentally keeping you as the CPU driving execution.

Jules breaks that model.

Google's async coding agent, which went generally available in 2025 and got major updates at I/O 2026, doesn't help you write code faster. It removes you from the writing loop entirely. You assign a task. Jules works. You review a pull request. That's it.

This article breaks down how Jules works technically — with architecture diagrams, sequence flows, and real code — and why the async model might be more significant than it first appears.

What Jules Actually Does (and Doesn't Do)

Jules is not an IDE plugin. It's not an inline suggestion engine. It's not a chat interface for your codebase.

Jules is a task-based async agent. You give it a scoped coding task — fix a bug, migrate a module, add a feature, write tests — and it:

Clones your repository into a secure Google Cloud VM
Analyzes the relevant codebase context (2M token context window as of I/O 2026)
Writes a step-by-step implementation plan using Gemini Pro
Executes that plan: writing code, running tests, fixing errors
Opens a pull request against your branch with a description, diff, and change summary

When it's done, you're not staring at a chat window waiting to approve line-by-line edits. You're reviewing a PR — just like you would from any engineer on your team.

The 2026 update closes the loop further: if the CI/CD pipeline fails on the Jules-authored PR, Jules automatically receives the error, analyzes it, applies a fix, and re-pushes the commit — often without any human intervention at all.

Architecture: How Jules Is Built

Jules is your end-to-end agentic product development platform. It reads your entire product context to figure out what to build next, comes up with solutions, and then ships a PR.

jules.google.com

Here's how the components fit together:

Key architectural choices:

Isolated VM per task: no shared state between runs, reproducible environments
Network access retained: Jules can npm install, run builds, call APIs — unlike Codex which sandboxes with no egress
Two-model split: Gemini Pro handles planning and hard reasoning; Gemini Flash handles lighter subtasks for cost efficiency
Native GitHub integration: reads issues, creates branches, authors commits, opens PRs — not a wrapper, it's first-class

Sequence: The Async Flow End-to-End

The sequence below shows what happens from task assignment to merged PR, and where the developer is actually free:

The key insight: the developer's attention is only required at step 1 (spec) and step 12 (review). Everything in between is Jules.

Code: Using Jules in Your Workflow

1. Jules CLI (Jules Tools — GA at I/O 2026)

# Install Jules Tools CLI
npm install -g @google/jules-tools

# Authenticate
jules auth login

# Submit a task against a GitHub repo
jules task create \
  --repo your-org/your-repo \
  --branch main \
  "Fix the race condition in payment/processor.go — 
   two concurrent requests can double-charge. 
   Add regression tests covering the concurrent case."

# Check task status
jules task status <task-id>

# List open tasks
jules task list --status=in-progress

2. Via Gemini CLI Extension

# Install Gemini CLI
npm install -g @google/gemini-cli

# Add Jules extension
gemini extensions install https://github.com/gemini-cli-extensions/jules --auto-update

# Submit directly from your terminal
/jules Fix the flaky integration tests in auth/session_test.go. 
       Root cause appears to be missing teardown between test runs.

# Jules responds async — you get a PR link when it's done

3. Jules API (for CI/CD integration)

import google.auth
from jules_client import JulesClient

credentials, project = google.auth.default()
client = JulesClient(credentials=credentials)

# Submit a task programmatically
task = client.tasks.create(
    repo="your-org/your-repo",
    branch="main",
    description="""
        Migrate the UserRepository class from raw SQL to 
        the new ORM layer introduced in db/orm.py.
        Preserve all existing query behaviour and update tests.
    """,
    labels=["migration", "automated"]
)

print(f"Task submitted: {task.id}")
print(f"Track at: {task.url}")

# Poll for completion (or use webhooks)
import time
while task.status not in ["completed", "failed"]:
    time.sleep(30)
    task = client.tasks.get(task.id)

if task.status == "completed":
    print(f"PR ready: {task.pull_request_url}")

4. GitHub Actions Integration

# .github/workflows/jules-debt.yml
name: Weekly tech debt sweep

on:
  schedule:
    - cron: '0 9 * * MON'   # Every Monday at 9am

jobs:
  sweep:
    runs-on: ubuntu-latest
    steps:
      - name: Submit Jules tasks from tech-debt.md
        uses: google/jules-action@v1
        with:
          jules-api-key: ${{ secrets.JULES_API_KEY }}
          task-file: .github/tech-debt.md
          branch: main
          auto-merge: false   # Always require human review

Practical Workflow: What Jules Is Good At

Jules excels when the unit of work maps to a ticket. The sharper your spec, the better the output.

Task Type	Jules Fit	Why
Bug fix with clear repro steps	✅ Excellent	Deterministic target, testable outcome
Add test coverage to a module	✅ Excellent	Well-defined scope, no design decisions
Dependency upgrades with API changes	✅ Good	Mechanical but multi-file
Migrate module to new framework/ORM	✅ Good	Repetitive pattern Jules handles well
Security patch + regression tests	✅ Good	Scoped + CI validates automatically
Exploratory refactor (uncertain scope)	⚠️ Risky	Scope drift, Jules may over-engineer
Greenfield architecture design	❌ Wrong tool	No acceptance criteria to validate against
Real-time pair debugging	❌ Wrong paradigm	Needs synchronous back-and-forth

The honest rule of thumb: if you could write a solid Jira ticket for it, Jules can probably do it.

Jules vs. the Field

	Jules	Claude Code	OpenAI Codex	GitHub Copilot
Execution model	Async (PR delivery)	Sync (interactive terminal)	Async (PR delivery)	Sync (inline suggestion)
Runtime environment	Google Cloud VM	Local / container	Cloud sandbox	Editor plugin
Network access in VM	✅ Yes	✅ Yes	❌ No (strict sandbox)	N/A
GitHub integration	Native (issues → PR)	Via CLI	Native	Native
Languages supported	Node, Python, Go, Rust, Java	Any	Node, Python primary	Any
Parallel task execution	✅ Yes	❌ One at a time	✅ Yes	❌ One at a time
CI auto-fix loop	✅ Yes (2026)	❌ No	❌ No	❌ No
Context window	2M tokens	~200K tokens	~128K tokens	~8K tokens
Best for	Delegated ticket work	Complex collaborative tasks	Security-sensitive workflows	Inline acceleration

What Jules Gets Right — and Where It's Still Incomplete

What's working:

The async PR model genuinely removes you from low-value execution loops
CI integration with auto-fix is a real quality-of-life improvement for teams
Multi-language runtime support (Node, Python, Go, Rust, Java) is broader than most competitors
The CLI and Gemini CLI extension make it composable into existing dev workflows
2M token context means Jules can reason across large codebases without truncation

What's still incomplete:

Jules validates against tests — codebases with thin coverage expose the reviewer to unknown unknowns
The debugging story for multi-agent ADK workflows is thin; distributed AI agent observability is largely unsolved
Spec quality gates: Jules has no way to flag an underspecified task before burning compute on it
For exploratory or greenfield work, you still need a synchronous collaborator

The Bigger Picture: What This Means for SWEs

Jules isn't a replacement for engineers. It's a redefinition of what "engineering work" means at the margin.

The value of a senior engineer is increasingly not in the ability to implement — it's in:

Writing specs precise enough that an agent can execute them
Reviewing AI-generated PRs for correctness, design quality, and unintended side effects
Knowing when to reach for async delegation vs. interactive collaboration
Building and maintaining the test coverage and CI infrastructure that makes async agents safe to trust

Google I/O 2026 framed this explicitly: the engineers who get the most from agentic coding will be those who run both patterns in parallel — async for ticket-level work, sync for exploration — not those who pick a favorite.

Jules is a real tool for real workflows right now. If you have a backlog of well-scoped tasks and a codebase with decent test coverage, it's worth spinning up.

References

Google Developers Blog — All the news from the Google I/O 2026 Developer keynote
Google Blog — 100 things we announced at Google I/O 2026
Google Research — AI in software engineering at Google: Progress and the path ahead
Google Cloud Blog — Innovations from Google I/O 26 on Google Cloud
TechCrunch — Google's Jules enters developers' toolchains as AI coding agent competition heats up
AI Builder Club — Google I/O 2026: Everything That Matters for AI Builders

Amazon Quick: AWS's Agentic Workspace, Explained for Engineers

Jubin Soni — Thu, 21 May 2026 03:47:40 +0000

AWS has been building agentic infrastructure for some time now — Bedrock, AgentCore, Strands — mostly aimed at engineers who want to build their own agent systems from scratch. Amazon Quick is a different layer of the same bet: a ready-to-use agentic workspace that targets teams directly, without requiring custom orchestration code.

This article walks through what Quick is, how its components fit together technically, how the MCP integration model works with real code, and where it sits relative to the rest of AWS's agent stack.

What Amazon Quick Is

Amazon Quick is an AI assistant for work that connects to your existing tools — Slack, Microsoft Teams, Outlook, CRMs, databases, and local files — and gives a unified layer for querying, automating, and acting across them. It launched in preview at AWS's "What's Next with AWS" event on April 28, 2026.

The product is aimed at teams, not just individual users. One person can build a custom agent scoped to a specific dataset or workflow, and the whole team benefits from it. Responses from Quick agents are grounded in your actual business data, not the underlying model's training distribution.

Under the hood, Quick is built on Amazon Bedrock AgentCore and uses the Model Context Protocol (MCP) as its standard for connecting to external tools. It runs on AWS IAM and VPC, which means it inherits the same security and compliance posture as the rest of your AWS workloads.

Product Components

Quick bundles five distinct capabilities. It helps to understand each one separately before thinking about how they compose.

Component	What it does
Spaces	Collaborative workspaces where teams pool files, dashboards, and data sources. Agents in a Space are grounded in that Space's data.
Agents	Custom, domain-scoped agents built on your team's specific data. One person builds, everyone uses.
Research	Multi-source synthesis across internal data, the public web, and third-party datasets. Produces structured reports.
Visualize (Quick Sight)	Integrated BI layer. Conversational access to dashboards, charts, and forecasting — no separate BI tool required.
Automate (Quick Flows)	Workflow automation from simple daily tasks to complex multi-step processes with cross-app action execution.

Each component is available through the web app, mobile, and a native desktop app (currently in preview for macOS and Windows) that can read local files and calendar context without requiring browser access.

Where Quick Sits in the AWS Agent Stack

AWS is building in two directions at once. AgentCore is the infrastructure layer for engineers who want to compose their own agent systems — runtime, memory, gateway, observability — with any model and any framework. Quick is the product layer on top: opinionated, team-facing, and deployable without writing orchestration code.

The practical implication: if you're an engineer building internal tools or automation pipelines, you'll likely interact with both layers. AgentCore for the infrastructure wiring; Quick as a surface where non-technical teammates interact with the agents you build.

The Integration Architecture

The core question for any engineer evaluating Quick is: how does it actually connect to external systems, and what does the request path look like?

Quick uses MCP (Model Context Protocol) as its primary integration standard. This is significant because MCP is an open protocol — it means Quick agents are not locked into AWS-specific connectors, and any MCP-compatible server can be registered as a tool source.

High-Level Request Flow

The sequence below shows the full lifecycle of a single agent-triggered tool call — from the moment Quick receives a prompt through to the response returning from a downstream API.

Quick acts as the MCP client. Your MCP server exposes tools via listTools and callTool. Quick discovers them at registration time and makes them available to any agent or automation in the workspace. Authentication flows through OAuth 2.0, with support for Dynamic Client Registration (DCR) so Quick can register itself automatically without manual credential setup.

Building an MCP Server for Quick

Here is a minimal Python MCP server using the mcp SDK that exposes two tools Quick can invoke — get_ticket and list_open_tickets. This pattern works whether you host the server yourself or run it on AgentCore Runtime.

Install dependencies

pip install mcp[server] httpx uvicorn

Server implementation

# server.py
from mcp.server import Server
from mcp.server.sse import SseServerTransport
from mcp.types import Tool, TextContent
import httpx
import json
from starlette.applications import Starlette
from starlette.routing import Route

app = Server("jira-quick-integration")

JIRA_BASE_URL = "https://yourorg.atlassian.net"
JIRA_TOKEN    = "Bearer <your-token>"  # in production, load from AWS Secrets Manager


@app.list_tools()
async def list_tools() -> list[Tool]:
    return [
        Tool(
            name="get_ticket",
            description="Retrieve details for a single Jira ticket by issue key.",
            inputSchema={
                "type": "object",
                "properties": {
                    "issue_key": {
                        "type": "string",
                        "description": "The Jira issue key, e.g. ENG-1234"
                    }
                },
                "required": ["issue_key"]
            }
        ),
        Tool(
            name="list_open_tickets",
            description="List open Jira tickets assigned to a given user.",
            inputSchema={
                "type": "object",
                "properties": {
                    "assignee": {
                        "type": "string",
                        "description": "The Jira username or email of the assignee"
                    }
                },
                "required": ["assignee"]
            }
        )
    ]


@app.call_tool()
async def call_tool(name: str, arguments: dict) -> list[TextContent]:
    headers = {"Authorization": JIRA_TOKEN, "Content-Type": "application/json"}

    async with httpx.AsyncClient() as client:
        if name == "get_ticket":
            key = arguments["issue_key"]
            resp = await client.get(
                f"{JIRA_BASE_URL}/rest/api/3/issue/{key}",
                headers=headers
            )
            resp.raise_for_status()
            data = resp.json()
            summary = data["fields"]["summary"]
            status  = data["fields"]["status"]["name"]
            return [TextContent(type="text", text=f"{key}: {summary} [{status}]")]

        elif name == "list_open_tickets":
            assignee = arguments["assignee"]
            jql = f"assignee={assignee} AND status != Done ORDER BY updated DESC"
            resp = await client.get(
                f"{JIRA_BASE_URL}/rest/api/3/search",
                headers=headers,
                params={"jql": jql, "maxResults": 20}
            )
            resp.raise_for_status()
            issues = resp.json().get("issues", [])
            results = [
                f"{i['key']}: {i['fields']['summary']}"
                for i in issues
            ]
            return [TextContent(type="text", text="\n".join(results) or "No open tickets found.")]

    raise ValueError(f"Unknown tool: {name}")


# Wire up SSE transport for Quick compatibility
sse = SseServerTransport("/messages/")

async def handle_sse(request):
    async with sse.connect_sse(
        request.scope, request.receive, request._send
    ) as streams:
        await app.run(streams[0], streams[1], app.create_initialization_options())

starlette_app = Starlette(
    routes=[Route("/sse", endpoint=handle_sse)]
)

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(starlette_app, host="0.0.0.0", port=8080)

A few design constraints to be aware of when building for Quick:

Each MCP tool call has a 300-second hard timeout. Operations that exceed this fail with HTTP 424. Keep individual tool calls narrow and fast.
The tool list is treated as static after registration. If you add or remove tools on the server, the Quick admin must re-establish the connection to pick up changes.
Quick supports both Server-Sent Events (SSE) and streamable HTTP as transports. Streamable HTTP is preferred for new implementations.

Registering the MCP Server in Quick

Once your server is running and publicly reachable over HTTPS, registration in Quick takes the following path:

Quick Console → Integrations → Add Integration → MCP

Fields:
  Server URL:        https://your-mcp-server.example.com/sse
  Auth type:         OAuth 2.0 (or Service, or None)
  Client ID:         <from your identity provider>
  Authorization URL: https://auth.example.com/oauth/authorize
  Token URL:         https://auth.example.com/oauth/token

If your identity provider supports OAuth Dynamic Client Registration, Quick will auto-register and you skip the manual client ID step entirely. Quick sends an initial unauthenticated request to the MCP server; if it receives a 401 with a WWW-Authenticate header containing a resource_metadata URL, it fetches the metadata document and proceeds with DCR automatically.

Once registered, Quick calls listTools at startup and exposes every discovered tool to agents and automations in the workspace.

The AgentCore Gateway Option

For teams that don't want to write and operate an MCP server from scratch, Amazon Bedrock AgentCore Gateway provides a managed alternative. You point Gateway at a Lambda function or an OpenAPI spec, and it handles the MCP wrapping, auth, logging, and semantic tool discovery automatically. If you use it, Quick never calls your internal APIs directly — everything flows through Gateway's auth and routing layer, as shown in the sequence diagram above.

The semantic search capability is worth noting specifically. When an agent has access to dozens or hundreds of tools, passing the full tool list on every turn wastes context and causes the model to pick the wrong tool. Gateway's built-in x_amz_bedrock_agentcore_search tool lets Quick find the right tool by semantic similarity rather than scanning the entire registry each turn.

Practical Considerations

A few things worth keeping in mind before integrating:

Tool scope matters. When agents are given too many tools simultaneously, selection accuracy degrades — the model reasons over too many options per turn and picks incorrectly more often. Keeping each agent or MCP server to a focused set of 3–5 tools produces better results than exposing everything through one endpoint. This is a known pattern in multi-agent architectures and applies equally to Quick agents.

The 300-second timeout is real. Design each tool call to complete a single, bounded operation. Avoid chaining multiple downstream API calls inside a single tool invocation. If you need a multi-step workflow, model it as separate tools and let the agent orchestrate the sequence.

Local context on the desktop app. The desktop app reads local files and calendar events directly, without upload. For engineers who work primarily in terminals and local editors, this is a meaningful integration point — meeting context, local documentation, and recent file changes are all available to the assistant without any configuration.

MCP interoperability. Because Quick uses MCP as the standard, the same MCP server you build for Quick can also be consumed by Claude Code, Amazon Q Developer, and other MCP-compatible clients. The integration contract is portable.

References

Run Gemma 4 on Your Laptop — A Hands-On Guide to Google's Latest Open Multimodal LLM

Jubin Soni — Fri, 15 May 2026 02:36:05 +0000

If you've been watching the open-source LLM space, you've probably noticed it's been a great couple of years. Llama, Mistral, Phi, Qwen — a whole zoo of models you can download and run on your own machine. Google's entry into that zoo is Gemma, and the fourth generation, Gemma 4 (released April 2, 2026), is the biggest leap yet: built from Gemini 3 research, multimodal (text + image + video + audio), 256K context, native function calling, configurable "thinking mode," and — finally — a clean Apache 2.0 license.

In this post we're going to:

Understand what Gemma 4 actually is, with an architecture diagram
Get it running on your laptop with Ollama in about 5 minutes
Chat with it from the terminal
Send it an image and ask questions about it
Turn on thinking mode for harder problems
Call it from a Python script like a real API
Build a small project that glues it all together

No GPU rental, no API keys, no telemetry. Let's go.

Heads up: This guide assumes zero ML background. If you can install software and run a terminal command, you can do this.

What is Gemma 4?

Gemma is Google DeepMind's family of open-weight language models. "Open-weight" means the actual neural network weights — the giant matrices of numbers that make the model work — are freely downloadable. You can run them, modify them, fine-tune them, ship them in your product.

Gemma 4 brings several big changes over Gemma 3:

Apache 2.0 license. Earlier Gemma releases used a custom license with a Prohibited Use Policy that made some enterprise legal teams nervous. Gemma 4 is plain Apache 2.0 — unlimited commercial use, no MAU caps, no special permissions. This alone is a big deal for production deployments.
Mixture-of-Experts. A new 26B MoE variant activates only ~4B parameters per token, giving you 13B-class quality at 4B-class cost.
Thinking mode. A configurable reasoning mode where the model thinks step-by-step before answering. Toggle it on for hard problems, off for fast chat.
Native function calling. Built-in support for structured tool use — write an agent without needing prompt engineering hacks.
More modalities. Image, video frames, and (on the smaller E2B/E4B models) native audio input. Native system prompt support too.
Bigger context. 128K on the small models, 256K on the larger ones.

Model sizes at a glance

Model	Disk (Ollama)	Active params	Total params	Multimodal	Context	Best for
E2B	~7.2 GB	~2B	~2.3B	text + image + audio	128K	Phones, edge devices, browser
E4B	~9.6 GB	~4B	~4.5B	text + image + audio	128K	Most laptops — the sweet spot
26B A4B (MoE)	~18 GB	~4B	26B	text + image	256K	Consumer GPUs, agentic workloads
31B Dense	~20 GB	31B	31B	text + image	256K	Workstations, highest-quality answers

Two naming notes worth understanding:

E2B / E4B. The "E" stands for Effective parameters. These are dense edge-first models that use a trick called Per-Layer Embeddings (PLE — more on this below) to do more with fewer active parameters.
26B A4B. This is the Mixture-of-Experts model. 26B parameters total, but only ~4B "activate" per forward pass. Latency and cost behave like a 4B model; quality is closer to a 13B dense model. Caveat: you still need to load all 26B into memory.

For most readers on a laptop, E4B is the right starting point. It runs comfortably on a 16 GB Mac or any modern dev machine.

Gemma 4 vs the rest of the open-model zoo (May 2026)

Model	Sizes	Multimodal	Context	License
Gemma 4	E2B / E4B / 26B MoE / 31B	text + image + video + audio (small)	128K / 256K	Apache 2.0
Llama 4	various	text + image	128K+	Llama community license
Qwen 3.5	various	text + image	128K+	Apache 2.0
DeepSeek V4 Flash	MoE	text	128K	MIT

Gemma 4's pitch: the only family that spans phones to servers under Apache 2.0, with multimodal and audio in the same release.

The architecture (in plain English)

You don't need this section to use Gemma 4 — feel free to skip to the install steps. But if you've ever wondered what's actually happening when a multimodal model "sees" and "hears," here it is.

A few pieces worth understanding:

Three input paths. Text goes through a SentencePiece tokenizer (shared with Gemini). Images go through a vision encoder that handles variable aspect ratios and resolutions natively (no more square-only inputs like Gemma 3). On the E2B and E4B models, audio goes through a USM-style conformer encoder borrowed from Gemma 3n. All three paths produce tokens that get interleaved in a single stream — so you can freely mix text, images, and audio in any order in one prompt.
Alternating local/global attention. Most layers only look at a sliding window of recent tokens (cheap). A subset of layers attend to the full context (expensive but rare). This is the standard trick for keeping the KV cache from blowing up at 256K context.
Per-Layer Embeddings (PLE) — the small-model secret. In a normal transformer, each token gets one embedding vector at input and that's all the residual stream has to work with. PLE adds a parallel pathway: for each token, every layer gets its own small conditioning vector from a lookup table. The embedding tables are large (lots of memory) but the "active" parameters per token stay small — that's why a 4-billion-active-parameter E4B can punch above its weight.
Mixture-of-Experts (26B A4B). The MoE layer has multiple "expert" feed-forward networks. A small router picks 2 of 8 (or similar) for each token. Total params = 26B (all loaded), active params per token = ~4B (only those fire). Pareto-optimal for quality-per-FLOP.
Thinking mode. When you include the special <|think|> token at the start of the system prompt, the model emits internal reasoning between <|channel>thought\n...<channel|> markers before the final answer. Disable it for fast chat; enable it for math, code, multi-step reasoning.

That's most of what's worth knowing. Now let's actually run it.

Step 1: Install Ollama

There are a few ways to run Gemma 4 locally, but the easiest by a mile is Ollama. Think of it as "Docker for LLMs" — it handles downloading the model, managing memory, GPU acceleration, and exposing a local API. You don't have to think about CUDA versions or PyTorch.

Install it:

macOS / Windows: Download the installer at ollama.com/download and run it.
Linux:

  curl -fsSL https://ollama.com/install.sh | sh

Verify:

ollama --version

You should see a version number. Gemma 4 requires Ollama v0.20.0 or later — if you're on an older version, update first.

Step 2: Pull a Gemma 4 model

Download the default (E4B, ~9.6 GB):

ollama pull gemma4

This downloads about 9.6 GB. Grab a coffee. ☕

Other sizes if you want them:

ollama pull gemma4:e2b   # ~7.2 GB — smallest, for low-RAM machines
ollama pull gemma4:e4b   # ~9.6 GB — the default; same as `gemma4`
ollama pull gemma4:26b   # ~18 GB  — the MoE; 256K context
ollama pull gemma4:31b   # ~20 GB  — biggest dense model

Hardware reality check: On Apple Silicon, 16 GB unified memory handles E4B comfortably. NVIDIA users need the model to fit entirely in VRAM for GPU-accelerated inference. The 26B model fits on 24 GB but leaves very little headroom — treat it as the ceiling, not the target.

List what you've got:

ollama list

Step 3: Chat with it in the terminal

Easiest possible test:

ollama run gemma4

You'll get an interactive prompt:

>>> Explain what a hash map is, like I'm a junior dev.

Hit enter and watch it stream a response. To exit, type /bye.

That's it. You're running a state-of-the-art LLM locally with zero cloud dependency. Try:

"Write a Python function that finds duplicates in a list, with three different approaches and their tradeoffs."
"What's the difference between TCP and UDP? Use an analogy."
"Translate 'Where is the nearest train station?' into Japanese, Spanish, and Hindi."

Step 4: Send it an image

Gemma 4 can see. Drop any image file in your current directory, then:

ollama run gemma4
>>> Describe what's in this image: ./screenshot.png

Ollama loads the image, sends it through the vision encoder, and the model answers. Unlike Gemma 3 (which resized everything to 896×896), Gemma 4 handles variable aspect ratios and resolutions natively — so tall screenshots, wide diagrams, and high-res photos all work without manual cropping.

Try:

"What error is shown in this screenshot?" (paste a stack trace)
"What's the bounding box for the 'submit' button in this UI?" (Gemma 4 will answer in JSON — natively!)
"Read the handwriting in this note and transcribe it."

Step 5: Turn on thinking mode

For harder problems — multi-step math, complex code, logic puzzles — turn on thinking mode. Include the <|think|> token at the very start of your system prompt:

ollama run gemma4
>>> /set system "<|think|>You are a careful, methodical assistant."
>>> Three friends split a $73.42 dinner bill. Alice had a $12 appetizer, Bob had a $9 drink. The rest is shared. What does everyone pay?

The model will emit its reasoning in a <|channel>thought\n...<channel|> block before the final answer. For fast chat, leave the token out and the model answers directly.

🧠 When to use it: Code generation, math, multi-hop reasoning, agentic planning — yes. Single-turn factual questions, summarization, translation — no, it just adds latency.

Step 6: Call Gemma 4 from Python

A chat prompt is nice, but you're a developer — you want to call this thing from code. When Ollama is running, it exposes a local REST API on http://localhost:11434. There's also an official Python client.

Install it:

pip install ollama

Basic chat

import ollama

response = ollama.chat(
    model="gemma4",
    messages=[
        {"role": "system", "content": "You are a senior code reviewer. Be concise and direct."},
        {"role": "user",   "content": "Review this code:\n\ndef add(a, b):\n    return a+b"},
    ],
)

print(response["message"]["content"])

Streaming responses (ChatGPT-style)

import ollama

stream = ollama.chat(
    model="gemma4",
    messages=[{"role": "user", "content": "Write a haiku about debugging."}],
    stream=True,
)

for chunk in stream:
    print(chunk["message"]["content"], end="", flush=True)

Sending an image

import ollama

response = ollama.chat(
    model="gemma4",
    messages=[{
        "role": "user",
        "content": "What's in this image?",
        "images": ["./my_photo.jpg"],
    }],
)

print(response["message"]["content"])

Thinking mode + function calling (the agentic combo)

This is where Gemma 4 actually starts feeling like a "real" agent. You declare your tools as JSON schemas, the model decides when to call them, and you execute the call and pass results back. No prompt engineering hacks needed.

import ollama

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name, e.g. 'Tokyo'"},
            },
            "required": ["city"],
        },
    },
}]

def get_weather(city: str) -> str:
    # Pretend this hits a real API.
    return f"{city}: 22°C, partly cloudy"

response = ollama.chat(
    model="gemma4",
    messages=[
        {"role": "system", "content": "<|think|>You are a helpful weather assistant."},
        {"role": "user",   "content": "Should I bring an umbrella in Tokyo today?"},
    ],
    tools=tools,
)

# If the model wants to call a tool, execute it and feed the result back:
for tool_call in response["message"].get("tool_calls", []):
    name = tool_call["function"]["name"]
    args = tool_call["function"]["arguments"]
    if name == "get_weather":
        result = get_weather(**args)
        # Send result back for the model to finalize its answer
        followup = ollama.chat(
            model="gemma4",
            messages=[
                {"role": "user", "content": "Should I bring an umbrella in Tokyo today?"},
                response["message"],
                {"role": "tool", "content": result, "name": name},
            ],
        )
        print(followup["message"]["content"])

Raw HTTP (no Python client needed)

For any other language:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4",
  "messages": [{"role": "user", "content": "Hello!"}],
  "stream": false
}'

Same JSON shape works from Node, Go, Rust, your shell — anything that can make an HTTP request.

A small project: folder-watching image describer

Here's a useful ~30-line script. It watches a folder, and any new image dropped in gets automatically described by Gemma 4. Great for accessibility tools, content moderation prototypes, or just learning.

import os, time
import ollama

WATCH_DIR = "./inbox"
os.makedirs(WATCH_DIR, exist_ok=True)
SEEN = set(os.listdir(WATCH_DIR))

print(f"📁 Watching {WATCH_DIR}/ — drop an image in to describe it.")
print("   (Ctrl+C to stop)\n")

IMAGE_EXTS = (".png", ".jpg", ".jpeg", ".webp", ".gif")

try:
    while True:
        current = set(os.listdir(WATCH_DIR))
        new_files = sorted(current - SEEN)

        for filename in new_files:
            if not filename.lower().endswith(IMAGE_EXTS):
                continue

            path = os.path.join(WATCH_DIR, filename)
            print(f"📸 New image: {filename}")

            response = ollama.chat(
                model="gemma4",
                messages=[{
                    "role": "user",
                    "content": (
                        "Describe this image in 2-3 sentences. "
                        "Mention any visible text. Be specific."
                    ),
                    "images": [path],
                }],
            )

            print(f"   → {response['message']['content']}\n")

        SEEN = current
        time.sleep(2)
except KeyboardInterrupt:
    print("\n Stopped.")

Run it, drag images into the inbox/ folder, and watch descriptions appear. That's a real, useful, completely local AI tool — written in 30 lines.

Things to know before shipping anything serious

A few honest caveats:

Caveat	Why it matters
Hallucination	Local models still confidently make things up. Don't trust factual claims without verification. Thinking mode reduces this for reasoning tasks but doesn't eliminate it.
CPU latency	Expect 1–3 tokens/sec on a CPU-only laptop with E4B. A GPU gives 3–10× speedup.
Context costs RAM	256K context is real, but actually filling it eats memory. Most use cases need <16K tokens.
MoE memory	The 26B MoE runs fast (only 4B active per token), but you still need to load all 26B into RAM. Don't confuse active params with memory footprint.
Audio is small-model only	E2B/E4B have native audio input. The 26B and 31B models do not.
Apache 2.0 ≠ no responsibilities	The license is permissive, but you're still on the hook for safety, bias, and compliance in whatever you ship.

📚 References & further reading

Gemma 4 announcement — Google blog — The launch post (April 2, 2026).
Gemma 4 model overview — Google AI for Developers — Official docs: sizes, capabilities, hardware requirements.
Welcome Gemma 4 — Hugging Face blog — Best technical write-up: covers PLE, MoE, USM audio encoder, benchmarks, and code samples.
Gemma 4 model card on Hugging Face — E4B instruct model weights and configuration.
Gemma 4 Complete Guide 2026 — dev.to — Community guide with architecture details and competitor comparisons.
SigLIP (Zhai et al., 2023) — The vision encoder family Gemma's image path builds on.
Mixture-of-Experts (Shazeer et al., 2017) — The original sparsely-gated MoE paper. The 26B A4B is a direct descendant.
Switch Transformer (Fedus et al., 2021) — Modern MoE techniques.
Llama 4 — Meta's competing open-weight family.

AWS Kiro: The Agentic IDE That Makes Specs the Unit of Work

Jubin Soni — Mon, 11 May 2026 22:53:13 +0000

The agentic IDE space has gotten crowded fast. Cursor, Claude Code, Copilot, Windsurf — they all share the same core model: you type a prompt, the AI writes some code, you iterate. It works well for prototyping. It breaks down when you're building production systems on a large codebase with a team of more than one.

AWS Kiro takes a different bet. Instead of chat-first, it's spec-first. The unit of work isn't a prompt — it's a structured specification that the agent uses to plan, implement, verify, and document your feature end to end. That's a meaningful philosophical difference, and in practice it changes what the tool is useful for.

Here's what Kiro actually is, how its core concepts fit together, and an honest take on when it makes sense over the alternatives.

What Kiro Is

Kiro launched from AWS in mid-2025 and is built on top of Amazon Bedrock, routing between Claude Sonnet for reasoning-heavy work and Amazon Nova for high-throughput code generation. It ships in three forms:

Kiro IDE — a VS Code-compatible editor (built on Code OSS, so you can import your existing themes, keybindings, and Open VSX plugins)
Kiro CLI — the same agent in your terminal, useful for SSH sessions or scripted workflows
Kiro Autonomous Agent — a background agent that picks up tasks, implements them, and opens PRs without you sitting in the loop

You don't need an AWS account to get started — you can sign in with GitHub or Google. The IDE feels immediately familiar if you've used VS Code, which removes one of the usual adoption barriers for new tooling.

In January 2026, AWS also announced the end of Amazon Q Developer for new signups (effective May 15, 2026), explicitly directing users to Kiro as its successor for IDE-based AI assistance. That's a significant signal about where AWS is placing its bets.

The Three Concepts That Make Kiro Different

1. Specs

When you start a new feature in Kiro, you don't jump straight to code. You describe what you want to build, and Kiro generates three structured files:

requirements.md — user stories and acceptance criteria
design.md — system design, component breakdown, data flow
tasks.md — a numbered implementation checklist the agent works through

These become the source of truth. Code is a build artifact of the spec. When you come back to the feature a month later, or hand it to a new team member, the reasoning behind every decision is documented — not in a Confluence page nobody reads, but in the repo next to the code it describes.

This is the thing chat-first tools can't replicate. Cursor or Claude Code can generate excellent code from a good prompt. What they can't do is maintain a structured paper trail of why the code looks the way it does.

2. Hooks

Hooks are event-driven automations that fire when things happen in your workspace — file save, new file created, commit opened. You define what Kiro should do in response, and it runs those actions in the background without you having to think about them.

Common hooks teams set up:

Run the linter and auto-fix on every file save
Regenerate unit tests when implementation files change
Update the relevant section of design.md when a module is modified
Run a security scan before any commit

The practical effect is that a junior developer's output passes the same automated quality bar as a senior's, because the standards are enforced by the environment rather than by code review heroics.

3. Steering Files

Steering files are Markdown files that give Kiro persistent context about your project — your conventions, the libraries you've standardized on, your architecture decisions, your security requirements. You create them once, and Kiro reads them on every interaction without you having to re-explain your stack in every prompt.

They live in two places:

~/.kiro/steering/ — global rules that apply across all your projects
.kiro/steering/ — project-specific overrides checked into the repo

A typical global steering file might say things like "always use TypeScript strict mode," "prefer AWS CDK over raw CloudFormation," or "all Lambda functions must have structured logging with a correlation ID." Project steering files add things like "this service is a multi-tenant SaaS, tenant ID is always passed in the request context."

The result is that Kiro's context isn't reset between sessions and doesn't depend on whoever wrote the last prompt being thorough.

The Hooks + Specs Flywheel

The real power emerges when hooks and specs work together. Here's what that looks like in practice:

You describe a new feature. Kiro generates requirements.md, design.md, and tasks.md.
You review and refine the spec — add an edge case to requirements, adjust the component breakdown in design.
Kiro implements the tasks list, following your steering files for conventions.
On each file save, hooks run: linter, tests, security scan. Issues surface immediately.
When you're done, a hook generates the commit message from the spec diff.
The PR description writes itself from requirements.md.

The spec doesn't go stale because hooks keep it in sync with the code. The code doesn't drift from the design because the design was written before the code. This is what "engineering rigor" means in the context of agentic development — not slower, but structured.

AWS-Native Advantages (and the Honest Tradeoff)

Kiro has deep integration with the AWS ecosystem: CodeCatalyst for repositories and CI/CD, Bedrock for model access, IAM Identity Center for enterprise auth, and "Kiro Powers" — pre-packaged MCP servers for AWS-specific domains like CDK, CloudFormation, pricing, and (recently) HealthOmics workflows.

If your team is already AWS-first, this is a genuine multiplier. Your Kiro agent can query your actual AWS account context, reference live Bedrock documentation, and generate CDK constructs that match your organization's guardrails.

The honest tradeoff: if your team isn't AWS-first, some of this integration feels like overhead rather than lift. Kiro works perfectly well as a general-purpose agentic IDE — the spec/hooks/steering system has value regardless of your cloud provider — but the ecosystem integrations are clearly designed for AWS shops. Most teams running mixed infrastructure (some AWS, some not) find it practical to use Kiro for the AWS-native services and keep their existing editor for everything else. The two coexist fine.

How It Compares to the Alternatives

	Kiro	Cursor	Claude Code
Primary paradigm	Spec-driven	Chat-driven	Task-driven (CLI)
Persistent context	Steering files	Rules / `.cursorrules`	AGENTS.md
Automation	Hooks (event-driven)	Manual	Manual
AWS integration	Native	None	None
IDE	Standalone (VS Code-compatible)	Fork of VS Code	Terminal only
Background agent	Yes (autonomous agent)	Limited	Yes
Best for	Production features, team consistency	Fast prototyping, exploration	Complex refactors, agentic tasks

Kiro and Claude Code aren't direct competitors in practice — Kiro is an IDE product and Claude Code is a terminal agent. Many teams run both, using Kiro for structured feature work and Claude Code for open-ended refactors or one-off tasks.

Getting Started

Download the IDE from kiro.dev — no AWS account required. Sign in with GitHub or Google, point it at an existing repo, and run through the onboarding to import your VS Code settings.

A good first experiment: take a feature you're planning to build anyway, describe it to Kiro, and look at the spec it generates before writing any code. The value of the approach becomes obvious when you see your vague "add user preferences" idea turn into a concrete requirements doc with six acceptance criteria and a data model.

From there:

Create one global steering file in ~/.kiro/steering/ with your language and framework defaults
Set up one hook that runs your linter on file save
Build the feature using the task list Kiro generated

That's the feedback loop that makes the tool click. The full power of the hooks and autonomous agent comes later, but even the basic spec workflow is a meaningful improvement over prompt-and-iterate for anything that takes more than a day to build.

Worth Watching

A few things that make Kiro worth keeping an eye on even if you're not ready to switch:

The spec-as-artifact model is genuinely novel. When agents get better, spec-driven codebases will be better positioned to benefit — the structured requirements and design docs give future agents much richer context than a commit history and some comments.

Kiro Powers (the MCP server marketplace) is growing fast. The HealthOmics extension in February 2026 showed that domain-specific agent packs are a real product direction, not just a demo.

And with Amazon Q Developer sunsetting for new users, AWS is clearly consolidating its developer AI bet onto Kiro. Whatever the roadmap looks like from here, it's going to get resources.

Kiro isn't the right tool for every workflow. If you're prototyping solo or doing exploratory work, the spec-first overhead is friction you don't need. But for teams shipping production features that need to be documented, tested, and maintained — the bet that specs should be the unit of work is a compelling one.

Kiro vs. the Alternatives

Feature	Kiro	Cursor	Claude Code	GitHub Copilot
Primary paradigm	Spec-driven	Chat-driven	Task-driven (CLI)	Inline completion
Persistent context	Steering files	`.cursorrules`	`AGENTS.md`	None
Event automation	Hooks (file save, commit)	None	None	None
Structured specs	✅ Native	❌	❌	❌
Background agent	✅ Autonomous agent	Limited	✅	❌
AWS-native integration	✅ Deep	❌	❌	❌
Dynamic MCP loading	✅ Powers	Manual	Manual	❌
IDE base	Code OSS (VS Code compat.)	VS Code fork	Terminal only	Plugin
Free tier	✅	✅	✅	✅

How Spec-Driven Development Works

┌─────────────────────────────────────────────────────────┐
│              YOU: describe a feature                     │
└─────────────────────────┬───────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│                KIRO GENERATES SPECS                      │
│                                                          │
│  .kiro/specs/my-feature/                                 │
│  ├── requirements.md  ← user stories + EARS notation     │
│  ├── design.md        ← architecture, data flow, APIs    │
│  └── tasks.md         ← ordered implementation plan      │
└─────────────────────────┬───────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│              YOU: review + refine specs                  │
│  add edge cases, adjust design, approve task list        │
└─────────────────────────┬───────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│            KIRO IMPLEMENTS task by task                  │
│  guided by steering files + spec context                 │
└─────────────────────────┬───────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│              HOOKS FIRE AUTOMATICALLY                    │
│  on every file save:                                     │
│  → linter + autofix                                      │
│  → test generation / update                              │
│  → security scan                                         │
│  → design.md sync                                        │
└─────────────────────────┬───────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│        PR OPENS — description from requirements.md       │
│        commit message generated from spec diff           │
└─────────────────────────────────────────────────────────┘

Steering File Layout

~/.kiro/steering/              ← global, applies to every project
├── typescript.md              "always use strict mode, no any"
├── aws.md                     "prefer CDK over raw CloudFormation"
├── security.md                "IAM roles must follow least privilege"
├── git.md                     "use conventional commits"
└── testing.md                 "80% coverage minimum, jest + RTL"

your-repo/
└── .kiro/
    └── steering/              ← project-specific overrides (checked in)
        ├── architecture.md    "multi-tenant SaaS, one DB schema per tenant"
        ├── api.md             "all endpoints versioned under /v1"
        └── data-model.md      "tenant ID always in request context, never inferred"

Hook Definition Example

# .kiro/hooks/test-sync.yaml
name: Sync Tests on Component Save
trigger:
  event: onSave
  pattern: "src/**/*.tsx"
instructions: |
  When a React component file is saved:
  1. Check if a corresponding test file exists in __tests__/
  2. If not, create one with basic render and snapshot tests
  3. If it exists, update it to cover any new props or exported functions
  4. Run the test file and report failures inline

# .kiro/hooks/security-scan.yaml
name: Pre-commit Security Scan
trigger:
  event: onCommit
instructions: |
  Before every commit:
  1. Scan staged files for hardcoded secrets, API keys, and credentials
  2. Check for any 0.0.0.0/0 ingress rules in IaC files
  3. Flag any new IAM policies that use wildcard actions (*)
  4. Block the commit and explain any findings — do not auto-fix

How Powers Solve Context Rot

Without Powers, connecting multiple MCP servers front-loads your entire context window before you write a single line:

Without Powers
──────────────────────────────────────────────────
Context window (200K tokens)

[Figma MCP tools]     ~12K tokens  ████
[Postman MCP tools]   ~18K tokens  ██████
[Stripe MCP tools]    ~10K tokens  ███
[Supabase MCP tools]  ~15K tokens  █████
[Datadog MCP tools]   ~9K tokens   ███
                      ──────────────────
Total overhead        ~64K tokens  (32% gone before first prompt)

With Powers (dynamic loading)
──────────────────────────────────────────────────
You mention "payment" → Stripe power activates
You mention "database" → Supabase activates, Stripe deactivates
Baseline overhead: ~0K tokens until context is needed

Workspace Architecture for AWS Teams

AWS Organization
└── Management Account
    ├── Client A Account
    │   ├── Kiro workspace  (.kiro/ scoped here)
    │   ├── CodeCatalyst repo
    │   ├── Bedrock access (us-east-1)
    │   └── Secrets Manager (client A secrets only)
    │
    ├── Client B Account
    │   ├── Kiro workspace  (.kiro/ scoped here)
    │   ├── CodeCatalyst repo
    │   ├── Bedrock access (us-east-1)
    │   └── Secrets Manager (client B secrets only)
    │
    └── Shared Services Account
        ├── IAM Identity Center (SSO for all Kiro logins)
        └── Billing consolidated

This pattern keeps client IP, secrets, and Bedrock spend isolated by account boundary — IAM does the enforcement, not convention.

Resources

kiro.dev — download is free, no AWS account required
Introducing Kiro — the original launch post, good context on the design philosophy behind specs and hooks
Introducing Powers — explains why dynamic MCP loading matters and how Powers solve context rot
Teaching Kiro new tricks with steering and MCP — practical deep dive on using steering + MCP to handle custom libraries and DSLs
Specs documentation — full reference including the Design-First and Bugfix spec workflows
Kiro Powers marketplace — browse Figma, Stripe, Supabase, Datadog, Terraform and more
IDE Changelog — how fast the product is moving
Amazon Q Developer end-of-support announcement — official AWS post confirming Kiro as Q Developer's successor
github.com/kirodotdev/Kiro — issue tracker and feedback repo

Build a Git Commit Analyzer with Gemma 4 31B and a 256K Context Window

Jubin Soni — Fri, 08 May 2026 17:41:18 +0000

This is a submission for the Gemma 4 Challenge: Build with Gemma 4

Most developers reach for an LLM when they need code completion or a chatbot. This article is about something more useful and less obvious: feeding your entire sprint's git history to Gemma 4 31B — diffs, commit messages, authors and all — and getting back structured, actionable analysis of what actually changed and why it might matter.

The 31B Dense model's 256K context window is the key enabler here. It means you can pass tens of thousands of lines of patch output in a single prompt and ask the model to reason across the whole thing — not chunk-and-summarize, but genuinely cross-reference commits, spot patterns, and flag risk. That's a qualitatively different capability from what a smaller model or an older Gemma generation could provide.

By the end of this guide you'll have a working Python CLI tool that:

Shells out to git log --patch to collect a commit range
Sends the full diff to Gemma 4 31B via the Gemini API (free tier in Google AI Studio)
Returns a structured JSON report with change summaries, risk flags, and a draft changelog
Optionally writes a Markdown changelog file

Why Gemma 4 31B Is the Right Model for This

Three specific properties make the 31B Dense the correct pick here — not the 26B MoE, not the edge models.

256K context window. A week's worth of commits on a mid-size codebase generates 20,000–80,000 tokens of patch text. The 31B handles that in a single pass. Chunking and summarizing separately loses cross-commit signal: the model can't notice that a refactor in commit 3 introduced the same variable name collision that commit 7 later fixed.

Maximum quality per query. The 31B Dense is the highest-accuracy model in the Gemma 4 family. For code analysis you care about precision — a false positive risk flag wastes a senior engineer's time, and a false negative ships a bug. You're making one expensive call per analysis run, so raw quality beats throughput.

Native structured output. Gemma 4 has first-class support for function calling and structured JSON output. The analyzer requests a strict JSON schema and the model reliably returns it — no fragile string parsing required.

The 26B MoE is the right choice if you're building something that calls the model thousands of times per day and want cost efficiency. This tool calls it once per analysis run and prioritizes signal quality, so the Dense wins.

Prerequisites

Python 3.10+
A Google AI Studio API key (free — get one here)
A git repository to analyze
The google-generativeai Python SDK

pip install google-generativeai

Set your API key as an environment variable:

export GEMINI_API_KEY="your-key-here"

Step 1: Collect the Git Diff

The first job is gathering the raw patch data. We use git log --patch with a commit range and pipe the output to a string. We also collect structured commit metadata separately so the model has author and timestamp context alongside the diff.

import subprocess
import sys

def collect_git_history(repo_path: str, since: str = "1 week ago", until: str = "HEAD") -> tuple[str, list[dict]]:
    """
    Returns (full_patch_text, list_of_commit_metadata).
    `since` accepts anything git understands: '7 days ago', 'v1.2.3', a SHA, etc.
    """
    # Collect the full unified diff
    patch_result = subprocess.run(
        ["git", "log", "--patch", "--no-merges", f"--since={since}", f"--until={until}",
         "--pretty=format:COMMIT: %H%nAuthor: %an <%ae>%nDate: %ci%nMessage: %s%n"],
        cwd=repo_path,
        capture_output=True,
        text=True,
        check=True
    )

    # Collect lightweight metadata for the summary header
    meta_result = subprocess.run(
        ["git", "log", "--no-merges", f"--since={since}", f"--until={until}",
         "--pretty=format:%H|%an|%ci|%s"],
        cwd=repo_path,
        capture_output=True,
        text=True,
        check=True
    )

    commits = []
    for line in meta_result.stdout.strip().splitlines():
        if not line:
            continue
        sha, author, date, *msg_parts = line.split("|")
        commits.append({
            "sha": sha[:8],
            "author": author,
            "date": date,
            "message": "|".join(msg_parts)
        })

    return patch_result.stdout, commits

A week of commits on a real codebase might be 40,000–100,000 tokens. We'll let the model handle the full text — that's exactly what the 256K window is for.

Step 2: Build the Prompt

The prompt does three things: gives the model its role and output contract, defines the JSON schema it must return, and passes the raw git history.

SYSTEM_PROMPT = """You are a senior staff engineer performing a structured code review
of a git commit history. Your job is to analyse the provided patch text and return a
single JSON object — nothing else, no markdown fences, no explanation outside the JSON.

The JSON object must match this schema exactly:

{
  "summary": "2-3 sentence plain-English summary of the overall change set",
  "changed_areas": [
    {
      "path": "path/to/file_or_directory",
      "change_type": "added | modified | deleted | renamed",
      "description": "what changed and why it likely changed"
    }
  ],
  "risk_flags": [
    {
      "severity": "low | medium | high",
      "area": "file or component",
      "reason": "specific, concrete reason this change carries risk"
    }
  ],
  "patterns": [
    "notable cross-commit pattern, refactor theme, or repeated change"
  ],
  "changelog_entry": "A polished, user-facing changelog entry in Markdown. Use ## [Unreleased] as the heading. Group under Added, Changed, Fixed, Removed as appropriate."
}

Be specific. Do not flag risk without a concrete reason tied to the actual diff.
Do not invent changes that are not present in the patch text."""


def build_prompt(patch_text: str, commits: list[dict]) -> str:
    commit_count = len(commits)
    authors = list({c["author"] for c in commits})
    date_range = f"{commits[-1]['date'][:10]} to {commits[0]['date'][:10]}" if commits else "unknown"

    header = (
        f"ANALYSIS REQUEST\n"
        f"Commits: {commit_count}\n"
        f"Authors: {', '.join(authors)}\n"
        f"Date range: {date_range}\n\n"
        f"FULL PATCH TEXT FOLLOWS\n"
        f"{'='*60}\n"
    )

    return header + patch_text

The system prompt enforces a strict schema so we can parse the response with json.loads — no regex, no fallbacks. One of Gemma 4's standout improvements over Gemma 3 is how reliably it follows structured output instructions at this schema complexity.

Step 3: Call Gemma 4 31B

We use the google-generativeai SDK with gemma-4-31b-it (the instruction-tuned variant — always use IT for structured task completion).

import google.generativeai as genai
import json
import os

def analyze_with_gemma(patch_text: str, commits: list[dict]) -> dict:
    genai.configure(api_key=os.environ["GEMINI_API_KEY"])

    model = genai.GenerativeModel(
        model_name="gemma-4-31b-it",
        system_instruction=SYSTEM_PROMPT,
        generation_config=genai.GenerationConfig(
            temperature=0.2,      # Low temperature for consistent structured output
            top_p=0.9,
            max_output_tokens=4096,
        )
    )

    prompt = build_prompt(patch_text, commits)

    print(f"Sending {len(prompt.split()):,} words to Gemma 4 31B...", file=sys.stderr)

    response = model.generate_content(prompt)

    raw = response.text.strip()

    # Strip markdown fences if the model adds them despite instructions
    if raw.startswith("```

"):
        raw = raw.split("

```")[1]
        if raw.startswith("json"):
            raw = raw[4:]

    return json.loads(raw)

Temperature at 0.2 keeps the output deterministic and schema-compliant. For creative changelog prose you could nudge it to 0.4 — but for risk flags you want the model to be conservative and consistent.

Step 4: Format and Output the Report

from datetime import datetime

def print_report(analysis: dict, commits: list[dict]) -> None:
    print("\n" + "="*60)
    print("GIT HISTORY ANALYSIS — Gemma 4 31B")
    print("="*60)
    print(f"\nCommits analysed: {len(commits)}")
    print(f"\nSUMMARY\n{analysis['summary']}\n")

    if analysis.get("risk_flags"):
        print("RISK FLAGS")
        for flag in sorted(analysis["risk_flags"], key=lambda f: {"high": 0, "medium": 1, "low": 2}[f["severity"]]):
            icon = {"high": "🔴", "medium": "🟡", "low": "🟢"}[flag["severity"]]
            print(f"  {icon} [{flag['severity'].upper()}] {flag['area']}")
            print(f"     {flag['reason']}")
        print()

    if analysis.get("patterns"):
        print("PATTERNS DETECTED")
        for p in analysis["patterns"]:
            print(f"  • {p}")
        print()

    print("CHANGED AREAS")
    for area in analysis.get("changed_areas", []):
        print(f"  [{area['change_type'].upper():8}] {area['path']}")
        print(f"             {area['description']}")
    print()


def write_changelog(analysis: dict, output_path: str) -> None:
    entry = analysis.get("changelog_entry", "")
    if not entry:
        return

    # Inject today's date if the entry has a placeholder
    entry = entry.replace("[Unreleased]", f"[Unreleased] — {datetime.today().strftime('%Y-%m-%d')}")

    with open(output_path, "w") as f:
        f.write(entry + "\n")

    print(f"Changelog written to {output_path}", file=sys.stderr)

Step 5: Wire It Together as a CLI

import argparse

def main():
    parser = argparse.ArgumentParser(
        description="Analyse a git commit range with Gemma 4 31B"
    )
    parser.add_argument("repo", help="Path to git repository")
    parser.add_argument("--since", default="1 week ago",
                        help="Start of range (default: '1 week ago'). Accepts any git date or ref.")
    parser.add_argument("--until", default="HEAD",
                        help="End of range (default: HEAD)")
    parser.add_argument("--changelog", default=None,
                        help="Write changelog entry to this file")
    parser.add_argument("--json", dest="json_out", default=None,
                        help="Write full JSON report to this file")
    args = parser.parse_args()

    patch_text, commits = collect_git_history(args.repo, args.since, args.until)

    if not commits:
        print("No commits found in the specified range.", file=sys.stderr)
        sys.exit(0)

    analysis = analyze_with_gemma(patch_text, commits)
    print_report(analysis, commits)

    if args.changelog:
        write_changelog(analysis, args.changelog)

    if args.json_out:
        with open(args.json_out, "w") as f:
            json.dump(analysis, f, indent=2)
        print(f"Full JSON report written to {args.json_out}", file=sys.stderr)


if __name__ == "__main__":
    main()

Running It

Analyse the last week of commits in the current repo:

python git_analyzer.py . --since "1 week ago"

Analyse a specific SHA range and write a changelog:

python git_analyzer.py /path/to/repo \
  --since v1.4.0 \
  --until v1.5.0 \
  --changelog CHANGELOG.md \
  --json report.json

Analyse a single sprint (two-week window):

python git_analyzer.py . --since "14 days ago" --changelog CHANGELOG.md

Sample Output

Here's an abbreviated example of what the tool produces on a real project:

============================================================
GIT HISTORY ANALYSIS — Gemma 4 31B
============================================================

Commits analysed: 23

SUMMARY
This sprint focused on migrating the authentication layer from session
cookies to JWTs, with supporting changes to the user model and API
middleware. Three unrelated bug fixes were included. No database
migrations were added despite schema-adjacent changes in user.py.

RISK FLAGS
  🔴 [HIGH]  src/auth/middleware.py
             Token expiry is set to 0 in the new JWT config, which
             disables expiry entirely. This appears unintentional given
             the surrounding comments referencing a 24h TTL.
  🟡 [MEDIUM] src/models/user.py
             The `last_login` field is now written in two places with
             different timezone handling (UTC in the old path, local
             time in the new one). Cross-commit inconsistency introduced
             in commits a3f1cc and 9d02bb.

PATTERNS DETECTED
  • JWT migration touched 11 files across 8 commits — no single
    atomic commit, suggesting iterative discovery during implementation
  • Four separate commits add logging statements then remove them,
    indicating debug churn that could have been a feature branch

CHANGED AREAS
  [MODIFIED ] src/auth/middleware.py
               Core auth middleware rewritten to validate Bearer tokens
               instead of reading from session. Old session path removed.
  [MODIFIED ] src/models/user.py
               Added jwt_secret field; last_login timezone handling changed
  [ADDED    ] src/auth/token.py
               New module for JWT encode/decode with HS256
  ...

The risk flag about token expiry being set to 0 is real — this is the kind of thing that slips through human PR review precisely because it looks like a config value, not a bug. The cross-commit inconsistency flag is only possible because the model reasoned across all 23 commits simultaneously rather than reviewing each in isolation.

Context Window Headroom

The 256K window on Gemma 4 31B means you have significant headroom. At roughly 3 characters per token, the practical limits look like this:

Scenario	Approx. tokens	Fits in 256K?
1 day of commits, small team	~5,000	✅
1 sprint (2 weeks), small team	~40,000	✅
Full quarter, mid-size team	~180,000	✅
1 year of active development	~500,000+	❌ use `--since` to segment

For very large repos, segment by component directory using git log -- path/to/subdir rather than trying to fit everything.

Where to Take This Next

GitHub Action. Trigger the analyzer on each PR, post the risk flags as a PR comment, and block merge if any high severity flags are found. One YAML file and a secrets entry gets you there.

Slack/Teams digest. Run on a cron, pipe the changelog entry to a webhook. Engineering managers get a plain-English weekly summary without reading git.

Fine-tuning. If your team consistently disagrees with certain risk classifications, collect those corrections as a small labeled dataset and fine-tune the model on Vertex AI or Colab. Gemma 4's Apache 2.0 license means there are no restrictions on using it as a fine-tuning base for internal tools.

Multi-repo analysis. Pass diffs from multiple services in the same prompt window. The 256K context means you can compare what changed across your backend, frontend, and infra repos in the same analysis run.

Why This Matters

The git history is one of the most information-dense artifacts a software team produces, and it's almost entirely ignored outside of git blame. Gemma 4 31B's context window is large enough to treat a sprint's history as a single document rather than a stream of individual events.

That shift in granularity changes what the model can do: it can notice that a change made on Tuesday was partially reverted on Thursday, that two different authors independently touched the same configuration key, or that a "refactor" commit introduced a subtle behavioral change buried in 400 lines of renames.

None of that is possible when each commit is reviewed in isolation.

Resources

Engineering LLMOps: Building Robust CI/CD Pipelines for LLM Applications on Google Cloud

Jubin Soni — Fri, 01 May 2026 23:48:44 +0000

The transition of Large Language Models (LLMs) from experimental notebooks to production-grade applications requires more than just a well-crafted prompt. As enterprises integrate Generative AI into their core workflows, the need for stability, scalability, and reproducibility becomes paramount. This is where LLMOps—the intersection of DevOps, Data Engineering, and Machine Learning—enters the frame.

Building a CI/CD pipeline for LLM-based applications on Google Cloud Platform (GCP) presents unique challenges. Unlike traditional software, LLM outputs are non-deterministic, making testing complex. Unlike traditional ML, the "model" is often a managed service (like Gemini) or a fine-tuned version of an open-source giant, shifting the focus from training to orchestration, prompt management, and RAG (Retrieval-Augmented Generation) infrastructure.

In this technical deep dive, we will explore how to architect a robust CI/CD pipeline for LLM applications using Google Cloud's suite of tools, ensuring your AI deployments are as reliable as your backend microservices.

The Evolution of the Pipeline: From DevOps to LLMOps

Traditional CI/CD focuses on code integrity, unit tests, and artifact deployment. LLMOps extends this by adding layers for prompt versioning, evaluation against golden datasets, and semantic monitoring.

On Google Cloud, the backbone of this workflow is Cloud Build for orchestration, Vertex AI for model management and evaluation, and Artifact Registry for versioning. The goal is to move away from manual testing in the Vertex AI Studio and toward an automated, repeatable process.

Core Components of the GCP LLM Stack

Vertex AI Model Garden & Model Registry: Centralized hubs for discovering and managing models.
Cloud Build: A serverless CI/CD platform that executes builds on GCP infrastructure.
Vertex AI Pipelines: Based on Kubeflow, these allow you to orchestrate complex ML workflows.
Cloud Run / GKE: For hosting the application logic or serving custom model containers.
Vertex AI Evaluation Service: Provides automated metrics for model performance (e.g., faithfulness, answer relevancy).

Architectural Blueprint: The LLM CI/CD Lifecycle

A robust pipeline must handle three distinct types of updates: changes to the application code, changes to the prompt templates, and updates to the retrieval data (in RAG systems).

The Workflow Logic

This flowchart illustrates the progression from code commit to production. The "Performance Gate" is the most critical addition in LLMOps. It prevents models that hallucinate or provide poor-quality answers from reaching the end user.

Continuous Integration: Beyond Unit Testing

In a standard application, O(1) or O(n) performance and logical correctness are the benchmarks. In LLM apps, we must test for semantic accuracy. CI for LLMs on GCP should include:

Prompt Linting: Checking for formatting and required variables in prompt templates.
Deterministic Testing: Testing the helper functions that format data for the LLM.
LLM-based Evaluation (LLM-as-a-judge): Using a stronger model (like Gemini 1.5 Pro) to grade the output of a smaller, faster model (like Gemini 1.5 Flash).

Practical Code: Automated Evaluation Script

Using the Vertex AI SDK, we can automate the evaluation of a prompt change during the CI phase. The following Python snippet demonstrates how to trigger an evaluation job that measures "fluency" and "safety."

import vertexai
from vertexai.generative_models import GenerativeModel
from vertexai.evaluation import EvalTask, PointwiseMetric

# Initialize Vertex AI
vertexai.init(project="your-project-id", location="us-central1")

# Define the evaluation metric (LLM-as-a-judge)
fluency_metric = PointwiseMetric(
    metric="fluency",
    metric_prompt_template="Rate the fluency of the following text from 1-5.",
)

def run_evaluation(candidate_model_output, reference_data):
    eval_task = EvalTask(
        dataset=reference_data,
        metrics=[fluency_metric],
        experiment="llm-app-v1-eval"
    )

    # Run the evaluation
    results = eval_task.evaluate(
        prompt_template="Summarize this text: {text}",
        model="google/gemini-1.5-flash"
    )

    return results.summary_metrics

# Example usage in a CI script
# if results.summary_metrics['fluency'] < 4.0:
#     sys.exit(1) # Fail the build

Data Management and Versioning

In LLM applications, especially those utilizing RAG, the data is as important as the code. Your pipeline must account for the versioning of the Vector Database index and the embeddings model. If you update your embeddings model (e.g., from Gecko v1 to v2), you must re-index your entire dataset. Failure to do so leads to a "schema mismatch" in semantic space, where the LLM cannot find the relevant context.

Technology Comparison: Serving Options on Google Cloud

Feature	Vertex AI Endpoints	Cloud Run	Google Kubernetes Engine (GKE)
Best For	Managed model serving	Lightweight AI APIs	Large-scale custom deployments
Auto-scaling	Built-in (to zero with some models)	Highly responsive to HTTP traffic	Complex scaling based on GPU usage
Cold Start	Medium	Low (Serverless)	High (unless using warm pools)
GPU Support	Seamlessly managed	Limited (via Sidecars)	Full control over GPU types
Pricing Model	Per-node-hour	Per-request/CPU-second	Cluster-based provisioning

Continuous Delivery: Deployment Strategies

Deploying LLMs requires a safety-first approach. Because LLM behavior can shift with new data or minor prompt tweaks, Canary deployments are essential. Vertex AI Endpoints facilitate this by allowing traffic splitting between multiple model versions.

Sequence of a Managed Deployment

This sequence ensures that if the new prompt version causes a spike in 400-level errors or results in lower semantic confidence scores, the pipeline can automatically roll back to the stable version.

Infrastructure as Code (IaC) with Terraform

To ensure the environment is reproducible, all GCP resources (Vertex AI Indexes, Endpoints, and Cloud Storage buckets) should be managed via Terraform. This prevents "configuration drift," where the staging environment differs from production.

resource "google_vertex_ai_endpoint" "llm_endpoint" {
  name         = "gemini-service-endpoint"
  display_name = "Gemini Service Endpoint"
  location     = "us-central1"
  project      = var.project_id
}

resource "google_cloudbuild_trigger" "llm_pipeline_trigger" {
  name = "deploy-llm-on-push"

  github {
    owner = "your-org"
    name  = "your-repo"
    push {
      branch = "^main$"
    }
  }

  filename = "cloudbuild.yaml"
}

Implementing a "PromptOps" Strategy

One of the most significant shifts in LLMOps is treating prompts as first-class citizens. Instead of hardcoding prompts in the application code, store them as versioned assets.

Branching Strategy for Prompts

Using a Git-based workflow for prompts allows prompt engineers to experiment without breaking the production application logic.

The Cloud Build Configuration

The following is an example of a cloudbuild.yaml file that orchestrates the entire process: running tests, performing model evaluation, and deploying to a staging environment.

steps:
  # Step 1: Install dependencies and run unit tests
  - name: 'python:3.10'
    entrypoint: /bin/sh
    args:
      - -c
      - |
        pip install -r requirements-test.txt
        pytest tests/unit

  # Step 2: Run Vertex AI Evaluation
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: 'python'
    args: ['scripts/evaluate_model.py']
    env:
      - 'PROJECT_ID=$PROJECT_ID'

  # Step 3: Build the application container
  - name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'us-central1-docker.pkg.dev/$PROJECT_ID/app-repo/llm-app:$SHORT_SHA', '.']

  # Step 4: Push to Artifact Registry
  - name: 'gcr.io/cloud-builders/docker'
    args: ['push', 'us-central1-docker.pkg.dev/$PROJECT_ID/app-repo/llm-app:$SHORT_SHA']

  # Step 5: Update Cloud Run Service
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: gcloud
    args: 
      - 'run'
      - 'deploy'
      - 'llm-service-staging'
      - '--image=us-central1-docker.pkg.dev/$PROJECT_ID/app-repo/llm-app:$SHORT_SHA'
      - '--region=us-central1'

images:
  - 'us-central1-docker.pkg.dev/$PROJECT_ID/app-repo/llm-app:$SHORT_SHA'

Monitoring and Feedback Loops

Once an LLM application is in production, the CI/CD pipeline doesn't stop. It transforms into a feedback loop. Google Cloud Monitoring and Cloud Logging can be used to track:

Token Usage: Monitoring costs to prevent budget overruns.
Latency: Tracking time-to-first-token (TTFT) and total response time.
Human-in-the-loop Feedback: Sending flagged responses back to a labeling task in Vertex AI for future fine-tuning.

Handling Non-Determinism

Because LLMs are non-deterministic, your monitoring tools should use statistical significance. Instead of a binary "pass/fail" for every request, look for distribution shifts in the "Helpfulness" score over a window of 1000 requests. If the mean score drops by more than two standard deviations, the pipeline should trigger a rollback or alert the engineering team.

Security and Governance in LLMOps

Security in the CI/CD pipeline for LLMs involves protecting the data used for RAG and the API keys for the model providers.

Secret Manager: Use GCP Secret Manager to store API keys and database credentials. Never hardcode these in your cloudbuild.yaml or application containers.
VPC Service Controls: For enterprises with strict data residency requirements, ensure that Vertex AI is used within a VPC Service Control perimeter to prevent data exfiltration.
IAM Granularity: Assign the least privilege roles. The Cloud Build service account needs roles/aiplatform.user to trigger evaluations but should not have permission to delete model registries.

Conclusion: The Path to Mature AI Delivery

Building a CI/CD pipeline for LLM applications on Google Cloud is an iterative journey. It begins with basic automation and evolves into a sophisticated system capable of semantic evaluation and automated rollbacks. By leveraging Vertex AI and Cloud Build, organizations can treat LLMs not as mysterious black boxes, but as manageable components of a robust software ecosystem.

The key to success lies in the "Performance Gate"—investing heavily in evaluation metrics early on will save hundreds of hours of manual debugging later. As the Generative AI landscape continues to evolve, those with the most resilient pipelines will be the ones who can innovate at the speed of the market without sacrificing reliability.

The Most Important Announcement at NEXT '26 Was a Sidecar

Jubin Soni — Sun, 26 Apr 2026 08:07:59 +0000

This is a submission for the Google Cloud NEXT Writing Challenge

Google Cloud NEXT '26 made 260 announcements. Most of the discussion has rightly gone to the headline acts: the Gemini Enterprise Agent Platform, 8th-gen TPUs, the Cross-Cloud Lakehouse, Agentic Defense.

Announcement #124 is not one of those.

It's titled "Predictive latency boost in GKE Inference Gateway." The official blurb says it cuts time-to-first-token by up to 70% by replacing heuristic guesswork with real-time capacity-aware routing — no manual tuning required. That sentence is engineered to slide past you.

Here's why I think it's the most consequential thing Google shipped this week.

The problem this is actually solving

If you've ever stood up a vLLM cluster on Kubernetes, you've felt this pain:

You have N replicas of the same model. A request lands. Your load balancer has to decide which pod gets it. The "obvious" answers all break:

Round-robin? Ignores that pod 3 is sitting on a 60-token KV cache and pod 7 is at 95% memory pressure.
Least-connections? Treats a 50-token prompt and a 50,000-token prompt as equivalent units of work. They are not.
Cache-aware (route to the pod with the prefix already cached)? Concentrates load. Cache-hot pods melt. Cache-cold pods sit idle.
Utilization-aware (route to the least-loaded pod)? Throws away the entire benefit of prefix caching by scattering related requests.

The standard production answer is what the Kubernetes Inference Gateway calls a "load+prefix scorer" — you give it weights like (prefix=1, queue=1, kv_cache=1) and tune them by hand. The weights you pick are wrong roughly five minutes after you pick them, because traffic shape changes. The weights that worked at 2pm don't work at 2am. The weights that worked for chat workloads don't work when your evals job kicks off.

Everyone running LLM inference at scale has built some version of "we tuned the scorer weights for our workload." Everyone has watched those weights silently rot.

What Google announced

Buried in the GKE keynote, Google linked to a research blog from the llm-d team describing the actual mechanism behind announcement #124. The architecture is shockingly simple — and that's the whole point.

                                  ┌──────────────────────────┐
                                  │   Inference Gateway      │
  request ──────────────────────► │   Endpoint Picker (EPP)  │
                                  └──────────┬───────────────┘
                                             │ "for each candidate pod,
                                             │  predict TTFT and TPOT"
                                             ▼
                                  ┌──────────────────────────┐
                                  │  Latency Predictor       │
                                  │  (XGBoost regression,    │
                                  │   sidecar to EPP)        │
                                  └──────────┬───────────────┘
                                             │ predictions
                                             ▼
                                  pod with best predicted latency wins
                                             │
                                             ▼
                                  ┌────────┐ ┌────────┐ ┌────────┐
                                  │ vLLM 1 │ │ vLLM 2 │ │ vLLM 3 │
                                  └────┬───┘ └────┬───┘ └────┬───┘
                                       │          │          │
                                       └──────────┼──────────┘
                                                  ▼
                                  ┌──────────────────────────┐
                                  │  Trainer sidecar         │
                                  │  observes completed      │
                                  │  requests, retrains      │
                                  │  on sliding window       │
                                  └──────────────────────────┘

There is no large model here. There is no Gemini call in the hot path. The "AI" is a small XGBoost regressor that predicts two numbers per candidate pod:

TTFT — time to first token (dominated by prefill)
TPOT — time per output token (dominated by decode)

It uses six features: KV cache utilization, input length, queue depth, running requests, prefix cache match percentage, and input tokens in flight. That's the whole input.

Then the scheduler routes to the pod with the best predicted outcome. If you provided latency SLOs in the request headers, it does best-fit packing — pick the pod with the least positive headroom, so the others stay free for harder requests later.

That's it. That's the announcement.

Why this matters more than anything else announced

Look at the production numbers from the llm-d post:

Strategy	E2E p50	TTFT p50	TTFT p95	TPOT p99
K8s round-robin baseline	15.98s	4.47s	24.04s	93ms
Load+Prefix `(1,1,1)`	16.42s	2.86s	18.06s	103ms
Load+Prefix `(3,2,2)` (hand-tuned for this workload)	13.42s	3.38s	16.78s	63ms
Predicted-latency	9.06s	0.97s	11.34s	53ms

The hand-tuned heuristic was specifically tuned by humans who looked at seven days of production traffic. The XGBoost model — which retrains on a 1ms-window sliding stratified bucket — beat it by 43% on E2E p50 and 70% on TTFT p50.

This is the part that should make every infrastructure engineer pay attention: the model didn't beat round-robin. It beat the best version of the thing your team is currently running.

The workload was Qwen3-480B on 13 servers with 8×H200 each, simulating realistic Poisson-distributed traffic with concurrency 1000 and ~94% peak prefix cache reuse. That's not a toy benchmark. That's what your stack looks like.

The deeper claim hiding in plain sight

Read this sentence carefully, because it's the actual thesis:

"Accelerator performance is fairly predictable when we account for [server] state and request characteristics."

This is a quietly heretical claim against the entire current direction of LLM ops tooling. A huge amount of effort right now goes into making serving systems more general — disaggregated prefill/decode, KV cache offloading to any filesystem, multi-tier caches across RAM/SSD/GCS (also at NEXT, announcement #125). The complexity is exploding.

The latency-predictor team's bet is the opposite: the system is already deterministic enough that a six-feature regression hits 5% MAPE. Most of what we call "tuning" is just humans doing worse-than-XGBoost approximations of a function that's actually quite learnable.

If that's true — and the production numbers say it is — then a lot of what gets sold as "AI infrastructure intelligence" is going to collapse into very small models that learn very narrow things online. Not LLMs. Not even deep learning. Boosted trees. Trained on the last few hundred completed requests. Retrained constantly.

The ironic punchline is that this announcement, which got dropped in a footnote at NEXT '26, may be a more honest preview of where production AI infrastructure is heading than the entire Gemini Enterprise Agent Platform keynote.

Trying it

You can run this today. The implementation is open-source under the Kubernetes Gateway API Inference Extension. The gateway is the K8s upstream component; what Google did at NEXT was bake it into GKE Inference Gateway as a managed feature.

Once installed, requests opt in via headers — and this is where the design choice gets clever:

curl -v $GW_IP/v1/completions \
  -H 'Content-Type: application/json' \
  -H 'x-prediction-based-scheduling: true' \
  -H 'x-slo-ttft-ms: 200' \
  -H 'x-slo-tpot-ms: 50' \
  -d '{
    "model": "Qwen/Qwen3-32B",
    "prompt": "what is the difference between Franz and Apache Kafka?",
    "max_tokens": 200,
    "stream": "true"
  }'

The two SLO headers are the part to dwell on. You're not telling the gateway how to route. You're telling it what you need, and letting it figure out the routing as a constrained optimization. x-slo-ttft-ms: 200 means "I need first token in 200ms or this is a degraded request." The scheduler computes headroom (predicted_ttft − slo_ttft) per pod and packs accordingly.

This is a real, observable shift in how we think about LLM ops: from imperative ("route to pod 3") to declarative ("meet this SLO"), the same shift that databases went through twenty years ago when query planners replaced hand-written joins.

The EPP exposes -v=4 log lines that let you watch the scorer think:

msg:"Running profile handler"   plugin:"slo-aware-profile-handler"
msg:"Pod score"   scorer_type:"slo-scorer"   pod_name:"vllm-...-9b4wt"   score:0.82
msg:"Picked endpoint"   selected_pod:"vllm-...-9b4wt"

Pair this with announcement #129 — autoscaling on custom metrics — and you have a closed loop: the predictor surfaces SLO headroom, the autoscaler reacts to headroom collapse before queue depth even spikes. Most autoscaling triggers fire after the system is already in pain. This one fires when the forecast says pain is 30 seconds away.

What I'd watch next

A few open questions that the announcement and the underlying paper don't fully resolve:

The model assumes a homogeneous accelerator pool. In real fleets you have H100s and H200s and B200s mixed together, with different price/performance curves. The team flagged this as future work; whoever solves it well wins the heterogeneous-GPU-cost-optimization market that nobody is talking about yet.

The trainer runs as a sidecar to the EPP and retrains continuously. At the QPS levels in the scaling table — 10,000 QPS needs 4 prediction servers — the cost of the routing decision starts to be non-trivial relative to the inference itself. There's a coordination cost story here that's missing from the blog post.

And the bigger question: this technique generalizes. The same XGBoost-on-six-features approach should work for autoscaling, for spot/on-demand routing decisions, for cache eviction policies, for batch scheduling. If Google ships predicted-latency primitives across the rest of GKE, the consequences are larger than a single-feature blog post implies.

Closing

The contest prompt asks for the announcement that "speaks to you." The honest answer for me is: the boring sidecar with the unglamorous name that takes a week of pain — the slow rot of hand-tuned scorer weights — and replaces it with something that retrains itself.

Everyone watching the keynote saw the agent demos. The serving runtime is where the actual money gets won or lost, and it's where six-feature regression beats a roomful of senior SREs with grafana dashboards. That's the announcement I think we'll be talking about in 18 months.

The agent layer makes for a better trailer. The runtime layer is the movie.

Sources & credits: Technical details, production benchmark numbers, and architecture diagram concept drawn from the llm-d project's "Predicted-Latency Based Scheduling for LLMs" post (March 2026) by Kaushik Mitra, Benjamin Braun, Abdullah Gharaibeh, and Clayton Coleman, and the Google Cloud NEXT '26 Wrap-Up (announcement #124). The opinions, framing, and analysis are mine. AI tools were used as a writing assistant; all technical claims trace to the linked primary sources.

What Does OpenClaw Take From You?

Jubin Soni — Sun, 26 Apr 2026 07:46:29 +0000

This is a submission for the OpenClaw Challenge.

TL;DR: The conversation about personal AI is almost entirely about what these agents give you. The harder question — and the one that determines whether the deal is actually good — is what they take. Here are three things personal AI is quietly absorbing, and what I think you should keep.

We all know the story around personal AI. It gives you time back. It automates. It amplifies. It handles your inbox while you sleep, summarizes your morning, drafts replies, books flights, files receipts, sends messages—so you don’t have to. The language never really changes: gain. More throughput. More execution. More delegation. More agency.

This framing is missing half the equation. Anything you offload, you stop doing. Anything you stop doing, you eventually stop being good at. And anything you stop being good at, you eventually stop noticing you used to be good at.

This is not a luddite essay. I want this technology to work. OpenClaw, specifically, is one of the more honest things in the personal AI space — file-first, locally hosted, legible memory, open-source ethos. If any agent framework is going to be defensible five years from now, it is probably this one. But that is exactly why the question matters more here than it does for some hosted SaaS chatbot. OpenClaw is not a toy. It is built to actually live in your life. And the things it is built to absorb are not random — they are a specific class of cognition that, until very recently, you did yourself.

So: what is in that class? Three things, because I think the conversation needs the vocabulary.

The first thing: the friction that makes you decide

It is tempting to treat every recurring annoyance in your life as something to automate away. Bills. Calendar conflicts. Inbox triage. Deadline tracking. Grocery lists. Household coordination. The agent handles them, the annoyance goes away, the win goes on the board.

Friction is not always a bug.

The reason you used to look at your bills before paying them is not that you enjoyed the experience. It is that the act of looking — even for two seconds — sometimes caught the thing that mattered. The duplicate charge. The subscription you forgot you had. The number that was higher than last month and signaled something upstream in your life was off. The five seconds of friction was a sampling pass on your own financial reality, run weekly, for free.

When you build an agent that summarizes the bills and tells you the total, you have not just removed the friction. You have removed the sampling pass. The summary will tell you what the agent thinks is interesting. It will not tell you what you would have thought was interesting if you had looked, because you no longer have the muscle to know.

This is not theoretical. It is the same pattern that GPS did to your sense of direction, that autocomplete did to your spelling, and that calculators did to your arithmetic. In each case the technology was net-positive. In each case something specific and unrecoverable was traded away. We made those trades half-consciously because we did not have a vocabulary for what was on the other side of the ledger.

The personal-agent generation is making bigger trades, faster, with even less vocabulary.

The second thing: the practice of small decisions

There is a category of decision that is too small to think about and too consequential to skip.

What to reply to that ambiguous Slack message. Whether the email from your landlord needs a same-day response or can wait until Monday. Whether the meeting your colleague proposed at 4 p.m. is one you should accept or politely deflect. Whether the calendar conflict your assistant just flagged is a real conflict or one of those situations where it is fine to be ten minutes late to the second thing.

Personal agents are very good at the first 80% of these decisions and quietly bad at the last 20%. The first 80% — the obvious cases — is where they shine and where the demos look great. The last 20% — the cases that require taste, social calibration, and an accurate model of the specific humans involved — is where they fail in ways that do not show up in any benchmark, because the failure mode is the agent did something locally reasonable that was globally wrong, and you did not notice until it was too late.

The deeper problem is that the small-decisions practice is how taste is built in the first place. You develop a sense for which Slack messages need a careful reply by replying to a thousand of them, badly at first, and getting feedback from how the relationship went. If your agent handles the first nine hundred and fifty, you arrive at message nine hundred and fifty-one with the calibration of a beginner.

The framing of "delegate the boring stuff and focus on the important stuff" assumes three things: that the boring stuff and the important stuff are clearly separated, that the boring stuff does not feed into the important stuff, and that you can train the agent on the boring stuff without losing access to the inputs that would have eventually made you good at the important stuff. None of these assumptions survive contact with how human skill actually develops.

The third thing: the silence in which you notice you were wrong

This one is harder to name and I think it is the most important.

Right now, when you have a thought that is incomplete, a plan that is half-formed, or an instinct that something is off, there is a natural waiting period. You sit with it. You go for a walk. You stare at the ceiling for an hour. Eventually, sometimes, the thing resolves. You realize the project you were excited about is actually a bad idea. You realize the email you drafted last night was angrier than you intended. You realize the person you were going to call does not actually need a call from you. They need space.

This kind of cognition does not happen in language. It happens in the gaps between language. It is what your nervous system does when nothing is asking it for output.

Personal AI agents are, by their nature, output machines. They want to be useful. They want to give you something. The honest, well-built ones — and OpenClaw is honest and well-built — are designed to be proactive, to surface things, to ping you with the briefing, to suggest the next step. The whole pitch is that they fill the gaps.

But the gaps were doing work.

The morning before you check your phone. The walk to the coffee shop where you have not yet asked the agent anything. The half-hour of unstructured staring before the meeting. These are not inefficiencies in your life that an agent should be optimizing away. They are the conditions under which your slower, more honest cognition can operate. Compressing them does not give you back time. It gives you back the same amount of time, minus the part of your mind that needed the silence to work.

This is the trade nobody in the personal AI space wants to look at directly, because looking at it threatens the entire growth story. If the value of the agent is partly a function of what it disrupts in your inner life, and if some of what it disrupts is irreplaceable, then the unbounded "delegate everything" pitch starts to look less like a productivity story and more like a deal you should sign carefully.

What to actually do

Use OpenClaw. I mean that. The category is real, the project is good, and the alternative — keeping your data with hosted platforms whose pricing pages will change without your consent — is worse on almost every axis.

But sign the deal carefully. The rule I would actually follow is the simplest one I can write down:

Pick the offloads where the friction is genuinely friction.
Keep the offloads where the friction is doing work.
Leave the gaps alone.

The first one is for things where the human cost is high and the cognitive value is zero. Receipt parsing. Standard meeting confirmations. Repetitive document formatting. Things that genuinely should have been a script.

The second one is for the things where the friction is the point. Read your own bills. Reply to your own ambiguous Slack messages, at least most of them. Look at your own calendar before you ask the agent to look at it. Treat the small-decisions practice like a gym membership — something you do not because you cannot afford the alternative, but because you understand what your body becomes if you stop using it.

The third one is the hardest, because the agent is built to fill the gaps and your brain is built to let it. The morning before your first meeting. The walk where you have not yet opened a chat. The half-hour of unstructured staring. Leave them alone. The silence is not a bug to be fixed. It is the thing keeping the rest of it alive.

Personal AI is going to be one of the largest technology shifts of the next decade, and OpenClaw is going to be in the middle of it. The question is not whether to participate. It is what you intend to keep, and what you are quietly agreeing to give up.

Most of the conversation right now is an accounting of the gains.

Somebody should account for the rest.

What's an offload you regret? Or one you almost made and pulled back from? I'd genuinely like to hear it.

DEV Community: Jubin Soni

I Built a VS Code Extension to Debug Azure AI Foundry Agents Without Leaving My Editor

The problem

What Foundry Trace Inspector does

Trajectories — the full span tree

Duration

Tokens

User View — readable conversation replay

Token & Cost chart

How it works under the hood

Setting it up

What's next

Try it

Further Reading

Amazon CodeWhisperer Q Developer Kiro: The Rise of Agentic Coding

The Abrupt End of Amazon Q Developer

Historical Context: From CodeWhisperer to Q Developer to Kiro

Architecture Comparison

Feature-by-Feature Breakdown

Production Code Example 1: Spec-Driven Microservice Scaffolding with Kiro

Production Code Example 2: Using Kiro Hooks for Automatic Test Generation

Production Code Example 3: Migrating Q Developer Inline Chat Workflows to Kiro

When to Migrate Now vs Wait

Performance & Productivity Metrics

Summary

References

Building a Vector Index in Azure AI Search: HNSW, Profiles, and RAG Retrieval

What is a RAG system, in short

Why we need vector search

How Azure AI Search performs vector search

The architecture of our RAG system

The data model

Hands-on tutorial

Step 1: Create an Azure AI Search service

Step 2: Install the Python libraries

Step 3: Create the vector index

Step 4: Embed and upload the documents

Step 5: Search using a query vector

Step 6: Build the RAG prompt

Using Azure OpenAI embeddings instead

A few practical notes

Conclusion

References

Amazon OpenSearch Vector Search Explained for RAG Systems

What is a RAG system, in short

Why we need vector search

How Amazon OpenSearch performs vector search

The architecture of our RAG system

The data model

Hands-on tutorial

Step 1: Run OpenSearch locally

Step 2: Create the k-NN index

Step 3: Connect, embed, and index the documents

Step 4: Search using a query vector

Step 5: Build the RAG prompt

Running this against Amazon OpenSearch Service

A few practical notes for production

Conclusion

References

Jules: Google's Async Coding Agent Is Changing How We Think About AI and Software Development

What Jules Actually Does (and Doesn't Do)

Architecture: How Jules Is Built

Sequence: The Async Flow End-to-End

Code: Using Jules in Your Workflow

1. Jules CLI (Jules Tools — GA at I/O 2026)

2. Via Gemini CLI Extension

3. Jules API (for CI/CD integration)

4. GitHub Actions Integration

Practical Workflow: What Jules Is Good At

Jules vs. the Field

What Jules Gets Right — and Where It's Still Incomplete

The Bigger Picture: What This Means for SWEs

References

Amazon Quick: AWS's Agentic Workspace, Explained for Engineers

What Amazon Quick Is

Product Components

Where Quick Sits in the AWS Agent Stack

The Integration Architecture

High-Level Request Flow

Building an MCP Server for Quick