DEV Community: Xavier Portilla Edo

Durable Genkit Flows with Temporal: Introducing the genkitx-temporal Plugin

Xavier Portilla Edo — Tue, 29 Sep 2026 14:23:24 +0000

Introduction

Anyone who has shipped a non-trivial Gen AI feature to production has hit the same wall: the happy path is fun to demo, but the unhappy path is brutal. Model providers throw 5xx errors. Rate limits kick in at the worst possible moment. A long-running agent gets halfway through a 12-step plan and the pod is restarted by Kubernetes. A user closes the tab in the middle of a streaming response and you have no idea what state the flow ended up in.

Genkit gives you a clean way to author those LLM-orchestrating pipelines as flows, but a Genkit flow, by itself, is just a function. It lives and dies with the process that invokes it. There is no durable history, no automatic retry policy, no built-in cancellation, no operations UI.

This is exactly the gap that Temporal was designed to close for general-purpose backend code, and it is the gap that the new genkitx-temporal plugin closes for Genkit flows.

In this article we will look at:

What the plugin actually does.
Why Temporal and Genkit are a particularly good match.
How to use it end to end: define, register, run.
When you should reach for it (and when you probably shouldn't).

What is `genkitx-temporal`?

genkitx-temporal is a Genkit plugin that lets you execute any Genkit flow inside a Temporal Workflow, transparently. You keep writing flows the way you always have; the plugin takes care of:

Registering the flow so a Temporal Worker can pick it up.
Wrapping each execution in a deterministic Temporal Workflow.
Running the non-deterministic LLM/tool/RAG work inside a Temporal Activity, so retries and timeouts are safe.
Giving you helpers to start workflows from any client.

Under the hood, the plugin ships a generic, deterministic workflow called runGenkitFlow that invokes a single Temporal Activity called runGenkitFlowActivity. The activity looks up your Genkit flow by name in an in-process registry and runs it inside a full Node environment, where all the messy, non-deterministic things (network calls, model calls, tool calls, RAG lookups) are perfectly fine.

┌────────────┐  start workflow   ┌──────────────────────┐
│  Client    │ ────────────────▶ │  Temporal Server     │
└────────────┘                   └─────────┬────────────┘
                                           │ task
                                           ▼
                                ┌──────────────────────┐
                                │  Worker process      │
                                │  ┌────────────────┐  │
                                │  │ runGenkitFlow  │  │  (workflow, sandboxed)
                                │  │       │        │  │
                                │  │       ▼        │  │
                                │  │ runGenkit-     │  │  (activity, full Node)
                                │  │ FlowActivity   │  │
                                │  │       │        │  │
                                │  │       ▼        │  │
                                │  │  your Genkit   │  │
                                │  │  flow (LLM…)   │  │
                                │  └────────────────┘  │
                                └──────────────────────┘

This split is the whole trick. Temporal Workflows are required to be deterministic so that they can be replayed from event history after a crash, deploy or scale event. LLM calls obviously are not deterministic. Putting the LLM work inside an Activity is the canonical Temporal pattern, and the plugin does it for you so you don't have to think about it.

Why Temporal and Genkit are a great match

It is easy to underestimate how much engineering is hiding behind "just call the model again if it fails". Once you start building real agents, the list of things you want from your runtime grows quickly. Genkit gives you the authoring primitives, and Temporal gives you the runtime guarantees. Together:

Automatic retries for transient errors. LLM providers regularly return 429s and 5xx. Tool calls hit the network. Vector stores time out. Temporal lets you express retry policies (exponential backoff, max attempts, non-retryable error types) declaratively, applied to every flow execution.
Durable history. Every event in your flow's execution is persisted by the Temporal Server. If your Worker pod is killed mid-flow, another Worker picks up the workflow exactly where it left off, with all prior activity results intact. No partial state, no double-charging the user, no orphan jobs.
Timeouts, heartbeats and cancellation. Long-running agents that browse, plan and call tools for minutes are a nightmare to control with plain HTTP. Temporal models start-to-close, schedule-to-close and heartbeat timeouts as first-class concepts, plus explicit cancellation semantics you can wire to a user closing a tab.
Operational visibility out of the box. The Temporal UI gives you a per-execution timeline of every workflow and activity. You can see inputs, outputs, retries, failures and stack traces, search by workflow id, terminate or signal running workflows. This complements the Genkit Developer UI nicely: Genkit's UI is your local debugging tool during development; Temporal's UI is your operational dashboard in staging and production.
Horizontal scalability for free. Need more throughput? Run more Worker processes pointing at the same task queue. The Temporal Server load-balances workflows and activities across them. Your flows did not need to change.
Long-running, human-in-the-loop friendly. Temporal workflows can sleep for days, wait for signals from external systems, and resume cleanly. Perfect for agents that need approval before executing a sensitive tool, or RAG pipelines that wait for a document indexing job to finish.

In other words, genkitx-temporal turns your Genkit flows from "smart functions inside a Node process" into first-class durable workloads without changing the way you write them.

Installation

npm install genkitx-temporal genkit

The Temporal SDK packages are peer-installed automatically as dependencies of the plugin.

You also need a running Temporal Server. For local development, the easiest path is:

brew install temporal
temporal server start-dev

This starts a Temporal Server on localhost:7233 and the UI on http://localhost:8233.

Usage end to end

The plugin exposes a small, focused API. Three things are enough to ship a durable Genkit flow: define it with the Temporal-aware helper, start a Worker, and execute it from a client.

1. Define a flow with `defineTemporalFlow`

defineTemporalFlow is a drop-in replacement for ai.defineFlow. The returned object is a normal Genkit flow, so you can still call it directly, expose it via the Developer UI, run it from tests, etc. The only difference is that it is also registered internally so a Temporal Worker can find it by name.

// flows.ts
import { genkit, z } from 'genkit';
import { googleAI } from '@genkit-ai/google-genai';
import { defineTemporalFlow, temporal } from 'genkitx-temporal';

export const ai = genkit({
  plugins: [
    googleAI(),
    temporal({ taskQueue: 'my-queue' }),
  ],
  model: googleAI.model('gemini-flash-latest'),
});

export const jokeFlow = defineTemporalFlow(
  ai,
  {
    name: 'jokeFlow',
    inputSchema: z.string(),
    outputSchema: z.string(),
  },
  async (subject) => {
    const { text } = await ai.generate(`Tell me a joke about ${subject}`);
    return text;
  },
);

A few things to notice:

You configure the plugin like any other Genkit plugin, passing the Temporal task queue (and optionally address, namespace, etc.).
The flow body is plain Genkit code. No Temporal-specific imports, no proxyActivities, no determinism gymnastics. The plugin handles all that for you.
The flow's name is also its Temporal registration key. Keep it unique within your Worker.

2. Start a Worker

Workers are the processes that actually execute your flows. A Worker imports your flows (so the registry is populated) and then calls startTemporalWorker:

// worker.ts
import './flows';   // side-effect import: registers the flows
import { startTemporalWorker } from 'genkitx-temporal';

startTemporalWorker({ taskQueue: 'my-queue' })
  .catch((e) => { console.error(e); process.exit(1); });

node ./dist/worker.js

You can run as many Worker processes as you want against the same task queue. Temporal will distribute work across them automatically. If a Worker dies mid-flow, another one picks up.

3. Execute a flow as a Temporal Workflow

From any client (an HTTP handler, a CLI, a cron job, another workflow), use executeTemporalFlow to start a workflow and await its result:

// client.ts
import { executeTemporalFlow } from 'genkitx-temporal';
import { jokeFlow } from './flows';

const result = await executeTemporalFlow(jokeFlow, 'cats', {
  taskQueue: 'my-queue',
});
console.log(result);

If you don't want to block on the result, use startTemporalFlow instead. It returns the raw Temporal WorkflowHandle, which lets you query, signal, or cancel the running workflow later. This is the building block for human-in-the-loop scenarios, scheduled flows, fan-out/fan-in patterns, and so on.

Configuration

temporal(options) and every helper accept the same connection options. Anything you don't pass falls back to environment variables, then to sensible defaults:

Option	Env var	Default
`address`	`TEMPORAL_ADDRESS`	`localhost:7233`
`namespace`	`TEMPORAL_NAMESPACE`	`default`
`taskQueue`	`TEMPORAL_TASK_QUEUE`	`genkit`

This makes it straightforward to run the same code locally against a dev server and in production against Temporal Cloud or a self-hosted cluster.

Advanced: combine with your own workflows and activities

The bundled runGenkitFlow workflow is enough for the common case. If you want to mix Genkit flows with your own existing Temporal workflows and activities, the plugin lets you bring your own:

await startTemporalWorker({
  taskQueue: 'my-queue',
  workflowsPath: require.resolve('./my-workflows'),
  activities: { ...require('./my-activities') },
});

// my-activities.ts
export { runGenkitFlowActivity } from 'genkitx-temporal/activities';
export async function myOtherActivity(/* ... */) { /* ... */ }

Re-exporting runGenkitFlowActivity keeps the built-in workflow working, so you can compose Genkit flows alongside hand-written activities for the parts of your system that aren't AI.

API summary

The full public surface is tiny:

temporal(options?) — the Genkit plugin.
defineTemporalFlow(ai, config, fn) — define a flow and register it for Temporal execution.
startTemporalWorker(options?) — start a Worker process.
executeTemporalFlow(flow, input, options?) — run a flow inside a Workflow and await the result.
startTemporalFlow(flow, input, options?) — same, but returns the WorkflowHandle for fire-and-forget / signalling.
runGenkitFlowActivity — the underlying activity, re-exported so you can combine it with your own activities.
registerTemporalFlow(name, flow) — manually register a flow that was defined elsewhere (useful when wrapping flows you don't own).

A small API surface is the point. The plugin is intentionally a thin bridge between two well-designed systems; it does not try to reinvent either of them.

When to reach for `genkitx-temporal`

Not every flow needs a durable runtime. A streaming chat response that takes 800ms and either succeeds or is retried by the user is fine running on a plain HTTP handler.

You will feel the benefits the moment your flows look like one of these:

Multi-step agents that orchestrate several model calls and tool invocations, where partial progress is expensive to throw away.
Long-running pipelines (document ingestion, batch summarization, fine-tuning prep) where individual steps can take minutes and the process must survive deploys.
Critical business workflows (refunds, account changes, contract generation) where you cannot afford to lose state or accidentally execute a step twice.
Human-in-the-loop agents that need to pause for approval, an external webhook, or a manual review before proceeding.
Anything you currently glue together with a job queue, a retry library, and a state machine. Temporal subsumes all three, and genkitx-temporal plugs Genkit straight into it.

If your team already runs Temporal for non-AI workloads, this plugin is a no-brainer: it lets your Gen AI features inherit all the operational maturity your platform team has already built around it.

How it pairs with the rest of the Genkit ecosystem

The thing I like the most about this plugin is that it composes cleanly with everything else Genkit gives you, not just one or two features:

Genkit Developer UI. Because defineTemporalFlow returns a normal Genkit flow, you can still iterate on it locally with the Genkit Developer UI: fast feedback loop during development, durable execution in production.
Genkit middleware. Middleware (in-call retries, fallbacks, skill injection, tool approval, prompt rewriting, etc.) runs inside the activity. You get two complementary layers of resilience: middleware for fine-grained in-call recovery, Temporal for whole-flow durability and replay.
Tools and function calling. Tools defined with ai.defineTool are just normal flow code from Temporal's perspective. Their calls, retries and outputs appear in both the Genkit trace and the Temporal event history.
RAG primitives. Retrievers, indexers, embedders and rerankers all live inside the activity. That means heavy ingestion jobs (chunking, embedding, upserting into a vector store) inherit Temporal's retry policies and survive restarts mid-batch.
Evaluators and datasets. Genkit's evaluators are flows like any other, so you can run eval jobs as Temporal Workflows, schedule them, fan them out across Workers, and inspect every run in the Temporal UI.
Prompts and Dotprompt. Versioned .prompt files, prompt registries and structured outputs all work unchanged. The flow body is plain Genkit.
Plugins and model providers. Any Genkit plugin (Google AI, Vertex AI, OpenAI, Anthropic, Ollama, local models, vector stores, etc.) plugs in as usual; the Temporal layer doesn't care which provider is on the other side of the call.
Telemetry and tracing. Genkit's OpenTelemetry traces continue to be emitted from inside the activity, so they show up in whatever observability backend you already use, alongside Temporal's own event history.
Deployment surfaces. Flows can still be exposed as HTTP endpoints, Cloud Functions, Firebase callable functions or Express handlers; executeTemporalFlow is just one more entry point, and you can mix and match (e.g. quick chat requests over HTTP, long agent runs through Temporal).
Multi-language story. Genkit is available in JS/TS, Go, Python (preview), Dart/Flutter (preview) and through a community Java SDK. This particular plugin targets JS/TS, but the architectural pattern — define your AI logic in Genkit, run it as a Temporal Workflow — is reusable across runtimes thanks to Temporal's polyglot SDKs.

Conclusion

Genkit is a great way to author Gen AI logic. Temporal is a great way to run any long-running, failure-prone workload. genkitx-temporal is the missing adapter between the two: a small, focused plugin that turns every Genkit flow into a durable, retryable, observable Temporal Workflow without asking you to rewrite a single line of business logic.

If you are building anything more ambitious than a single-turn chat endpoint, give it a try:

Source: github.com/xavidop/genkitx-temporal
Docs: xavidop.github.io/genkitx-temporal
Runnable example: examples/test-app

Your future on-call self will thank you the first time a model provider has a bad afternoon and your agents keep humming along regardless.

Why a Local Debugging Tool is Non-Negotiable for Building AI Apps: Genkit Developer UI vs Vercel AI SDK DevTools vs Mastra Studio

Xavier Portilla Edo — Tue, 29 Sep 2026 14:23:17 +0000

Introduction

If you have ever shipped a backend service without a debugger, a hot-reload dev server or decent logs, you remember how painful it was. Now imagine doing it in a world where your "function" is a non-deterministic black box that can rewrite its own output every time you call it. Welcome to building Gen AI applications.

The single biggest productivity multiplier I have found in the last two years of shipping AI apps is a good local debugging tool. Not a hosted dashboard. Not a cloud trace viewer with a 30-second propagation delay. A local tool that runs next to your code, picks up your edits, lets you replay a request with a tweaked prompt, and shows the full trace, tool call by tool call. We will get into the local-vs-hosted trade-off later in the article.

This article is about why that matters and how the three leading JS/TS Gen AI frameworks approach it:

They are three very different products solving overlapping problems, and each one makes different trade-offs that are worth understanding before you commit to one.

Shift-left, applied to Gen AI

"Shift-left" comes from the world of testing and security: the earlier in the development lifecycle you catch a problem, the cheaper it is to fix. Bugs found in production cost orders of magnitude more than bugs found while you're typing.

Gen AI apps make this principle existential, not just economical, because the failure modes are weirder:

A prompt regression doesn't show up as a stack trace. It shows up as the wrong tone, a hallucinated fact or a tool call that misfires once every twenty runs.
A new model version can quietly change behavior across thousands of code paths with no warning.
A tool that returns slightly different JSON can cause silent downstream breakage.

The only way to keep your sanity is to collapse the feedback loop to seconds. You want to be able to:

Change a prompt or a piece of orchestration code.
Re-run that exact unit, with that exact input.
See the model's input, its output, every tool call, every retry, every token used.
Compare it to the previous run.
Decide if it's better, worse or different.

If any one of those steps requires deploying, redeploying, opening a cloud console or grep'ing logs, you have already lost. Cycle time is the metric. A debugging tool is what makes that cycle time low enough to iterate productively.

Local vs hosted

It is worth being explicit about this trade-off, because all three tools in this article are primarily local:

A local debugging tool runs on your machine, next to your code, with zero network latency between your edits and what you see. Iteration is fast, traces are immediate, and there is no risk of leaking prompts/responses to a third party. The downside is that it's just for you.
A hosted observability platform (Langfuse, LangSmith, cloud-native APM tools, etc.) is shared by your team, persists data long-term and is essential for production monitoring. The trade-off is propagation delay, configuration overhead and, in some cases, data residency concerns.

These are complementary, not competing. The argument here is that the local part of the loop is the one that's still often missing, and that's where shift-left lives.

What "good" looks like

After working with all three of the tools below, I would summarize the qualities of a good Gen AI debugging tool as:

Zero-config or near-zero-config — it should pick up your code, not the other way around.
Live reload — edit your code, save, hit "run" again. No restart.
Interactive runners — invoke any prompt, tool, model, or higher-level primitive (agent, workflow, etc.) directly with arbitrary input.
Full traces — every step of the generation loop, with input/output/usage at each node.
Replay and modify — re-run any past trace with a tweak.
Minimal code changes — the framework should be observable by default, ideally without you having to remember to wrap each model call.

Let's see how each tool stacks up.

Genkit Developer UI

The Genkit team treats the Developer UI as a first-class part of the framework. It ships with the genkit-cli package and requires no code changes in your application to attach to your running process.

You install the CLI once globally:

npm install -g genkit-cli

And then start your app under it:

genkit start -- npx tsx --watch src/index.ts

That's it. Genkit attaches to your running Node process, discovers every flow, prompt, model, tool, retriever, indexer, embedder and evaluator you have defined, and exposes them all in a local web app at http://localhost:4000.

Telemetry API running on http://localhost:4033
Genkit Developer UI: http://localhost:4000

What you actually get inside:

Interactive runners for every primitive. Flows, prompts, tools, models, retrievers, indexers, embedders and evaluators all get an interactive panel where you fill out the input (validated against the Zod schema) and hit run.
Full traces with the entire generation graph: every model call, every tool invocation, every middleware, with input/output/usage tokens.
Live reload. Combined with --watch, edits to your code show up without restarting the UI.
Prompt iteration. Tweak prompts and re-run inline.
Evals. Run evaluators against datasets directly from the UI.
Open by default. Add -o to auto-open in your browser.

What stands out here is that observability comes for free. You do not wrap your model and you do not add a middleware just to see traces. Anything defined with the Genkit primitives is automatically introspected and traced, which makes the "let me see what's happening" cost essentially zero.

If you also use the middleware system, every middleware also shows up in the trace, which makes debugging things like retries and fallbacks a lot easier.

Vercel AI SDK DevTools

Vercel introduced the AI SDK DevTools more recently, and it is a meaningful step up from having nothing. The design philosophy is different from Genkit's: it is a middleware-based capture tool that focuses tightly on the language-model layer, rather than an integrated development environment for the whole AI app.

You opt in by wrapping your model:

import { wrapLanguageModel, gateway } from 'ai';
import { devToolsMiddleware } from '@ai-sdk/devtools';

const model = wrapLanguageModel({
  model: gateway('anthropic/claude-sonnet-4.5'),
  middleware: devToolsMiddleware(),
});

And then run the viewer in another terminal:

npx @ai-sdk/devtools
# open http://localhost:4983

What it captures:

Input parameters and prompts.
Output content and tool calls.
Token usage and timing.
Raw provider request/response payloads.
Multi-step interactions are grouped into "runs" with multiple "steps".

Things to keep in mind when comparing it with the others:

It is opt-in per model. Every model you want to observe has to be wrapped explicitly. In a real codebase with many models, this usually means a shared factory or repeated wrapping. If you forget on one path, you have a blind spot.
It focuses on the language-model layer. Anything happening above that layer (your own application logic, custom orchestration, business steps) is not captured unless you add it yourself.
It is a viewer, not an interactive playground. You cannot "re-run this with a different prompt" from the UI; you go back to your code, edit and re-call.
It does not enumerate AI primitives because, in the AI SDK, those are just functions in your codebase. There is nothing to enumerate at runtime.
Storage is a JSON file (.devtools/generations.json). The team is clear that this is local-only, never for production, and the middleware automatically appends .devtools to your .gitignore.

The upside is that it is probably the quickest of the three to drop into an existing project: install the package, wrap a model, run the viewer. If you live inside Next.js and the AI SDK and you mainly want a "what just happened?" panel for your model calls, it does the job well. If you are debugging a multi-step agent with custom orchestration, you will probably want to combine it with something that sees more of the picture.

Mastra Studio

Mastra Studio is the most ambitious of the three in terms of surface area. It is bundled with mastra dev and runs at http://localhost:4111, doubling as a local development tool and a deployable team console (you can ship Studio to production for non-developers).

Out of the box, it gives you:

Agents tab — chat with your agents, hot-swap models, tweak temperature/top-p, view traces, attach scorers.
Workflows tab — visualize workflows as graphs, run them step by step with custom inputs, watch the active step in real time, inspect tool calls and JSON outputs.
Processors tab — see input/output processors and guardrails wired to each agent.
Tools tab — run tools in isolation to debug them.
Workspaces tab — file browser into the agent's workspace filesystem, with a Skills tab listing discovered skills.
MCP servers tab — list attached MCP servers and their tools.
Request context — set runtime variables that flow into agent instructions/tools through DI, with schema-driven forms.
Evaluation suite — Scorers, Datasets and Experiments tabs to run datasets through agents/workflows, attach scorers and compare experiments side-by-side.
Observability with traces and logs.
Editor integration for non-technical teammates to iterate on agents and version every change without redeploying.

This is more than a debugging tool — it is closer to a development environment plus a team-facing console. The natural trade-off is that Studio is tightly coupled to Mastra's primitives (Agents, Workflows, Processors, Workspaces), so it shines the most when your application is structured the Mastra way.

If you build non-trivial agentic systems and you want evaluation and dataset management baked into the same tool, Studio is genuinely very good. For a quick "see my prompt and the model's response" loop, it can feel like more than you need.

Side-by-side comparison

Capability	Genkit Developer UI	Vercel AI SDK DevTools	Mastra Studio
Setup	`genkit start --` your code	`wrapLanguageModel(...)` + `npx @ai-sdk/devtools`	`mastra dev`
Code changes required	None (auto-discovers)	Yes (wrap each model)	None (auto-discovers Mastra primitives)
Live reload	Yes (`--watch`)	N/A (viewer only)	Yes
Interactive runners	Flow, prompt, model, tool, retriever, indexer, embedder, evaluator	None (viewer only)	Agent chat, workflow runner, tool runner
Traces	Full generation graph	Per-model-call run/step	Workflow + agent traces
Re-run / replay	Yes, from any primitive	No (must edit code and re-call)	Yes, including workflow step-through
Dataset / eval UI	Evaluators panel	Not built-in	Datasets + Experiments + Scorers
Workspace / files browser	No	No	Yes
MCP server browser	Not first-class	No	Yes
Deployable to team	No (local-only)	No (local-only)	Yes (Studio on Mastra platform)
Languages	JS/TS (Go, Python, Dart all have local tooling too)	JS/TS only	JS/TS only
Style	Auto-discovery of primitives	Per-model middleware capture	All-in-one studio for the agent lifecycle

A quick note on the Vercel column: simplicity of integration is a real feature for many teams, and it is fair to call this the lightest-touch option. The trade-off is that observability is opt-in per model, so you have to be deliberate to avoid blind spots.

When I would pick which

After using all three, this is roughly how I'd think about it:

Building a serious agent or multi-step pipeline with tools, retries, fallbacks and evals → Genkit Developer UI is hard to beat on pure iteration speed. Zero-config tracing of the full pipeline, every primitive runnable, and middleware shows up in the trace for free.
Working primarily inside Next.js with the Vercel AI SDK and you mainly want better visibility into model calls → Vercel AI SDK DevTools is a quick win. Lightweight setup and it does what it says on the tin; pair it with something else if you need higher-level orchestration views.
Building agentic systems with workflows, datasets and scorers, and you want a UI your PM or domain expert can also use → Mastra Studio. The evaluation suite and the deployability are real differentiators.
Multi-language stack (JS/TS plus Go, Python or Dart) → Genkit is the only one of the three with first-party multi-language support; the local tooling story is consistent across runtimes.

The moral of the story: pick a debugging tool, any debugging tool

In a normal backend project, you can sometimes get away with a weak dev loop because the code is deterministic. Bad UX, but it works.

In an AI project, the code is not deterministic. Every iteration is also a small experiment. If your loop is slow or your visibility is poor, you don't just iterate slower, you iterate worse, because you can't tell whether your changes are improvements or regressions. Shift-left in this world is not a virtue, it is a survival strategy.

A good local debugging tool:

Turns "I think the prompt got better" into "I can see the trace, the tokens, the eval score, side by side".
Catches regressions before they reach a user.
Lets you onboard new engineers in hours instead of days, because they can see the system.
Lets non-engineers (PMs, designers, domain experts) participate in iteration when the UI is friendly enough — Mastra Studio is explicit about this.

If you are starting a Gen AI project today and you are picking a framework, the local tooling story should weigh as much as the model abstraction or the tool API. It is the difference between writing AI code and engineering AI features.

Conclusion

All three tools are good in their lane:

Genkit Developer UI is the most introspective of the three and requires zero code changes to be useful. It's the one I reach for when I want to move fast and see every step of the pipeline.
Vercel AI SDK DevTools is the lightest-touch option to drop into an existing AI SDK app, with a clear focus on the model-call layer. Think of it as a focused viewer rather than a full development environment.
Mastra Studio is the most ambitious, closer to a full IDE-meets-console for agentic systems, with first-class evaluation, datasets and a deployable team UI. It's at its best when your app is structured around Mastra's primitives.

Pick the one that matches the shape of your project, but please pick something. Building Gen AI without a local debugging tool in 2026 is engineering with the lights off.

Further reading:

Top Gen AI Frameworks for Go in 2026: A Hands-On Comparison

Xavier Portilla Edo — Tue, 29 Sep 2026 13:41:18 +0000

Introduction

Go has quietly become the language of AI infrastructure. MCP servers, agent runtimes, model gateways, Kubernetes operators that talk to LLMs: a large share of that plumbing is written in Go, because Go is what you reach for when something has to be a small static binary that handles thousands of concurrent connections without drama. What was missing for a long time was the framework layer on top of the raw provider SDKs. In 2026 that gap is closed, and the question is no longer "Go or Python?" but "which Go framework?".

This article covers the five frameworks a Go developer will actually run into in 2026: Genkit Go, Eino, Google ADK Go, tRPC-Agent-Go, and LangChainGo. Four of them are alive and moving fast. The fifth, LangChainGo, is the one most Go developers heard of first, and it is now effectively unmaintained. It stays in the list because you will find it in existing codebases, and you need to know what that means for you.

Same rules as the Java and JavaScript/TypeScript editions: this is not a benchmark post. Tokens per second go stale in weeks. This is a developer experience and architecture comparison, with working code and the real output of that code for every framework, all against the same model, gemini-3.8-flash, on Go 1.27.

Versions used: Genkit Go v1.13.1, Eino v0.9.20, Google ADK Go v2.4.0, tRPC-Agent-Go v1.11.2, LangChainGo v0.1.14. APIs as of September 2026.

Genkit Go

History and Direction

Genkit was announced by Google at I/O 2024 as a TypeScript-first framework. The Go SDK arrived only three months later, with v0.1.0 in August 2024, and unlike the Java SDK, which is a community effort, Genkit Go is an official, first-party SDK maintained by the same google team that ships the TypeScript runtime. It reached 1.0 in September 2025 and sits at v1.13.1 as of September 2026, on a cadence of roughly one minor release per month.

The direction is parity with TypeScript plus Go-native ergonomics: generics instead of Zod schemas, context.Context everywhere, plain net/http handlers for flows, and an agents package, still marked experimental, with session stores, multi-agent delegation, artifacts, and a Skills middleware. The monorepo (genkit-ai/genkit) has 6.4k stars and 80 commits in the last 30 days across all SDKs.

If you want the long-form introduction, start with Stop Using Python for Your Gen AI Apps, Use Go and Genkit Instead.

What Makes Genkit Go Stand Out

Genkit Go gives you three levels of abstraction in a single SDK: direct model calls, typed flows, and agents, all traced by the same OpenTelemetry pipeline and all visible in the same Dev UI. On top of that sits a middleware layer (retry, fallback, tool approval, filesystem, skills) that wraps any of the three. No other Go framework gives you that combination.

Supported languages: Go 1.25+ for this SDK. The same framework exists for TypeScript, Python, Dart/Flutter, and, unofficially, Java.

Vanilla Generation

import (
    "github.com/genkit-ai/genkit/go/ai"
    "github.com/genkit-ai/genkit/go/genkit"
    "github.com/genkit-ai/genkit/go/plugins/googlegenai"
)

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithDefaultModel("googleai/gemini-3.8-flash"))

text, err := genkit.GenerateText(ctx, g,
    ai.WithPrompt("Explain the CAP theorem in two sentences."))

The CAP theorem states that a distributed data system can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are inevitable in the real world, systems must ultimately choose between returning the most up-to-date data (Consistency) or ensuring every request receives a response (Availability).

Typed Flows: Observable Pipelines

Flows are the heart of Genkit. A flow is a named, typed, traceable unit that you can call in-process, expose as an HTTP endpoint with one line, and run from the Dev UI. The JSON schema for the output comes from the Go struct, and GenerateData returns the struct, not a string you have to parse.

type TranslateRequest struct {
    Text           string `json:"text"`
    TargetLanguage string `json:"target_language"`
}

type TranslateResponse struct {
    Translation      string `json:"translation"`
    DetectedLanguage string `json:"detected_language"`
}

translate := genkit.DefineFlow(g, "translateText",
    func(ctx context.Context, req TranslateRequest) (*TranslateResponse, error) {
        out, _, err := genkit.GenerateData[TranslateResponse](ctx, g,
            ai.WithPrompt("Translate %q to %s and detect the source language.",
                req.Text, req.TargetLanguage))
        return out, err
    })

// Call it in-process...
out, err := translate.Run(ctx, TranslateRequest{Text: "Buenos días", TargetLanguage: "English"})
fmt.Printf("%+v\n", *out)

// ...or expose it over HTTP (and to the Dev UI).
mux := http.NewServeMux()
mux.HandleFunc("POST /translateText", genkit.Handler(translate))
log.Fatal(server.Start(ctx, "127.0.0.1:3400", mux))

{Translation:Good morning DetectedLanguage:Spanish}
INFO server listening addr=127.0.0.1:3400

curl -X POST http://127.0.0.1:3400/translateText \
  -H 'Content-Type: application/json' \
  -d '{"data":{"text":"Buenos días","target_language":"English"}}'
# {"result":{"translation":"Good morning","detected_language":"Spanish"}}

Tools and Agents

Tools are typed functions. The input schema is inferred from the struct, and the tool loop (model asks, tool runs, model answers) happens inside Generate.

type WeatherInput struct {
    City string `json:"city"`
}

weather := genkit.DefineTool(g, "getWeather", "Returns the current weather for a city.",
    func(ctx *ai.ToolContext, in WeatherInput) (string, error) {
        return fmt.Sprintf("Sunny, 24C in %s", in.City), nil
    })

resp, err := genkit.Generate(ctx, g,
    ai.WithPrompt("What's the weather like in Tokyo?"),
    ai.WithTools(weather))
fmt.Println(resp.Text())

The weather in Tokyo is currently sunny and 24°C.

When you need memory across turns, the experimental agents package adds sessions on top of the same tools. Enable it with genkit.WithExperimental():

import (
    aix "github.com/genkit-ai/genkit/go/ai/exp"
    "github.com/genkit-ai/genkit/go/ai/exp/localstore"
    genkitx "github.com/genkit-ai/genkit/go/genkit/exp"
)

advisor := genkitx.DefineAgent(g, "travelAdvisor",
    aix.InlinePrompt{
        ai.WithSystem("You are a helpful travel advisor. Use tools to give accurate advice. Be brief."),
        ai.WithTools(weather),
    },
    aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()))

first, err := advisor.RunText(ctx, "Should I pack a jacket for my trip to Lisbon?")
second, err := advisor.RunText(ctx, "Which city did I ask about?",
    aix.WithSessionID[any](first.SessionID))

It is currently sunny and 24°C (75°F) in Lisbon. You won't need a heavy coat, but packing a light jacket or sweater is recommended for cooler evenings and coastal breezes.
You asked about Lisbon.

Multi-agent delegation, cross-vendor fallback, and tool approval are covered in Building Agents in Go: Where Genkit Shines vs ADK 2.0.

The Dev UI: Same Power as TypeScript

The same Genkit Developer UI that TypeScript developers use works with Go, with no code changes. Install the CLI and start your Go program through it:

npm install -g genkit
genkit start -- go run .

Genkit Developer UI: http://localhost:4000

The flow above shows up as /flow/translateText next to every Gemini model the plugin registered, and you get:

Flow runner: execute any flow with a custom input and inspect the typed output.
Trace explorer: OpenTelemetry traces for every Generate and flow call, with latency, token counts, and the exact prompts.
Model playground: call any registered model directly.
Tool testing: run tools in isolation.
Dotprompt editor: edit .prompt files live.

This is the single biggest advantage Genkit has over every other framework in this list: a zero-config, local trace explorer that replaces Langfuse or LangSmith during development.

Provider Support

Official plugins in the Go module: Google AI (Gemini), Vertex AI, Anthropic, Ollama, and an OpenAI-compatible family (compat_oai) with ready-made packages for OpenAI, xAI, DeepSeek, OpenRouter, Kimi, DashScope, and Z.ai. Vector stores: localvec (local, no service needed), PostgreSQL/pgvector, AlloyDB, Pinecone, Weaviate, and Firebase Firestore. Plus MCP (client and server), evaluators, and an A2UI plugin for streaming UI components.

Pros and Cons

✅ Pros	❌ Cons
Best-in-class Dev UI with local trace explorer	Agents package still marked experimental
Multi-level abstractions: vanilla, flows, agents	Fewer vector stores than LangChainGo
Middleware: retry, fallback, tool approval	Go 1.25+ required
Official Google SDK, monthly releases
Typed output from Go structs, no parsing
Same framework in TS, Python, Dart, Java

Eino

History and Direction

Eino (pronounced "aino") comes from ByteDance's CloudWeGo team, the group behind Kitex and Hertz, the RPC and HTTP frameworks that run a large part of ByteDance's Go services. It was open-sourced in December 2024 after internal use, and it is now the most-starred LLM framework in Go at 13.1k stars, ahead of LangChainGo. Releases land every few days: v0.9.20 on September 20, 2026, with v0.10.0 already at alpha 35.

Eino's own description is that it draws from LangChain and Google ADK but follows Go conventions. The framework splits in two: compose, for orchestration (Chain, Graph, Workflow), and adk, an Agent Development Kit that borrows Google ADK's vocabulary of agents, runners, and events. Components (models, tools, retrievers, indexers, loaders) live in a separate repo, eino-ext, together with callback handlers for Langfuse, LangSmith, CozeLoop, and ByteDance's APMPlus.

The 2026 roadmap is all agents: DeepAgent (planning plus sub-agents), agent middleware, interrupt and resume for human-in-the-loop, skills, A2UI streaming components, and a turn loop with preemption. The English documentation on cloudwego.io is now a full eleven-chapter course, a big change from 2025 when most of it was Chinese-first.

What Makes Eino Stand Out

Orchestration with automatic stream handling. Every node in an Eino graph can be invoked or streamed, and the framework concatenates, boxes, merges, and copies streams as data moves between nodes. You build the graph once and get both Invoke and Stream for free. Graph inputs and outputs are generic types, so the compiler checks that the nodes fit together. Cross-cutting concerns go through callbacks at fixed points (OnStart, OnEnd, OnError, plus the streaming variants) that apply to components, graphs, and agents alike.

Supported languages: Go only. The module declares go 1.18, the lowest bar in this list. One thing to know: Eino depends on ByteDance's sonic JSON library, and on Go 1.27 it prints a warning at startup and falls back to encoding/json. Harmless, but noisy.

ChatModel

Eino does not own the provider client. You create the genai client yourself and hand it to the component.

import (
    "github.com/cloudwego/eino-ext/components/model/gemini"
    "github.com/cloudwego/eino/schema"
    "google.golang.org/genai"
)

client, err := genai.NewClient(ctx, &genai.ClientConfig{APIKey: os.Getenv("GEMINI_API_KEY")})
cm, err := gemini.NewChatModel(ctx, &gemini.Config{Client: client, Model: "gemini-3.8-flash"})

msg, err := cm.Generate(ctx, []*schema.Message{
    schema.UserMessage("Explain the CAP theorem in two sentences."),
})
fmt.Println(msg.Content)

The CAP theorem states that a distributed data store can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are practically inevitable in distributed systems, architects must choose between keeping data strictly up-to-date (Consistency) or ensuring the system remains responsive (Availability) during a network failure.

Chains, Graphs, and Callbacks

A chain is a linear pipeline. Here the callback handler is attached at invocation time and fires for the chain, the template, and the model.

import (
    "github.com/cloudwego/eino/callbacks"
    "github.com/cloudwego/eino/components/prompt"
    "github.com/cloudwego/eino/compose"
)

template := prompt.FromMessages(schema.FString,
    schema.SystemMessage("You are a translator. Reply with the translation only."),
    schema.UserMessage("Translate {text} to {language}."))

chain := compose.NewChain[map[string]any, *schema.Message]()
chain.AppendChatTemplate(template).AppendChatModel(cm)
runnable, err := chain.Compile(ctx)

tracer := callbacks.NewHandlerBuilder().
    OnStartFn(func(ctx context.Context, info *callbacks.RunInfo, in callbacks.CallbackInput) context.Context {
        log.Printf("start %s %s", info.Component, info.Type)
        return ctx
    }).
    OnEndFn(func(ctx context.Context, info *callbacks.RunInfo, out callbacks.CallbackOutput) context.Context {
        log.Printf("end   %s %s", info.Component, info.Type)
        return ctx
    }).
    Build()

out, err := runnable.Invoke(ctx,
    map[string]any{"text": "Buenos días", "language": "English"},
    compose.WithCallbacks(tracer))
fmt.Println(out.Content)

start Chain
start ChatTemplate Default
end   ChatTemplate Default
start ChatModel Gemini
end   ChatModel Gemini
end   Chain
Good morning

Graphs are where Eino earns its name. Nodes are typed, edges are explicit, and plain Go functions slot in as lambda nodes:

graph := compose.NewGraph[map[string]any, string]()
_ = graph.AddChatTemplateNode("template", template)
_ = graph.AddChatModelNode("generate", cm)
_ = graph.AddLambdaNode("shout", compose.InvokableLambda(
    func(ctx context.Context, msg *schema.Message) (string, error) {
        return strings.ToUpper(strings.TrimSpace(msg.Content)), nil
    }))

_ = graph.AddEdge(compose.START, "template")
_ = graph.AddEdge("template", "generate")
_ = graph.AddEdge("generate", "shout")
_ = graph.AddEdge("shout", compose.END)

runnable, err := graph.Compile(ctx)
out, err := runnable.Invoke(ctx, map[string]any{"product": "a Go framework for LLM apps"})

BUILD PRODUCTION-GRADE LLM APPS AT GO SPEED.

Branches, parallel fan-out, and sub-graphs use the same API, and a compiled graph can be wrapped as a tool and handed to an agent.

Eino ADK: Agents and Workflows

Tools are inferred from a Go struct with jsonschema tags. The agent runs the ReAct loop and emits one event per step, so you see the tool call, the tool result, and the final answer as separate messages.

import (
    "github.com/cloudwego/eino/adk"
    "github.com/cloudwego/eino/components/tool"
    "github.com/cloudwego/eino/components/tool/utils"
)

type WeatherInput struct {
    City string `json:"city" jsonschema:"description=The city to look up"`
}

weather, err := utils.InferTool("get_weather", "Returns the current weather for a city.",
    func(ctx context.Context, in WeatherInput) (string, error) {
        return fmt.Sprintf("Sunny, 24C in %s", in.City), nil
    })

advisor, err := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{
    Name:        "travel_advisor",
    Description: "Helps with trip planning and weather-based advice.",
    Instruction: "You are a helpful travel advisor. Use tools to give accurate advice. Be brief.",
    Model:       cm,
    ToolsConfig: adk.ToolsConfig{
        ToolsNodeConfig: compose.ToolsNodeConfig{Tools: []tool.BaseTool{weather}},
    },
})

runner := adk.NewRunner(ctx, adk.RunnerConfig{Agent: advisor})
iter := runner.Query(ctx, "Should I pack a jacket for my trip to Lisbon?")
for {
    event, ok := iter.Next()
    if !ok {
        break
    }
    if event.Err != nil {
        log.Fatal(event.Err)
    }
    if event.Output == nil || event.Output.MessageOutput == nil {
        continue
    }
    msg, _ := event.Output.MessageOutput.GetMessage()
    fmt.Printf("[%s] %s\n", msg.Role, msg.Content)
}

[assistant]
[tool] Sunny, 24C in Lisbon
[assistant] It is currently sunny and 24°C (75°F) in Lisbon. You won't need a heavy coat, but packing a light jacket or cardigan is a good idea for breezy coastal evenings.

The first assistant event is the tool call itself, which is why its content is empty. Multi-agent workflows are one constructor away:

pipeline, err := adk.NewSequentialAgent(ctx, &adk.SequentialAgentConfig{
    Name:        "content_pipeline",
    Description: "Research, then write.",
    SubAgents:   []adk.Agent{researcher, writer},
})

[researcher] 1. Created at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released in 2009.
2. Statically typed, compiled language with built-in garbage collection.
3. Features native, lightweight concurrency through goroutines and channels.

[writer] Created at Google by Robert Griesemer, Rob Pike, and Ken Thompson, the Go programming language was released in 2009. It is a statically typed, compiled language that features built-in garbage collection and native, lightweight concurrency through goroutines and channels.

NewParallelAgent and NewLoopAgent complete the set, and DeepAgent adds planning and delegation on top.

Developer Tooling

Eino's answer to observability during development is the Eino Dev plugin for GoLand and VS Code, backed by the devops module in eino-ext. It renders your chains and graphs visually inside the editor and lets you start execution from any node with mock input. It is an IDE feature rather than a browser Dev UI, so there is no trace explorer for a running service. For that you attach one of the callback handlers (Langfuse, LangSmith, CozeLoop, APMPlus).

Provider Support

Model components in eino-ext: OpenAI, Claude, Gemini, DeepSeek, Qwen, Ollama, OpenRouter, Ark (Volcano Engine), and Qianfan (Baidu), plus "agentic" variants of the main ones for the newer agentic message format. Retrievers and indexers: Elasticsearch (7, 8, 9), Milvus, Qdrant, Redis, OpenSearch, Dify, and VikingDB. Tools: MCP, Google Search, Bing, DuckDuckGo, SearXNG, Wikipedia, HTTP request, command line, browser use, and sequential thinking.

Pros and Cons

✅ Pros	❌ Cons
Most active independent Go LLM framework (13.1k stars)	Still 0.x with frequent alphas; expect API churn
Typed graph orchestration with automatic streaming	No browser Dev UI; visual debugging lives in the IDE plugin
Callback aspects for logging, tracing, and metrics	Verbose: config structs and explicit error checks everywhere
ADK-style agents plus DeepAgent and interrupt/resume	ByteDance-flavoured corners (Ark, VikingDB, CozeLoop)
Broad component ecosystem in `eino-ext`	`sonic` dependency warns on the newest Go toolchains
Runs on Go 1.18+	Chinese-first history still shows in some docs and examples

Google ADK Go

History and Direction

ADK, the Agent Development Kit, was announced at Google Cloud Next 2025 with Python as its first language. The Go SDK made its first tagged release, v0.1.0, in November 2025, went 1.0 on March 23, 2026, and shipped a new major module, google.golang.org/adk/v2, on June 30, 2026. It is at v2.4.0 (September 11, 2026), has 8.8k stars, and logged over 100 commits in the last 30 days: the highest commit volume in this list. It is an official Google product with the Python, Java, Kotlin, and TypeScript SDKs as siblings.

The design is the same as its siblings: everything is an agent, a workflow agent, or a tool. Gemini is the native model, with openaimodel for anything speaking the OpenAI wire format and an Apigee model for enterprise gateways. Deployment gravity points at Vertex AI Agent Engine, Cloud Run, and GKE, with Pub/Sub and Eventarc triggers built into the launcher. The minimum Go version is 1.26.6, the highest bar of the five.

A deeper look at the low-level differences with Genkit is in Genkit Go vs ADK Go 2.0.

ADK Go's Position: Agent-Only, Enterprise-Grade

ADK Go is an agent framework. There is no vanilla generation primitive and no flow abstraction: you always build an agent, give it to a runner, and iterate the events. In exchange you get sessions, memory, artifacts, workflow agents, agent-to-agent (A2A) protocol support, MCP toolsets, tool confirmation, and an embedded web UI, all first party.

Supported languages: Go 1.26.6+. Same framework in Python (primary), Java, Kotlin, and TypeScript.

import (
    "google.golang.org/adk/v2/agent"
    "google.golang.org/adk/v2/agent/llmagent"
    "google.golang.org/adk/v2/model/gemini"
    "google.golang.org/adk/v2/runner"
    "google.golang.org/adk/v2/tool"
    "google.golang.org/adk/v2/tool/functiontool"
    "google.golang.org/genai"
)

model, err := gemini.NewModel(ctx, "gemini-3.8-flash", &genai.ClientConfig{
    APIKey: os.Getenv("GOOGLE_API_KEY"),
})

weather, err := functiontool.New(functiontool.Config{
    Name:        "get_weather",
    Description: "Returns the current weather for a city.",
}, func(_ agent.Context, in WeatherInput) (WeatherOutput, error) {
    return WeatherOutput{Report: fmt.Sprintf("Sunny, 24C in %s", in.City)}, nil
})

advisor, err := llmagent.New(llmagent.Config{
    Name:        "travel_advisor",
    Model:       model,
    Instruction: "You are a helpful travel advisor. Use tools to give accurate advice. Be brief.",
    Tools:       []tool.Tool{weather},
})

r, err := runner.NewInMemory("travel-app", advisor)

msg := genai.NewContentFromText("Should I pack a jacket for my trip to Lisbon?", genai.RoleUser)
for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) {
    if err != nil {
        log.Fatal(err)
    }
    if event.LLMResponse.Content == nil || !event.IsFinalResponse() {
        continue
    }
    for _, part := range event.LLMResponse.Content.Parts {
        fmt.Print(part.Text)
    }
}

It is currently sunny and warm at 24°C (75°F) in Lisbon. You won't need a heavy jacket during the day, but packing a light jacket or sweater is recommended for cooler evenings and coastal breezes.

Two things to notice. The runner returns a Go 1.23 iterator, so you range over events with the error as the second value, which is the most idiomatic event loop of the five. And the function tool is generic: input and output structs give the schema, nothing is spelled out twice.

Multi-Agent Orchestration

Workflow agents compose LLM agents. The OutputKey writes an agent's answer into session state, and the next agent reads it back through a {placeholder} in its instruction.

import "google.golang.org/adk/v2/agent/workflowagents/sequentialagent"

researcher, err := llmagent.New(llmagent.Config{
    Name:        "researcher",
    Model:       model,
    Instruction: "List three key facts about the user's topic. Be terse.",
    OutputKey:   "facts", // saved into session state
})
writer, err := llmagent.New(llmagent.Config{
    Name:        "writer",
    Model:       model,
    Instruction: "Write a two-sentence brief using these facts:\n{facts}",
})

pipeline, err := sequentialagent.New(sequentialagent.Config{
    AgentConfig: agent.Config{
        Name:      "content_pipeline",
        SubAgents: []agent.Agent{researcher, writer},
    },
})

[researcher] * **Origin:** Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released as open source in 2009.
* **Concurrency:** Features built-in, lightweight concurrency primitives known as goroutines and channels, managed by its own runtime.
* **Ecosystem Role:** Statically typed and compiled, it is the primary language behind core cloud-native technologies, including Docker and Kubernetes.

[writer] Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson and released as open source in 2009, Go is a statically typed, compiled language featuring built-in, lightweight concurrency primitives known as goroutines and channels that are managed by its own runtime. Owing to these strengths, it has become the primary language powering core cloud-native technologies, including Docker and Kubernetes.

parallelagent and loopagent follow the same pattern.

The Dev UI: ADK Web, Embedded in Your Binary

The full launcher turns any agent into a CLI with console and web modes. The web mode takes sub-launchers, and webui serves the Angular ADK Web app straight out of your Go binary via go:embed, so there is nothing to install:

import (
    "google.golang.org/adk/v2/cmd/launcher"
    "google.golang.org/adk/v2/cmd/launcher/full"
)

l := full.NewLauncher()
config := &launcher.Config{AgentLoader: agent.NewSingleLoader(advisor)}
if err := l.Execute(ctx, config, os.Args[1:]); err != nil {
    log.Fatalf("%v\n\n%s", err, l.CommandLineSyntax())
}

go run . web -port 8080 api webui

Web servers starts on http://localhost:8080
       api:  you can access API using http://localhost:8080/api

The root URL redirects to /ui/, the Agent Development Kit Dev UI, and /api/list-apps answers ["assistant"]. The launcher also wires a2a, pubsub, and eventarc sub-servers, and an -otel_to_cloud flag exports traces to Google Cloud. Flags go before the sub-launcher names, or the sub-launcher rejects them.

Vertex AI Gravity

You can run ADK Go anywhere. But the session and memory services with persistence are the in-memory ones, a database-backed session store, and Vertex AI; the evaluation tooling, the managed scaling, and the telemetry defaults all assume Google Cloud. Same trade-off as ADK Java and ADK Python: the best experience is on GCP.

Pros and Cons

✅ Pros	❌ Cons
Official Google support, highest commit velocity	Agent-only: no vanilla generation, no flows
Idiomatic Go: generics for tools, iterators for events	Go 1.26.6+ required
Embedded ADK Web UI, no extra install	Strong pull toward Vertex AI and GCP
Sequential, parallel, and loop workflow agents	Gemini-first; other providers only through OpenAI-compatible endpoints
A2A, MCP toolsets, tool confirmation, artifacts	No vector store components; RAG goes through tools or Vertex
Same mental model as ADK Python/Java/TS	Config structs get long for real agents

tRPC-Agent-Go

History and Direction

tRPC-Agent-Go is Tencent's entry, built by the team behind tRPC, the RPC framework that underpins much of Tencent's backend. Development started in May 2025, the first public release (v0.0.1) landed in August 2025, and the project reached 1.x at the end of December 2025. It is now at v1.11.2 (August 20, 2026), with 1.8k stars and 56 commits in the last 30 days. The README credits Tencent Yuanbao, Tencent Video, Tencent News, IMA, and QQ Music with real-world validation, and I have no reason to doubt it.

The framework is unapologetically ADK-shaped: llmagent, runner, chainagent, parallelagent, cycleagent, sessions, memory, OutputKey, {placeholder} instructions. If you know ADK, you already know most of it. What it adds on top is where it gets interesting: a GraphAgent that is a faithful LangGraph in Go (state schemas, conditional edges, checkpoints, time travel), an AG-UI server for streaming agent events to frontends, A2A, MCP, Anthropic-style Agent Skills with sandboxed execution, agent self-evolution, an evaluation package, model failover and hedging, prompt caching, and OpenTelemetry traces and metrics with Langfuse examples. RAG is covered by a knowledge module with embedders for OpenAI, Gemini, Ollama, and Hugging Face, and stores from in-memory and SQLite-vec up to pgvector, Elasticsearch, Milvus, Qdrant, and TcVector.

What Makes tRPC-Agent-Go Stand Out

The GraphAgent. Nobody else in Go ships a LangGraph-equivalent state graph as a first-class agent type that plugs into the same runner, sessions, and telemetry as the LLM agents.

Supported languages: Go only, go 1.21+. Model providers: OpenAI and anything OpenAI-compatible (DeepSeek, Qwen, and friends through variants), Anthropic, Gemini, Bedrock, Ollama, Hugging Face, and Hunyuan.

import (
    "trpc.group/trpc-go/trpc-agent-go/agent/llmagent"
    "trpc.group/trpc-go/trpc-agent-go/model"
    "trpc.group/trpc-go/trpc-agent-go/model/gemini"
    "trpc.group/trpc-go/trpc-agent-go/runner"
    "trpc.group/trpc-go/trpc-agent-go/tool"
    "trpc.group/trpc-go/trpc-agent-go/tool/function"
)

type weatherReq struct {
    City string `json:"city" jsonschema:"description=The city to look up,required"`
}

type weatherRsp struct {
    Report string `json:"report"`
}

func getWeather(ctx context.Context, req weatherReq) (weatherRsp, error) {
    return weatherRsp{Report: fmt.Sprintf("Sunny, 24C in %s", req.City)}, nil
}

m, err := gemini.New(ctx, "gemini-3.8-flash") // reads GEMINI_API_KEY / GOOGLE_API_KEY

weather := function.NewFunctionTool(getWeather,
    function.WithName("get_weather"),
    function.WithDescription("Returns the current weather for a city."))

advisor := llmagent.New("travel_advisor",
    llmagent.WithModel(m),
    llmagent.WithInstruction("You are a helpful travel advisor. Use tools to give accurate advice. Be brief."),
    llmagent.WithTools([]tool.Tool{weather}))

r := runner.NewRunner("travel-app", advisor)
events, err := r.Run(ctx, "user-1", "session-1",
    model.NewUserMessage("Should I pack a jacket for my trip to Lisbon?"))
for evt := range events {
    if evt.Error != nil {
        log.Fatalf("%s", evt.Error.Message)
    }
    if evt.Response == nil || len(evt.Response.Choices) == 0 {
        continue
    }
    if evt.IsFinalResponse() {
        fmt.Println(evt.Response.Choices[0].Message.Content)
    }
}

It is currently sunny and warm in Lisbon at 24°C (75°F), so you will not need a heavy coat. However, packing a light jacket or sweater is recommended for cooler, breezy evenings near the coast.

Events arrive on a channel and use OpenAI's response shape (Choices[0].Message or Choices[0].Delta when streaming), which is familiar but means digging through nested structs for every message.

Chains

The chain agent is ADK's sequential agent under another name, OutputKey and {facts} included:

import (
    "trpc.group/trpc-go/trpc-agent-go/agent"
    "trpc.group/trpc-go/trpc-agent-go/agent/chainagent"
)

researcher := llmagent.New("researcher",
    llmagent.WithModel(m),
    llmagent.WithInstruction("List three key facts about the user's topic. Be terse."),
    llmagent.WithOutputKey("facts"))
writer := llmagent.New("writer",
    llmagent.WithModel(m),
    llmagent.WithInstruction("Write a two-sentence brief using these facts:\n{facts}"))

pipeline := chainagent.New("content_pipeline",
    chainagent.WithSubAgents([]agent.Agent{researcher, writer}))

[researcher] 1. Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released in 2009.
2. Statically typed, compiled language featuring automatic garbage collection.
3. Native concurrency support using lightweight threads called goroutines and channels.

[writer] Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson and released in 2009, Go is a statically typed, compiled language featuring automatic garbage collection. It is designed with native concurrency support, utilizing channels and lightweight threads called goroutines.

GraphAgent: LangGraph in Go

A state graph with a routing node, a conditional edge, and two LLM nodes. Short questions go to the quick node, long ones to the deep node.

import (
    "trpc.group/trpc-go/trpc-agent-go/agent/graphagent"
    "trpc.group/trpc-go/trpc-agent-go/graph"
)

sg := graph.NewStateGraph(graph.MessagesStateSchema())
sg.AddNode("route", func(ctx context.Context, state graph.State) (any, error) {
    return nil, nil // pure routing node, no state change
})
sg.AddLLMNode("quick", m, "Answer in one short sentence.", nil)
sg.AddLLMNode("deep", m, "Answer thoroughly. Follow the length the user asks for.", nil)

sg.SetEntryPoint("route")
sg.AddConditionalEdges("route",
    func(ctx context.Context, state graph.State) (string, error) {
        input, _ := state[graph.StateKeyUserInput].(string)
        if len(strings.Fields(input)) > 8 {
            return "deep", nil
        }
        return "quick", nil
    },
    map[string]string{"quick": "quick", "deep": "deep"})
sg.SetFinishPoint("quick")
sg.SetFinishPoint("deep")

g, err := sg.Compile()
router, err := graphagent.New("router", g)
r := runner.NewRunner("graph-app", router)

Asked "What is Go?" and then a request for two paragraphs on goroutines and channels:

Go is an ancient strategy board game, as well as an open-source programming language developed by Google.

Unlike operating system threads that require megabytes of memory and expensive kernel-level context switching, goroutines are extremely lightweight user-space constructs that start with just a few kilobytes of stack space. [...]

The same graph package gives you checkpoints, interrupts for human input, sub-graphs, map-reduce fan-out, per-node callbacks, and a visualization export. It is the most complete workflow engine of the five, and it is the reason to look at this framework even if you never use its LLM agents.

Pros and Cons

✅ Pros	❌ Cons
GraphAgent: a real LangGraph for Go	Smallest community of the five (1.8k stars)
ADK mental model without the GCP gravity	No Dev UI; observability means OpenTelemetry plus Langfuse
AG-UI, A2A, MCP, Agent Skills, evaluation	Huge option surface (`llmagent` alone has dozens)
Model failover and hedging built in	OpenAI-shaped event structs are verbose to consume
OpenTelemetry traces and metrics first party	Tencent-flavoured corners (Hunyuan, tRPC integration)
Runs on Go 1.21+	Documentation partly Chinese-first

LangChainGo

History and Direction

LangChainGo was the first. Travis Cline started it in February 2023, weeks after LangChain Python went viral, and for two years it was the default answer to "how do I call an LLM from Go". It has 9.7k stars, 17 LLM providers, and 15 vector stores, and the LangChain names (chains, prompts, agents, memory, outputparser) make it instantly readable to anyone coming from Python.

And it has stalled. The last release, v0.1.14, is from October 20, 2025. The last commit to main is from January 11, 2026. It is still 0.1.x after three and a half years. There are 243 open issues and 174 open pull requests. The repository is not archived, but nobody is merging.

That would be survivable if the code kept working. It does not:

The default model is retired. googleai.New(ctx) with no options targets gemini-2.0-flash, and Google turned that model off:

  default model: gemini-2.0-flash
  err: googleapi: Error 404: This model models/gemini-2.0-flash is no longer available. Please update your code to use models/gemini-3.6-flash for the latest features and improvements.

The Gemini provider sits on a dead SDK. llms/googleai is built on google/generative-ai-go, whose support ended on November 30, 2025.
Streaming is broken on Gemini. The chunks arrive, and then the call returns an error, so every caller that checks errors treats the request as failed:

  One, two, three, four, five.
  err: error in stream mode: invalid character ']' looking for beginning of value

No middleware, retry, fallback, or OpenTelemetry anywhere. The callback handler is seventeen observe-only methods with no way to change the request. Tools are Call(ctx, string) (string, error): no schema, no typed arguments.

The full teardown, with a capstone app built in both frameworks, is in LangChainGo vs Genkit Go: Where Genkit Shines.

LangChainGo's Position: Legacy

None of this means the basics stopped working. Pin a current model and the classic API still runs:

import (
    "github.com/tmc/langchaingo/llms"
    "github.com/tmc/langchaingo/llms/googleai"
)

llm, err := googleai.New(ctx, googleai.WithDefaultModel("gemini-3.8-flash"))

out, err := llms.GenerateFromSinglePrompt(ctx, llm,
    "Explain the CAP theorem in two sentences.")

The CAP theorem states that a distributed data store can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance.

Because network partitions are inevitable in real-world systems, architects must fundamentally choose between serving the most up-to-date data (Consistency) or keeping the system operational (Availability) when a network failure occurs.

import (
    "github.com/tmc/langchaingo/chains"
    "github.com/tmc/langchaingo/prompts"
)

prompt := prompts.NewPromptTemplate(
    "Translate {{.text}} to {{.language}}. Reply with the translation only.",
    []string{"text", "language"})
chain := chains.NewLLMChain(llm, prompt)

out, err := chains.Predict(ctx, chain, map[string]any{
    "text": "Buenos días", "language": "English",
})

Good morning

Tools are strings in, strings out. The agent is a ReAct loop driven by prompt text:

import (
    "github.com/tmc/langchaingo/agents"
    "github.com/tmc/langchaingo/tools"
)

type WeatherTool struct{}

func (WeatherTool) Name() string        { return "get_weather" }
func (WeatherTool) Description() string { return "Returns the current weather. Input: a city name." }
func (WeatherTool) Call(ctx context.Context, city string) (string, error) {
    return fmt.Sprintf("Sunny, 24C in %s", city), nil
}

agent := agents.NewOneShotAgent(llm, []tools.Tool{WeatherTool{}}, agents.WithMaxIterations(3))
out, err := chains.Run(ctx, agents.NewExecutor(agent), "What's the weather like in Tokyo?")

The weather in Tokyo is currently sunny with a temperature of 24°C.

If you are on LangChainGo today, the move is not urgent, but it is inevitable: pin your models, avoid streaming on Gemini, and plan the migration. The concept-by-concept guide is in Migrating from LangChainGo to Genkit Go.

Pros and Cons

✅ Pros	❌ Cons
Largest catalog: 17 providers, 15 vector stores	Unmaintained since January 2026, still 0.1.x
LangChain vocabulary, easy for Python teams	Gemini provider on a retired SDK; streaming broken
Conversation memory is ergonomic	No middleware, retry, fallback, or OpenTelemetry
Runs on Go 1.24+	Tools are untyped strings
Many existing tutorials and examples	Every vector store needs an external service

Head-to-Head Comparison

Developer Experience

Framework	DX Highlights	Shortcomings
Genkit Go	Dev UI with trace explorer. Typed output from structs. Flows are one line from HTTP.	Agents still experimental
Eino	Typed graphs, automatic streaming, callbacks everywhere.	Verbose config structs; 0.x churn
ADK Go	Iterator-based event loop, generic tools, embedded web UI.	Agent-only; Go 1.26.6
tRPC-Agent-Go	GraphAgent, ADK mental model, resilience built in.	OpenAI-shaped events; option overload
LangChainGo	Familiar names, biggest catalog.	Unmaintained; broken Gemini streaming

Abstraction Levels

Genkit Go is the only framework here with all three levels in one SDK: vanilla generation, typed flows, and agents. Eino covers generation and orchestration, plus agents through its ADK module. ADK Go and tRPC-Agent-Go are agent frameworks: every call goes through an agent and a runner. LangChainGo has chains and agents, but no typed pipeline and no session-aware agent.

Observability

Framework	Local Dev	Production
Genkit Go	Dev UI with trace explorer	OpenTelemetry export, Google Cloud plugin, any OTEL collector
Eino	Eino Dev plugin for GoLand and VS Code (graph rendering, node debugging)	Callback handlers: Langfuse, LangSmith, CozeLoop, APMPlus
ADK Go	Embedded ADK Web UI	OpenTelemetry to Google Cloud (`-otel_to_cloud`)
tRPC-Agent-Go	Events on a channel, no UI	OpenTelemetry traces and metrics, Langfuse
LangChainGo	Logging callbacks only	Nothing built in

Provider Support

Framework	Model Providers	Vector Stores
Genkit Go	Gemini (Google AI, Vertex AI), Anthropic, Ollama, OpenAI-compatible family (OpenAI, xAI, DeepSeek, OpenRouter, Kimi, DashScope, Z.ai)	localvec, pgvector, AlloyDB, Pinecone, Weaviate, Firestore
Eino	OpenAI, Claude, Gemini, DeepSeek, Qwen, Ollama, OpenRouter, Ark, Qianfan	Elasticsearch, Milvus, Qdrant, Redis, OpenSearch, Dify, VikingDB
ADK Go	Gemini, OpenAI-compatible, Apigee	None as components (Vertex AI Search via tools)
tRPC-Agent-Go	OpenAI-compatible, Anthropic, Gemini, Bedrock, Ollama, Hugging Face, Hunyuan	In-memory, SQLite-vec, pgvector, Elasticsearch, Milvus, Qdrant, TcVector
LangChainGo	17, including OpenAI, Anthropic, Google AI, Vertex, Bedrock, Cohere, Mistral, Ollama	15, all external services

How Go Does It Feel?

Framework	Min Go	Tools	Streaming	Style
Genkit Go	1.25	Generic, schema from struct	Callback per chunk	Functional options (`ai.With...`)
Eino	1.18	Generic (`utils.InferTool`)	`StreamReader`, automatic across nodes	Config structs plus builder chains
ADK Go	1.26.6	Generic (`functiontool.New`)	`iter.Seq2` over events	Config structs
tRPC-Agent-Go	1.21	Generic plus `jsonschema` tags	Channel of events	Functional options
LangChainGo	1.24	Untyped strings	Callback per chunk (fails on Gemini)	Interfaces plus option funcs

Only ADK Go uses range-over-func iterators for its event loop. Genkit and Eino use generics the most naturally; LangChainGo predates all of it and shows its age.

Maintenance Snapshot

Numbers as of September 21, 2026:

Framework	Stars	Latest Release	Last Commit	Commits, Last 30 Days
Genkit Go	6.4k (monorepo)	go/v1.13.1, Sep 3, 2026	Sep 21, 2026	80 (all SDKs)
Eino	13.1k	v0.9.20, Sep 20, 2026	Sep 20, 2026	7 on `main`, plus five v0.10 alphas
ADK Go	8.8k	v2.4.0, Sep 11, 2026	Sep 21, 2026	100+
tRPC-Agent-Go	1.8k	v1.11.2, Aug 20, 2026	Sep 21, 2026	56
LangChainGo	9.7k	v0.1.14, Oct 20, 2025	Jan 11, 2026	0

Stars measure the past. The last two columns measure the present, and they tell the story.

Which Framework Should You Choose?

Choose Genkit Go if:

You want to iterate on your AI fast and get feedback with less back and forth. The Dev UI is the best local development loop in Go, by a distance.
You need vanilla calls, typed flows, and agents in the same service.
Provider neutrality matters: mixing Gemini, Claude, OpenAI, and Ollama should be a one-line change.
Your team also writes TypeScript, Dart, or Java and wants one framework story.

Choose Eino if:

Your application is a real graph: branches, parallel fan-out, sub-graphs, and streaming across all of it.
You want the most active independent community in Go and are fine living on 0.x.
Callback-based instrumentation into Langfuse, LangSmith, or your own APM is what you already do.
You need ByteDance-scale orchestration patterns, or you run on Volcano Engine.

Choose Google ADK Go if:

You are building multi-agent systems and Google Cloud is your runtime.
You want official Google support, the embedded ADK Web UI, and A2A interoperability.
Your organization already uses ADK in Python or Java and wants Go services with the same shape.
The newest Go toolchain is not a problem for you.

Choose tRPC-Agent-Go if:

You want LangGraph-style state graphs with checkpoints and interrupts, in Go, plugged into agents.
You like the ADK mental model but do not want the GCP gravity.
AG-UI frontends, Agent Skills, evaluation, and OpenTelemetry out of the box matter more than a Dev UI.
You can live with a smaller community and a large API surface.

Stay on LangChainGo only if:

You already run it in production, your models are pinned, and you do not stream through Gemini.
Even then, plan the migration. Nothing is being merged.

Conclusion

Go's Gen AI framework landscape in 2026 is richer than most Go developers realise, and it has a clear shape. Genkit Go is the most versatile choice and the best developer experience, with the Dev UI as a category of its own. Eino is the ambitious independent, the framework to pick when orchestration is the hard part of your problem. Google ADK Go is the enterprise agent runtime, at its best on Google Cloud. tRPC-Agent-Go is the dark horse, with the best workflow engine of the five hiding behind a modest star count. And LangChainGo is the framework that got everyone started and now needs a farewell plan.

If I had to pick one for a new Go service today with no cloud commitment, it would be Genkit Go, for the same reason as in TypeScript and Java: the combination of multiple abstraction levels, provider neutrality, and local observability shortens the loop between an idea and a working feature more than anything else on this list.

Whatever you pick, pick it deliberately. Go finally has the tooling. Use it.

Last updated: September 2026. Framework versions referenced: Genkit Go v1.13.1, Eino v0.9.20 (with eino-ext gemini v0.1.36), Google ADK Go v2.4.0, tRPC-Agent-Go v1.11.2, LangChainGo v0.1.14. Go 1.27.1, model gemini-3.8-flash.

How to Migrate from LangChainGo to Genkit Go

Xavier Portilla Edo — Tue, 29 Sep 2026 13:40:34 +0000

Introduction

If you built a Go AI service in the last two years, there is a good chance it runs on LangChainGo. It was the obvious choice: the name was familiar, the concepts came straight from Python, and it had the biggest provider catalog in the Go ecosystem.

That choice has aged badly. LangChainGo's last release was v0.1.14 in October 2025, its last commit to main was January 2026, and it is still on 0.1.x after three and a half years. Meanwhile the ecosystem moved: models were retired, SDKs were replaced, and APIs changed shape. LangChainGo did not move with them, and parts of it are broken today as a result.

Genkit Go is where that workload belongs now. It is GA, it is on v1.13.1, and it is actively maintained by Google.

This guide is the complete migration path: every LangChainGo concept mapped to its Genkit equivalent, with before and after code for each one. Follow it top to bottom and you can move a real service in an afternoon.

If you want the case for why rather than the how, read the companion article: LangChainGo vs Genkit Go: where Genkit shines.

Why Migrate Now

Three things in LangChainGo are broken today, and none of them will be fixed, because nobody is upstream to fix them.

1. The default model no longer exists. DefaultOptions() in llms/googleai/option.go hardcodes gemini-2.0-flash, which Google retired. A fresh googleai.New(ctx) with no options fails on the first call:

googleapi: Error 404: This model models/gemini-2.0-flash is no longer available.
Please update your code to use models/gemini-3.6-flash for the latest features
and improvements.

2. The Google provider runs on an end-of-life SDK. llms/googleai imports github.com/google/generative-ai-go, which Google replaced with google.golang.org/genai. Its README is explicit:

End-of-Life Date: All support for this repository (including bug fixes) will permanently end on November 30, 2025.

3. Streaming is broken on every current Gemini model. Because of that dead SDK, any streaming call through the Google provider fails:

error in stream mode: invalid character ']' looking for beginning of value

This fails on gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash and gemini-3.6-flash. Non-streaming calls still work, and the OpenAI provider streams fine, so this is specific to Google on the legacy SDK.

The trap: observability breaks your agents

This one catches people out, so it is worth knowing before you debug it the hard way.

LangChainGo turns on streaming automatically whenever you attach a callbacks handler. From chains/options.go:

if opts.StreamingFunc == nil && opts.CallbackHandler != nil {
    opts.StreamingFunc = func(ctx context.Context, chunk []byte) error {
        opts.CallbackHandler.HandleStreamingFunc(ctx, chunk)
        return nil
    }
}

agents/mrkl.go does the same thing. Since streaming is broken, attaching the only supported observability hook is also what breaks your agent. Same agent, same tool, one line different:

WITHOUT callbacks handler: err=<nil> out="The current year is 2026."
WITH    callbacks handler: err=error in stream mode: invalid character ']' looking for beginning of value

On LangChainGo with Gemini today, you can have working agents or you can have instrumentation. Not both.

What You Need

Go 1.24+ (this guide was written against Go 1.27)
A Google AI API key, free from Google AI Studio
Optionally an OpenAI API key, if you want cross-vendor fallback
The Genkit CLI, for the Developer UI:

npm install -g genkit

Add Genkit to your module:

go get github.com/firebase/genkit/go@latest

The Concept Map

This is the whole translation table. Keep it open while you work.

LangChainGo	Genkit Go	What changes
`llms.Model` + provider `New()`	plugin + `genkit.Init`	Models become string references like `googleai/gemini-3.5-flash`
`llms.GenerateFromSinglePrompt`	`genkit.GenerateText`	Direct swap
`prompts.PromptTemplate`	`ai.WithPrompt` or a `.prompt` file	Dotprompt files keep prompts outside the binary
`chains.LLMChain` + `chains.Run`	`genkit.DefineFlow`	A typed Go function instead of a `map[string]any`
`chains.NewSequentialChain`	ordinary Go code	Just call the next function
`outputparser.NewDefined[T]`	`genkit.GenerateData[T]`	Real schema instead of prompt-and-parse
`tools.Tool` interface	`genkit.DefineTool[In, Out]`	Typed arguments with a JSON schema
`agents.NewOneShotAgent` (ReAct)	`ai.WithTools`, or `genkitx.DefineAgent`	Native function calling instead of text parsing. Genkit also has a real agent primitive with sessions
`memory.NewConversationBuffer`	`resp.History()`, or an agent session store	Explicit history, or `aix.WithSessionStore` if you want the buffer back
`embeddings` + `vectorstores`	`localvec` or a vector store plugin	Genkit has a local store, LangChainGo does not
`callbacks.Handler`	built-in tracing	Delete the handler entirely
nothing	`middleware.Retry`, `middleware.Fallback`	Retry and fallback stop being your code

Step 1: Set Up Genkit

In LangChainGo, each provider is a concrete struct you construct and hold:

llm, err := googleai.New(ctx,
    googleai.WithDefaultModel("gemini-3.5-flash"),
    googleai.WithDefaultEmbeddingModel("gemini-embedding-001"))
if err != nil {
    log.Fatal(err)
}
defer llm.Close()

In Genkit, plugins register with one runtime object and models become string references:

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithDefaultModel("googleai/gemini-3.5-flash"))

That difference matters more than it looks. Registering a second vendor is one more plugin:

g := genkit.Init(ctx, genkit.WithPlugins(
    &googlegenai.GoogleAI{},
    &openai.OpenAI{},
))

Now both vendors live in the same registry and a model is just a name, which is what makes swapping and falling back a configuration change later on.

Step 2: Migrate Your Model Calls

A plain text generation is close to a direct swap.

Before:

out, err := llms.GenerateFromSinglePrompt(ctx, llm,
    "In one sentence: what is Go's error handling philosophy?")

After:

out, err := genkit.GenerateText(ctx, g,
    ai.WithPrompt("In one sentence: what is Go's error handling philosophy?"))

If you need the full response rather than just the text, use genkit.Generate, which gives you resp.Text(), resp.History() and usage data.

Step 3: Turn Chains into Flows

A LangChainGo chain wraps a prompt template and takes a map[string]any:

Before:

prompt := prompts.NewPromptTemplate(
    "Triage this support ticket.\n{{.format}}\n\nTicket: {{.ticket}}",
    []string{"format", "ticket"},
)
chain := chains.NewLLMChain(llm, prompt)
raw, err := chains.Predict(ctx, chain, map[string]any{
    "format": parser.GetFormatInstructions(),
    "ticket": ticket,
})

A Genkit flow is a typed Go function. Your input and output types are the contract and the compiler enforces them:

After:

triage := genkit.DefineFlow(g, "triage", func(ctx context.Context, ticket string) (*Triage, error) {
    out, _, err := genkit.GenerateData[Triage](ctx, g,
        ai.WithPrompt("Triage this support ticket: %s", ticket))
    return out, err
})

out, err := triage.Run(ctx, "I was charged twice for order A-1029.")

This step deletes the most code, because most of what a chain does is pass a map around. Go does not need that.

Sequential chains have no equivalent, and do not need one. chains.NewSequentialChain becomes calling the next function:

summary, err := summarize.Run(ctx, doc)
if err != nil {
    return nil, err
}
return classify.Run(ctx, summary)

Step 4: Migrate Structured Output

Prioritise this step. In LangChainGo, structured output is prompt engineering. In Genkit it is a schema.

Before: generate format instructions from your struct, paste them into the prompt, parse the text back.

type Ticket struct {
    Category string `json:"category" describe:"one of: billing, bug, feature, other"`
    Severity int    `json:"severity" describe:"1 (low) to 5 (critical)"`
    Summary  string `json:"summary" describe:"one sentence"`
}

parser, err := outputparser.NewDefined(Ticket{})
// ...put parser.GetFormatInstructions() into the prompt...
out, err := parser.Parse(raw)

After: one call, with the schema sent to the model as a real response schema.

type Ticket struct {
    Category string `json:"category" jsonschema:"enum=billing,enum=bug,enum=feature,enum=other"`
    Severity int    `json:"severity" jsonschema_description:"1 (low) to 5 (critical)"`
    Summary  string `json:"summary" jsonschema_description:"one sentence"`
}

out, _, err := genkit.GenerateData[Ticket](ctx, g,
    ai.WithPrompt("Triage this support ticket: %s", ticket))

Note the tag change: describe: becomes jsonschema_description:, and enums move into a jsonschema: tag where the model actually enforces them.

Step 5: Migrate Your Tools

The LangChainGo tool interface is Call(ctx context.Context, input string) (string, error). One string in, one string out, no schema. A tool that needs typed arguments has to describe its schema in prose and unmarshal by hand:

Before:

func (RefundTool) Name() string { return "issue_refund" }

func (RefundTool) Description() string {
    return `Issues a refund. Input MUST be a JSON object string with exactly these keys:
{"order_id": string, "amount_cents": integer}. Example: {"order_id":"A-1","amount_cents":2500}`
}

func (RefundTool) Call(ctx context.Context, input string) (string, error) {
    input = strings.TrimSpace(input)
    input = strings.TrimPrefix(input, "```json")
    input = strings.TrimPrefix(input, "```")
    input = strings.TrimSuffix(input, "```")
    var a RefundArgs
    if err := json.Unmarshal([]byte(strings.TrimSpace(input)), &a); err != nil {
        return fmt.Sprintf("error: input was not valid JSON (%v). Got: %s", err, input), nil
    }
    return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil
}

In Genkit the tool function is generic over its argument type and the schema comes from the struct:

After:

type RefundArgs struct {
    OrderID     string `json:"order_id"`
    AmountCents int    `json:"amount_cents"`
}

refund := genkit.DefineTool(g, "issue_refund", "Issues a refund for an order.",
    func(ctx *ai.ToolContext, a RefundArgs) (string, error) {
        return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil
    })

Delete the fence stripping, the unmarshalling and the prose schema. The model receives a real JSON schema and the arguments arrive typed.

Step 6: Replace the Agent

A LangChainGo ReAct agent drives tools by parsing the model's text output for Action: and Action Input: lines, wrapped in an executor with an iteration cap:

Before:

agent := agents.NewOneShotAgent(llm, []tools.Tool{RefundTool{}}, agents.WithMaxIterations(6))
out, err := chains.Run(ctx, agents.NewExecutor(agent),
    "Refund order A-1029 for 25 dollars and 50 cents.")

In Genkit, native function calling means a one-shot agent is just a generate call with tools attached:

After:

out, err := genkit.GenerateText(ctx, g,
    ai.WithPrompt("Refund order A-1029 for 25 dollars and 50 cents."),
    ai.WithTools(refund))

No executor, no iteration cap, no text parsing. Genkit runs the tool loop for you.

When you want a real agent, Genkit has one

The generate call above replaces a OneShotAgent. For anything that has to hold a conversation, Genkit Go ships an actual agent primitive with sessions and persisted state, which LangChainGo has no equivalent for:

import (
    aix "github.com/firebase/genkit/go/ai/exp"
    "github.com/firebase/genkit/go/ai/exp/localstore"
    genkitx "github.com/firebase/genkit/go/genkit/exp"
)

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithDefaultModel("googleai/gemini-2.5-flash"),
    genkit.WithExperimental())

store := localstore.NewInMemorySessionStore[State]()

agent := genkitx.DefineAgent(g, "taskAgent",
    aix.InlinePrompt{
        ai.WithSystem("Manage a task list. Use tools when changing tasks."),
        ai.WithTools(addTask),
    },
    aix.WithSessionStore(store),
    aix.WithDescription[State]("Task management assistant"),
)

Three constructors are available: DefineAgent for an inline prompt, DefinePromptAgent to wrap a prompt already in the registry, and DefineCustomAgent to replace the prompt loop with your own code. Options cover a session store, WithStateTransform to reshape state on the way out to a client, and WithStreamTransform for chunks. You drive one with Run, RunText, or Connect for a streaming connection.

These live in the exp packages. Agents are experimental in Genkit Go v1.13.1, and calling genkitx.DefineAgent without genkit.WithExperimental() on genkit.Init panics with a message telling you so. The API may change between minor releases, so weigh that before putting one in production.

Step 7: Migrate RAG

Check this step before planning your timeline, because it has an infrastructure consequence.

LangChainGo ships fifteen vector stores: AlloyDB, Azure AI Search, Bedrock Knowledge Bases, Chroma, CloudSQL, Dolt, MariaDB, Milvus, MongoDB, OpenSearch, pgvector, Pinecone, Qdrant, Redis and Weaviate. Every one of them needs an external service running. There is no in-memory or local option, and the pull request adding one has been open since February 2024. If you needed a local store, you wrote it yourself.

Genkit ships localvec, a file-backed local store, so the same thing is three calls:

After:

embedder := googlegenai.GoogleAIEmbedder(g, "gemini-embedding-001")
store, retriever, err := localvec.DefineRetriever(g, "policy",
    localvec.Config{Dir: "/tmp/gkvec", Embedder: embedder}, nil)
if err != nil {
    return err
}

// Index your documents once.
localvec.Index(ctx, docs, store)

// Retrieve at request time.
found, err := genkit.Retrieve(ctx, g, ai.WithRetriever(retriever), ai.WithTextDocs(ticket))

Then hand the documents straight to the model:

out, _, err := genkit.GenerateData[Triage](ctx, g,
    ai.WithDocs(found.Documents...),
    ai.WithPrompt("Triage this ticket using the policy documents: %s", ticket))

If you were on a hosted store, Genkit has Pinecone, Weaviate, Postgres and AlloyDB plugins.

Step 8: Migrate Memory

LangChainGo keeps conversation history in a buffer attached to the chain:

Before:

c := chains.NewConversation(llm, memory.NewConversationBuffer())
chains.Run(ctx, c, "My name is Xavi and I work in Go.")
chains.Run(ctx, c, "What is my name and what language do I use?")

Genkit passes history explicitly:

After:

resp, err := genkit.Generate(ctx, g, ai.WithPrompt("My name is Xavi and I work in Go."))
if err != nil {
    return err
}

resp, err = genkit.Generate(ctx, g,
    ai.WithMessages(resp.History()...),
    ai.WithPrompt("What is my name and what language do I use?"))

For a plain generate call this is the one mapping where LangChainGo is less typing. The trade is that Genkit has no hidden buffer to keep in sync, which is what you want the moment your conversation state lives in Redis or Postgres and your handlers are stateless.

If you want the buffer back, that is what an agent's session store is for. Give the agent a store and pass a session id, and the history is the framework's problem again:

store := localstore.NewInMemorySessionStore[State]()

agent := genkitx.DefineAgent(g, "taskAgent",
    aix.InlinePrompt{
        ai.WithSystem("Manage a task list. Use tools when changing tasks. Be terse."),
        ai.WithTools(addTask),
    },
    aix.WithSessionStore(store))

for _, turn := range []string{
    "Add 'buy milk' to my list.",
    "What did I just ask you to add?",
} {
    out, err := agent.RunText(ctx, turn, aix.WithSessionID[State]("demo-session"))
    if err != nil {
        return err
    }
    fmt.Printf("> %s\n%s\n\n", turn, out.Message.Text())
}

> Add 'buy milk' to my list.
Task added.

> What did I just ask you to add?
You asked me to add 'buy milk' to your list.

Nothing in that loop passes the first turn into the second. The session id does it. localstore.NewInMemorySessionStore is the one that ships for local development; the SessionStore interface is what you implement to put sessions in Redis or Postgres.

So the honest version of this row is: Genkit gives you both ends. Explicit resp.History() when you want stateless handlers, and a store-backed session when you want the buffer.

Step 9: Delete Your Resilience Code

This is the step that usually surprises people with how much it removes.

LangChainGo has no middleware layer. callbacks.Handler is seventeen methods that all return nothing:

HandleText(ctx, text)
HandleLLMGenerateContentStart(ctx, ms)
HandleLLMGenerateContentEnd(ctx, res)
HandleToolStart(ctx, input)
// ...and thirteen more

You can watch a call. You cannot wrap it, retry it, rewrite its request or substitute a model, because there is no next(). There is no retry, no fallback and no OpenTelemetry anywhere in the library, so if your service has those, you wrote them.

In Genkit they are configuration:

After:

out, _, err := genkit.GenerateData[Triage](ctx, g,
    ai.WithModel(googlegenai.GoogleAIModelRef("gemini-3.5-flash", nil)),
    ai.WithDocs(found.Documents...),
    ai.WithTools(refund),
    ai.WithPrompt("Triage this ticket and act on it: %s", ticket),
    ai.WithUse(
        &middleware.Retry{MaxRetries: 2},
        &middleware.Fallback{Models: []ai.ModelRef{openai.ModelRef("gpt-5-mini", nil)}},
    ),
)

g := genkit.Init(ctx, genkit.WithPlugins(
    &googlegenai.GoogleAI{}, &openai.OpenAI{}, &middleware.Middleware{},
))

Point the primary at a dead model and the request heals onto the other vendor on its own:

WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND"
{Category:billing Severity:2 PolicyQuote:Duplicate charges must be refunded in full within 5
business days, no manager approval required. Action:Issued full refund of $25.50 (2550 cents)
for the duplicate charge on order A-1029.}

Gemini went down, OpenAI picked it up, the tool still ran, and the typed struct came back intact.

Delete your callbacks handler too. Genkit traces flows, generate calls, tool calls and retrieval automatically. Run the Developer UI to see them:

genkit start -- go run .

Step 10: Wrap the Entrypoints

Finally, expose your flows over HTTP. Where LangChainGo left you writing net/http handlers around chains, Genkit turns a flow into an endpoint:

mux := http.NewServeMux()
mux.HandleFunc("POST /triage", genkit.Handler(triage))
log.Fatal(server.Start(ctx, "127.0.0.1:3400", mux))

The flow's input and output types define the request and response shapes, so the whole path is typed end to end.

Before You Commit

The migration is not free. Check these three things first:

Provider and vector store catalog. LangChainGo has seventeen model providers and fifteen vector stores. Genkit Go covers Google AI, Vertex, OpenAI, Anthropic, Ollama and the OpenAI-compatible vendors, plus Pinecone, Weaviate, Postgres and AlloyDB. If you are on Dolt, MariaDB or Bedrock Knowledge Bases as a vector store, confirm you have a path.

Conversation buffers. There is no drop-in ConversationBuffer. You pass history explicitly, as in Step 8.

Specialised chains. Helpers like chains.NewSQLDatabaseChain and the map-reduce summarisation chains have no one-to-one equivalent. In Genkit they are Go functions you write, which is usually shorter than the chain was, but it is work.

Conclusion

The migration is smaller than it looks, because most of a LangChainGo program is plumbing that Go does not need: maps passed between chains, format instructions pasted into prompts, JSON unmarshalled out of tool strings, and a resilience layer you wrote yourself. Typed flows, typed tools, schema-backed output and middleware delete all four.

The reason to start now is not ergonomics. It is that the library your service depends on has a retired default model, an end-of-life SDK underneath its Google provider, broken streaming, and an observability hook that breaks the agents it is supposed to observe. Those are not problems you can wait out.

For the side-by-side numbers and the full picture, read the companion: LangChainGo vs Genkit Go: where Genkit shines.

Further reading:

Genkit documentation
Genkit Go plugins
LangChainGo repository
Google Gen AI SDK for Go, the replacement for the SDK LangChainGo still uses
Stop using Python for GenAI, use Genkit Go
Mastering Genkit: Go Edition

LangChainGo vs Genkit Go: Where Genkit Shines

Xavier Portilla Edo — Tue, 29 Sep 2026 13:40:25 +0000

Introduction

There are two realistic choices for building AI services in Go today: LangChainGo, the Go port of the Python framework everyone knows, and Genkit Go, Google's open-source Gen AI framework.

LangChainGo is still the first result most Go developers find. It has 9,677 stars and the biggest provider catalog in the ecosystem. It is also eight months without a commit, eleven months without a release, and still on 0.1.x after three and a half years.

This article compares them the way that actually matters: by building the same production service in both and looking at what each one costs you. Not a feature checklist, a real application with real requirements.

If you have already decided and want the migration mechanics, that is the companion article: How to migrate from LangChainGo to Genkit Go.

The State of Both Projects

	LangChainGo	Genkit Go
Latest release	`v0.1.14`, October 2025	`v1.13.1`
Last commit to `main`	January 2026	active
Age and version	3.5 years, still `0.1.x`	GA since `v1.0`
Open issues	242
Open pull requests	171, oldest from June 2023
Stars	9,677

LangChainGo is not archived, and "dead" overstates it. But eleven months without a release is long enough for the ecosystem to move underneath a library, and it has.

What Is Broken in LangChainGo Today

Three concrete things, all of which affect a service in production right now.

The quickstart returns a 404. DefaultOptions() hardcodes gemini-2.0-flash, which Google retired. A fresh googleai.New(ctx) fails on the first call.

The Google provider runs on an end-of-life SDK. llms/googleai imports github.com/google/generative-ai-go, whose README states support ended on November 30, 2025.

Streaming is broken on every current Gemini model. gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash and gemini-3.6-flash all fail the same way:

error in stream mode: invalid character ']' looking for beginning of value

The OpenAI provider streams fine, so this is the Google provider on the legacy SDK, not the library everywhere.

There is a fourth consequence that follows from the third, and it is the nastiest one. LangChainGo installs a streaming function automatically whenever you attach a callbacks handler, in chains/options.go and again in agents/mrkl.go. Since streaming is broken, turning on observability breaks your Gemini agents:

WITHOUT callbacks handler: err=<nil> out="The current year is 2026."
WITH    callbacks handler: err=error in stream mode: invalid character ']' looking for beginning of value

Feature by Feature

Capability	LangChainGo	Genkit Go
Structured output	Format instructions in the prompt, parse the text back	`genkit.GenerateData[T]`, real response schema
Tools	`Call(ctx, string) (string, error)`, no schema	`genkit.DefineTool[In, Out]`, typed with JSON schema
Agents	ReAct, driven by parsing `Action:` lines	Native function calling, plus an agent primitive with sessions (experimental)
Composition	`Chain` interfaces over `map[string]any`	Typed Go functions (`DefineFlow`)
Retry	Write it yourself	`middleware.Retry`
Cross-vendor fallback	Write it yourself	`middleware.Fallback`
Observability	17 observe-only callback methods	Built-in tracing plus a Developer UI
OpenTelemetry	None	Built in
Local vector store	None, all 15 stores need a server	`localvec`, file-backed
Streaming on Gemini	Broken	Works
HTTP endpoint	`net/http` boilerplate	`genkit.Handler(flow)`

The single most important row is retry and fallback, and the reason is architectural rather than cosmetic. A Genkit middleware wraps a call and receives a next(), so it can decide to call again, or call something else. LangChainGo's callbacks.Handler returns nothing from every method. You can watch a call, you cannot change it. That missing seam is why every resilience concern ends up in your application code.

What We Are Building

A support-ticket triage service. Every requirement here is something a real team asks for:

Retrieve the relevant policy documents for an incoming ticket.
Let the model call a tool to act on it.
Return a typed struct, not free text.
Retry transient model failures.
Survive a vendor outage by falling back to a different provider.
Trace every step, so somebody can debug it at 3am.

The Genkit Go Version

The whole service, in one file:

g := genkit.Init(ctx, genkit.WithPlugins(
    &googlegenai.GoogleAI{}, &openai.OpenAI{}, &middleware.Middleware{},
))

// RAG store.
embedder := googlegenai.GoogleAIEmbedder(g, "gemini-embedding-001")
store, retriever, err := localvec.DefineRetriever(g, "policy",
    localvec.Config{Dir: "/tmp/gkvec", Embedder: embedder}, nil)
localvec.Index(ctx, docs, store)

// A tool the model can call.
refund := genkit.DefineTool(g, "issue_refund", "Issues a refund for an order.",
    func(ctx *ai.ToolContext, a struct {
        OrderID     string `json:"order_id"`
        AmountCents int    `json:"amount_cents"`
    }) (string, error) {
        return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil
    })

// The flow.
triage := genkit.DefineFlow(g, "triage", func(ctx context.Context, ticket string) (*Triage, error) {
    found, err := genkit.Retrieve(ctx, g, ai.WithRetriever(retriever), ai.WithTextDocs(ticket))
    if err != nil {
        return nil, err
    }
    out, _, err := genkit.GenerateData[Triage](ctx, g,
        ai.WithModel(googlegenai.GoogleAIModelRef("gemini-3.5-flash", nil)),
        ai.WithDocs(found.Documents...),
        ai.WithTools(refund),
        ai.WithPrompt("Triage this ticket using the policy documents, and act on it: %s", ticket),
        ai.WithUse(
            &middleware.Retry{MaxRetries: 2},
            &middleware.Fallback{Models: []ai.ModelRef{openai.ModelRef("gpt-5-mini", nil)}},
        ),
    )
    return out, err
})

73 lines, and it returns exactly what you asked for:

{Category:billing Severity:1 PolicyQuote:Duplicate charges must be refunded in full within
5 business days, no manager approval required. Action:Issued a refund of $25.50 (2550 cents)
to resolve the duplicate charge on order A-1029.}

The interesting part is which requirements are not code. Retry and fallback are two entries in a list. Tracing does not appear at all, because Genkit instruments flows, generate calls, tool calls and retrieval itself. The RAG store is three calls because localvec ships in the box.

The LangChainGo Version

Requirements 1, 2 and 3 port directly. Requirements 4, 5 and 6 do not exist in the library, so they become your code:

File	Lines	Why it exists
`main.go`	110	the actual application
`memstore.go`	68	all 15 vector stores need an external service, so the local one is yours
`resilience.go`	58	no middleware layer, so retry and fallback are yours
`tracer.go`	36	observability is a 17-method interface to implement
total	272

None of those three extra files contain interesting code. memstore.go is cosine similarity and a sort. tracer.go is six methods that log and eleven empty ones that exist only to satisfy the interface:

// Everything below exists only to satisfy the interface.
func (t *tracer) HandleText(context.Context, string)               {}
func (t *tracer) HandleLLMStart(context.Context, []string)         {}
func (t *tracer) HandleLLMError(context.Context, error)            {}
func (t *tracer) HandleChainStart(context.Context, map[string]any) {}
// ...seven more

resilience.go wraps the primary model to add retries and a fallback, and it has a sharp edge worth seeing:

clean := make([]llms.CallOption, 0, len(opts))
for _, o := range opts {
    var probe llms.CallOptions
    o(&probe)
    // Model names are vendor-specific, and gpt-5-mini rejects 'stop'
    // outright, which the ReAct agent always sets. Both must go.
    if probe.Model != "" || len(probe.StopWords) > 0 {
        continue
    }
    clean = append(clean, o)
}
return r.backup.GenerateContent(ctx, msgs, clean...)

CallOption is an opaque function, so to find out what an option does you have to apply it to a probe struct and inspect the result. Miss the stop words and the fallback fails with Unsupported parameter: 'stop' is not supported with this model.

Both versions work and produce equivalent results. So far this is a story about volume, not capability. The outage test is where that changes.

What Happens When a Model Goes Down

Point the primary at a retired model in both versions, which is a real 404 from the live API.

Genkit heals and finishes:

WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND"
WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND"
{Category:billing Severity:2 PolicyQuote:Duplicate charges must be refunded in full within
5 business days, no manager approval required. Action:Issued full refund of $25.50 ...}

Two calls fail over, once for the initial generate and once for the turn after the tool call. The typed struct survives a vendor switch.

LangChainGo heals at the model layer, then breaks one layer up:

2026/09/11 20:01:45 primary failed (googleapi: Error 404 ...), falling back
2026/09/11 20:01:54 PARSE FAILED: input text should start with ```json and end with ```

The hand-written fallback did its job. The output parser then rejected the answer, because outputparser.Defined.Parse hard-requires code fences:

const opening = "```json"
const closing = "```"
if text[:len(opening)] != opening || text[len(text)-len(closing):] != closing {
    return target, fmt.Errorf("input text should start with %s and end with %s", opening, closing)
}

Gemini tends to emit those fences. OpenAI does not. The typed output layer is quietly coupled to one vendor's formatting habit, and the coupling surfaces at the exact moment you switch vendors, which is the worst possible time.

That same function has a second problem. When the response is shorter than seven characters, it does not return an error, it panics:

input "ok"  PANIC: runtime error: slice bounds out of range [:7] with length 2
input ""    PANIC: runtime error: slice bounds out of range [:7] with length 0

An empty completion or a refusal becomes an uncaught panic in your request handler.

Dependency Weight

The same typed-extraction program, each in its own clean module:

	LangChainGo	Genkit Go
Modules in the build graph	62	41
Binary size	38.0 MB	28.7 MB

The LangChainGo-only half of that list is worth reading: the deprecated generative-ai-go, plus cloud.google.com/go/vertexai and aiplatform, plus sprig, gonja (a Jinja2 template engine), go.starlark.net (the calculator tool evaluates expressions in Starlark), logrus, tiktoken-go, pkg/errors and json-iterator. A Python-shaped dependency tree, in Go.

Where Genkit Go Shines

Pulling the thread through all of it, the difference comes down to four things Genkit has and LangChainGo does not.

Middleware is the first. Because a Genkit middleware wraps a call and can decide to call it again or call something else, resilience becomes library code. Retry, cross-vendor fallback and tool approval are list entries instead of wrapper types you maintain forever.

Types are the second. Typed flows, typed tools and schema-backed output mean the compiler carries the contract. That is not just tidier: it is why the Genkit service survived the failover and the LangChainGo one did not. Nothing in the Genkit path was slicing strings out of a model response to find the data.

Plugins are the third. A model is a string reference in a registry, so swapping Gemini for GPT is a configuration change. In LangChainGo, two providers are two concrete structs of two different types, which is why a hand-written fallback has to strip vendor-specific call options and still gets it wrong the first time.

The Developer UI is the fourth, and it is the one you feel every day rather than during an incident. LangChainGo has no equivalent to it at all.

One command starts it next to your app:

genkit start -- go run .

It reads the registry your code built, so nothing has to be declared twice. On the triage service it found the flow, the tool, 39 models and 3 embedders without any configuration:

Flows (1)      triageTicket
Tools (1)      issue_refund
Models (39)    googleai/gemini-2.5-flash, ...
Embedders (3)  googleai/gemini-embedding-001, ...

Open a flow and the input form is generated from your Go struct. A Ticket with Subject and Body fields gives you this, pre-filled and ready to edit:

{
  "body": "",
  "subject": ""
}

Press Run and you get the typed output back as a tree, not a blob of text:

{ "category": "Billing", "urgency": 4, "action": "Refund issued" }

The part that earns its place is the trace underneath. Every run is a span tree with per-span timings, and the tool call is a span of its own:

triageTicket                        4.64s
├─ generate                         2.52s
│  ├─ googleai/gemini-2.5-flash     1.67s
│  ├─ issue_refund                    1ms
│  └─ generate (2)                  846ms
│     └─ googleai/gemini-2.5-flash  844ms
└─ generate                         2.12s
   └─ googleai/gemini-2.5-flash     2.12s

You can read the whole shape of the run off that: the first generate called the tool, the model went back for a second turn with the tool result, and a separate generate did the typed extraction. Nobody wrote a line of logging for it.

Failures land in the same place. A run that hit a real Gemini constraint showed the error inline with a full Go stack trace, and marked every span in the tree that was affected:

Error 400, Message: Function calling with a response mime type: 'application/json'
is unsupported, Status: INVALID_ARGUMENT, Details: []

That error is worth knowing on its own, and it is model-specific rather than a Genkit limitation. Running the same tools-plus-typed-output call across five models makes the line clear:

gemini-2.5-flash         FAIL  Function calling with a response mime type: 'application/json' is unsupported
gemini-3-flash-preview   OK
gemini-3.5-flash         OK
gemini-3.6-flash         OK
gemini-3.8-flash         OK

Gemini 2.5 refuses function calling combined with a JSON response mime type, so on that model a tool call and a typed extraction have to be two steps. Every Gemini 3 model lifted the restriction, which is why the triage service above runs both in one call on gemini-3.5-flash. The useful part is that the Developer UI told me which span failed and why, instead of leaving me to bisect it.

The sidebar also carries Traces, Evaluations and Datasets, so the same UI covers running an action by hand, reading what it did, and building a dataset out of it. In LangChainGo, the equivalent of all of this is a callbacks.Handler you write yourself, which returns nothing from every method and, as shown above, breaks your Gemini agents when you attach it.

Underneath all four, someone is shipping. When gemini-2.0-flash was retired, that was a 404 to work around in LangChainGo and a fallback that healed itself in Genkit.

What LangChainGo Still Does Well

This is not a demolition. There are real reasons it is still in use:

Provider catalog. Seventeen model providers including Bedrock, Cloudflare, Ernie, HuggingFace, Maritaca, Mistral, watsonx and llamafile. Genkit's plugin set is narrower.
Vector store catalog. Fifteen integrations, including Dolt, MariaDB and OpenSearch, which Genkit has no equivalent for.
Conversation memory is less typing. chains.NewConversation(llm, memory.NewConversationBuffer()) beats passing resp.History() around by hand. Genkit's answer is an agent session store, which is more capable but sits behind an experimental flag, so on a plain generate call LangChainGo is still terser.
The ReAct agent works. On a sequential two-tool task it succeeds consistently. The cost is the code you own, not flakiness.
Familiarity. If your team already thinks in chains, memories and retrievers from Python, the vocabulary transfers directly.

If you depend on a provider or a vector store only LangChainGo supports, that is a real reason to stay. Weigh it against an unmaintained dependency rather than against line counts.

The Scorecard

Non-blank, non-comment lines.

Requirement	Genkit Go	LangChainGo
Full service: RAG + tool + typed output + retry + fallback + tracing	73, one file	272, four files
Behaviour under a vendor outage	heals, typed output intact	heals at model layer, output parser fails
Tool with two typed arguments	31	55 (hand-written unmarshalling)
Retry and cross-vendor fallback	2 config lines	58 lines, vendor-specific options to strip
Local vector store for RAG	ships as `localvec`	68 lines, no local option exists
Observability	built in, zero lines	36 lines, and it breaks Gemini agents
Streaming on current Gemini models	works	broken on all four tested
Quickstart with no options	works	404, default model retired
Modules / binary for the same program	41 / 28.7 MB	62 / 38.0 MB

Conclusion

If you are starting a Go AI project today, use Genkit Go. That part is not close.

If you already have LangChainGo in production, the decision is about risk rather than elegance. The library is eight months without a commit, its Google provider is built on an SDK that reached end of life in November 2025, its quickstart returns a 404, its streaming is broken on every current Gemini model, and turning on observability breaks the agents it is meant to observe. Those are not problems that resolve themselves.

The clearest signal is the outage test. Both versions were supposed to survive a vendor failure, and technically both did at the model layer. Only one came back with the data the caller asked for, because in the other one the layer above was matching on code fences.

Further reading:

Genkit Flows as senro Steps: Extremely Powerful AI Pipelines in Go

Xavier Portilla Edo — Tue, 29 Sep 2026 13:39:43 +0000

Introduction

A Genkit flow is an excellent unit of AI work. It is typed, traced, testable, and genkit.DefineFlow gives you something you can call from anywhere.

What a flow is not is a unit of orchestration. The moment you have twelve documents to summarise, a flow whose output feeds two other flows, or a model call that fails on the third of forty items, you end up writing the same scaffolding again: a worker pool, a retry loop, a way to avoid re-running the eleven items that already succeeded, logs you can actually read, and some way to find out what happened at three in the morning.

senro is a pipeline engine where a step already has all of that: retries that distinguish infrastructure from workload, an action cache that skips work it has already done, an append-only event stream, and a live attach protocol. I covered it in the introduction and the monorepo article.

This article shows how to combine them. Put a Genkit flow inside a senro step and every model call becomes a graph node with its own log, state, retry policy and cache entry.

Every example below ran against the real Gemini API with googleai/gemini-2.5-flash, on Genkit Go v1.13.1 and senro v1.4.0. The summaries are what the model actually returned.

What you need

go get github.com/firebase/genkit/go
go get github.com/xavidop/senro

The example is a small corpus of release notes in docs/*.md that gets summarised one document at a time, then merged into a digest.

Step 1: the flow

Nothing senro-specific here. This is an ordinary Genkit flow:

// SummarizeInput is what the summarize flow takes.
type SummarizeInput struct {
    Path string `json:"path"`
    Text string `json:"text"`
}

// Summary is what it returns.
type Summary struct {
    Path    string `json:"path"`
    Summary string `json:"summary"`
}

func defineFlows(g *genkit.Genkit) *core.Flow[SummarizeInput, Summary, struct{}] {
    return genkit.DefineFlow(g, "summarize",
        func(ctx context.Context, in SummarizeInput) (Summary, error) {
            text, err := genkit.GenerateText(ctx, g,
                ai.WithSystem("You summarize release notes for an operator. "+
                    "Answer in one sentence, no preamble."),
                ai.WithPrompt(fmt.Sprintf("Summarize these release notes:\n\n%s", in.Text)))
            if err != nil {
                return Summary{}, err
            }
            return Summary{Path: in.Path, Summary: text}, nil
        })
}

Called on its own it does what you expect:

The billing reconciliation job was moved to a queue worker, accelerating invoice
settlement from up to 24 hours to within two minutes.

Step 2: wrap the flow in a step

senro has two kinds of step: a command, and a registered Go function. A Genkit flow is a Go function, so it is the second kind:

// summarize is the registered flow. A func step runs in the pipeline's own
// process, so a package-level handle is all the wiring the two need.
var summarize *core.Flow[SummarizeInput, Summary, struct{}]

type SummarizeParams struct {
    Path string `json:"path"`
    Name string `json:"name"`
}

func init() { senro.RegisterFunc("ai/summarize", RunSummarize) }

// RunSummarize runs the flow and writes the summary into the one workspace
// this step mounts.
func RunSummarize(ctx senro.Ctx, p SummarizeParams) error {
    dir, ok := ctx.Workspace("doc-" + p.Name)
    if !ok {
        return fmt.Errorf("step %s mounts no doc-%s workspace", ctx.StepID(), p.Name)
    }

    text, err := os.ReadFile(filepath.Join(string(dir), "doc.md"))
    if err != nil {
        return err
    }

    res, err := summarize.Run(ctx, SummarizeInput{Path: p.Path, Text: string(text)})
    if err != nil {
        return err
    }

    fmt.Fprintf(ctx.Stdout(), "%s\n", res.Summary)
    return os.WriteFile(filepath.Join(string(dir), "summary.txt"), []byte(res.Summary+"\n"), 0o644)
}

Three details make this work smoothly:

senro.Ctx embeds context.Context, so it passes straight into summarize.Run with no adapter. Genkit's tracing and cancellation behave exactly as they always did.
ctx.Stdout() is the step's real log stream, recorded and redacted like a command's output. Writing to os.Stdout instead reaches your terminal and no log file.
ctx.Workspace(name) hands the function the same path a mount gives a command, on every executor. The function body does not change when the step later runs in a container or a pod.

RegisterFunc registers the function once, from an init. The name is the function's identity: it is what the plan records and what feeds the step's cache key, so renaming it invalidates the cache exactly as renaming a command would. Parameters must be JSON-serializable, and decoding is strict, so a renamed field fails loudly instead of running with a zero value.

Step 3: build the graph

Because the pipeline is a Go program, fanning out over a corpus is a for loop:

func main() {
    ctx := context.Background()

    g := genkit.Init(ctx,
        genkit.WithPlugins(&googlegenai.GoogleAI{}),
        genkit.WithDefaultModel("googleai/gemini-2.5-flash"))
    summarize = defineFlows(g)

    docs, err := filepath.Glob("docs/*.md")
    if err != nil {
        log.Fatal(err)
    }
    sort.Strings(docs)

    p := senro.New("ai")
    w := p.Workflow("summarize")

    var names, ids []string
    var wss []*senro.WorkspaceRef
    corpus := map[string]string{}

    for _, doc := range docs {
        text, err := os.ReadFile(doc)
        if err != nil {
            log.Fatal(err)
        }
        name := strings.TrimSuffix(filepath.Base(doc), ".md")
        names = append(names, name)
        ids = append(ids, "summarize/"+name)
        corpus[name] = string(text)

        // One workspace per document, so one document changing does not
        // invalidate the other summaries' cache keys.
        wss = append(wss, senro.Workspace("doc-"+name, senro.Scope(senro.ScopeRun)))
    }

    seed := w.Step("seed", senro.Func("ai/seed", SeedParams{Docs: corpus}))
    for i, ws := range wss {
        seed.Mount(ws.At("/doc-"+names[i], senro.RW))
    }

    for i, name := range names {
        // One AI call, one node: its own log, state, retry and cache entry.
        w.Step(ids[i], senro.Func("ai/summarize", SummarizeParams{
            Path: docs[i],
            Name: name,
        })).
            Needs("seed").
            Mount(wss[i].At("/doc", senro.RW)).
            Retry(3, retry.OnInfra()).
            Pure().
            Inputs(artifact.File("doc.md")).
            Outputs(artifact.File("summary.txt"))
    }

    digestStep := w.Step("digest", senro.Func("ai/digest", DigestParams{Names: names})).
        Needs(ids...)
    for i, ws := range wss {
        digestStep.Mount(ws.At("/doc-"+names[i], senro.RO))
    }

    if err := senro.Run(ctx, p); err != nil {
        log.Print(err)
        os.Exit(1)
    }
}

seed writes each document into a workspace of its own. Then one summarize step per document, running in parallel. Then digest, which waits for all of them and merges the results.

That loop produces this graph, rendered straight from the run's own plan.json:

pipeline: ai
  │
  ├─ wave 1                seed                  func
  │
  ├─ wave 2 (3 parallel)   summarize/alpha       func · retry 3/infra
  │                        summarize/beta        func · retry 3/infra
  │                        summarize/gamma       func · retry 3/infra
  │
  └─ wave 3                digest                needs summarize/alpha, summarize/beta,
                                                 summarize/gamma · func

Three documents give three parallel nodes. Thirty would give thirty, from the same for loop, with no change to the code.

Step 4: run it

Run the pipeline binary directly, the way any Go program runs:

go run .

Or hand the package to the senro CLI, which builds it, execs it and attaches automatically. You get a terminal UI on a TTY and plain streaming lines anywhere else:

senro run .

seed             -> succeeded
summarize/beta   -> succeeded
summarize/alpha  -> succeeded
summarize/gamma  -> succeeded
digest           -> succeeded

- alpha: The billing reconciliation job now uses a queue worker for invoice settlement
  within two minutes, replacing the nightly cron.
- beta: The search index now rebuilds incrementally in under a second without blocking
  writes, replacing the old full rebuild and requiring operators to delete the
  `rebuild-search` cron job.
- gamma: Service gamma's public API now features per-tenant rate limiting set at 100
  requests per second, which may cause HTTP 429 errors for existing bursting integrations.

Three real model calls in parallel, each with its own log file, and a merge step that ran once they were all done.

While that is running, a second terminal can watch it, or steer it:

senro attach          # terminal UI
senro ui              # browser view, prints a one-time link
senro attach --run <id>   # reopen a finished run from disk

Every run also lands in runs/<id>/, so senro runs lists what happened and each step's stdout is a file you can read afterwards.

One thing to know when you switch between go run and senro run. A func step's cache key includes the binary digest, so the first run after changing how the binary is built re-runs every func step once, then caches normally from there. That is correct behaviour rather than a bug: the function's body lives in the binary, and senro cannot tell whether it changed without hashing it.

Step 5: stop paying for the same call twice

Each summarize step declares Pure(). Run the pipeline again with nothing changed:

seed             -> succeeded
summarize/beta   -> cached
summarize/alpha  -> cached
summarize/gamma  -> cached
digest           -> succeeded

cached is a real state, not a label. Those three steps did not execute: no model call, no tokens, no latency. Their recorded outputs were restored from the content-addressed store, so digest still read three summaries and produced the same result.

Now change one document out of three:

seed             -> succeeded
summarize/alpha  -> cached
summarize/beta   -> cached
summarize/gamma  -> succeeded
digest           -> succeeded

One document changed, one model call made. And senro will tell you which component of the key moved:

$ senro cache explain --run <id> summarize/gamma
MISS  summarize/gamma  key d3eb7450 (previous f68652ad)
  ✗ input_digests: docs/gamma.md  6750cb09 → ef0fdb64
  ✗ workspace_digests: doc-gamma  16774a4e → d43e13c8
  ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity,
    tool_versions, version unchanged

For a few dozen documents this is a convenience. For a nightly re-index over ten thousand where one file changed, it is the difference between a bill you notice and one you do not.

Note func_identity in that list. For a func step the cache key covers the binary digest, the registered name and the parameter digest. So changing your prompt, your model or the flow's code all correctly invalidate the cache. It is not keyed on the input file alone, which is the mistake a hand-rolled cache usually makes.

Give each call its own workspace

This is the one structural rule to follow when caching model calls.

Inputs feeds the input_digests component of the key, but Mount feeds workspace_digests, which is the digest of the entire mounted workspace. Narrow Inputs do not narrow it.

Mount one shared tree across a fan-out and every step's key depends on every document, so changing one document re-runs all of them and the cache buys you nothing. One workspace per unit of work, as in the loop above, keeps each key dependent on its own input.

Step 6: let a model explain a failure

senro has a second place a model belongs: explaining a failed step. contrib/genkitanalyzer is a ready-made failure analyzer built on Genkit:

go get github.com/xavidop/senro/contrib/genkitanalyzer

It is a nested module with its own go.mod, so senro itself never pulls in an AI SDK. If you never install it, your dependency graph never hears about Genkit.

g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}))

p := senro.New("analyze")
w := p.Workflow("build")
w.Step("migrate", exec.Command("sh", "-c",
    `echo "dial tcp 10.0.3.7:5432: connect: connection refused" >&2; exit 1`))

err := senro.Run(ctx, p,
    senro.WithAnalyzer(
        genkitanalyzer.New(g, genkitanalyzer.Model("googleai/gemini-2.5-flash")),
        senro.AnalyzerName("genkit"),
        senro.AnalyzeTimeout(20*time.Second)))

That is the whole integration. When the step fails, an analysis.proposed event lands in the run's ledger:

{
 "id": "migrate@1",
 "analyzer": "genkit",
 "duration_ns": 8344562458,
 "summary": "The 'migrate' step failed because its command explicitly reported a database
             connection refusal error and exited.",
 "detail": "... The error message \"dial tcp 10.0.3.7:5432: connect: connection refused\"
            typically indicates that a client attempted to establish a TCP connection to a
            server at 10.0.3.7 on port 5432, but the server actively rejected the connection.
            This could be due to the database server not running, not listening on the
            specified address/port, or being blocked by a firewall ..."
}

Because it is an event, the explanation shows up in the TUI, in events.jsonl, and in anything else folding the run.

What the combination gives you

Genkit gives you the model call: typed, traced, provider-agnostic, testable. senro gives you everything around it.

Concern	What senro already does for a step
Ordering	A dependency edge, not a place in a loop
Outcome	Ten states, so a call that passed on the second attempt is `recovered`, not silently a success
Retries	`retry.OnInfra()` retries a dropped connection, never a model answer you did not like
Logs	One log file per step per attempt, redacted
Cost	An action cache keyed on the prompt, parameters, model and binary
Observability	One append-only event stream you can attach to live or replay later
Failure analysis	An analyzer gated behind a human or an explicit policy

You can build every one of those yourself. It is never quite the same twice, and it is rarely tested well.

Conclusion

The pattern is small enough to summarise in four lines:

Define the flow with genkit.DefineFlow as you always would.
Register a func step with senro.RegisterFunc that calls flow.Run(ctx, ...).
Give each call its own workspace and declare Pure() with Inputs and Outputs.
Add senro.WithAnalyzer if you want failures explained.

From there the graph is a Go program, so the loop over your corpus is a for loop and the logic that decides what runs is testable with go test.

senro on GitHub
Genkit, and the Go documentation
The last article in this series: Debugging senro pipelines

Debugging senro Pipelines: Replay a Run, Shell Into a Live Step, Audit a Cache

Xavier Portilla Edo — Tue, 29 Sep 2026 13:39:35 +0000

Introduction

The usual way to debug a CI failure is to add echo "HERE" to a YAML file, push, wait four minutes, read forty thousand lines of log, and repeat until you understand something that would have taken thirty seconds at a shell prompt. The machine that produced the failure is gone, so there is nowhere else to look.

senro is built so that loop is not necessary. This article is the field guide: what to run when a pipeline goes wrong, in roughly the order you should run it. It follows the introduction and the monorepo guide.

Every output below comes from a real broken run.

The one idea underneath all of it

Every observable fact about a run is an event, appended in order, never rewritten. The live terminal UI, the browser UI, and reading the file a week later all build their view by folding the same list through the same function.

So there is no difference between watching a run and investigating one afterwards. It is the same protocol and the same client, reading a socket in one case and a file in the other. senro has no separate offline mode that could be worse than the online one.

Every run writes a directory:

runs/20260911T105325-4410b2c052/
├── events.jsonl      the append-only ledger, the source of truth
├── plan.json         the immutable graph that was executed
├── run.json          id, pipeline, start time
├── logs/<step>/<attempt>/{stdout,stderr}
├── ws/<workspace>    the workspaces this run used
└── work/<step>       each step's sandbox

Note logs/<step>/<attempt>/. Attempts get separate directories, so a step that failed twice and passed on the third try keeps all three sets of logs.

Reading the graph that ran

plan.json is the graph the engine executed, and it is plain JSON with no dependency beyond the standard library behind it. When a run did not do what you expected, the first question is often whether the plan was what you expected, and jq answers it in one line:

jq -r '.nodes[] | . as $n | (($n.needs // ["(root)"])[] | "\(.) -> \($n.id)")' runs/<id>/plan.json

(root) -> checkout
checkout -> vet
checkout -> test
vet -> build
test -> build

This is worth reaching for before anything else on a fan-out, where the plan is built from your repository rather than written by hand. A missing unit, a phantom unit, or an edge you did not mean to declare shows up here rather than three steps into a run.

1. Find the run

$ senro runs -n 6
RUN ID                      PIPELINE  STATUS     STARTED              DURATION
20260911T110526-57739ce41a  slow      running    2026-09-11 20:05:26  14s
20260911T110510-de69b88fe7  slow      cancelled  2026-09-11 20:05:10  9s
20260911T105408-1d19454e86  ship      failed     2026-09-11 19:54:08  0s
20260911T105325-4410b2c052  ci        failed     2026-09-11 19:53:25  1s
20260911T105314-bc1a4e9d3d  ci        succeeded  2026-09-11 19:53:14  1s
20260911T105259-8fc9b32e31  ci        succeeded  2026-09-11 19:52:59  1s

-n caps how many are printed. This answers "what ran" without already knowing a run ID.

2. Replay the run

senro attach --run 20260911T105325-4410b2c052 --ui=plain

checkout stderr | Cloning into '.'...
checkout stderr | done.
checkout succeeded
vet succeeded
test stdout | ?     senrolab/ci [no test files]
test stdout | --- FAIL: TestHello (0.00s)
test stdout |     greet_test.go:7: got "HELLO, senro"
test stdout | FAIL
test stdout | FAIL  senrolab/greet  0.376s
test failed
build skipped_upstream_failed
run failed

This is the same client that watches a live run, reading recorded events instead of a socket. On a TTY you get the full interactive terminal UI, with the same key bindings, on a run that finished yesterday. --follow tails a run from disk with no socket needed.

The last two lines carry most of the diagnosis. test failed is a step that ran and returned non-zero. build skipped_upstream_failed is a step that never ran because of it. Those are different states, so nobody has to read upwards through a log wondering whether build broke too.

Five of the ten states are the ones you meet while debugging:

State	What it tells you
`failed`	It ran, returned non-zero, retries are exhausted
`timed_out`	An attempt outlived the `Timeout` you declared
`panicked`	A Go function step panicked; the stack is in its stderr
`skipped_upstream_failed`	It never ran, something it needs broke
`recovered`	It passed, but not on the first attempt

recovered is not a failure and it is the most useful of the five. A run full of recovered steps is a passing run telling you your infrastructure is flaky, and it stays visible instead of being collapsed into "green".

If you only want the states, the ledger is one command away:

grep step.finished runs/<id>/events.jsonl | jq -r '"\(.step) -> \(.payload.state)"'

checkout -> succeeded
vet -> succeeded
test -> failed
build -> skipped_upstream_failed

3. Look at the files the step had

The log says what the step printed. Often you need what the step was looking at. Workspaces are snapshotted into a content-addressed store, so a failed run's directories survive it:

$ senro ws ls 20260911T105325-4410b2c052
src    sha256:f37f35faa76ae965f92e410a877b7c0c90da14cd926711fde5b0a2137be1c5cf  13 files  3.2 KiB

$ senro ws pull 20260911T105325-4410b2c052 src /tmp/broken-src
$ ls /tmp/broken-src
ci  cmd  go.mod  go.sum  greet  probe

That is the exact tree the failing step saw, on your laptop, from a run on a machine that may no longer exist.

ws pull prints what a snapshot does and does not preserve, which matters exactly once, when a permission looks wrong: modes are normalized to 0644 or 0755, mtimes are fixed at the epoch, and uid, gid, extended attributes, ACLs, hard links and devices are not stored at all. That normalization is what makes a snapshot's digest depend only on content, not on which machine produced it.

For "it worked yesterday", compare two runs:

$ senro ws diff <run-a> <run-b> src
+ added   - removed   M content changed   P mode changed   K kind changed

workspace "src"
  a  20260911T105314-bc1a4e9d3d  sha256:3c3503f85c99ced855d5564624e31f350608c5b029410f65811b5898bc54b81a
  b  20260911T105325-4410b2c052  sha256:f37f35faa76ae965f92e410a877b7c0c90da14cd926711fde5b0a2137be1c5cf
  - bin             dir
  - bin/app         0755  2.3 MiB  53d0659f
  M greet/greet.go  113 B -> 113 B  57930022 -> 6bd8b782
  0 added, 2 removed, 1 modified, 0 mode, 0 kind, 12 unchanged

greet/greet.go changed, and the binary is missing because the build never ran. ws ls and ws diff read stored indexes without downloading any file contents, so both are instant on a tree of any size. Use ws pull when a workspace's last state came from a cache hit, since a cache entry stores a body digest rather than an index.

4. Get inside a step that is still running

While a run is live, open a session on one of its steps:

$ senro shell --pid 36343 --step slow -- ls -la
senro shell: this session runs against pipes: no prompt, no line editing, no job control.
Type a command and press enter; ^D ends the session. Pass --tty for a real terminal.
total 16
drwxr-xr-x  4 xavierportillaedo  wheel  128 Sep 11 20:05 .
drwxr-xr-x  3 xavierportillaedo  wheel   96 Sep 11 20:05 ..
-rw-r--r--  1 xavierportillaedo  wheel    4 Sep 11 20:05 a.txt
-rw-r--r--  1 xavierportillaedo  wheel    4 Sep 11 20:05 b.txt

$ senro shell --pid 36343 --step slow -- cat a.txt
one

That is the step's own workspaces, read-only, at the paths the step sees them, on the step's own executor. Not a copy and not a reconstruction. If the step is running in a pod, the session is in that pod.

Pair it with a breakpoint to stop the run before a step and inspect exactly what it was about to run against. Add --tty for a real terminal with a prompt and line editing, which the local, container and Kubernetes executors host.

Two deliberate limits: no secrets are delivered into a session, and a finished run has no engine to host one, so use ws pull for a run that is over.

5. Attach to a live run you did not start watching

If your pipeline called attach.Listen, you can join a run in progress from any terminal:

$ senro attach --pid 36343 --ui=plain
prepare succeeded
slow stdout | tick 1
slow stdout | tick 2
slow stdout | tick 3
slow started
slow stdout | tick 4
slow stdout | tick 5

Attaching three seconds in replays what already happened, reports where the run currently is, then follows it live. The client does not need to have been there from the start.

From there, attach is not read-only. You can cancel, pause and resume the run, retry, skip or re-run from a step, set a breakpoint, and force a workspace snapshot. senro ui serves the same thing as a browser page on loopback with a one-time link, offering the same control operations except the shell.

6. Ask why the cache did what it did

Caching bugs are the worst class of build bug, because the symptom is that something did not happen. senro cache explain answers it component by component.

A hit:

$ senro cache explain --run <id> build
HIT  build  key a52a9b6c

A miss, after one file changed:

MISS  build  key 0e769472 (previous a52a9b6c)
  ✗ input_digests: ci/main.go  ee7563f3 → 4f7d4315
  ✗ input_digests: greet/greet.go  59d2873d → 57930022
  ✗ input_digests: probe/main.go  added
  ✗ workspace_digests: src  f377ce04 → de2069cc
  ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity,
    tool_versions, version unchanged

Which components changed, both digests for each, which path was added, and an explicit list of what stayed the same.

A surprising miss is almost always a component you forgot about, so it is worth knowing all eleven: command, env, secrets (identity, never values), executor_class, platform, input_digests, workspace_digests, mount_shape, step_shape, func_identity and version.

The one that catches people is workspace_digests: every mounted workspace enters the key in full. Narrow Inputs do not narrow it. A step that mounts a big shared tree misses on any change anywhere in that tree, however precisely its inputs were declared.

7. Check whether a step lied about being pure

Pure() is a promise only you can make. senro does not sandbox network access, so it cannot tell that a tool which also SSHes into production is impure. What it can do is audit the claim afterwards:

senro verify --recheck-pure --run <id> --rerun

It re-executes the run's cached pure steps in throwaway trees restored from the workspace content each step's own cache key records, and compares what they produce against what the cache stored. Nothing runs without --rerun, deliberately: the premise is that a purity claim may be false, so its safety corollary is not assumed either. No cache entry is ever written.

Pointed at a go build step:

NONDETERMINISTIC build  key a52a9b6ce84a  entry from run 20260911T105243-8b696eb1d0
  ✗ output     bin/app   cached cf1663a0  re-run 739ebb3d  again e86c2b3d
  ✗ workspace  src       cached 18160d77  re-run 07c2a015  again a7e507af
  declared inputs   glob:**/*.go, file:go.sum
  declared outputs  file:bin/app
  the two re-runs disagreed with each other as well as with the entry, so this step does not
  produce the same bytes twice from the same input; that is a reproducibility problem (an
  archive embedding a build timestamp looks exactly like this) and it is NOT evidence about
  purity either way

1 nondeterministic of 1 cached Pure() step(s)
1 step(s) cannot reproduce their own output twice, so their entries hold bytes a re-run would
not produce. That is worth fixing and it is not evidence about purity.

It ran the step twice, noticed the two re-runs disagreed with each other and not only with the cache, and declined to draw the wrong conclusion. The build is not byte-reproducible, which is real and worth fixing, and that is separate from whether the step reaches the network. A cruder tool would report "impure" and send you hunting a bug that is not there.

verify exits 0 whether or not it finds anything, because a finding is an answer rather than a failed run. Use --fail-on-mismatch in CI if you want the opposite.

8. Repeat the run instead of rediscovering it

senro rerun --run <id>

rerun re-executes the plan a previous run recorded, from plan.json rather than by rebuilding your pipeline. That is the distinction: it repeats a run instead of re-discovering one, so a fan-out that would find different units today still runs the graph that actually failed. Unchanged steps come from the action cache.

--step re-runs one step and everything below it. --regenerate asks generators for a fresh subgraph, and it is a separate verb because that run may do different work than the one it repeats.

Failures that are not a step's fault

Two categories look nothing alike and are worth recognising immediately.

Build-time refusals

senro.Run builds first, so a dangling Needs, a duplicate step id, an empty command or a plan that would leak a credential comes back before anything executes. The messages do the diagnosis:

plan: step "summarize/alpha" is a func step and sets WorkDir "/tree", which it never runs in:
a func step on the coordinator runs in the coordinator's own process, where the working
directory is process-global, so honouring it would move every step running alongside it.
Reach files through ctx.Workspace(name) [...]

plan: step "summarize/alpha" is Pure() with no Inputs, so its cache key would not change when
its sources do: declare them with Inputs(artifact.Glob(...))

engine: step "leak" puts the value of secret "RegistryToken" in command argument 1; a command
argument is visible in ps(1), in shell history and in auditd execve records, where senro cannot
redact it, so senro refuses to run rather than leak it. Deliver it as a file instead [...]

Each names what you wrote, why it cannot hold, and what to write instead. Read them rather than searching.

Exit code with no output

A step fails with exit code 127 and both stdout and stderr are zero bytes. Exit 127 normally means "command not found", but the real cause here is usually the working directory: the process never started, so it never wrote anything.

The common trigger is a WorkDir that is a subpath of a mount. On the local executor a mount path is a logical name, and senro translates WorkDir into the sandbox only when it matches a mount point exactly:

func CmdDir(workDir string, mounts []Mount) string {
    for _, mt := range mounts {
        if mt.At == workDir {
            return ""
        }
    }
    return workDir
}

So with a workspace mounted at /tree, WorkDir("/tree") works and WorkDir("/tree/services/api") is passed through as a literal host path that does not exist. Keep WorkDir on the mount point and let the tool do the moving:

exec.Command("go", "-C", u.Dir, "build", "-o", "/dev/null", "./...")

The general rule: an exit code with no output means the process never started, so check the working directory and the executable path before you look at the command itself.

Writes through a read-only mount

engine: step "build[unit=services/api]" wrote through its read-only mount of workspace "tree"
(c9271303 became 832bb638); a read-only mount that changes makes every cache key computed from
it wrong

That is go build dropping a compiled binary next to the source. On the container and Kubernetes executors the kernel refuses the write; on local and SSH, senro.RO is checked after the fact. The failure is the point, because a read-only mount that silently changed would poison every cache key computed from it.

Instrument it yourself

None of the above requires the CLI. The event stream is a public contract in github.com/xavidop/senro/api, which depends on nothing outside the standard library, and you can fold it in your own process:

senro.Run(ctx, p, senro.WithSink(mySink))

That is how the trace exporter works, and it is how you push run facts into whatever you already use. Anything the TUI can conclude, your code can conclude from the same events.

Conclusion

The checklist, in order:

senro runs to find the run.
senro attach --run <id> to replay it, and read the states, not just the log.
jq over plan.json when you need to check the graph that actually ran.
senro ws pull for the files it left behind, senro ws diff for what changed since the run that worked.
senro shell --step <id> if it is still running, ideally stopped on a breakpoint.
senro cache explain when the surprise is that something did not happen.
senro verify --recheck-pure --rerun when you suspect a cached result is wrong.
senro rerun to repeat the recorded plan rather than build a new one.

None of these require pushing a commit, and none require the machine that produced the failure to still exist.

senro is at github.com/xavidop/senro, MIT licensed. The other articles in this series:

Building a Real Pipeline with senro: Monorepos, Caching and Triggers

Xavier Portilla Edo — Tue, 29 Sep 2026 13:38:52 +0000

Introduction

In the introduction to senro I covered the model: a pipeline is a Go program, it builds an immutable plan, and the engine executes that plan while anything can attach to the event stream.

This article is the practical follow-up. It covers the five things you will actually write on your first real pipeline: getting your code into a step, caching a step so it never runs the same work twice, fanning one step out over every module in a monorepo, narrowing that fan-out to only what a change touches, and deciding whether a webhook should start a run at all.

Everything below was run against senro v1.4.0.

A pipeline that builds a real repository

Here is the shape of a Go CI pipeline in senro. Check the code out into a workspace, vet and test it in parallel, then build:

// Command ci checks the repo out into a workspace, then vets, tests and
// builds it.
package main

import (
    "context"
    "log"
    "os"

    "github.com/xavidop/senro"
    "github.com/xavidop/senro/artifact"
    "github.com/xavidop/senro/exec"
    "github.com/xavidop/senro/retry"
)

func main() {
    ctx := context.Background()

    repo := os.Getenv("REPO_URL")
    home := os.Getenv("HOME")

    p := senro.New("ci")

    // One directory for this run, mounted by every step that needs the code.
    src := senro.Workspace("src", senro.Scope(senro.ScopeRun))

    verify := p.Workflow("verify")

    verify.Step("checkout", exec.Command("git", "clone", "--depth", "1", repo, ".")).
        WorkDir("/src").
        Mount(src.At("/src", senro.RW)).
        Retry(3, retry.OnInfra())

    verify.Step("vet", exec.Command("go", "vet", "./...")).
        Needs("checkout").
        WorkDir("/src").
        Mount(src.At("/src", senro.RO)).
        Env("HOME", home)

    verify.Step("test", exec.Command("go", "test", "./...")).
        Needs("checkout").
        WorkDir("/src").
        Mount(src.At("/src", senro.RO)).
        Env("HOME", home)

    verify.Step("build", exec.Command("go", "build", "-o", "bin/app", "./cmd/app")).
        Needs("vet", "test").
        WorkDir("/src").
        Mount(src.At("/src", senro.RW)).
        Env("HOME", home).
        Pure().
        Inputs(artifact.Glob("**/*.go"), artifact.File("go.sum")).
        Outputs(artifact.File("bin/app")).
        Retry(3, retry.OnInfra())

    if err := senro.Run(ctx, p); err != nil {
        log.Print(err)
        os.Exit(1)
    }
}

test stdout | ?     senrolab/ci [no test files]
test stdout | ?     senrolab/cmd/app    [no test files]
test stdout | ok    senrolab/greet  0.360s

The graph those Needs calls describe, rendered from the run's own plan.json:

pipeline: ci
  │
  ├─ wave 1                checkout          retry 3/infra
  │
  ├─ wave 2 (2 parallel)   test
  │                        vet
  │
  └─ wave 3                build             needs vet, test · retry 3/infra

Two things in there are not obvious coming from a CI system that hands you a checkout and a shell.

Workspaces: how code reaches a step

Every step runs in an isolated, empty working directory. Nothing hands you a checkout, because "hand the step a checkout" means four different things across a local process, a container, a pod and an SSH host.

A Workspace is a named, versioned directory that steps mount. senro snapshots it into a content-addressed store when a step that mounts it finishes, and restores it for the next step that needs it. So the code gets into the run the way it gets into any CI runner: the checkout step clones it, and the src workspace carries the result to the other three.

Mount modes matter:

Mode	Use it when
`senro.RW`	The step writes into the workspace, including its declared `Outputs`
`senro.RO`	The step only reads

In the pipeline above, checkout is RW because it writes the tree, vet and test are RO because they only read it, and build is RW because its output lands there.

ws.At(path, mode) always needs a mode, Mount accumulates so one step can mount several things, and two mounts at one path are refused at build time.

Workspaces come in three scopes:

Scope	What you get
`senro.ScopeRun`	The default. One directory for this run, shared by every step that mounts it
`senro.ScopePersistent`	One directory on this machine that outlives the run. Requires explicit bounds
`senro.ScopeStep`	One fresh directory per step, shared with nobody

Declaring the environment

Build() adds nothing to a step's environment. Not a HOME, not a GOPATH, nothing. The local executor supplies its own PATH and that is all.

This is deliberate: two developers on the same commit produce the same plan because nothing about their shells leaks into it, and the environment is one of the eleven components a cache key is computed from. In practice it means tools that need something get told about it:

.Env("HOME", home)

Env takes one pair per call, and it is not variadic on purpose: Env("A", "1", "B") would compile and mean nothing.

Caching a step with `Pure()`

Pure() is a promise: given these inputs, this command produces these outputs, and nothing else matters. senro hashes the inputs, the command, the environment, the workspaces, the mount shape and six other things into a key. On a second run with the same key the step is not executed at all and its recorded outputs are restored.

Running the pipeline above twice:

RUN ID                      PIPELINE  STATUS     STARTED              DURATION
20260911T105259-8fc9b32e31  ci        succeeded  2026-09-11 19:52:59  1s
20260911T105243-8b696eb1d0  ci        succeeded  2026-09-11 19:52:43  2s

$ senro cache explain --run 20260911T105259-8fc9b32e31 build
HIT  build  key a52a9b6c

The event stream records it as a state, not a label:

{"type":"cache.hit","step":"build",
 "payload":{"key":"sha256:a52a9b6c...","from_run":"20260911T105243-8b696eb1d0"}}
{"type":"step.finished","step":"build","payload":{"state":"cached","cached":true}}

Touch a source file and the same command tells you exactly why it has to run again:

MISS  build  key 0e769472 (previous a52a9b6c)
  ✗ input_digests: ci/main.go  ee7563f3 → 4f7d4315
  ✗ input_digests: greet/greet.go  59d2873d → 57930022
  ✗ input_digests: probe/main.go  added
  ✗ workspace_digests: src  f377ce04 → de2069cc
  ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity,
    tool_versions, version unchanged

One rule about the key is worth memorising, because it decides how you structure a fan-out:

Mount puts the whole mounted workspace into the cache key. Inputs narrows what you declare as sources, but it does not narrow the workspace component. A step that mounts a big shared tree has a key that depends on that entire tree.

The practical consequence: when you cache expensive per-unit work, give each unit its own workspace instead of sharing one.

Read-only mounts are enforced

A step that writes through an RO mount fails, and says so:

engine: step "build[unit=services/api]" wrote through its read-only mount of workspace "tree"
(c9271303 became 832bb638); a read-only mount that changes makes every cache key computed from
it wrong

That one is go build dropping a compiled binary next to the source. The fix is go build -o /dev/null ./..., which is what a build check wanted anyway.

How the rule is enforced depends on the executor, and it is worth knowing which you are relying on:

Executor	What `senro.RO` does	When a write is caught
Container	A real read-only bind mount	At the write. It fails
Kubernetes	`readOnly` on the volume mount	At the write. It fails
Local	Nothing at mount time	Right afterwards, when the workspace is found changed
SSH	Nothing on the far side	On read-back. The write is reported, not carried home

On local and SSH, read-only is a request senro checks after the fact rather than a rule the kernel enforces. Keep sensitive input out of any workspace a step could overwrite by mistake.

Monorepos: fan out with `Expand`

Expand(id, graph) adds one step per unit that a graph discovers. Add a module, get a step, and nobody has to write it.

Here is a Go workspace with four modules and a dependency chain three deep:

libs/log  <-  libs/config  <-  services/api
libs/log  <-  services/worker

verify.Expand("build", gowork.Modules()).
    MaxParallel(4).
    Needs("checkout").
    Affected(src).
    Template(func(u senro.Unit) *senro.StepBuilder {
        // WorkDir is the mount point itself, and go -C moves into the unit.
        return senro.NewStep(exec.Command("go", "-C", u.Dir, "build", "-o", "/dev/null", "./...")).
            WorkDir("/tree").
            Mount(tree.At("/tree", senro.RO)).
            Env("HOME", home)
    })

checkout                     -> succeeded
build[unit=libs/log]         -> succeeded
build[unit=libs/config]      -> succeeded
build[unit=services/worker]  -> succeeded
build[unit=services/api]     -> succeeded

One Expand call, four nodes in the plan:

pipeline: mono (no change filter)
  │
  ├─ wave 1                checkout
  │
  └─ wave 2 (4 parallel)   build[unit=libs/config]
                           build[unit=libs/log]
                           build[unit=services/api]
                           build[unit=services/worker]

Template is called once per unit in sorted order and must return a fresh *senro.StepBuilder, built with senro.NewStep. Each unit gives you:

On `Unit`	What it is
`u.ID`, `u.Name`	The unit's stable identity
`u.Dir`	The unit's directory, relative to the root
`u.Base()`	The last path segment: `"web"` for `"apps/web"`
`u.Sources()`	Every file under the unit, ready for `.Inputs(...)` on a `Pure()` template

Child ids come from the expansion id and the unit (build[unit=libs/log]), never a name the template picks, so the same repository always builds the same graph and a re-run reconstitutes the same children.

Two bounds are worth setting. MaxParallel(n) limits how many children of this expansion run at once. MaxNodes(n) rejects an expansion wider than n at build time, default 500, so a scheduler never discovers mid-run that it has forty thousand sandboxes to hold open.

Expansion happens once, at Build(). Every unit is discovered and written into the plan before the first step starts.

The eight unit graphs

Graph	Discovers
`glob.Dirs` / `glob.Files`	One unit per matching directory, or per directory containing a matching file
`gowork`	Go workspace modules
`cargo`	Rust workspace crates
`jswork`	npm, pnpm or yarn workspaces
`maven`, `gradle`	JVM modules
`pyproject`	Python projects
`bazel`	Bazel packages

glob just matches paths. The others read the ecosystem's own manifests, which is what makes the next section work.

Running only what a change affects

Affected(src) narrows the fan-out to the units a change reaches: the ones owning a changed file, and everything that depends on them, at any depth.

The same pipeline, told that one file changed:

=== changed: services/api/main.go ===
   build[unit=services/api]

=== changed: libs/config/config.go ===
   build[unit=libs/config]
   build[unit=services/api]

=== changed: libs/log/log.go ===
   build[unit=libs/config]
   build[unit=libs/log]
   build[unit=services/api]
   build[unit=services/worker]

The third case is the point. services/api does not import libs/log. It imports libs/config, which does. That transitive hop is why reading manifests beats globbing paths, and it is why Affected over a glob graph is refused at build time: a path does not know what imports it, and senro would rather say so than quietly run everything.

Unaffected units are not skipped, they are not in the plan at all. That is visible in the plan itself. Same pipeline, same Expand, told that libs/config/config.go changed:

pipeline: mono (libs/config changed)
  │
  ├─ wave 1                checkout
  │
  └─ wave 2 (2 parallel)   build[unit=libs/config]
                           build[unit=services/api]

Two nodes instead of four, decided at build time. libs/log and services/worker are not present to be skipped, so a re-run of this recorded plan reconstitutes exactly these two.

The change source is pluggable:

Source	What it is
`change.FromTrigger(ev)`	What the webhook that started the run recorded
`change.Paths("a/x.go", ...)`	A literal list, for your own logic and for tests
`change.Everything()`	Every unit runs
`change.Ignoring(src, "docs/**")`	The above, minus paths you do not care about

change.Paths() with no arguments means nothing changed, not everything. Use change.Everything() for that. They are different answers.

Keep the run directory out of the tree

senro writes each run to runs/<id>/ under the working directory. That directory holds the run's workspaces, and a workspace usually holds a checkout of the repository. Unit graphs walk the filesystem for manifests, pruning only .git and node_modules, so a second run will discover the first run's checkout as a second copy of every module:

build[unit=runs/20260911T105558-08e7daff14/ws/tree/services/api]

Pin the run directory somewhere outside the tree you fan out over:

senro.Run(ctx, p, senro.WithDir("/var/lib/senro/runs/"+runID))

Adding runs/ to .gitignore does not help, because the unit graph is walking the filesystem, not asking git.

Triggers: the pipeline decides whether to run

Instead of a CI service deciding whether to invoke your pipeline, the pipeline binary is its own matcher. Hand it a webhook delivery and it decides whether the event is its business:

err = senro.Run(ctx, pipeline(ev),
    senro.WithTrigger(ev,
        // A push to main reports mode "all", which change.FromTrigger passes
        // straight through: everything builds there.
        trigger.OnPush(trigger.Branches("main")),
        // Everything else is narrowed to what it touched.
        trigger.OnPush(),
        trigger.OnPullRequest(trigger.Actions("opened", "synchronize")),
    ))

A push to a topic branch that touched services/api/main.go runs one step. A tag push, which this pipeline does not want, runs nothing and says why:

$ go run ./examples/monorepo --trigger-event /tmp/ev-tag.json
senro: trigger: no trigger matched the event: tag refs/tags/v1.2.3
  push(branches=[main]): only answers push events, this was tag
  push: only answers push events, this was tag
  pull_request(actions=[opened synchronize]): only answers pull_request events, this was tag
exit status 78

Exit code 78 is EX_CONFIG, the conventional "this was not my business". One line per matcher explaining why it declined, which is a better answer than a GitHub Actions on: block that silently did not fire.

GitHub, GitLab, Bitbucket and Gitea ship, plus your own provider. And the event file is optional: trigger.FromRequest verifies and parses a delivery straight off the wire, per source, so the pipeline binary can be the HTTP endpoint itself.

Because the matcher is a binary, you can test a trigger decision locally against a saved payload before it ever reaches a runner.

Handlers: cleanup and evidence

OnFailure runs its handlers once retries are exhausted and the step still failed. Always runs whatever the outcome. A handler can be a Go function, and it knows what it is cleaning up after:

func Collect(ctx senro.Ctx, _ ReportParams) error {
    f, ok := ctx.Failure()
    if !ok {
        return errors.New("ship/collect only makes sense as a handler")
    }
    fmt.Fprintf(ctx.Stdout(), "%s ended %s (exit %d) on attempt %d\n",
        f.Step, f.State, f.ExitCode, f.Attempt)
    fmt.Fprintf(ctx.Stdout(), "log tail: %q\n", f.LogTail)
    return nil
}

// in the pipeline:
build.Step("package", exec.Command("sh", "-c", "echo packaging; exit 1")).
    OnFailure(senro.Handler("collect", senro.Func("ship/collect", ReportParams{}))).
    ContinueOnError()

package ended failed (exit 1) on attempt 1
log tail: "packaging\n"

The log tail arrives with the failure, so a handler that classifies errors never has to open a file. Handlers run in the order listed, and a handler that fails never masks the step's own failure: the recorded cause stays the original step.

Use senro.Handler(id, action) rather than a workflow's Step. Passing a real step to OnFailure is rejected at build time, because that step would otherwise run twice. A handler also gets its parent's workspaces read-only at the same paths automatically, so Mount on one is refused.

Conclusion

Put the pieces together and the argument is not about syntax.

The fan-out is a function called once per unit, so what your build does is testable with go test. The Affected graph reads your ecosystem's manifests, so a change to a shared library rebuilds the things that transitively depend on it and nothing else. The cache tells you which of eleven key components changed when it misses. The trigger decision happens in a binary you can run locally against a saved payload. And the whole thing is a Go module with a go.sum, reviewed like the rest of your code.

None of that is impossible in YAML. It is that in YAML each one is a plugin, a marketplace action, a service feature or four hundred lines of bash, and here they are library calls in a language with a compiler.

Next in this series:

Genkit flows as senro steps: AI pipelines where every model call is a graph node with its own retry, log and cache entry.
Debugging senro pipelines: replaying a finished run, shelling into a live step, and auditing a cache.

senro lives at github.com/xavidop/senro.

Introducing senro: A Pipeline Engine You Program in Go

Xavier Portilla Edo — Tue, 29 Sep 2026 13:38:43 +0000

Introduction

Every CI configuration I have ever maintained follows the same arc. It starts as twenty lines of YAML. Then a step needs to run only on main, so there is an if. Then two steps run in parallel and a third needs both, so there is a needs block. Then a flaky registry breaks the build weekly, so there is a retry, which also retries the failing test, which is exactly wrong. Then someone needs a loop over six services, the templating language does not have one, and the whole thing collapses into a single run: block holding four hundred lines of bash.

At that point the CI configuration is not configuration. It is a program, written in a language with no functions, no types, no tests and no debugger, executed by a service that can only tell you whether it went green.

senro (線路, Japanese for railway track) is my answer to that. It is an open-source pipeline engine you program instead of configure: your Go code builds an immutable graph of workflows and steps, senro resolves it into a plan, executes it, and exposes one append-only event stream that a terminal UI, a browser page or a script can attach to, live, to watch and steer the run.

CI/CD is the most familiar thing to build with it, but not the boundary. A data pipeline, a batch job, an infrastructure rollout or a release script is built the same way: steps with dependencies, retries, failure handlers and one event stream.

This article is the tour. Every command and every output in it was run against senro v1.4.0 on Go 1.27.1.

The premise

CI systems ask you to describe real programming in a data format. So the logic ends up in shell scripts the YAML merely invokes, and you lose everything a language gives you at exactly the layer that decides whether your build ships.

senro starts from the other end: the pipeline is the Go program. For the thing that decides whether your code ships, you get the compiler, types, go test, and a debugger.

That has practical consequences before it has philosophical ones. A loop over six services is a for loop. Shared configuration between two pipelines is a function. A tricky bit of branching logic is a function with a unit test. None of those need a plugin, a marketplace action or a templating dialect.

Install

The CLI:

brew install xavidop/tap/senro
# or
go install github.com/xavidop/senro/cmd/senro@latest

The library, which is all a pipeline actually needs:

go get github.com/xavidop/senro

Heads up: senro requires Go 1.26 or newer, on Linux or macOS. Windows is deliberately unsupported: attach's security boundary is a kernel peer-credential check with no Windows equivalent implemented, and senro fails closed rather than advertising a feature it cannot secure.

Released binaries for linux and darwin on amd64 and arm64, with checksums, SBOMs and SLSA provenance, are on the releases page.

Getting started

A pipeline is a Go package with a main that builds a senro.Pipeline and calls senro.Run. This one is a diamond: one step, then two in parallel, then one that waits for both.

// Command first is the smallest senro pipeline that says something.
package main

import (
    "context"
    "log"
    "os"

    "github.com/xavidop/senro"
    "github.com/xavidop/senro/exec"
    "github.com/xavidop/senro/retry"
)

func main() {
    ctx := context.Background()

    p := senro.New("first")

    ci := p.Workflow("ci")
    ci.Step("fetch", exec.Command("echo", "fetching dependencies"))
    ci.Step("lint", exec.Command("echo", "linting")).Needs("fetch")
    ci.Step("test", exec.Command("echo", "testing")).Needs("fetch")
    ci.Step("build", exec.Command("echo", "building")).
        Needs("lint", "test").
        Retry(3, retry.OnInfra())

    if err := senro.Run(ctx, p); err != nil {
        log.Print(err)
        os.Exit(1)
    }
}

Run it with the CLI, which builds the package, execs it and attaches automatically. On a TTY you get a terminal UI; anywhere else you get plain lines:

senro run ./first

fetch stdout | fetching dependencies
fetch succeeded
lint stdout | linting
lint succeeded
test stdout | testing
test succeeded
build stdout | building
build succeeded
run succeeded

Those four Step calls describe a graph, and the graph is what the engine actually runs. Rendered from the run's own plan.json:

pipeline: first
  │
  ├─ wave 1                fetch
  │
  ├─ wave 2 (2 parallel)   lint
  │                        test
  │
  └─ wave 3                build             needs lint, test · retry 3/infra

lint and test run at the same time because nothing orders them against each other, and build waits for both because it said so. You never wrote a scheduler.

The vocabulary is five words, and the rest of the API is a detail of one of them:

Term	What it is
`Pipeline`	A name and a set of workflows: `senro.New("ci")`
`Workflow`	A named group of steps: `p.Workflow("ci")`
`Step`	One action: a command, or a registered Go function
`Plan`	The validated, immutable graph that `Build()` resolves a pipeline into
Event stream	The append-only record of everything a run did

Step ids are unique across the whole pipeline, not per workflow, because a plan is flat.

That Retry(3, retry.OnInfra()) is worth a sentence on its own. It allows up to three attempts, but only when the environment failed the step: a dropped connection, a killed process, a registry hiccup. It never retries a command that ran and returned non-zero. Retrying a dropped SSH connection is sane. Retrying a failing test until it passes deletes what it just told you.

Definition, plan, execution

This is the one concept worth internalising, because everything else follows from it.

Your code does not execute anything. It builds a description. Build() resolves that description into an immutable, validated Plan, and the engine executes the plan, never your code:

plan, err := p.Build() // a validated snapshot
if err != nil {
    log.Fatal(err)
}
fmt.Println("about to run", plan.Digest())
if err := senro.RunPlan(ctx, plan); err != nil {
    log.Fatal(err)
}

senro.Run does the Build() for you. Because it builds first, mistakes surface before anything executes: a dangling Needs, a duplicate step id or an empty command comes back as an error, not as a half-finished run that already deployed something.

The refusals are specific enough to act on directly. A func step that declares a working directory it can never honour:

plan: step "summarize/alpha" is a func step and sets WorkDir "/tree", which it never runs in:
a func step on the coordinator runs in the coordinator's own process, where the working directory
is process-global, so honouring it would move every step running alongside it. Reach files through
ctx.Workspace(name), which hands a function the same path a mount gives a command, on every executor

A step that opts into caching without saying what it reads:

plan: step "summarize/alpha" is Pure() with no Inputs, so its cache key would not change when
its sources do: declare them with Inputs(artifact.Glob(...))

Each one names what you wrote, why it cannot hold, and what to write instead.

The separation is also what lets a second process watch a run. An attached terminal reads the same Plan the engine is executing, which it could never reconstruct by watching your function calls go by.

One event stream, and nothing else

Every observable fact about a run is an api.Event, appended in order, never rewritten. Real lines from a runs/<id>/events.jsonl, trimmed for width:

{"seq":21,"type":"cache.hit","run":"20260911T105259-8fc9b32e31","step":"build",
 "payload":{"key":"sha256:a52a9b6c...","from_run":"20260911T105243-8b696eb1d0"}}
{"seq":22,"type":"ws.restored","step":"build","payload":{"name":"src","digest":"sha256:18160d77..."}}
{"seq":23,"type":"step.finished","step":"build","attempt":1,
 "payload":{"state":"cached","duration_ns":28378625,"cached":true}}

The live terminal UI, the browser UI, and reading that file a week later all build their view by folding the same list through one function. Three things follow:

If the UI shows a step as cached, event 23 arrived. There is no other way for it to know.
Replaying the file offline gives you the screen the TUI showed at the time.
Your own code can fold the same events with senro.WithSink and reach the same conclusions.

There is no "the UI is out of date" state in senro, because there is nothing for it to be out of date with.

Attaching to a run

A pipeline can open a socket that a second process attaches to while it runs. One call, and one option:

att, err := attach.Listen(ctx, attach.Options{Bind: attach.AutoUnixSocket})
if err != nil {
    log.Fatal(err)
}
defer att.Close()

log.Printf("attach with: senro attach --pid %d", os.Getpid())

if err := senro.Run(ctx, p, senro.WithAttach(att)); err != nil {
    os.Exit(1)
}

Attaching to a pipeline three seconds into a ten second step, from a second terminal:

senro attach --pid 36343 --ui=plain

prepare succeeded
slow stdout | tick 1
slow stdout | tick 2
slow stdout | tick 3
slow started
slow stdout | tick 4
slow stdout | tick 5

The client replays what already happened, reports where the run currently is, then follows it live. It does not need to have been there from the beginning, because the event stream is the record and the client just folds it.

Three renderers read that same stream:

Command	What you get
`senro attach`	A terminal UI, or plain streaming lines with `--ui=plain`
`senro ui`	A browser view on loopback, a Go client compiled to WebAssembly, with a one-time link
`senro attach --run <id>`	A finished run, reopened from disk over the same protocol

And attach is not read-only. You can cancel, pause and resume a run, retry, skip and re-run from a step, set breakpoints, and open an interactive shell inside a live step's sandbox.

If attach.Listen is not in your main, senro.Run costs exactly what the engine costs: no attach server, no extra goroutine.

Ten states, not a boolean

A step ends in exactly one of ten states, because "did it pass" hides information a build system should be surfacing.

State	How a step ends there
`succeeded`	Passed without ever failing
`recovered`	Failed at least once, then passed on retry
`cached`	A `Pure()` step hit the action cache and was skipped entirely
`failed`	Ran, failed, exhausted its retries
`timed_out`	An attempt outlived the step's `Timeout`
`cancelled`	The run was cancelled before it could finish
`panicked`	A Go function step panicked; the panic was caught, not fatal
`skipped_upstream_failed`	Something it depends on failed
`skipped_condition`	Its `When` condition was not met at run start
`skipped_manual`	An operator took it out of a live run

Two distinctions do most of the work.

recovered is not succeeded. A build that needed three attempts and one that needed one are different facts. Collapsing them is how flaky infrastructure stays invisible for months. A run full of recovered steps is a passing run that is telling you something.

skipped_condition is not skipped_upstream_failed. Gate a workflow to a branch:

deploy := p.Workflow("deploy", senro.When(senro.Branch("main")))
deploy.Step("apply", exec.Command("echo", "deploying"))

On a feature branch:

apply    -> skipped_condition
report   -> succeeded
package  -> failed
sign     -> succeeded
RUN      -> failed

apply is recorded as gated off, not as broken. Had nothing else failed, the run would have ended succeeded: the deploy was not supposed to fire on a feature branch, and it did not. Your pull request stays green without a special case.

That run ended failed because package genuinely failed. sign still ran and succeeded, because package declared ContinueOnError(). ContinueOnError is about surviving a failure, and it deliberately does not rescue a skipped_condition step, which produced nothing to run against.

Two kinds of step

A command is one kind. The other is a registered Go function:

type ReportParams struct {
    Env string `json:"env"`
}

func init() { senro.RegisterFunc("ship/report", Report) }

func Report(ctx senro.Ctx, p ReportParams) error {
    fmt.Fprintf(ctx.Stdout(), "run %s, step %s, attempt %d, env %s\n",
        ctx.RunID(), ctx.StepID(), ctx.Attempt(), p.Env)
    return nil
}

// in the pipeline:
build.Step("report", senro.Func("ship/report", ReportParams{Env: "staging"}))

run 20260911T105408-1d19454e86, step report, attempt 1, env staging

Both kinds are built, scheduled, retried, cached and handled by exactly the same code. A function receives a senro.Ctx in place of a working directory and an argv:

Method	What it gives you
`ctx.Workspace(name)`	A mounted workspace's path, the same one a command's mount resolves to
`ctx.Secret(name)`	A delivered secret's file path, never the value
`ctx.Stdout()`, `ctx.Stderr()`	The step's real log streams, redacted and recorded
`ctx.RunID()`, `ctx.StepID()`, `ctx.Attempt()`	This invocation's identity in the event stream
`ctx.Failure()`	What this function is cleaning up after, when it runs as a handler

ctx.Attempt() is what an idempotency key needs to know before you retry against a remote API. And senro.Ctx embeds context.Context, so it passes straight into any library call that takes one.

A panic inside a func step is caught and reported as the state panicked rather than taking the run down. Panics are not retried.

Where a step runs

The executor belongs to the workflow, chosen with senro.On(...). There are four:

Target	Package	Runs every step of the workflow
`senro.Local()`	built in, the default	as processes on the coordinator's own machine
`container.Image(ref)`	`executor/container`	in a container on a local Docker daemon
`k8s.Pod(ref, k8s.Namespace(ns))`	`executor/k8s`	as a pod in a Kubernetes cluster
`ssh.Host(dest)`	`executor/ssh`	as a process on a remote machine, via your own `~/.ssh/config`

A step behaves the same wherever it lands, because retries, Timeout, ContinueOnError, the state taxonomy, handlers, workspace snapshots, the action cache, secret delivery and log redaction all live above the executor layer. No target re-implements them, so none of them can differ between targets.

Func steps work on all four. A Go function's body only exists inside your compiled binary, and a plan is JSON, so running one elsewhere means moving the binary rather than the plan. senro stages a copy of your pipeline binary on the target and re-enters it there, and ctx.Workspace(...) returns paths on the target, so the same function body works either way. Run senro func check to find out whether your module cross-compiles before a run tells you on a Friday.

If your build does not include a given executor, targeting it is rejected at Build(). It never quietly falls back to running locally.

Secrets are files, not strings

Credentials are a typed struct resolved by mamori, handed to senro.WithSecrets. A step asks for one by field name:

type Config struct {
    RegistryToken secret.String `source:"env:NPM_TOKEN"`
}

cfg, err := mamori.Load[Config](ctx)

w.Step("use", exec.Command("sh", "-c",
    `echo "the variable holds: $NPM_TOKEN"; echo "the file holds: $(cat "$NPM_TOKEN")"`)).
    SecretEnv("NPM_TOKEN", "RegistryToken")

senro.Run(ctx, p, senro.WithSecrets(cfg))

With NPM_TOKEN=super-secret-value, that step's log:

the variable holds: /var/folders/1p/ppx8j26s6jjghmfp1nv9p1zc0000gn/T/senro-secret-880958012/RegistryToken
the file holds: [REDACTED]

The environment variable holds a path, not the token, because argv is world-readable in /proc and environment values leak into crash dumps and child processes. A file has an owner and a mode, and it disappears with the sandbox. And when the step deliberately printed the file's contents, the log shows [REDACTED], because every stream is redacted on the way out.

What senro refuses is more interesting than what it redacts. A plan that would route a resolved value through argv, an environment value, WorkDir, Inputs, Outputs or a mount is rejected before the run starts:

w.Step("leak", exec.Command("echo", cfg.RegistryToken.Reveal()))

senro: engine: step "leak" puts the value of secret "RegistryToken" in command argument 1;
a command argument is visible in ps(1), in shell history and in auditd execve records, where
senro cannot redact it, so senro refuses to run rather than leak it. Deliver it as a file
instead: SecretEnv("VAR", "RegistryToken"), then read "$VAR" as a path in the step

Nothing ran. Inside a func step the same credential arrives through ctx.Secret("RegistryToken"), which is also a path.

Webhooks and triggers: the binary decides whether to run

In most CI systems, the decision to start a build belongs to the service, and you describe it in an on: block you cannot run locally. In senro that decision belongs to your program: the pipeline binary is its own webhook matcher.

Hand it a delivery and it decides whether the event is its business:

err = senro.Run(ctx, pipeline(ev),
    senro.WithTrigger(ev,
        trigger.OnPush(trigger.Branches("main")),
        trigger.OnPush(),
        trigger.OnPullRequest(trigger.Actions("opened", "synchronize")),
    ))

A push to a topic branch matches and the pipeline runs. A tag push does not, and nothing runs:

$ go run ./examples/monorepo --trigger-event /tmp/ev-tag.json
senro: trigger: no trigger matched the event: tag refs/tags/v1.2.3
  push(branches=[main]): only answers push events, this was tag
  push: only answers push events, this was tag
  pull_request(actions=[opened synchronize]): only answers pull_request events, this was tag
exit status 78

Exit code 78 is EX_CONFIG, the conventional "this was not my business", and you get one line per matcher saying why it declined. That beats staring at a YAML on: block that silently did not fire, and because it is a binary you can test the decision on your laptop against a saved payload before it ever reaches a runner.

GitHub, GitLab, Bitbucket and Gitea ship, plus your own provider through a small interface.

The event is also more than a yes or no. It carries what changed, so change.FromTrigger(ev) feeds a monorepo fan-out that runs only the units the push actually reaches. That is the subject of the next article.

Or be the endpoint yourself

You do not need an event file, or a CI service in front of you at all. trigger.FromRequest works out which source sent the delivery, verifies its signature, and parses it into the same event a file would have produced. One line changes:

ev, err := trigger.FromRequest(r, trigger.Secret(hookSecret))  // instead of LoadEvent(path)

So the pipeline binary can be the endpoint GitHub posts to:

func (s *server) webhook(w http.ResponseWriter, r *http.Request) {
    ev, err := trigger.FromRequest(r, trigger.Secret(s.secret))
    if err != nil {
        httpError(w, err)
        return
    }

    w.WriteHeader(http.StatusAccepted)

    // Not r.Context(): it is cancelled when the response is written.
    go senro.Run(context.Background(), pipeline(),
        senro.WithTrigger(ev, trigger.OnPush(trigger.Branches("main"))))
}

Two things worth copying from that: reply 202 before the run finishes, because a webhook sender times out in seconds and a pipeline takes minutes, and never hand r.Context() to senro.Run, because it is cancelled the moment the response is written. trigger.Secret covers all four providers, which authenticate differently. See Run it as a server.

Two caches, for two different jobs

Both are opt-in, and neither is on unless you ask.

The action cache: skip a step entirely

Mark a step Pure() and declare its inputs. senro hashes those inputs plus the command plus the environment into a key. On a second run with the same key the step is not executed at all and its recorded outputs are restored:

verify.Step("build", exec.Command("go", "build", "-o", "bin/app", "./cmd/app")).
    Pure().
    Inputs(artifact.Glob("**/*.go"), artifact.File("go.sum")).
    Outputs(artifact.File("bin/app"))

$ senro cache explain --run <id> build
HIT  build  key a52a9b6c

Change one file and it rebuilds, and the reason is a command rather than a guess:

MISS  build  key 0e769472 (previous a52a9b6c)
  ✗ input_digests: greet/greet.go  59d2873d → 57930022
  ✗ workspace_digests: src  f377ce04 → de2069cc
  ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity,
    tool_versions, version unchanged

Which of the eleven key components changed, both sides of each digest, and an explicit list of what did not.

Nothing is cached by default, because Pure() is a promise only you can make. A tool that also SSHes into production is not pure and senro cannot tell. What it can do is audit the claim afterwards with senro verify --recheck-pure.

The scratch cache: start warm, never be wrong

The other job is the mutable directories tools keep for their own speed: ~/.npm, ~/.cargo, a Go module cache. You want yesterday's copy if there is one and you do not care if there is not:

gomod := senro.ScratchCache("gomod",
    senro.Key(`gomod-{{ hashFiles "go.sum" }}`),
    senro.RestoreKeys("gomod-"))

verify.Step("test", exec.Command("go", "test", "./...")).
    Mount(gomod.At("/root/go/pkg/mod"))

A scratch cache is restored best-effort by key and never enters a cache key. RestoreKeys is the fallback, so a go.sum change misses the exact key but still starts from the last module cache rather than from nothing. A miss costs time and nothing else, and a stale entry cannot make a build produce the wrong answer, because nothing downstream is keyed on it.

One store, so caches are shareable

Artifacts, workspaces, cached results and staged binaries all live in one content-addressed store, normalized so the same bytes hash the same way on any machine. Point SENRO_REMOTE_CACHE at an S3-compatible bucket or an OCI registry and a fresh CI runner starts warm on what another machine already built. An unreachable store degrades the run to local disk, and never fails the run.

Three rules to know before your first pipeline

senro's execution model differs from CI systems that hand you a checkout and a shell. Three consequences are worth knowing up front rather than discovering.

1. A step does not start in your repository. Every step runs in an isolated, empty working directory. Code reaches a step through a workspace that an earlier step populates, usually with a git clone. That is what makes a step mean the same thing in a pod as on your laptop.

2. A step gets exactly the environment you declared. Build() adds nothing, not even a PATH; the local executor supplies its own and that is all. Two developers on the same commit therefore produce the same plan, since nothing about their shells leaks into it, and the environment is one of the components a cache key is computed from. Tools that need HOME, GOMODCACHE or anything else get it from .Env(...).

3. On the local executor a mount path is a logical name. WorkDir is translated into the sandbox only when it matches a mount point exactly. WorkDir("/src") against a mount at /src works; WorkDir("/src/services/api") is passed through as a real host path. Keep WorkDir on the mount point and let the tool move: go -C services/api build ./....

When to reach for senro (and when not)

Reach for it when your pipeline logic has outgrown what YAML can honestly express: real branching, loops over units, recovery, anything you want to unit test, anything where you want to attach to a running build and look inside it. Reach for it when you want a build that behaves identically on a laptop and on a runner, and a failure report that distinguishes ten outcomes instead of two.

Skip it when your pipeline is genuinely twenty lines of YAML and always will be. GitHub Actions is excellent at twenty lines of YAML, and a Go module and a binary is a lot of machinery for a problem you do not have. Skip it too if you need Windows runners, because that is a decision rather than a gap.

senro is a library first: no global state, no os.Exit, no reading your argv, attach is one opt-in call, and the wire contract lives in github.com/xavidop/senro/api with no dependency beyond the standard library. You can embed it and never run the CLI at all.

Wrapping up

senro is MIT licensed and lives at github.com/xavidop/senro. The full documentation is in the repository under site/, runnable examples are in examples/, and an agent skill ships in-repo for AI coding tools.

Three more articles continue this series:

Building a real pipeline with senro: monorepo fan-out, running only what a change affects, caching and triggers.
Genkit flows as senro steps: AI pipelines in Go where each model call is a graph node with its own retry, log and cache entry.
Debugging senro pipelines: replaying a finished run, shelling into a live step, and auditing a cache.

If you build something with it, I would like to hear about it.

Building Agents in Go: Where Genkit Shines vs ADK 2.0

Xavier Portilla Edo — Tue, 29 Sep 2026 13:38:01 +0000

Introduction

In my previous comparison of Genkit Go and ADK Go 2.0 I covered the everyday tasks. This article is about the agent layer, and instead of comparing features one by one, I did what you would actually do at work: I built something complex in one framework and then tried to build the same thing in the other.

Same rules as always: every program here compiled and ran. Genkit Go v1.11.0 and ADK Go v2.1.0, gemini-3-flash-preview plus gpt-5-mini, 2026-08-05. Where I cite internals, they come from the released module sources.

One caveat upfront so it frames everything: Genkit's agents API sits behind genkit.WithExperimental() and can change between minor releases, while ADK's agent stack is GA. That is a real ADK advantage. Everything else in this article is about what you can build.

The build: a research team you can fire and forget

Here is the spec. It is deliberately the kind of thing a real product asks for:

An orchestrator agent that coordinates a researcher and a writer.
The sub-agents save their work as files (findings.md, draft.md) that end up in one place.
The system survives a model outage: if the orchestrator's primary model fails, it falls back to a different vendor.
Dangerous tools need human approval before they run.
The client fires the task and disconnects; the result waits server-side.

In Genkit this is one program of 90 lines, and it ran on the first try. The interesting part is that requirements 1, 2 and 3 are not code at all. They are three entries in a middleware list:

orchestrator := genkitx.DefineAgent(g, "orchestrator",
    aix.InlinePrompt{
        // The primary model is BROKEN on purpose.
        ai.WithModelName("googleai/gemini-nonexistent-model"),
        ai.WithSystem("You coordinate a team. First delegate research to the researcher, then delegate a short draft to the writer. Finally reply with a one-sentence status."),
        ai.WithUse(
            &middleware.Fallback{Models: []ai.ModelRef{
                ai.NewModelRef("openai/gpt-5-mini", nil),      // req 3: cross-vendor healing
            }},
            &middlewarex.Agents{                                // req 1: the team
                Agents:           []aix.AgentRef{researcher.Ref(), writer.Ref()},
                MaxDelegations:   4,
                ArtifactStrategy: middlewarex.ArtifactStrategySession,
            },
            &middlewarex.Artifacts{Readonly: true},             // req 2: read the team's files
        ),
    },
    aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()),
)

The researcher and writer are each ~8 lines: an InlinePrompt with a Gemini model, a system prompt, and &middlewarex.Artifacts{} so they get write_artifact tools. Requirement 5 is the client side:

conn.Send(&aix.AgentInput{
    Message: ai.NewUserTextMessage("Research why Go suits AI agents, then have the writer produce a short draft."),
    Detach:  true,
})
out, _ := conn.Output() // returns immediately
// later, from anywhere:
snap, _ := orchestrator.GetSnapshot(ctx, out.SnapshotID)

And the run, verbatim:

detached after 1ms, snapshot fc70253d-55f8-4890-a587-a6aa74d69438
settled: completed after 58s
artifact "researcher_1/findings.md" (5041 chars) from researcher
artifact "writer_2/draft.md" (3840 chars) from writer
orchestrator: I delegated the research task to the researcher (findings.md saved)
and the short draft to the writer (draft.md saved). Status: research and draft
are complete and saved as researcher_1/findings.md and writer_2/draft.md.

Read that output again, because a lot happened in those 58 seconds while no client was connected: the orchestrator's broken Gemini model failed on every turn and healed onto OpenAI every time, it delegated to two Gemini-powered sub-agents through auto-generated delegate_to_* tools, their artifacts were merged into the orchestrator's session with source metadata, and the whole thing settled into a snapshot I could read later from another process. A multi-vendor agent team, running unattended, in 90 lines.

Why this is easy: middleware composes with agents

The capstone works because of one architectural decision: in Genkit, a middleware is a unit that can inject tools, wrap the model call, and wrap tool execution, and the same unit plugs into a plain Generate call or an agent's prompt with the same ai.WithUse(...) line. Capabilities become list entries.

The shelf in v1.11.0's plugins/middleware: Retry, Fallback, ToolApproval, Filesystem (sandboxed file tools), Skills, plus the experimental Agents (delegation) and Artifacts. Two of them deserve their own demos, both of which I ran.

Cross-vendor fallback is configuration

The plugin ecosystem and the middleware system meet here. Any model from any plugin can be a fallback target for any other, so surviving a vendor outage is this, and nothing else:

g := genkit.Init(ctx, genkit.WithPlugins(
    &googlegenai.GoogleAI{},
    &openai.OpenAI{},
))

text, err := genkit.GenerateText(ctx, g,
    ai.WithModelName("googleai/gemini-nonexistent-model"), // broken
    ai.WithPrompt("Say hi in three words."),
    ai.WithUse(
        &middleware.Retry{MaxRetries: 1},
        &middleware.Fallback{Models: []ai.ModelRef{
            ai.NewModelRef("openai/gpt-5-mini", nil),
        }},
    ),
)

My run: without the middleware, Error 404, Message: models/gemini-nonexistent-model is not found. With it: Hello there, friend. from OpenAI. 37 lines total, and the same two lines work inside an agent, which is exactly how the capstone's orchestrator heals itself.

Human approval is configuration too

ToolApproval wraps tool execution and interrupts any tool not on its allowlist. The tool itself knows nothing about approvals:

resp, _ := genkit.Generate(ctx, g,
    ai.WithPrompt("Delete the files in /tmp/scratch please."),
    ai.WithTools(deleteFiles),
    ai.WithUse(&middleware.ToolApproval{}), // nothing pre-approved
)
// resp.FinishReason == "interrupted"

part, _ := deleteFiles.RestartWith(p,
    ai.WithResumedMetadata[DeleteInput](map[string]any{"toolApproved": true}))
final, _ := genkit.Generate(ctx, g,
    ai.WithMessages(resp.History()...), ai.WithToolRestarts(part), /* ... */)

My run: finish reason: interrupted, then after approval, OK. I've deleted the files in /tmp/scratch. Governance became a deployment decision: you can add approval to an existing agent without touching a single tool.

The same build in ADK 2.0, piece by piece

Now the other side, requirement by requirement, with everything verified against the v2.1.0 sources or actually built.

The team (req 1): ADK has this, GA. SubAgents with the auto-injected transfer_to_agent tool, or agenttool for function-call delegation. No complaints; this is ADK's home turf.

Approval (req 4): ADK has this too, GA, and the declaration is even simpler. functiontool.Config{RequireConfirmation: true} is one flag. I ran the full round-trip in both frameworks and they land within one line of each other (102 vs 101 lines). The difference is the shape: ADK's wire flow is a synthetic adk_request_confirmation function call answered with a map[string]any{"confirmed": true} payload, while Genkit types the pause payload and the resume payload as Go structs and validates every resume against history. Pick your poison: a flag with an untyped protocol, or a typed protocol you wire yourself.

Artifacts (req 2): partial. ADK has an artifact service (in-memory or GCS) and a loadartifactstool. What it does not have is the merging story: sub-agent artifacts flowing into the parent session with source attribution is what ArtifactStrategySession gave me for free.

Cross-vendor fallback (req 3): you build it yourself, and it bites. ADK has no middleware and its callbacks cannot wrap the model call, so the seam is writing a custom model.LLM decorator. I built it: buffer the primary's output, and on failure call the backup. There is a trap here, and my first version hit it. The backup vendor rejected the call with:

"The requested model 'gemini-nonexistent-model' does not exist."

ADK's LLMRequest carries the model name inside the request, so my OpenAI backup was asked to run the Gemini model. The decorator also has to rewrite the request:

backupReq := *req
backupReq.Model = f.backup.Name()

After that fix it worked (Hi there, friend.). Final tally: 86 lines of decorator and ceremony versus two lines of configuration, and the 86-line version is per-agent, static, one-level-deep, and has to handle mid-stream failure buffering itself. This is what "no middleware" costs in practice.

Fire and forget (req 5): not possible. The v2.1.0 source is unambiguous. Runner.Run is a pull-based iterator: nothing executes unless the client consumes events, breaking the loop unwinds the stack, cancelling the context aborts the work, and the REST server passes the request context straight into the run. There is no context.WithoutCancel anywhere in the runner or agent paths. ADK can park at an interrupt and survive a process restart by replaying the session event log, which is genuinely robust, but pausing at a boundary is not continuing to work after the client leaves.

So the capstone scorecard: two requirements ADK meets as well or better (team, approval flag), one it meets partially (artifacts), one it makes you hand-build with sharp edges (fallback), and one it cannot do (detach). And each of the pieces it does have arrives through a different mechanism: a config field here, a toolset there, a plugin bundle, a hand-written decorator. Genkit's version is one composition model for all of them.

The pieces in detail

The capstone leaned on agent features that deserve their own numbers. All of these ran, in both frameworks where both have them.

Typed session state

A Genkit agent's state is a Go struct. A tool mutates it, every change streams to the client as a JSON-Patch, and the client reads it back typed:

type GameState struct {
    Score int      `json:"score"`
    Moves []string `json:"moves"`
}

// inside a tool
if s := aix.SessionFromContext[GameState](ctx); s != nil {
    s.UpdateCustom(func(st GameState) GameState {
        st.Score += in.Points
        st.Moves = append(st.Moves, in.Move)
        return st
    })
}

My run streamed [{"op":"replace","path":"","value":{"moves":["e4"],"score":10}}] and conn.Custom() returned score=20 moves=[e4 Nf3] as a struct. Prompts read the same state with {{@state.score}}.

ADK's session state is Get(string) (any, error) / Set(string, any): a map, type assertions at every read, no schema. It works (my run: score=20 (int)), and instruction strings can template state with {score}, but nothing checks anything at compile time. The 2.0 graph engine does have typed parameter binding, but only for workflow function nodes, not for session state in a chat agent.

Custom agent loops

Both frameworks let you write a fully custom agent. In ADK you cannot implement the agent.Agent interface directly, since it is sealed and the compiler rejects any attempt with missing method internal; the sanctioned path is agent.New(agent.Config{Run: ...}), a full execution override, and it ran in my tests.

The difference is what you write. Genkit's SessionRunner hands you a turn loop with history, per-turn tracing, snapshot writes and panic recovery built in; my draft-then-self-critique agent is 57 lines. The ADK version of the same agent is 96 lines, because a Run override means producing raw session.Events: dig the question out of the event log, call the raw model layer, construct and yield events yourself.

Agents defined in .prompt files

DefinePromptAgent binds an agent to a .prompt file by name; the model, config and persona are data, not Go:

pirate := genkitx.DefinePromptAgent(g, "pirate",
    aix.WithSessionStore[any](localstore.NewInMemorySessionStore[any]()),
)

26 lines of Go, and my run answered: Rewrite 'em in Python only if ye want yer agents movin' as slow as a barnacle-covered scow in a dead calm. Non-engineers edit the persona without touching code, and the same dotprompt format works across Genkit's other language SDKs. ADK's equivalent is an Instruction string or an InstructionProvider where you load files yourself; its YAML agent-config system exists but lives in an internal/ package only ADK's own CLI can reach.

Where ADK's agent stack is ahead

The list I would actually weigh on the other side, verified in the same source dive:

GA stability, against everything above being on Genkit's experimental track.
The graph engine as the runtime: every ADK agent runs as a node in the workflow engine, and the node kit (typed function nodes, routing, per-node schema validation, retries, nested graphs) is public API.
Schema-validated human input in workflows: a graph node can declare a JSON schema for the human's answer, validated on resume. For form-like HITL inside a pipeline, that is ahead of anything Genkit ships.
Task isolation: ModeTask sub-agents get their own conversation scope with an auto-injected finish_task tool and scope-filtered history.
Live audio: RunLive with modalities, speech config and transcription has no Genkit Go equivalent.

The scorecard

Non-blank lines including imports. Every program ran.

Experiment	Genkit Go	ADK Go 2.0
The capstone: team + artifacts + fallback + detach	90	not buildable (fallback DIY, detach impossible)
Cross-vendor fallback	37 (configuration)	86 (hand-written decorator, request-rewrite gotcha)
Approval on an existing tool	45 (middleware config)	flag exists (GA), untyped wire protocol
Typed state, 2 turns, live patches	80, typed	96, map + casts
Custom draft-critique agent	57	96
Agent from a .prompt file	26	no equivalent
Undo the last exchange	41	no API: append-only event log
Context caching	40, one method call	lifecycle DIY with a raw genai client

Conclusion

The agent layer is where Genkit's two architectural bets pay off together. The plugin ecosystem means models from different vendors, local runtimes and vector stores are interchangeable references. The middleware system means capabilities (delegation, artifacts, fallback, retries, approval, sandboxed files, skills) are composable list entries that work identically in a one-shot generate and inside an agent. Stack those two on top of typed sessions, snapshots and detach, and a fire-and-forget multi-vendor agent team is a 90-line program. I know because I ran it, and I also know what the same spec costs in ADK because I built the pieces: one requirement hand-written with real sharp edges, and one that cannot be built at all.

ADK's agent stack remains the right pick when its strengths are your requirements: GA guarantees, graph workflows with schema validation, task isolation, live audio, and the Google Cloud platform around it. But if the question is "how hard is it to build something complex", the answer I measured is: in Genkit, complexity composes; in ADK, each capability is its own project.

Further reading:

Genkit Go vs ADK Go 2.0: a Hands-On Look at Boilerplate and Low-Level Control

Xavier Portilla Edo — Tue, 29 Sep 2026 13:37:54 +0000

Introduction

Genkit and ADK (Agent Development Kit) are both Google-backed Gen AI frameworks with first-class Go SDKs, and I keep getting the same question: what is actually different when you sit down and write the code?

This is not a "which one is better" article. They point at different problems: ADK is an agent orchestration framework (agents, sessions, multi-agent transfer, and since 2.0 a graph-based workflow engine), while Genkit is a Gen AI application framework (model calls, flows, structured output, middleware). But they overlap on the everyday tasks every Go developer does (call a model, get typed output, stream, add tools, intercept requests), and on those tasks the developer experience diverges a lot.

So instead of comparing docs, I wrote the same eight programs twice and ran every single one against a live Gemini backend:

One prompt → one answer
A raw model call with no framework machinery
Structured output into a Go struct
Streaming
Tool calling
Intercepting / wrapping the model call
A custom model backend
Multi-turn conversation and HTTP serving

Everything below compiled and ran on Genkit Go v1.11.0 and ADK Go v2.1.0 (the google.golang.org/adk/v2 module, GA since June 30, 2026) with gemini-3-flash-preview, on 2026-08-05. Where I quote framework internals, they come from those released versions, not from the main branch. Nothing in this article is speculated from documentation.

TL;DR

Task	Genkit Go	ADK Go 2.0
One prompt → text	`genkit.GenerateText(...)`: 21 lines, 2 error paths	model + agent + runner + session + event loop: 49 lines, 8 error paths
Structured output	`genkit.GenerateData[Recipe](...)`, schema inferred from struct tags, typed result	hand-written `genai.Schema` tree, raw JSON text back, unmarshal it yourself
Streaming	callback receives deltas, final response is ready-assembled	SSE partial events; the final event repeats the full text, you deduplicate by hand
Tools	typed Go func, loop runs inside `Generate`	typed Go func (genuinely elegant), but agent/runner/session around it
Wrap the model call	`WithUse` middleware with a real `next()`, applied per call	fixed-signature callbacks (no wrapping) or decorate the `model.LLM` interface, per agent
Local models (Ollama)	official plugin, first-class provider	no native support; OpenAI-compatible adapter with `BaseURL`
Custom model backend	one function, works instantly with `Generate`	two-method interface (the smallest contract in either SDK), but invoking it needs the full stack
Multi-turn	automatic via agents + session stores (experimental track); `resp.History()` for plain generate calls	automatic via the GA session services
Multi-agent	delegation via the `Agents` middleware (experimental track)	first-class: sub-agents, transfer, workflow graph engine
Serve over HTTP	`genkit.Handler(flow)` on plain `net/http`	full REST platform API (sessions, SSE, events) + embedded Web UI
Ecosystem	16 Go plugins: multi-vendor models, vector stores, telemetry, MCP	Google Cloud integrations (Vertex, GCS, BigQuery, Apigee, Cloud Run) + OpenAI-compatible + MCP

The pattern that emerged: Genkit's primitive is the model call; ADK's primitive is the agent. When the thing you need is a model call, Genkit hands it to you directly and ADK makes you rent the whole agent apparatus. When the thing you need is a long-lived, multi-agent system, ADK's apparatus stops being boilerplate and starts being the product.

The mental model

Genkit: the model call is the unit

Genkit Go has exactly one setup object (*genkit.Genkit) and everything else is a package-level function call:

g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}))

text, err := genkit.GenerateText(ctx, g,
    ai.WithModelName("googleai/gemini-3-flash-preview"),
    ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."),
)

That is the complete program (plus imports and an error check). Init doesn't even return an error: configuration problems panic at startup, and from there on there is a single err to handle per generation. My runnable version is 21 non-blank lines with 2 error-handling lines.

ADK: the agent is the unit

ADK has no "just generate" entry point at the framework level. The unit of execution is an agent, and an agent runs inside a runner, and a runner requires a session service. This is the minimal equivalent program:

model, err := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{
    APIKey: os.Getenv("GOOGLE_API_KEY"),
})
// if err != nil ...

a, err := llmagent.New(llmagent.Config{
    Name:        "assistant",
    Model:       model,
    Instruction: "You are a helpful assistant.",
})
// if err != nil ...

r, err := runner.NewInMemory("hello-app", a) // in-memory session/artifact/memory services
// if err != nil ...

msg := genai.NewContentFromText("Why is Go a great language for building AI agents? One short sentence.", genai.RoleUser)

var text string
for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) {
    if err != nil { /* ... */ }
    if event.LLMResponse.Content == nil {
        continue
    }
    if event.IsFinalResponse() {
        for _, part := range event.LLMResponse.Content.Parts {
            text += part.Text
        }
    }
}

49 non-blank lines, 8 error-handling lines for the same output. And to be fair, runner.NewInMemory is already ADK being nice: the explicit form makes you wire runner.Config{AppName, Agent, SessionService, AutoCreateSession} yourself.

Count the concepts you must hold for one prompt: a model, an agent with a Name (mandatory), an app name, a user ID, a session ID (both meaningless for a one-shot call, both mandatory positional arguments), a RunConfig, an event iterator, an IsFinalResponse() predicate, and manual concatenation of Content.Parts. There is no .Text() helper on events. Even ADK's own console launcher spends ~45 lines turning events into printable text (cmd/launcher/console/console.go), because every consumer re-implements this loop.

None of this is bad design: sessions, users and apps are exactly the right vocabulary for a deployed multi-agent platform. It is simply the wrong altitude when what you wanted was an answer to a prompt, and there is no lower altitude offered (almost; see the escape hatch below).

Structured output: where the gap is widest

This was the experiment with the biggest difference, and the one where I initially got ADK wrong in an instructive way.

Genkit

You describe the shape as a Go type. The JSON Schema is inferred from the struct (via reflection over json/jsonschema tags), constrained decoding is enabled by default, and you get the struct back, parsed and typed:

type Recipe struct {
    Name        string   `json:"name"`
    PrepMinutes int      `json:"prep_minutes"`
    Ingredients []string `json:"ingredients"`
}

recipe, _, err := genkit.GenerateData[Recipe](ctx, g,
    ai.WithModelName("googleai/gemini-3-flash-preview"),
    ai.WithSystem("You generate recipes."),
    ai.WithPrompt("A simple tortilla de patatas recipe"),
)

fmt.Println(recipe.Name, recipe.PrepMinutes, recipe.Ingredients)

Output from my run:

Classic Spanish Tortilla de Patatas (15 min): 5 ingredients

28 lines total. The schema definition is the Go type.

ADK

ADK's llmagent.Config.OutputSchema takes a *genai.Schema: a hand-built tree, field by field, with a hand-maintained Required list. There is no "infer from struct" path for agent output (curiously, ADK does have schema inference, but only for tool inputs):

recipeSchema := &genai.Schema{
    Type: genai.TypeObject,
    Properties: map[string]*genai.Schema{
        "name":         {Type: genai.TypeString},
        "prep_minutes": {Type: genai.TypeInteger},
        "ingredients":  {Type: genai.TypeArray, Items: &genai.Schema{Type: genai.TypeString}},
    },
    Required: []string{"name", "prep_minutes", "ingredients"},
}

a, err := llmagent.New(llmagent.Config{
    Name:         "chef",
    Model:        model,
    Instruction:  "You generate recipes.",
    OutputSchema: recipeSchema,
})

The schema enforcement worked perfectly in my runs: Gemini returned valid JSON every time. The interesting part is what happens next. My first attempt read the event's Output field, which looks exactly like what you want. It was always nil.

Digging into the released v2.1.0 source: the LlmAgent does parse the structured output internally, and does attach it to the event, but the runner deliberately strips it before the event reaches you. In runner/runner.go, every non-partial agent reply gets cloned with clone.Output = nil. The parsed value exists to feed ADK 2.0's graph workflow engine (typed routing between nodes), not for you. As a consumer of Runner.Run, you receive raw JSON as text and parse it yourself:

var jsonText string
for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) {
    // ... error handling, nil checks, IsFinalResponse, Parts loop ...
    jsonText += part.Text
}

var recipe Recipe
if err := json.Unmarshal([]byte(jsonText), &recipe); err != nil { /* ... */ }

(If you set OutputKey, the same raw string is also copied into session state, still not parsed for you.)

Two more sharp edges I hit in the source while verifying this:

The OutputSchema doc comment in v2.1.0 still warns: "when this is set, agent can only reply and cannot use any tools, such as function tools, RAGs, agent transfer, etc." The implementation has grown workarounds (on some backends it injects a synthetic set_model_response tool; on Vertex AI with Gemini ≥2.0 it can use native constrained output alongside tools), which means combining tools and output schema behaves differently depending on backend and model name, silently.
Final tally: 68 lines vs Genkit's 28, and the typed half of "typed output" is entirely yours to write.

Streaming: deltas vs partial events

Both frameworks stream. The difference is who assembles the result.

Genkit gives you a callback that receives deltas, and the final response arrives already assembled:

resp, err := genkit.Generate(ctx, g,
    ai.WithModelName("googleai/gemini-3-flash-preview"),
    ai.WithPrompt("Count from one to ten in words, one per line."),
    ai.WithStreaming(func(ctx context.Context, chunk *ai.ModelResponseChunk) error {
        fmt.Print(chunk.Text())
        return nil
    }),
)

My run printed streamed chars: 48, final response chars: 48. Chunks and final response account for the same text exactly once each.

ADK streams by setting agent.RunConfig{StreamingMode: agent.StreamingModeSSE} and then flags chunk events with LLMResponse.Partial. The gotcha: after all the partial events, ADK emits a final non-partial event that repeats the entire text. My run: partial (streamed) chars: 48, final event chars: 48. Print events naively and you output everything twice. Every ADK consumer must carry this filter:

for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{
    StreamingMode: agent.StreamingModeSSE,
}) {
    // ...
    if event.LLMResponse.Partial {
        fmt.Print(part.Text) // the streamed deltas
    }
    // the final event repeats the full text: skip it, or you print it twice
}

This isn't hypothetical: ADK's own example code and console launcher both contain hand-rolled deduplication (if text != prevText in cmd/launcher/console/console.go). It's a small thing, but it's a small thing every single ADK streaming consumer writes, and no Genkit consumer does. As a bonus, Genkit's GenerateDataStream[T] even streams partially-parsed typed structs (your Recipe filling in field by field), which has no ADK equivalent.

Tool calling: closer than I expected

Here I expected a bigger gap and didn't find one: ADK's function tools are genuinely well designed. Both frameworks let a tool be an ordinary typed Go function, with the JSON Schema inferred from the input struct:

// Genkit
weather := genkit.DefineTool(g, "getWeather", "Gets the current weather for a city",
    func(ctx *ai.ToolContext, in WeatherInput) (string, error) {
        return fmt.Sprintf("22C and sunny in %s", in.City), nil
    })

resp, err := genkit.Generate(ctx, g,
    ai.WithModelName("googleai/gemini-3-flash-preview"),
    ai.WithPrompt("What is the weather in Valencia right now?"),
    ai.WithTools(weather),
)

// ADK: also nice! Generics, schema inferred, type params never spelled out.
weatherTool, err := functiontool.New(functiontool.Config{
    Name:        "get_weather",
    Description: "Gets the current weather for a city",
}, func(_ agent.Context, in WeatherInput) (WeatherOutput, error) {
    return WeatherOutput{Report: fmt.Sprintf("22C and sunny in %s", in.City)}, nil
})

Both ran the tool and produced the right answer. The differences are around the edges:

Ceremony: the Genkit program is 29 lines; the ADK one is 66, because the tool still needs the agent/runner/session stack around it.
Where the loop lives: Genkit's tool loop runs inside Generate (with parallel tool execution and a WithMaxTurns bound), so tools work in a plain one-shot call. In ADK the loop belongs to the agent flow, which is consistent with its philosophy, but it means there is no way to use ADK tools without an agent.
A public-API gotcha: ADK's public tool.Tool interface is only Name()/Description()/IsLongRunning(). The executable contract (Declaration(), Run()) lives in an internal/ package you cannot reference, so you can't hand-implement a runnable tool against a public interface; you go through functiontool.New. Genkit's equivalent (ai.ToolFunc + DefineTool) is fully public.
Composability limit: ADK documents (in its own examples/tools/multipletools) that you cannot mix built-in Gemini tools like geminitool.GoogleSearch{} with custom function tools on one agent; the workaround is wrapping each in a sub-agent.
Genkit also has tool interrupts/resume as a stable API (pause generation from inside a tool, get an Interrupts() list, resume with RespondWith/RestartWith). ADK's equivalents are tool confirmation flows wired through its HITL machinery.

Getting low-level

You can go deeper in Genkit than in ADK, and its middleware system is the reason: Genkit exposes the model call, the tool executions and the whole generation loop as wrappable layers, while ADK exposes fixed callback points and one wrappable interface. Here is what that means in code.

Wrapping the model call

Genkit middleware is a real onion: you get the request and a next() you control. Timing, retry, caching and request rewriting are all one closure, applied per call:

timing := ai.MiddlewareFunc(func(ctx context.Context) (*ai.Hooks, error) {
    return &ai.Hooks{
        WrapModel: func(ctx context.Context, params *ai.ModelParams, next ai.ModelNext) (*ai.ModelResponse, error) {
            start := time.Now()
            resp, err := next(ctx, params)
            log.Printf("model call: %d messages in, took %s",
                len(params.Request.Messages), time.Since(start))
            return resp, err
        },
    }, nil
})

resp, err := genkit.Generate(ctx, g, /* ... */, ai.WithUse(timing))

Ran, printed model call: 1 messages in, took 3.323s. There are WrapGenerate and WrapTool hooks at the other two altitudes (the whole tool loop, and each tool execution), middleware can inject tools, and the built-ins (Retry, Fallback, tool approval, filesystem sandbox) compose the same way.

ADK has callbacks, not middleware: BeforeModelCallback and AfterModelCallback with fixed signatures. They can short-circuit (return a response from Before and the model is never called, the documented caching hook) and they can replace the response, but they cannot wrap. There is no next(). To time a call I had to smuggle state between two separate functions:

var start time.Time

before := func(ctx agent.Context, req *model.LLMRequest) (*model.LLMResponse, error) {
    start = time.Now()
    return nil, nil // nil means "continue to the model"
}
after := func(ctx agent.Context, resp *model.LLMResponse, respErr error) (*model.LLMResponse, error) {
    log.Printf("took %s", time.Since(start))
    return nil, nil
}

This works (ran, took 1.631s) but it's a workaround, it's attached per-agent rather than per-call, and anything genuinely wrap-shaped (hedged requests, per-attempt timeouts, fallbacks across models) has to move to a different mechanism entirely: decorating the model. Which brings me to the thing ADK gets really right:

Custom model backends: local Gemma on Ollama

Nobody plugs in an echo model, so no toy examples here. The realistic scenario is a backend the framework didn't bless for you: a local model server, an internal gateway, a provider without an official plugin. I tested two levels of this: running Google's Gemma locally on Ollama, and writing a backend from scratch.

Genkit ships an official Ollama plugin, so the local model is a first-class provider:

oll := &ollama.Ollama{ServerAddress: "http://localhost:11434"}
g := genkit.Init(ctx, genkit.WithPlugins(oll))

gemma := oll.DefineModel(g, ollama.ModelDefinition{Name: "gemma3n:e4b", Type: "chat"}, nil)

resp, err := genkit.Generate(ctx, g,
    ai.WithModel(gemma),
    ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."),
)

23 lines, ran on the first try against gemma3n:e4b on my Mac, and the local model gets the same Generate, flows, middleware and structured output as any cloud model. The plugin covers embedders too.

ADK has no Ollama integration of its own. The path that works is the OpenAI-compatible adapter pointed at Ollama's OpenAI endpoint:

gemma, err := openaimodel.NewModel(ctx, "gemma3n:e4b", &openaimodel.ClientConfig{
    BaseURL: "http://localhost:11434/v1",
    APIKey:  "ollama", // any non-empty value; Ollama ignores it
})

This also ran (47 lines with the usual agent, runner and session around it), and it is a documented path: the package README lists Ollama, LM Studio and vLLM. The difference is the shape of the support. In Genkit the local model is a provider with its own plugin; in ADK it rides on OpenAI compatibility, so anything Ollama offers outside that API surface is out of reach, and then you are writing the backend yourself, which is the next scenario.

And when there is no plugin at all

When the backend has no plugin and no OpenAI-compatible endpoint (an internal LLM gateway, a niche provider), you write the integration yourself. I did, in both frameworks: the same minimal OpenAI chat/completions client, roughly 45 lines of identical plain-net/http glue, run against the real OpenAI API.

ADK's model.LLM interface is the smallest custom-backend contract I've seen in any framework:

type LLM interface {
    Name() string
    GenerateContent(ctx context.Context, req *LLMRequest, stream bool) iter.Seq2[*LLMResponse, error]
}

Two methods, and this interface is also ADK's genuine wrapping seam: ADK's own examples ship a resilientModel decorator doing retries and per-attempt timeouts this way. My OpenAI-backed model.LLM worked, with two costs on top of the shared HTTP glue. First, the vocabulary: requests and responses are google.golang.org/genai types, so you translate both ways, and some of it is non-obvious. The agent's Instruction arrives in Config.SystemInstruction, not in Contents; miss that and your backend silently drops the system prompt. (ADK's own OpenAI adapter is four files of translation code, which tells you how deep that rabbit hole goes. Also, parts of ADK sniff behavior from llm.Name(), so a renaming wrapper can silently change how output schemas are handled.) Second, the invocation: those two beautiful methods still need the agent, runner and session stack around them to run. Total: 116 lines.

The Genkit version of the same backend is the HTTP glue, a capability declaration, and nothing else:

func callOpenAI(ctx context.Context, req *ai.ModelRequest, cb func(context.Context, *ai.ModelResponseChunk) error) (*ai.ModelResponse, error) {
    body := oaRequest{Model: "gpt-5-mini"}
    for _, m := range req.Messages {
        role := string(m.Role) // "system" | "user" | "model"
        if role == "model" {
            role = "assistant"
        }
        body.Messages = append(body.Messages, oaMessage{Role: role, Content: m.Text()})
    }
    // ... plain net/http POST to api.openai.com/v1/chat/completions ...
    return &ai.ModelResponse{
        FinishReason: ai.FinishReasonStop,
        Message:      ai.NewModelTextMessage(out.Choices[0].Message.Content),
    }, nil
}

gpt := genkit.DefineModel(g, "myco/gpt-5-mini", &ai.ModelOptions{
    Supports: &ai.ModelSupports{Multiturn: true, SystemRole: true},
}, callOpenAI)

resp, err := genkit.Generate(ctx, g,
    ai.WithModel(gpt),
    ai.WithSystem("You are a helpful assistant."),
    ai.WithPrompt("Why is Go a great language for building AI agents? One short sentence."),
)

Total: 79 lines, and Generate calls it directly; the model immediately works with structured output, streaming, middleware and flows. The messages arrive as provider-neutral structs (ai.Message, with system prompt as a regular system-role message), so the translation is one role-mapping loop instead of a genai round-trip. One honest note: my first attempt failed at runtime because I passed nil options, since a Genkit model must declare what it Supports. That cost me one extra struct literal, and it's the flip side of a real feature: the framework validates every request against the declared capabilities and even simulates a system prompt for models that don't support one.

The raw call escape hatch

I didn't want to write "you can't go low-level in ADK" without trying it myself, so I tried. model.LLM is public, and this compiles and runs (33 lines in my experiment):

llm, _ := gemini.NewModel(ctx, "gemini-3-flash-preview", &genai.ClientConfig{APIKey: key})

for resp, err := range llm.GenerateContent(ctx, &model.LLMRequest{
    Contents: []*genai.Content{genai.NewContentFromText(prompt, genai.RoleUser)},
}, false) {
    // resp.Content.Parts ...
}

But notice what you're holding: this is the bare genai transport with an ADK-shaped envelope. No instruction templating, no tool loop, no callbacks, no structured output processing, no telemetry: all of that lives inside the agent flow you just bypassed. There are no helpers (Text(), message builders) at this layer, and no example in the entire ADK repo uses it; only tests do. It's an escape hatch, not a supported altitude.

That's the crux of the low-level story: in Genkit, dropping down doesn't cost you the framework. Generate is the low-level call: raw messages in, with middleware, tools, schemas and tracing still active. In ADK, the abstraction is a cliff: you're either at agent altitude with everything, or at transport altitude with nothing.

Where ADK's ceremony pays off

I also ran the experiments where I expected ADK's extra structure to earn its keep. It mostly did.

Multi-turn conversation. Both frameworks carry the conversation for you; neither makes you thread history by hand.

In ADK, I called r.Run twice with the same session ID and the second turn knew my name and city. The session service is the conversation, and swapping InMemoryService for the database or Vertex AI implementation brings persistence with it.

In Genkit, the agents API does the same job with a session store. I ran the equivalent two turns on v1.11.0:

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithExperimental(),
)

assistant := genkitx.DefineAgent(g, "assistant",
    aix.InlinePrompt{
        ai.WithModelName("googleai/gemini-3-flash-preview"),
        ai.WithSystem("You are a helpful assistant."),
    },
    aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()),
)

first, err := assistant.RunText(ctx, "My name is Xavi and I live in Valencia. Say hi in one sentence.")
// ...
second, err := assistant.RunText(ctx, "Where do I live? Answer with just the city name.",
    aix.WithSessionID[any](first.SessionID))

Both frameworks answered turn 2: Valencia in my runs, with zero history plumbing on either side. And Genkit's session layer goes further than a message log: session snapshots, detach/resume (the client disconnects, the server keeps working, you resume from a snapshot) and artifacts, with in-memory and file stores shipping today plus a SessionStore interface for your own persistence. For conversation state, the two frameworks are at parity. (If you are outside an agent, composing plain Generate calls, ai.WithMessages(resp.History()...) threads history in one line, but that is a choice, not a limitation.)

The one difference I would plan around is the maturity label. ADK's session services are GA and are the default path; Genkit's agents API is on the experimental track behind genkit.WithExperimental() and can change between minor releases, which is also why the rest of this article compares the stable surfaces. If you are comfortable riding the experimental track, Genkit gives you all of it today; if you need the stability guarantee, ADK has it.

Multi-agent. ADK's sub-agents get an auto-injected transfer_to_agent tool, there are sequential/parallel/loop workflow agents, and 2.0's headline feature is a full graph workflow engine with typed routing between nodes, retries and human-in-the-loop pauses.

Genkit delegates too. The Agents middleware injects one delegate_to_<name> tool per sub-agent, discovers each sub-agent's description automatically for the orchestrator's prompt, forwards recent conversation history, and can merge sub-agent artifacts back into the orchestrator's session. I ran an orchestrator delegating to a poet sub-agent:

orchestrator := genkitx.DefineAgent(g, "orchestrator",
    aix.InlinePrompt{
        ai.WithModelName("googleai/gemini-3-flash-preview"),
        ai.WithSystem("You are a coordinator. Delegate writing tasks to the right sub-agent using its delegation tool, then return its result verbatim."),
        ai.WithUse(&middlewarex.Agents{
            Agents:         []aix.AgentRef{poet.Ref()},
            MaxDelegations: 2,
        }),
    },
    aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()),
)

43 lines, worked on the first run (I got my haiku back through the orchestrator), and the full researcher-plus-engineer version is in the official basic-agents sample. Same maturity note as sessions: this is the experimental track.

And ADK's workflow layer, the sequential/parallel/loop agents and the 2.0 graph engine? Genkit's answer is that the orchestration language is Go itself. Flows compose: a flow calls other flows, sequence is code order, parallel is a goroutine group, a loop is a loop, and every flow stays traced and individually servable. I ran a brief-writing flow that fans out to two sub-flows in parallel and then synthesizes:

brief := genkit.DefineFlow(g, "briefFlow",
    func(ctx context.Context, topic string) (string, error) {
        // Parallel: plain Go concurrency.
        var facts, structure string
        eg, egCtx := errgroup.WithContext(ctx)
        eg.Go(func() error {
            var err error
            facts, err = research.Run(egCtx, topic)
            return err
        })
        eg.Go(func() error {
            var err error
            structure, err = outline.Run(egCtx, topic)
            return err
        })
        if err := eg.Wait(); err != nil {
            return "", err
        }

        // Sequential: the next line of code.
        return genkit.GenerateText(ctx, g,
            ai.WithModelName(model),
            ai.WithPrompt("Write a two-sentence brief about %s using these facts:\n%s\nand this outline:\n%s", topic, facts, structure))
    })

53 lines including both sub-flows, ran on stable APIs, and genkit.Handler(brief) serves it over HTTP like any other flow. So the difference between the two workflow stories is representation, not capability. ADK reifies the pipeline as a data structure (edges, typed routes) executed by an engine that owns retries, pauses and resumability at the graph level; Genkit keeps the pipeline as ordinary Go and covers those concerns with its own pieces (tool interrupts for human-in-the-loop, the retry middleware, and durable execution through integrations like Temporal). Pick by taste: an inspectable graph object, or code you read top to bottom.

The serving platform. Genkit's HTTP story is its philosophy in miniature. A flow becomes a handler on plain net/http:

mux := http.NewServeMux()
mux.HandleFunc("POST /greetingFlow", genkit.Handler(greetingFlow))
server.Start(ctx, "127.0.0.1:9090", mux)

$ curl -X POST localhost:9090/greetingFlow -d '{"data":"Xavi"}'
{"result":"Hello, Xavi!"}

ADK's adkrest.NewServer is also a plain http.Handler (nice), but what it mounts is a platform: app listing, session CRUD, SSE runtime, artifacts, debug/trace endpoints. My curl transcript needed two calls (create the session, then run) and got back the full event envelope with usage metadata, state deltas and node info. On top of that, ADK's launcher embeds the entire ADK Web UI into your binary via go:embed: go run . web webui gives you a chat UI with no npm and no separate process, which is honestly great. (Genkit's Developer UI is a separate CLI process, though it's also more than a chat UI: it traces every generate call, middleware hop and tool execution.)

These aren't contradictions of the boilerplate findings; they're the same design measured on its own terrain. ADK's five constructors are the price of a system where conversations persist, agents transfer to each other, and a REST platform falls out of a config struct.

The plugin ecosystems

One more thing I checked in the released modules, because it changes the decision for real projects: what each framework integrates with out of the box.

Genkit Go v1.11.0 ships sixteen plugins in the module: model providers for Google (Gemini API and Vertex AI), Anthropic, OpenAI plus any OpenAI-compatible endpoint, and Ollama; vector stores for AlloyDB, PostgreSQL/pgvector, Pinecone, Weaviate and a local dev store; Firebase and Google Cloud telemetry; MCP; evaluators; and the middleware pack. And that is the Go module alone, before you count the wider JS ecosystem the framework shares its design with.

ADK Go v2.1.0's integrations are a coherent Google Cloud story: Gemini and Vertex models, the OpenAI-compatible adapter, Apigee; sessions in memory, in a SQL database or on Vertex AI; GCS artifacts; Vertex AI memory; BigQuery agent analytics; MCP toolsets; and deploy tooling for Cloud Run and Agent Engine. What it does not have is a provider ecosystem beyond that: no Anthropic plugin, no vector store integrations, nothing aimed at other clouds.

If your stack is pure Google Cloud, ADK's set covers it end to end, deploy included. The moment you need Anthropic next to Gemini, a Postgres vector store, or observability outside Google Cloud, Genkit has the plugin and ADK expects you to build it.

The scorecard

Numbers from my runnable programs (non-blank lines including imports; error-handling lines in parentheses):

Experiment	Genkit Go	ADK Go 2.0
Hello world	21 (2)	49 (8)
Raw model call	same as hello	33 (4), undocumented path
Structured output → typed struct	28 (2)	68 (9)
Streaming	27 (2)	54 (8)
One tool	29 (2)	66 (10)
Time the model call	34 (2)	61 (8)
Local Gemma via Ollama, invoked	23 (2)	47 (8)
Custom model backend (real OpenAI), invoked	79 (5)	116 (13)
Multi-turn (2 turns)	37 (4) with agents + sessions; 30 (4) with plain `Generate`	51 (8)

Line counts are a blunt instrument; the interesting part is what the extra lines are. In every ADK program they are the same lines: construct model, construct agent, construct runner, invent user/session IDs, iterate events, filter partials, concatenate parts. It's a fixed tax, roughly 25-30 lines, charged on every program regardless of whether sessions and events are relevant to it.

A concrete decision guide

I would reach for Genkit Go when:

The unit of my product is a model call or a flow: an API endpoint, a pipeline step, a CLI, a backend feature.
I want typed structured output without maintaining schema trees by hand.
I want to wrap, retry, fall back, cache or rewrite model calls with composable per-call middleware.
I'm integrating a custom or non-Google model backend and want the framework features to keep working on top of it.
I value the Dev UI's per-call tracing during development.
I want sessions and agents too, and I'm fine with the experimental track while that API stabilizes.
My stack mixes vendors: Anthropic or OpenAI next to Gemini, Ollama locally, or Postgres/Pinecone/Weaviate for retrieval. The plugin shelf already covers them.

I would reach for ADK Go 2.0 when:

The unit of my product is a persistent, session-based agent: a chat product, a support agent, anything where conversation state is the point, and I want it on GA APIs today.
I need multi-agent coordination: sub-agent transfer, workflow agents, or 2.0's graph engine with typed routing and HITL pauses.
I want the serving platform for free: session REST API, SSE, the embedded Web UI, A2A, Cloud Run / Agent Engine deploy tooling.
My stack is Gemini/Vertex-first and the genai-typed vocabulary is a feature, not a translation burden.

And these compose better than you'd think: nothing stops you from building your model-centric services with Genkit flows and standing up an ADK agent where you genuinely need the agent platform. They're both idiomatic modern Go (both lean on iter.Seq2 and generics nicely), and they're solving different layers of the same problem.

Conclusion

After running everything, the facts are these: Genkit lets you get lower-level, and ADK charges more boilerplate. In detail:

The boilerplate is real and remarkably consistent: ADK charges a fixed ~25-30 line agent tax on every program, because there is no supported altitude between "full agent" and "bare transport". Genkit's Generate occupies exactly that missing middle: raw request access with the framework still on.
The low-level gap is sharpest in structured output (typed structs vs hand-built schema trees plus DIY parsing, and a runner that deliberately strips the parsed value it already has) and in middleware (a real next() vs fixed-signature callbacks).
But ADK's ceremony is not waste: it buys GA session persistence, multi-agent transfer, a graph engine and an embedded serving platform, and some of its individual designs (the two-method model.LLM, generic function tools) are excellent Go.

Pick the altitude that matches your product, and don't confuse "more setup" with "more capable", or "less setup" with "less serious." They're different tools that happen to share a language and a model provider.

Further reading:

Hot-Reloading Kubernetes Secrets and ConfigMaps in Go with mamori (No Pod Restarts)

Xavier Portilla Edo — Tue, 29 Sep 2026 13:37:11 +0000

Introduction

If you run Go services on Kubernetes, you know the ritual. You update a Secret or a ConfigMap, and then you do one of these:

kubectl rollout restart deployment/my-app and eat the disruption.
Install Reloader so it restarts your pods for you when a Secret changes.
Mount the Secret as a volume and cross your fingers that your app re-reads the file (spoiler: most apps read it once at boot).

All three share the same flaw: the way you "pick up" a config change is to throw away the process and start a new one. For a rotating database password, that means dropping every in-flight request, cold connection pools, and a latency spike — all to change one string.

I built mamori partly because of this exact frustration. mamori watches your Kubernetes Secrets and ConfigMaps with native informers, and when a value changes it reconciles the new value into your running Go process — typed, validated, and atomically swapped — without restarting anything.

New to mamori? Start with the launch article, Introducing mamori: Typed, Watchable Config & Secrets for Go, for the full picture. This post is the Kubernetes-focused deep cut.

The two schemes you'll use

mamori's Kubernetes provider gives you two ref schemes:

Ref	Reads from	Notes
`k8s-secret://<namespace>/<name>#<key>`	a `Secret`	value is base64-decoded for you
`k8s-cm://<namespace>/<name>#<key>`	a `ConfigMap`	plain string values

Drop the #key and you get the whole Secret/ConfigMap as a JSON map, which you can decode into a nested struct with flatten. A few concrete examples:

k8s-secret://prod/db-creds#password   # one key from a Secret, base64-decoded
k8s-secret://prod/tls#ca.crt          # a certificate as []byte
k8s-cm://prod/app-config#log_level    # one key from a ConfigMap
k8s-cm://prod/app-config              # the whole ConfigMap as a JSON map

Setup

The provider lives in its own module. Note the import path is providers/k8s (not providers/kubernetes):

go get github.com/xavidop/mamori
go get github.com/xavidop/mamori/providers/k8s

Register it with a blank import — same convention as database/sql drivers. Its init() wires the k8s-secret and k8s-cm schemes into mamori:

import _ "github.com/xavidop/mamori/providers/k8s"

In-cluster, the provider uses the pod's mounted ServiceAccount automatically. The only thing you owe it is RBAC: it needs to get, list, and watch the resources you reference in the namespaces you reference.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: mamori-reader
  namespace: prod
rules:
  - apiGroups: [""]
    resources: ["secrets", "configmaps"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: mamori-reader
  namespace: prod
subjects:
  - kind: ServiceAccount
    name: my-app
    namespace: prod
roleRef:
  kind: Role
  name: mamori-reader
  apiGroup: rbac.authorization.k8s.io

The watch verb is the important one — it's what lets mamori use an informer instead of hammering the API server with polls.

Load once, then watch forever

Describe what you need as a struct. mamori resolves every ref, applies defaults, and validates — all before your main gets a config:

package main

import (
    "context"
    "log"

    "github.com/xavidop/mamori"
    "github.com/xavidop/mamori/secret"

    _ "github.com/xavidop/mamori/providers/k8s"
)

type Config struct {
    DBPassword secret.String `source:"k8s-secret://prod/db-creds#password"`
    LogLevel   string        `source:"k8s-cm://prod/app-config#log_level" default:"info" validate:"oneof=debug info warn error"`
    MaxConns   int           `source:"k8s-cm://prod/app-config#max_conns" default:"10" validate:"gte=1,lte=500"`
    CACert     []byte        `source:"k8s-secret://prod/tls#ca.crt"`
}

func main() {
    cfg, err := mamori.Load[Config](context.Background())
    if err != nil {
        log.Fatal(err)
    }
    _ = cfg // fully resolved, defaulted, validated
}

Notice DBPassword is a secret.String, not a plain string. That type redacts itself in logs, fmt, and JSON — so a stray log.Printf("%+v", cfg) prints [REDACTED] instead of leaking the password. You only ever see the plaintext when you explicitly call .Reveal().

Now the part that matters. Swap Load for Watch, and the moment someone runs kubectl edit secret db-creds, your process reacts:

w, err := mamori.Watch[Config](ctx,
    mamori.OnChange(func(ev mamori.Change[Config]) {
        if ev.Changed("DBPassword") {
            // Rebuild the pool with the rotated password — live, no restart.
            pool.Rotate(ev.New.DBPassword.Reveal())
        }
        for _, f := range ev.Fields {
            log.Printf("reconciled %s: %s -> %s", f.Path, f.OldVersion, f.NewVersion)
        }
    }),
    mamori.OnError(func(err error) { metrics.Inc("config_error") }),
)
if err != nil {
    log.Fatal(err)
}
defer w.Close()

cfg := w.Get() // lock-free snapshot; always the last VALID config

w.Get() is lock-free and always returns the last valid configuration. If someone pushes a ConfigMap with log_level: banana, the validate:"oneof=..." tag rejects it: the update is dropped atomically, Get() keeps returning the good value, and OnError receives a *ValidationError. A bad edit never takes your service into a broken state.

Informers, not polling

This is the difference between mamori and the mount-a-file-and-poll approach. The Kubernetes provider implements native watching backed by the client-go informer machinery, so change detection is push-based — the API server tells your process the instant a Secret changes, rather than your process asking every N seconds.

Practically, that means:

Near-instant reconciliation. No polling interval to tune, no min(sync, interval) latency floor like the kubelet's ~60s volume refresh.
No API-server hammering. One watch connection per resource, shared, instead of a GET loop per pod.
Graceful fallback. If a native watch fails to start for some reason, mamori automatically falls back to polling with jittered intervals, so you never lose reconciliation entirely.

Compare that to the alternatives:

Approach	How a change is picked up	Restart?
`kubectl rollout restart`	you, manually	yes
Reloader / Stakater	operator restarts the pod	yes
Secret mounted as volume	kubelet re-syncs file (~60s), if your app re-reads it	no, but you must re-read
mamori	informer pushes change → validated atomic swap → callback	no

Whole ConfigMaps and JSON blobs

Sometimes you keep a structured blob in one key. Decode it straight into a nested struct with flatten:

type Config struct {
    Redis RedisConfig `source:"k8s-secret://prod/redis-conn" flatten:"json"`
}

type RedisConfig struct {
    Addr     string        `mapstructure:"addr"`
    Password secret.String `mapstructure:"password"`
    DB       int           `mapstructure:"db"`
}

That reads the whole redis-conn Secret (a JSON document), decodes it, and — because Password is a secret.String — keeps the password redacted throughout.

A note on TLS certificates

Certificates rotate too, and a cert reload is the perfect no-restart use case. Because file and Secret updates for certs usually want to apply immediately (no coalescing delay), you can set ?debounce=0 on the ref:

type TLS struct {
    Cert []byte       `source:"k8s-secret://prod/tls#tls.crt?debounce=0"`
    Key  secret.Bytes `source:"k8s-secret://prod/tls#tls.key?debounce=0"`
}

Wire the OnChange callback to swap the *tls.Certificate your server hands out in its GetCertificate hook, and you've got zero-downtime cert rotation from a Kubernetes Secret.

When you don't need this

Be honest with yourself: if your config is a log level from a ConfigMap that only ever changes on deploy, you don't need runtime watching — a plain Load at boot is fine, and mamori is happy to do just that. And if you already mount a Secret as a file and genuinely re-read it, mamori's built-in file:// provider (with fsnotify) covers that too, without any Kubernetes RBAC at all.

mamori earns its keep the moment a value changes while your process is running and you'd rather rotate a pool than recycle a pod.

Wrapping up

Kubernetes already tells you the instant a Secret changes — via informers. mamori just connects that signal to a typed, validated, redacted struct inside your Go process, so a rotation becomes a callback instead of a rollout.

go get github.com/xavidop/mamori/providers/k8s

Full introduction: Introducing mamori
Docs: mamorigo.dev/docs
Source: github.com/xavidop/mamori

mamori is MIT-licensed and currently at v0.1.0, so issues and feedback are very welcome. Happy shipping!

DEV Community: Xavier Portilla Edo

Durable Genkit Flows with Temporal: Introducing the genkitx-temporal Plugin

Introduction

What is genkitx-temporal?

Why Temporal and Genkit are a great match

Installation

Usage end to end

1. Define a flow with defineTemporalFlow

2. Start a Worker

3. Execute a flow as a Temporal Workflow

Configuration

Advanced: combine with your own workflows and activities

API summary

When to reach for genkitx-temporal

How it pairs with the rest of the Genkit ecosystem

Conclusion

Why a Local Debugging Tool is Non-Negotiable for Building AI Apps: Genkit Developer UI vs Vercel AI SDK DevTools vs Mastra Studio

Introduction

Shift-left, applied to Gen AI

Local vs hosted

What "good" looks like

Genkit Developer UI

Vercel AI SDK DevTools

Mastra Studio

Side-by-side comparison

When I would pick which

The moral of the story: pick a debugging tool, any debugging tool

Conclusion

Top Gen AI Frameworks for Go in 2026: A Hands-On Comparison

Introduction

Genkit Go

History and Direction

What Makes Genkit Go Stand Out

Vanilla Generation

Typed Flows: Observable Pipelines

Tools and Agents

The Dev UI: Same Power as TypeScript

Provider Support

Pros and Cons

Eino

History and Direction

What Makes Eino Stand Out

ChatModel

Chains, Graphs, and Callbacks

Eino ADK: Agents and Workflows

Developer Tooling

Provider Support

Pros and Cons

Google ADK Go

History and Direction

ADK Go's Position: Agent-Only, Enterprise-Grade

Multi-Agent Orchestration

The Dev UI: ADK Web, Embedded in Your Binary

Vertex AI Gravity

Pros and Cons

tRPC-Agent-Go

History and Direction

What Makes tRPC-Agent-Go Stand Out

Chains

GraphAgent: LangGraph in Go

Pros and Cons

LangChainGo

History and Direction

LangChainGo's Position: Legacy

Pros and Cons

Head-to-Head Comparison

Developer Experience

Abstraction Levels

Observability

Provider Support

How Go Does It Feel?

Maintenance Snapshot

Which Framework Should You Choose?

Conclusion

How to Migrate from LangChainGo to Genkit Go

Introduction

Why Migrate Now

The trap: observability breaks your agents

What You Need

The Concept Map

What is `genkitx-temporal`?

1. Define a flow with `defineTemporalFlow`

When to reach for `genkitx-temporal`

Caching a step with `Pure()`

Monorepos: fan out with `Expand`