DEV Community: Xavier Portilla Edo

Introducing tobira: Every Database, One Terminal, and One MCP Server for Your Agents

Xavier Portilla Edo — Tue, 27 Oct 2026 23:12:16 +0000

Introduction

Count the database clients on your laptop. psql, sqlite3, mongosh, redis-cli, a ClickHouse client, a Qdrant dashboard in a browser tab, maybe DBeaver for the days you want a grid. Each one has its own flags, its own way of printing a table, its own config file with a password in it, and its own idea of what "read-only" means. None of them has any idea what an MCP server is, so the moment you want Claude Code or Cursor to look at a database, you are either pasting query results into the chat or wiring up a one-off MCP server for that one engine.

tobira (扉, Japanese for door) is my answer to that. It is one Go binary that is, at the same time:

one CLI: tobira query runs a statement on any of 31 databases and prints a table, JSON, NDJSON or CSV, with exit codes a script can trust;
one terminal UI: browse, filter, edit rows, run queries and tail changes with the same keys everywhere;
one AI assistant: tobira ask proposes a query or answers a question by running read-only queries itself, on Gemini, Claude, OpenAI, Vertex AI or a local Ollama model;
one MCP server: tobira mcp exposes the connections you name to Claude Code, Claude Desktop, Cursor or VS Code, read-only by default;
one agent skill: npx skills add xavidop/tobira teaches your coding agent every command, flag and driver.

And underneath all five there is one read-only guard that the query text cannot lift, which is what makes it sane to hand a database to a model in the first place.

This article is the technical tour. Everything in it ran against tobira 1.4.1 on Go 1.27.

Why: one contract instead of five clients

Every database client is a vertical slice: its own connection handling, its own output, its own shell. tobira turns that into horizontal layers. Every database is a driver that implements one small Go contract. The CLI, the TUI, the AI assistant and the MCP server are written once, against that contract, not once per database.

	Without tobira	With tobira
Clients	`psql`, `mongosh`, `redis-cli`, a vector DB dashboard, a GUI	one binary, `tobira`
Scripting	each client's output format and exit codes	one `--format` flag (table, JSON, NDJSON, CSV) and exit codes 0 to 4 on every driver
Passwords	in DSNs, shell history and GUI config files	`env:`, `file://`, dotenv or OS keyring references, never a literal
Read-only	a database user, if someone set one up	a hard guard per driver plus whatever the server offers
AI	paste results into a chat	`tobira ask`, with schema context and read-only tools
Agents	a custom MCP server per engine, or none	`tobira mcp --expose <names>`, one server for every driver
Audit	nothing	`tobira history`, including what each MCP client ran

The driver contract is deliberately tiny. A Driver opens connections, and a Conn can Query, Ping, Close, list Namespaces and Collections and Describe a collection. Everything else (browse with paging, inline editing, explain, transactions, completion, export, change feeds, vector search, time-series reads) is an optional capability, an interface the connection may or may not implement, discovered at runtime:

if b, ok := driver.As[driver.Browser](conn); ok { ... }

Adding a capability never breaks a driver. The TUI asks each connection what it offers and shows only that: :tx appears where there are transactions, :similar on vector stores, :chart on time series.

The result is 31 drivers in the binary today: SQLite, PostgreSQL (with pgvector), MySQL, MariaDB, SingleStore, CockroachDB, Redshift, Greenplum, SQL Server, Oracle, libSQL/Turso, Cloudflare D1, ClickHouse, BigQuery, Snowflake, Vertica, MongoDB, Firestore, Cosmos DB, Redis, DynamoDB, Cassandra, Neo4j, Elasticsearch, OpenSearch, InfluxDB, Qdrant, Pinecone, Weaviate, Chroma and Milvus, plus DuckDB as an external plugin. Each driver is its own Go module, so the root module never imports a database SDK, and a plugin can be written in any language that speaks gRPC.

What We Are Building

A small shop database in SQLite with customers and orders, and then every door into it: tobira query from a script, the read-only guard trying to stop a delete, tobira ask answering a revenue question on Gemini, tobira mcp serving the database to a headless Claude Code session that is asked to delete rows and cannot, and finally tobira's driver layer embedded in a 60-line Go program.

SQLite is the demo because it needs no server. Every command below works the same on PostgreSQL, MongoDB or Qdrant: swap the DSN.

What You Need

tobira 1.4.1 or newer (Go 1.26+ if you build from source).
sqlite3 to seed the demo file.
A GEMINI_API_KEY for the AI steps (or an OpenAI or Anthropic key, or Ollama running locally).
Claude Code, Cursor, Claude Desktop or VS Code for the MCP step.

Step 1: Install and seed the demo

brew install xavidop/tap/tobira        # macOS
# Windows: scoop bucket add xavidop https://github.com/xavidop/scoop-bucket && scoop install tobira
# Linux: .deb, .rpm, .apk or a tarball from https://github.com/xavidop/tobira/releases
tobira version

tobira 1.4.1 (efcb4bb9bb162ede77bf811e31278b6614b7a2fc, 2026-10-23T17:20:10Z)

Seed a database:

sqlite3 shop.db <<'SQL'
create table customers (id integer primary key, name text not null, email text, country text not null, created_at text not null);
create table orders (id integer primary key, customer_id integer not null references customers(id), total_cents integer not null, status text not null, created_at text not null);
insert into customers values (1,'Ada Lovelace','ada@example.com','GB','2026-09-02'),(2,'Grace Hopper','grace@example.com','US','2026-09-10'),(3,'Linus Torvalds',NULL,'FI','2026-09-15'),(4,'Margaret Hamilton','margaret@example.com','US','2026-10-01'),(5,'Ken Thompson','ken@example.com','US','2026-10-20');
insert into orders values (1,1,4999,'paid','2026-10-21'),(2,1,1250,'paid','2026-10-22'),(3,2,8900,'refunded','2026-10-22'),(4,3,2300,'paid','2026-10-23'),(5,4,15000,'pending','2026-10-24'),(6,2,720,'paid','2026-10-25'),(7,5,3100,'paid','2026-10-26');
SQL

Then name the connection in a config file. Connections are read-write by default; this one is read-only on purpose, because it is the one the AI and the agent will get:

# config.yaml  (default location: ~/.config/tobira/config.yaml)
connections:
  - name: shop
    driver: sqlite
    dsn: sqlite:./shop.db
    read_only: true
ai:
  provider: googleai            # googleai, vertexai, anthropic, openai or ollama
  model: gemini-3.8-flash
  api_key: env:GEMINI_API_KEY   # a secret reference, never the key itself
  send_rows: ask                # ask, always or never

A literal API key or password in that file is a load error, not a warning. Passwords are env:NAME, file:///path, dotenv://.env#KEY or keyring:NAME references, resolved by mamori when the connection opens, and the resolved value is held as a driver.Secret whose String, Format, LogValue and MarshalJSON all print [redacted]. Only Reveal() returns it, and only the driver calls that.

tobira doctor checks all of it, as JSON when stdout is not a terminal, so it belongs in CI:

tobira --config config.yaml doctor

{"section":"connections","name":"shop","status":"ok","detail":"reachable in 1ms (read-only open, driver sqlite)"}
{"section":"refs","name":"ai.api_key","status":"ok","detail":"env ref resolves"}
{"section":"ai","name":"model","status":"ok","detail":"gemini-3.8-flash"}
{"section":"mcp","name":"mcp","status":"ok","detail":"not configured; nothing is exposed"}
{"section":"permissions","name":"config","status":"warn","detail":"0644 is group-readable, world-readable; run chmod 600 config.yaml"}

Doctor exits non-zero on that warning. chmod 600 config.yaml and it is green.

Step 2: One query command for scripts and CI

tobira query takes a connection name or a raw DSN, a statement in the database's own language, and prints the result:

tobira --config config.yaml query shop \
  "select c.name, count(o.id) as orders, sum(o.total_cents)/100.0 as total
   from customers c join orders o on o.customer_id=c.id group by c.id order by total desc"

name               orders  total
─────────────────  ──────  ─────
Margaret Hamilton  1       150
Grace Hopper       2       96.2
Ada Lovelace       2       62.49
Ken Thompson       1       31
Linus Torvalds     1       23

5 rows · <1ms

The table is the terminal default. Pipe it anywhere and the default becomes JSON, with typed columns and stats, so jq never has to guess:

tobira --config config.yaml query shop "select id, name, country from customers where country='US'" --format json

{
  "columns": [{"name": "id", "type": "int"}, {"name": "name", "type": "string"}, {"name": "country", "type": "string"}],
  "rows": [
    {"id": 2, "name": "Grace Hopper", "country": "US"},
    {"id": 4, "name": "Margaret Hamilton", "country": "US"},
    {"id": 5, "name": "Ken Thompson", "country": "US"}
  ],
  "stats": {"rows_affected": -1, "duration_ms": 0, "truncated": false}
}

--format csv and --format ndjson do what they say, --output orders.csv streams every row of a result of any size into a file, --explain prints the plan instead of running the query, -f - reads the statement from stdin, and --timeout bounds the whole call, connecting included.

The exit codes are the same on every driver, which is what makes the command usable in a pipeline:

Code	Meaning
`0`	Success
`1`	Usage error: bad flags, unknown connection, bad config
`2`	Could not open the connection: unreachable, bad credentials, unresolved secret
`3`	Any other query error: syntax, not found, timeout
`4`	Read-only violation, permission denied, or refused by policy

Errors go to stderr as one error: <message> line, always through the redactor, so a DSN password never ends up in a CI log.

Step 3: The read-only guard

This is the piece everything else leans on. Try a write on the shop connection:

tobira --config config.yaml query shop "delete from orders"; echo "exit=$?"

error: sqlite: query: only SELECT, WITH, VALUES, EXPLAIN and read-only PRAGMA statements are allowed on a read-only connection: read-only violation
exit=4

The same --read-only flag works on a bare DSN with no config at all, and smuggling a write behind a read does not work either, because the SQL drivers take exactly one statement per query:

tobira query sqlite:./shop.db "update orders set status='paid'" --read-only; echo "exit=$?"
# exit=4
tobira --config config.yaml query shop "select 1; delete from orders"; echo "exit=$?"
# error: sqlite: query: multiple statements are not supported yet: invalid query
# exit=3

The guard is not a regex in front of the connection. Each driver declares CapReadOnly in its capabilities, and that declaration is a contract: a query sent with ReadOnly must fail with ErrReadOnly and change nothing, whatever the text says. Each driver enforces it with the strongest mechanism its database has, with a statement check in front of that:

SQLite: the file is opened on a read-only handle, PRAGMA query_only=1 is re-asserted before every statement on the pooled connection (earlier text could have lifted it), ATTACH is disabled through SQLITE_LIMIT_ATTACHED=0, and the statement must start with an allow-listed keyword.
PostgreSQL: every call runs inside its own BEGIN READ ONLY and always ends with ROLLBACK on a fresh context, so a cancelled query can never leave a transaction open. Because BEGIN READ ONLY does not stop COPY ... TO PROGRAM, DO blocks or lo_export from touching the server's filesystem, those are refused lexically before they run, and the lexer is run with every string-quoting dialect PostgreSQL can be configured with.
CockroachDB and Redshift: an allow-list, since EXPORT, BACKUP, UNLOAD and crdb_internal functions reach outside the database without writing to it.
MySQL: an allow-list of statement kinds, then a read-only transaction on its own connection, then a post-check that the statement stayed inside it (MySQL 8.0 to 8.4).
MongoDB: the command document is checked as sent against an allow-list of read commands; $out, $merge and write-concern fields are refused.
Redis: an allow-list of read commands checked against the server's own COMMAND INFO flags, so a renamed command is still refused.
Vector stores: queries never write, and anything that would send your data to an outside embedding service from the server side (Weaviate nearText, Milvus TEXTEMBEDDING, Qdrant document inputs) is refused on a read-only path too.

Every driver except SingleStore, which has no read-only transaction and therefore refuses read-only connections outright rather than pretending, declares the guard. The security page lists, per driver, the one thing the guard cannot see: code that already lives on the server, such as user-defined functions or linked servers. The guard stops accidental and model-generated writes. A least-privilege database user is still the real boundary, and the two together are what you want before Step 5.

Step 4: Ask in plain language

With the ai: section in place, tobira ask has three modes. The default proposes a query and runs nothing:

tobira --config config.yaml ask shop "which customers spent the most, with their order count" --collection orders

```sqlite
SELECT c.id, c.name, COUNT(o.id) AS order_count, SUM(o.total_cents) AS total_spent_cents FROM customers c JOIN orders o ON c.id = o.customer_id GROUP BY c.id, c.name ORDER BY total_spent_cents DESC
```

This query aggregates orders by customer, calculating the total amount spent (in cents) and the total number of orders placed. It then sorts the customers in descending order by total spend so the top-spending customers appear first.

googleai/gemini-3.8-flash · 483 in / 118 out · 5.4s

--run proposes and then runs the proposal read-only, exactly as tobira query --read-only would:

tobira --config config.yaml ask shop "the three newest paid orders with the customer name" --run --format table

id  customer_name   total_cents  status  created_at
──  ──────────────  ───────────  ──────  ──────────
7   Ken Thompson    3100         paid    2026-10-26
6   Grace Hopper    720          paid    2026-10-25
4   Linus Torvalds  2300         paid    2026-10-23

3 rows · <1ms

--answer is the agentic mode: the model gets the read-only tools (list_collections, describe, sample, query, explain) and loops until it has an answer. --send-rows lets the row data reach the model for this run; without it, under send_rows: ask, the model gets column names and counts only.

tobira --config config.yaml ask shop \
  "how much revenue did we get in October 2026 from paid orders, and from which country does most of it come?" \
  --answer --send-rows

In October 2026, total revenue from paid orders was **12,369 cents** (**$123.69**), and most of it came from **GB** (6,249 cents / $62.49).

| Country | Revenue (cents) | Revenue ($) |
|---|---|---|
| **GB** | 6,249 | $62.49 |
| **US** | 3,820 | $38.20 |
| **FI** | 2,300 | $23.00 |

Queries run:
  SELECT DISTINCT status FROM orders (3 rows)
  SELECT created_at FROM orders WHERE created_at LIKE '2026-10%' LIMIT 5 (5 rows)
  SELECT SUM(total_cents) AS total_cents, ROUND(SUM(total_cents) / 100.0, 2) AS total_dollars FROM orders WHERE status = 'paid' AND created_at >= '2026-10-01' AND created_at < '2026-11-01' (1 row)
  SELECT c.country, SUM(o.total_cents) AS total_cents, ... GROUP BY c.country ORDER BY total_cents DESC (3 rows)
  ...

googleai/gemini-3.8-flash · 13.1k in / 812 out · 24.5s

Notice what the model did first: it checked which status values exist and what the dates look like before trusting its own assumptions. That is the schema context at work. Each request carries the collection in scope, its foreign-key neighbours and the other collection names, with columns, types, keys and indexes, but never DDL (defaults and comments can hold secrets) and never sample rows unless send_rows allows it.

A few design decisions worth knowing, because they are the difference between a demo and something you run against a real database:

The assistant is built on Genkit Go and lives in its own module, so the core of tobira has no model SDK. The four flows (askFlow, answerFlow, explainResultsFlow, fixErrorFlow) are Dotprompt files embedded in the binary, and a file of the same name in ~/.config/tobira/prompts/ replaces the built-in one without a rebuild. genkit start -- tobira ai dev opens them in the Genkit Dev UI.
The query tools are only offered on connections whose driver declares the read-only guard, and every call carries the read-only flag, a 30 second timeout and a row cap, whatever the model asks for. The driver stops a write, not the model's manners.
The tools reach only the connection of the conversation. Any other name gets the same unknown connection error as one that does not exist.
Prompt injection from data is made hard structurally: every piece of data is wrapped in a fence named with a random per-request nonce (<data-NONCE kind="schema">), the prompt says only fenced text is data, and Dotprompt's role markers are broken in every input so a row cannot open a new message.
The only tool that can write, execute, exists in the TUI only, only after :ai execute on on that connection, and asks you to confirm every statement, after you have scrolled to the end of it.

The same assistant is a panel in the TUI (a opens it, ctrl+t switches between propose and answer, F5 loads the proposal into the editor and runs it, f on a failed run asks the AI to fix the query).

Step 5: Hand it to your agent with `tobira mcp`

This is the part I think changes how you work. tobira mcp is a Model Context Protocol server over stdio (or Streamable HTTP with --http), and it serves the exact same tools the assistant uses, on the connections you name and nothing else:

claude mcp add tobira -- tobira mcp --expose shop

Or, in a .cursor/mcp.json, .vscode/mcp.json or claude_desktop_config.json:

{"mcpServers": {"tobira": {"command": "tobira", "args": ["mcp", "--expose", "shop"]}}}

Nothing is exposed until you name it. tobira mcp with no --expose and no mcp.expose in the config exits with code 1 rather than serving everything. On a raw JSON-RPC session the server advertises describe, explain, list_collections, list_connections, list_namespaces, query and sample, plus similar when an exposed driver can do vector search. execute is not in the list at all unless a connection is in allow_writes:

--- tools/call list_connections
[{"name":"shop","driver":"sqlite","kind":"sql","dialect":"sqlite","read_only":true,"status":"idle"}]

--- tools/call describe {"connection":"shop","collection":"orders"}
{"namespace":"main","collection":"orders","type":"table","fields":[{"name":"id","type":"INTEGER","nullable":true,"auto_increment":true},{"name":"customer_id","type":"INTEGER","nullable":false},{"name":"total_cents","type":"INTEGER","nullable":false},{"name":"status","type":"TEXT","nullable":false},{"name":"created_at","type":"TEXT","nullable":false}],"primary_key":["id"],"foreign_keys":[{"fields":["customer_id"],"references":"main.customers","ref_fields":["id"]}]}

--- tools/call query {"connection":"shop","query":"select status, count(*) as n, sum(total_cents) as cents from orders group by status"}
{"columns":[{"name":"status","type":"TEXT"},{"name":"n","type":"int"},{"name":"cents","type":"int"}],"rows":[["paid",5,12369],["pending",1,15000],["refunded",1,8900]],"truncated":false}

--- tools/call query {"connection":"shop","query":"delete from orders where status = 'refunded'"}
sqlite: query: only SELECT, WITH, VALUES, EXPLAIN and read-only PRAGMA statements are allowed on a read-only connection: read-only violation
(isError: true)

--- tools/call execute {"connection":"shop","statement":"delete from orders"}
{"code": -32602, "message": "tool 'execute' not found: tool not found"}

--- tools/call query {"connection":"prod","query":"select 1"}
unknown connection "prod": call list_connections for the names
(isError: true)

Every collection is also an MCP resource at tobira://<connection>/<namespace>/<collection>/schema, so a client can attach a schema to its context without spending a tool call.

Now the real test: a headless Claude Code session, with tobira as its only MCP server, asked to do something it must not be able to do.

claude -p --mcp-config mcp.json --strict-mcp-config --allowedTools "mcp__tobira__*" \
  "Using only the tobira MCP tools: which customer has the highest paid revenue in the shop database, and how much? Then try to delete the refunded orders and tell me exactly what happened."

**Highest paid revenue:** Ada Lovelace (customer id 1) has the most, at 6,249 cents, which is **$62.49**.
I counted only orders with status `paid`, and the currency is my assumption since the schema doesn't
name one. The next highest are Ken Thompson at $31.00, Linus Torvalds at $23.00 and Grace Hopper at $7.20.

**Deleting the refunded orders:** The delete did not go through, and nothing was changed.
- **What I ran:** `DELETE FROM orders WHERE status='refunded'`, sent through the tobira `query` tool.
- **What came back:** `sqlite: query: only SELECT, WITH, VALUES, EXPLAIN and read-only PRAGMA statements
  are allowed on a read-only connection: read-only violation`.
- **Why:** The `shop` connection is flagged `read_only: true`. The tobira tools have no write tool, so
  there was no other way to delete.
- **What I saw before the attempt:** The only refunded order is 1 row, worth 8,900 cents ($89.00).
  I didn't re-query afterwards, but the refusal means it should still be there.

I didn't try to get around the read-only setting. If you want the delete done, you'd need a writable
connection or to run the statement yourself.

The agent explored the schema, wrote a correct join, got the right answer, attempted the delete, was refused by the driver, and reported it honestly. And every one of those calls is in the local history, with the client's name and version from its initialize request:

tobira history --mcp --format table --limit 7

at                    conn  source  client               ok     rows  ms  query
────────────────────  ────  ──────  ───────────────────  ─────  ────  ──  ────────────────────────────────────────────
2026-10-27T23:02:32Z  shop  mcp     claude-code/2.1.308  false  0     0   DELETE FROM orders WHERE status='refunded'
2026-10-27T23:02:29Z  shop  mcp     claude-code/2.1.308  true   4     0   SELECT c.id, c.name, SUM(o.total_cents) paid_cents FROM cus…
2026-10-27T23:02:28Z  shop  mcp     claude-code/2.1.308  true   3     0   SELECT status, COUNT(*) n, SUM(total_cents) cents FROM orde…
2026-10-27T23:02:26Z  shop  mcp     claude-code/2.1.308  true   0     1   describe {"collection":"orders","connection":"shop"}
2026-10-27T23:02:25Z  shop  mcp     claude-code/2.1.308  true   0     0   describe {"collection":"customers","connection":"shop"}
2026-10-27T23:02:22Z  shop  mcp     claude-code/2.1.308  true   0     11  list_collections {"connection":"shop"}
2026-10-27T23:02:18Z        mcp     claude-code/2.1.308  true   0     2   list_connections

The history is a local SQLite file with mode 0600, the last 10,000 entries, credentials masked, and tobira history clear plus a retention setting when you want less of it.

The policy knobs, all in the mcp: section or as flags:

mcp:
  expose: [shop, cache]       # the only connections a client can see
  allow_writes: [cache]       # a subset of expose; execute exists only for these
  max_rows: 200               # 1 to 1000, per tool result
  send_rows: always           # always or never (no "ask": nobody is there to answer)
  token: env:TOBIRA_MCP_TOKEN # HTTP mode only

Two things to internalise. First, send_rows defaults to always here, unlike the assistant's ask, because an MCP client runs unattended; if the data must not reach a model, do not expose the connection. Second, execute has no per-call confirmation, because there is nobody to confirm; allow writes only on a scratch database an agent is allowed to break.

HTTP mode (--http 127.0.0.1:8765) is for clients that cannot start a process. It binds to loopback unless you pass --insecure-bind, needs a bearer token on every request, and checks Host and Origin against DNS rebinding.

Step 6: Teach the agent the CLI itself

The MCP server gives an agent tools. The agent skill gives it knowledge: a SKILL.md that covers every command, flag and exit code, the driver model, each driver's DSN schemes and capabilities, the config and secrets rules, the MCP setup and how to write a driver. Install it into your project with the skills CLI:

npx skills add xavidop/tobira

After that, Claude Code, Cursor or Copilot know that tobira query returns exit 4 on a read-only violation, that SingleStore has no read-only guard, that go install ...@latest does not work for the CLI module because it depends on its sibling modules, and that tobira doctor prints JSON off a terminal. The skill is the same file for every agent, and it is versioned with the project, so it never drifts from the binary.

MCP for the data, a skill for the tool. That combination is what makes tobira agent-ready rather than merely AI-flavoured: the agent can both query your databases and drive the CLI correctly in a script it writes for you.

Step 7: Use it as a library, or add a database

The driver layer is importable. github.com/xavidop/tobira/api is the stable, stdlib-only surface (type aliases over the driver package), and each driver is a module you link in. A private registry, a read-only open, a streamed query and a refused write in one small program:

package main

import (
    "context"
    "errors"
    "fmt"
    "strings"

    "github.com/xavidop/tobira/api"
    "github.com/xavidop/tobira/drivers/sqlite"
)

func main() {
    ctx := context.Background()
    reg := api.NewRegistry()
    if err := reg.Add(sqlite.New()); err != nil {
        panic(err)
    }
    conn, err := reg.Open(ctx, api.ConnConfig{DSN: "sqlite:./shop.db", ReadOnly: true})
    if err != nil {
        panic(err)
    }
    defer conn.Close()

    fmt.Println("capabilities:", api.CapabilitiesOf(conn))
    run(ctx, conn, "select name, country from customers order by id")
    run(ctx, conn, "delete from customers") // refused: the connection is read-only
}

func run(ctx context.Context, conn api.Conn, text string) {
    rs, err := conn.Query(ctx, api.Query{Text: text, ReadOnly: true, Limit: 3})
    if err != nil {
        fmt.Printf("%s -> %v (read-only: %t)\n", text, err, errors.Is(err, api.ErrReadOnly))
        return
    }
    defer rs.Close()
    for {
        row, ok, err := rs.Next(ctx)
        if err != nil || !ok {
            break
        }
        var cells []string
        for _, v := range row {
            cells = append(cells, v.String())
        }
        fmt.Println(strings.Join(cells, "\t"))
    }
    fmt.Printf("(truncated: %t)\n", rs.Stats().Truncated)
}

capabilities: [browser mutator explainer transactor completer]
Ada Lovelace    GB
Grace Hopper    US
Linus Torvalds  FI
(truncated: true)
delete from customers -> sqlite: query: only SELECT, WITH, VALUES, EXPLAIN and read-only PRAGMA statements are allowed on a read-only connection: read-only violation (read-only: true)

Values are a tagged union (driver.Value) rather than any, so a row can cross a plugin boundary unchanged. Results stream, so a large result never has to fit in memory. Errors are *driver.Error values classified by sentinels (ErrReadOnly, ErrNotFound, ErrSyntax, ErrUnsupported) you match with errors.Is.

Adding a database is the same contract in the other direction. A driver module is a Go package that implements Driver and Conn, registers from init like database/sql, and passes the drivertest conformance kit in strict mode, which probes a declared read-only guard with real writes. The in-repo example is about 150 lines. An external plugin is that same driver served over gRPC through hashicorp/go-plugin with one line of main:

func main() { sdk.Serve(memdemo.New()) }

go build -o tobira-driver-memdemo .
tobira driver install ./tobira-driver-memdemo --sha256 "$(shasum -a 256 tobira-driver-memdemo | cut -d' ' -f1)"
tobira driver info memdemo

tobira driver install requires the checksum and never runs the binary. Connections and transactions live inside the plugin behind unguessable handles, a panic in one call fails only that call, and if tobira dies the plugin notices within a second and exits. The wire protocol is plain protobuf over gRPC, so a driver can be written in any language. DuckDB ships this way because it needs cgo and tobira's releases do not.

Conclusion

The thing that makes tobira more than a nicer psql is the layering. One driver contract means the CLI, the TUI, the assistant and the MCP server are each written once and work on 31 databases. One read-only guard, declared and enforced per driver, means every one of those doors can be opened for a model without a leap of faith. One history means you can see, afterwards, exactly what the model or the agent ran.

If you only take one thing from this article, take Step 5: tobira mcp --expose <name> on a read_only: true connection, then ask your coding agent a question about your data. It is the same tool you use at the keyboard, with the same guard, and it took one line to set up.

tobira is MIT licensed and lives at github.com/xavidop/tobira. The best contribution is a driver for a database you use; open an issue first and start from the example module.

A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)

Xavier Portilla Edo — Wed, 21 Oct 2026 11:44:29 +0000

Introduction

The previous article put Genkit flows in Kubernetes pods with senro and let jev, a decision model, route a support ticket. The decisions there were small: which team, how urgent, send or hold.

This one hands jev something bigger to judge: software. A software factory takes a feature request and produces a Go module: a design, a plan, the code, a green build, passing tests and a pull request description. Gemini writes every one of those. jev decides, at every gate, whether the work moves on, goes back, or stops for a human. Each stage runs as a pod, each stage is a Genkit flow, and the whole thing is a few hundred lines of Go.

Everything below ran against the real Gemini API (gemini-3.8-flash) and the real TypeSafe API (jev-latest), on Genkit Go v1.14.0, senro v1.4.5 and a local kind cluster. The code is in genkit-k8s-flows-jev-software-factory.

Why a factory needs a judge, not a second writer

The usual way to review model-written code is to ask another chat model. It works, up to a point. The reviewer writes a paragraph, you parse the paragraph, the paragraph is sometimes wrong in the same way the code is, and a well-written pull request description can talk the reviewer into "looks good to me".

A decision model does not write. It takes a state (the request, the plan, the code, the test log) and answers typed questions with probabilities, in a few hundred milliseconds. There is no prose to parse and nothing to be talked into: BuildGreen is a number, and a red test log drags it down whatever the pull request says.

	Chat model as reviewer	Decision model as reviewer
Verdict	A paragraph you parse	`Choice[CodeVerdict]`: `ship`, `rework` or `human`, with a probability each
Evidence	Mentioned, maybe	A `Noul` per question: covers the request, tests are meaningful, build is green
Feedback for the next round	Whatever it felt like writing	Each question that scored "yes" is one line of feedback
Latency and cost	Seconds, thousands of output tokens	Milliseconds, tokens for the state only
Can be persuaded by the author	Yes	No: it never reads instructions, only state

The split that falls out of this is the whole design: Gemini builds, jev decides. And senro is what gives each build step a pod, a workspace, a log, a retry policy and a secret, and each decision a place in the graph.

What we are building

A feature request goes in. A reviewed Go module comes out, or a brief for a human saying why not.

The factory is three senro pipelines and a small state machine over them:

Run	Step	Where	Who	What it leaves behind
`design`	`clean`	coordinator	none	an empty workspace with `request.md` in it
`design`	`design`	pod	Gemini and jev, in a loop	`design.json`, `DESIGN.md` and one record per review round
`design`	`plan`	pod	Gemini	`plan.json`, `PLAN.md`
`implement`	`implement`	pod	Gemini	the module's files
`implement`	`vet`, `test`	pod	`go vet`, `go test`	`vet.log`, `test.log` and their exit codes
`implement`	`review`	pod	jev	`review.json`: ship, rework or human
`rework`	`fix`, then `vet`, `test`, `review`	pod	Gemini, Go, jev	the same, after a fix
`finish`	`pr` then `publish`	pod, coordinator	Gemini	`PR.md` and the module under `out/<request>/src`
`finish`	`brief` then `notify`	pod, coordinator	Gemini	`BRIEF.md` for a human

jev decides three times. Inside the design step, it reviews each design Gemini writes and the loop revises until jev says ready, says reject, or three rounds are up. After test, it reviews the code with the logs in hand and picks ship, rework or human. And the state machine in main turns each verdict into the next senro run, with rework bounded at two.

What you need

Go 1.27 or newer, Docker, kind and kubectl
GEMINI_API_KEY and TYPESAFE_API_KEY in your environment
the senro CLI if you want the terminal UI (optional)

go get github.com/firebase/genkit/go@v1.14.0
go get github.com/xavidop/senro@v1.4.5
go get github.com/xavidop/mamori

Step 1: a cluster and an image with a Go toolchain

The cluster setup is the one from the previous article: kind, a namespace, a ServiceAccount with the permissions senro's Kubernetes executor needs (including get on pods/exec), and three environment variables, since senro never reads your kubeconfig. The repo's scripts/cluster-up.sh does it and source scripts/cluster-env.sh exports them.

The one difference is the image. A func step needs only sh and tar in the pod, but this factory also runs go vet and go test as plain command steps, so every step uses the official Go image, pinned to its digest as senro requires:

const stepImage = "docker.io/library/golang:1.27-alpine@sha256:f2c872bcee949b7be9c490efd02b48a8a2fbf947cdf533f1c58525ec9d8170fd"

func pods() senro.ExecutorTarget {
    return k8s.Pod(stepImage,
        k8s.Namespace("factory"),
        k8s.Resources(
            map[string]string{"cpu": "500m", "memory": "512Mi"},
            map[string]string{"cpu": "2", "memory": "2Gi"},
        ))
}

Your pipeline binary is still not in that image. senro cross-compiles it and sends it into each pod, exactly as before.

Step 2: the design loop, with jev inside the flow

The first gate is the one that saves the most money: a bad design produces bad code, a green build and a confident pull request. So the design is reviewed before anything is planned, and the review is a loop inside one Genkit flow.

The questions are a Go type. Every yes/no question is one a revision can act on: a "yes" becomes a line of feedback for the next round.

type DesignVerdict string

func (DesignVerdict) Criteria() map[DesignVerdict]string {
    return map[DesignVerdict]string{
        "ready":  "Complete, unambiguous, scoped to the request, and testable as described",
        "revise": "The right direction, with gaps a revision can close",
        "reject": "The request is too vague or too large to design from; a human has to clarify it first",
    }
}

type Risk int

func (Risk) Levels() []string { return []string{"Low", "Medium", "High"} }

type DesignReview struct {
    Verdict        systemonex.Choice[DesignVerdict] `json:"verdict" jsonschema_description:"Is this design ready to implement?"`
    RequestClear   systemonex.Noul                  `json:"request_clear" jsonschema_description:"Does the request say what to build precisely enough that two engineers would build the same thing?"`
    ScopeCreep     systemonex.Noul                  `json:"scope_creep" jsonschema_description:"Does the design add things the request did not ask for?"`
    Underspecified systemonex.Noul                  `json:"underspecified" jsonschema_description:"Is any component described too loosely to implement without guessing: missing fields, signatures, edge cases or error behaviour?"`
    MissingTests   systemonex.Noul                  `json:"missing_tests" jsonschema_description:"Does the test plan leave any requirement of the request untested?"`
    Risk           systemonex.Score[Risk]           `json:"risk" jsonschema_description:"How risky is implementing this design as written?"`
}

And the loop. Gemini designs; jev reviews; if the verdict is revise, Gemini gets the previous design plus the feedback and tries again:

f.Design = genkit.DefineFlow(g, "designFeature", func(ctx context.Context, request string) (DesignResult, error) {
    var res DesignResult
    feedback := ""
    for round := 1; round <= maxDesignRounds; round++ {
        prompt := "Request:\n" + request
        if round > 1 {
            prompt += "\n\nYour previous design:\n" + asJSON(res.Design) + "\n\nReviewer feedback:\n" + feedback
        }
        d, _, err := genkit.GenerateData[Design](ctx, g,
            ai.WithModelName("googleai/gemini-3.8-flash"),
            ai.WithSystem("You are a senior Go engineer writing a short design for exactly what the request asks. "+
                "Stay inside the request: no extra features. Components are the Go types and functions you will create. "+
                "The test plan names concrete, deterministic test cases. Be brief."),
            ai.WithPrompt("%s", prompt))
        if err != nil {
            return res, err
        }
        res.Design = *d

        out, resp, err := genkit.GenerateData[DesignReview](ctx, g,
            ai.WithModelName("typesafe/jev-latest"),
            ai.WithSystem("The state has a feature request and a design written for it."),
            ai.WithPromptParts(ai.NewDataPart(map[string]any{"request": request, "design": res.Design})))
        if err != nil {
            return res, err
        }

        // jev's verdict, then our rules on top.
        verdict := string(out.Verdict.Choice)
        var notes []string
        if out.RequestClear.Probability < 0.4 {
            verdict = "reject" // nobody should design from a request two engineers would read differently
        }
        if out.ScopeCreep.Probability >= 0.5 {
            notes = append(notes, "remove everything the request did not ask for")
        }
        if out.Underspecified.Probability >= 0.5 {
            notes = append(notes, "specify every component fully: fields, signatures, edge cases and error behaviour")
        }
        if out.MissingTests.Probability >= 0.5 {
            notes = append(notes, "add a test case for every requirement in the request")
        }
        if verdict == "ready" && len(notes) > 0 {
            verdict = "revise"
        }
        res.Rounds = append(res.Rounds, DesignRound{Round: round, Verdict: verdict,
            Ready: out.Verdict.Probabilities["ready"], Feedback: notes /* ... */})
        res.Verdict = verdict
        if verdict != "revise" {
            break
        }
        feedback = fmt.Sprintf("The reviewer rated the design %q (probability of ready %.2f). Do the following:\n- %s",
            verdict, out.Verdict.Probabilities["ready"], strings.Join(notes, "\n- "))
    }
    return res, nil
})

Two things to notice. The verdict is jev's, but the rules around it are yours: a request that two engineers would read differently is rejected whatever the design looks like, and a design with a flagged problem is not ready even if jev leaned that way. And the feedback is built from the questions, not from prose: a decision model never writes a sentence, so the sentences live in your code, one per question, and the next round gets exactly the ones that applied.

The whole loop is one flow, so it is one senro step, one pod, one log, with every round recorded in design.json.

Step 3: plan, implement, and let Go have its say

The plan and the implementation are ordinary Gemini calls with typed outputs: a Plan with files and steps, then a Code with every file in full. The step writes the files into the workspace, refusing anything that would escape it:

type Code struct {
    Files []File `json:"files"`
    Notes string `json:"notes,omitempty"`
}

func writeCode(ctx senro.Ctx, code Code) error {
    root := wsPath(ctx, ".")
    for _, f := range code.Files {
        rel := filepath.Clean(f.Path)
        if rel == "." || strings.HasPrefix(rel, "..") || filepath.IsAbs(rel) || strings.HasPrefix(rel, factoryDir) {
            return fmt.Errorf("refusing to write outside the module: %q", f.Path)
        }
        path := filepath.Join(root, rel)
        if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
            return err
        }
        if err := os.WriteFile(path, []byte(f.Content), 0o644); err != nil {
            return err
        }
        fmt.Fprintf(ctx.Stdout(), "wrote %s (%d bytes)\n", rel, len(f.Content))
    }
    return nil
}

Then the compiler and the tests run, as command steps in the same pod image, with no model anywhere near them:

func goStep(w *senro.WorkflowBuilder, id, after, cmd string, ws *senro.WorkspaceRef) *senro.StepBuilder {
    script := "mkdir -p .factory; " + cmd + " > .factory/" + id + ".log 2>&1; echo $? > .factory/" + id + ".exit; " +
        "cat .factory/" + id + ".log; exit $(cat .factory/" + id + ".exit)"
    return w.Step(id, exec.Command("sh", "-c", script)).
        Needs(after).
        Mount(ws.At("/repo", senro.RW)).
        WorkDir("/repo").
        Env("HOME", "/tmp").
        Env("GOCACHE", "/tmp/gocache").
        Env("GOFLAGS", "-mod=mod").
        Env("GOPROXY", "off").
        Env("GOTOOLCHAIN", "local").
        Env("CGO_ENABLED", "0").
        Timeout(stepTimeout).
        Retry(3, retry.OnInfra()).
        ContinueOnError()
}

goStep(code, "vet", "implement", "go vet ./...", ws)
goStep(code, "test", "vet", "go test -count=1 ./...", ws)

Each one leaves its output and exit code under .factory/ in the workspace, so the reviewer can read them. GOPROXY=off and GOTOOLCHAIN=local keep the pod honest: the request said standard library only, and the pod cannot download anything to pretend otherwise. And ContinueOnError is the important one: a red test is a failed step, but the review still runs. jev gets to see why the build failed, which is exactly the evidence a rework needs.

Step 4: the code review gate

The second gate reads everything a human reviewer would open: the request, the plan, every source file, and what go vet and go test printed.

type CodeVerdict string

func (CodeVerdict) Criteria() map[CodeVerdict]string {
    return map[CodeVerdict]string{
        "ship":   "Builds, tests pass, does what the request asked, no obvious defects",
        "rework": "Fixable problems: failing or missing tests, a missed requirement, sloppy code",
        "human":  "Something a model should not decide: security, data loss, or the request was misunderstood",
    }
}

type Quality int

func (Quality) Levels() []string { return []string{"Poor", "Acceptable", "Good", "Excellent"} }

type CodeReview struct {
    Verdict         systemonex.Choice[CodeVerdict] `json:"verdict" jsonschema_description:"Should this change ship?"`
    Quality         systemonex.Score[Quality]      `json:"quality" jsonschema_description:"How good is the code?"`
    CoversRequest   systemonex.Noul                `json:"covers_request" jsonschema_description:"Does the implementation do everything the request asked for?"`
    TestsMeaningful systemonex.Noul                `json:"tests_meaningful" jsonschema_description:"Do the tests exercise the behaviour the request cares about, rather than just compile?"`
    BuildGreen      systemonex.Noul                `json:"build_green" jsonschema_description:"According to the logs and exit codes, did go vet and go test both pass?"`
}

The flow sends the whole ReviewInput as a data part, reads the answers, and applies the factory's rules on top. A red build never ships, whatever the overall verdict:

f.Review = genkit.DefineFlow(g, "reviewChange", func(ctx context.Context, in ReviewInput) (Review, error) {
    out, resp, err := genkit.GenerateData[CodeReview](ctx, g,
        ai.WithModelName("typesafe/jev-latest"),
        ai.WithSystem("The state has the feature request, the implementation plan, every source file, "+
            "and the output and exit codes of go vet and go test. Exit code 0 means success."),
        ai.WithPromptParts(ai.NewDataPart(in)))
    if err != nil {
        return Review{}, err
    }
    r := Review{
        Verdict:         string(out.Verdict.Choice),
        Confidence:      out.Verdict.Confidence,
        Quality:         out.Quality.Label(),
        CoversRequest:   out.CoversRequest.Probability,
        TestsMeaningful: out.TestsMeaningful.Probability,
        BuildGreen:      out.BuildGreen.Probability,
        Model:           systemonex.ResponseInfo(resp).Model,
    }
    if in.VetExit != 0 {
        r.Reasons = append(r.Reasons, "go vet failed")
    }
    if in.TestExit != 0 {
        r.Reasons = append(r.Reasons, "go test failed")
    }
    if r.CoversRequest < 0.6 {
        r.Reasons = append(r.Reasons, "the implementation does not cover the request")
    }
    if r.TestsMeaningful < 0.5 {
        r.Reasons = append(r.Reasons, "the tests do not exercise the requested behaviour")
    }
    if out.Quality.Level() == 0 {
        r.Reasons = append(r.Reasons, "code quality is poor")
    }
    if r.Verdict == "ship" && len(r.Reasons) > 0 {
        r.Verdict = "rework"
    }
    return r, nil
})

The Reasons are what the fix flow gets, together with the logs. Again: the decision model decides, your code says it in words.

Step 5: three pipelines and a state machine

The module under construction lives in a persistent workspace, so every run of one request works on the same tree. senro insists a persistent workspace is bounded, and refuses one without both bounds:

func repo() *senro.WorkspaceRef {
    return senro.Workspace("factory-repo",
        senro.Scope(senro.ScopePersistent),
        senro.MaxAge(24*time.Hour),
        senro.MaxSize(256<<20))
}

The three pipelines are short. Every AI step gets the same shape through a helper, with only the secrets it needs:

func aiStep(s *senro.StepBuilder, ws *senro.WorkspaceRef, secrets ...string) *senro.StepBuilder {
    s = s.Mount(ws.At("/repo", senro.RW)).Timeout(stepTimeout).Retry(3, retry.OnInfra())
    for _, name := range secrets {
        s = s.SecretEnv("SENRO_"+name, name)
    }
    return s
}

func buildPipeline(outDir string, rework bool) *senro.Pipeline {
    name, fn := "implement", "factory/implement"
    if rework {
        name, fn = "rework", "factory/fix"
    }
    p := senro.New(name)
    ws := repo()

    code := p.Workflow("code", senro.On(pods()))
    aiStep(code.Step(name, senro.Func(fn, NoParams{})), ws, "Gemini")
    goStep(code, "vet", name, "go vet ./...", ws)
    goStep(code, "test", "vet", "go test -count=1 ./...", ws)
    aiStep(code.Step("review", senro.Func("factory/review", NoParams{})), ws, "TypeSafe").Needs("test")

    handoff := p.Workflow("handoff", senro.Needs("code"), senro.On(senro.Local()))
    handoff.Step("collect", senro.Func("factory/collect", DirParams{Dir: outDir})).
        Mount(ws.At("/repo", senro.RO))

    return p
}

finish declares both endings and lets a run parameter pick one, the same senro.When pattern as the previous article:

func finishPipeline(outDir, reason string) *senro.Pipeline {
    p := senro.New("finish")
    ws := repo()

    ship := p.Workflow("ship", senro.On(pods()), senro.When(senro.ParamIs("verdict", "ship")))
    aiStep(ship.Step("pr", senro.Func("factory/pr", NoParams{})), ws, "Gemini")

    publish := p.Workflow("publish", senro.Needs("ship"), senro.On(senro.Local()),
        senro.When(senro.ParamIs("verdict", "ship")))
    publish.Step("publish", senro.Func("factory/publish", DirParams{Dir: outDir})).
        Mount(ws.At("/repo", senro.RO))

    escalate := p.Workflow("escalate", senro.On(pods()), senro.When(senro.ParamIs("verdict", "human")))
    aiStep(escalate.Step("brief", senro.Func("factory/brief", BriefParams{Reason: reason})), ws, "Gemini")
    // ...and a notify workflow on the coordinator that collects BRIEF.md
    return p
}

And main is the state machine. Each run ends with a record jev produced, and the record picks the next run:

state, reason, reworks := "design", "", 0
for {
    switch state {
    case "design":
        if err := run(designPipeline(request, out)); err != nil {
            exit(err)
        }
        var d DesignResult
        read(out, "design.json", &d)
        if d.Verdict == "ready" {
            state = "implement"
        } else {
            state, reason = "human", fmt.Sprintf("the design review ended %q after %d round(s)", d.Verdict, len(d.Rounds))
        }

    case "implement", "rework":
        // A red vet or test is a failed step, so the run is partial; the
        // review still ran (ContinueOnError) and its verdict is what counts.
        if err := run(buildPipeline(out, state == "rework")); err != nil && !partial(err) {
            exit(err)
        }
        var r Review
        read(out, "review.json", &r)
        switch r.Verdict {
        case "ship":
            state = "ship"
        case "rework":
            if reworks >= *maxRework {
                state, reason = "human", fmt.Sprintf("jev asked for rework %d times: %v", reworks+1, r.Reasons)
            } else {
                reworks++
                state = "rework"
            }
        default:
            state, reason = "human", fmt.Sprintf("the code review ended %q: %v", r.Verdict, r.Reasons)
        }

    case "ship", "human":
        if err := run(finishPipeline(out, reason), senro.WithParams(senro.Params{"verdict": state})); err != nil {
            exit(err)
        }
        return
    }
}

Every arrow in the state diagram is one of those cases, and every run is a directory under runs/ with its plan, its events and one log per step.

Step 6: run it

The first request is a token-bucket rate limiter, the kind of package every service ends up with:

# Feature request: a token-bucket rate limiter package

We need a small, dependency-free Go package `ratelimit` (module path
`github.com/acme/ratelimit`) that other services can import.

- `New(rate float64, burst int) *Limiter`: `rate` is tokens added per second,
  `burst` is the bucket capacity.
- `(*Limiter) Allow() bool`: takes one token if available, otherwise returns false
  without blocking.
- `(*Limiter) AllowN(n int) bool` for batches.
- Safe for concurrent use from many goroutines.
- Time must be injectable so tests are deterministic (no sleeping in tests).
- Standard library only. Tests included, run with `go test ./...`.

go run . -request requests/token-bucket.md

The first run is the design. Read back with senro attach --run <id> --ui=plain:

clean succeeded
design stdout | round 1: jev-1.13.0 decided "revise" (p_ready 0.44) request_clear=0.62 scope_creep=0.36 underspecified=0.56 missing_tests=0.38 risk=Low
design stdout |   - specify every component fully: fields, signatures, edge cases and error behaviour
design stdout | round 2: jev-1.13.0 decided "ready" (p_ready 0.57) request_clear=0.57 scope_creep=0.49 underspecified=0.28 missing_tests=0.34 risk=Low
design succeeded
plan stdout | module github.com/acme/ratelimit, 3 files, 11 steps
plan succeeded
collect succeeded
run succeeded

Two rounds. jev found the first design underspecified (0.56), the feedback said so in one sentence, and the second design named every field, signature and edge case. The ready probability moved from 0.44 to 0.57, and the design that came out of it is the one Gemini implemented; this is its component list in DESIGN.md:

## Components

- type Limiter struct { mu sync.Mutex; rate float64; burst float64; tokens float64;
  last time.Time; now func() time.Time } - concurrency-safe state tracking ...
- func New(rate float64, burst int) *Limiter - public constructor delegating to
  newWithClock using time.Now; clamps negative rate to 0.0 and negative burst to 0 ...
- func newWithClock(rate float64, burst int, now func() time.Time) *Limiter - internal
  constructor enabling time injection for tests; ... panics if now is nil.
- func (l *Limiter) Allow() bool - convenience method returning l.AllowN(1) ...
- func (l *Limiter) AllowN(n int) bool - thread-safe token deduction; edge cases: returns
  true immediately if n <= 0 without modifying state; returns false immediately if
  n > int(l.burst); advances tokens by elapsed.Seconds() * l.rate capped at l.burst ...

The plan turns that into three files and eleven steps, and the second run executes it.

implement stdout | wrote go.mod (42 bytes)
implement stdout | wrote ratelimit.go (1371 bytes)
implement stdout | wrote ratelimit_test.go (3172 bytes)
implement succeeded
vet succeeded
test stdout | ok    github.com/acme/ratelimit   0.009s
test succeeded
review stdout | jev-1.13.0 decided "ship" (confidence 0.66) quality=Excellent covers_request=0.87 tests_meaningful=0.95 build_green=0.98
review succeeded
collect succeeded
run succeeded

Gemini wrote the module, go vet and go test ran in the pod against it, and jev read the three files and the logs and said ship, with the build at 0.98 and the tests at 0.95. The module it reviewed:

// AllowN reports whether n tokens can be consumed immediately.
func (l *Limiter) AllowN(n int) bool {
    l.mu.Lock()
    defer l.mu.Unlock()

    if n <= 0 {
        return true
    }
    if float64(n) > l.burst {
        return false
    }

    curr := l.now()
    elapsed := curr.Sub(l.last)
    if elapsed > 0 {
        l.tokens += elapsed.Seconds() * l.rate
        if l.tokens > l.burst {
            l.tokens = l.burst
        }
        l.last = curr
    }

    if l.tokens >= float64(n) {
        l.tokens -= float64(n)
        return true
    }
    return false
}

with six tests beside it: clamping, a nil clock panic, burst capacity, AllowN edge cases, deterministic replenishment against an injected clock, and a race test over concurrent goroutines. No sleeping anywhere, as the request demanded.

The third run has both endings declared and runs one:

pr stdout | # Implement token-bucket rate limiter package
pr stdout | ## Summary
pr stdout | Introduces `github.com/acme/ratelimit`, a thread-safe, zero-dependency token-bucket rate limiter. ...
pr stdout | ## Tests
pr stdout | ok      github.com/acme/ratelimit   0.009s
pr stdout | ## Risk
pr stdout | Low. Mutex contention under high concurrency is mitigated by keeping critical sections limited to basic in-memory arithmetic. ...
pr succeeded
publish stdout | published 3 files to .../out/token-bucket/src
publish succeeded
brief skipped_condition
notify skipped_condition
run succeeded

Three senro runs, seven pods, two Gemini designs, two jev design reviews, one implementation, one vet, one test, one jev code review and one pull request, in a little over a minute and a half. out/token-bucket/ holds every record and the module under src/.

Step 7: when jev says no

The second request is the kind every team gets:

# Request

The app feels slow lately, especially in the afternoon. Can you make it faster?
Nothing specific, just whatever you think is best. Thanks.

go run . -request requests/make-it-faster.md

clean succeeded
design stdout | round 1: jev-1.13.0 decided "reject" (p_ready 0.13) request_clear=0.05 scope_creep=0.87 underspecified=0.70 missing_tests=0.79 risk=Low
design stdout |   - remove everything the request did not ask for
design stdout |   - specify every component fully: fields, signatures, edge cases and error behaviour
design stdout |   - add a test case for every requirement in the request
design succeeded
plan stdout | design ended "reject", nothing to plan
plan succeeded
collect succeeded
run succeeded

Gemini did what a chat model does with a vague request: it invented a feature, in this case pprof routes and a latency middleware. jev put the probability that the request is clear at 0.05, and the factory's rule turned that into reject in the first round, before a plan, a line of code or a second model call. The finish run then takes the other ending:

pr skipped_condition
publish skipped_condition
brief stdout | ### What was asked
brief stdout | A user submitted a vague performance complaint stating the app "feels slow lately, especially in the afternoon" and asked to "make it faster" without providing specific endpoints, metrics, or reproduction steps.
brief stdout | ### Why it stopped
brief stdout | The design was rejected on Round 1 (Readiness: 13%, Clarity: 5%). The factory attempted to introduce Go `net/http/pprof` diagnostic routes and custom in-memory latency middleware, triggering a rejection for high scope creep (87%) ...
brief stdout | ### What a human should decide
brief stdout | 1. **Clarify the issue:** Contact the user to identify affected routes, user workflows, and target latency SLOs.
brief stdout | 2. **Authorize observability work:** Determine if the team should prioritize a dedicated ticket for production profiling/metrics ...
brief succeeded
notify succeeded
run succeeded

The brief is written by Gemini, from the record jev produced. The two models keep their jobs even in the failure path.

A request jev cannot be talked into

The third request is a glob matcher: *, **, ?, character classes with ranges and negation, backslash escapes, and an error for an unterminated class. It is precise, it is testable, and the factory never got to write it:

clean succeeded
design stdout | round 1: jev-1.13.0 decided "revise" (p_ready 0.39) request_clear=0.68 scope_creep=0.35 underspecified=0.49 missing_tests=0.32 risk=Medium
design stdout | round 2: jev-1.13.0 decided "revise" (p_ready 0.39) request_clear=0.65 scope_creep=0.40 underspecified=0.34 missing_tests=0.22 risk=Medium
design stdout | round 3: jev-1.13.0 decided "revise" (p_ready 0.47) request_clear=0.59 scope_creep=0.66 underspecified=0.33 missing_tests=0.20 risk=Medium
design stdout |   - remove everything the request did not ask for
design succeeded

Gemini's design transpiles the glob into a Go regular expression. jev never called it ready, the ready probability never crossed 0.5 in three rounds, and in two of those rounds none of the specific questions crossed the line either, so the feedback Gemini got was the generic "tighten the approach". The loop cannot converge on a complaint it has no question for, and after three rounds the factory stops and writes the brief:

## What was asked
Implement a standard-library Go package (`github.com/acme/glob`) providing `Compile(pattern string)
(*Glob, error)` and `(*Glob) Match(path string) bool`. ...

## Why it stopped
The design review failed after 3 rounds, ending in a "revise" verdict (`p_ready` plateaued at 0.46).
Automated reviewers flagged persistent specification risks and scope creep regarding the chosen
approach (transpiling globs into Go `regexp`), specifically around complex regex metacharacter
escaping and separator handling in negated character classes.

## What a human should decide
1. **Architecture:** Approve regex transpilation (accepting edge-case escaping risks) or mandate a
   dedicated AST/rune-based glob parser.
2. **Path boundaries:** Explicitly define exact semantics for `**` edge cases and `/` exclusions
   inside negated classes.

That brief is the factory working as designed. A decision model only answers the questions you ask, and when it keeps saying no without any of them firing, the honest move is to stop and ask a person, with the evidence attached. The fix for the next version of the factory is one more field on DesignReview: a question about the approach itself, so that "this should be a parser, not a regex" becomes a sentence Gemini can act on.

What the combination gives you

Concern	Who handles it
Design, plan, code, fix, pull request text	Gemini, through `genkit.GenerateData` and `GenerateText`
Is the design ready, should the code ship	jev, through the same `GenerateData` call with a decision type
Feedback for the next round	Your code, one sentence per question jev answered "yes"
Compiling and testing	`go vet` and `go test`, as command steps in the pod, with no model involved
A red build reaching the reviewer	`ContinueOnError` on the Go steps
The module between runs	A bounded persistent workspace
Which ending runs	`senro.When(senro.ParamIs("verdict", ...))`
Credentials	mamori resolves, senro delivers a file per pod, per step
Evidence	`runs/<id>/` per run, plus `.factory/` records that travel with the code

Conclusion

A software factory is not one big agent. It is a handful of small, typed calls with a judge between them, and the judge is the part most people leave out. jev makes that part cheap: a design review is one call that answers six typed questions in a few hundred milliseconds, and the questions are Go fields you choose. Put the loop inside a Genkit flow, the flow inside a senro step, the step in a pod, and the verdict in a run parameter, and the factory is a state machine you can read in one screen.

The code: genkit-k8s-flows-jev-software-factory
The previous article: Genkit flows on Kubernetes, routed by a decision model
Decision models in Genkit Go and Flows
senro's Kubernetes executor and persistent workspaces

Genkit Flows on Kubernetes, Routed by a Decision Model: senro + jev in Go

Xavier Portilla Edo — Wed, 21 Oct 2026 11:44:21 +0000

Introduction

Genkit flows as senro steps showed how to put a Genkit flow inside a senro step and get retries, caching and an event stream for every model call. Every step in that article ran on your own machine.

This one moves the steps into Kubernetes and adds a second kind of model. Gemini writes. jev decides. jev is a decision model: it does not generate text, it evaluates a state against typed questions and answers each one with a probability, in a few hundred milliseconds. Genkit Go 1.14 reaches it through the System One plugin, and a decision is one genkit.GenerateData call whose output type is the question set.

That makes jev the right model to sit at the switch points of a pipeline. In this guide it decides twice: which department's workflow runs for a support ticket, and whether the reply Gemini drafted is safe to send.

Everything below runs against the real Gemini API (gemini-3.8-flash) and the real TypeSafe API (jev-latest), on Genkit Go v1.14.0, senro v1.4.5 and a local kind cluster on Kubernetes 1.37. The code is in genkit-k8s-flows-jev.

Why a decision model decides and a chat model writes

Asking a chat model "which team should own this ticket, answer with one word" works until it does not: you get prose around the word, a department that is not on the list, no idea how sure the model was, and a second or two of latency for a decision that should take milliseconds. A decision model is built for exactly that call.

	Chat model as router	Decision model as router
Answer	Text you parse	A typed field: `Choice[Dept]`, `Noul`, `Score[L]`
Options	In the prompt, hoped for in the output	In the Go type, enforced by the schema
Confidence	None, or whatever the model says about itself	A probability per option, calibrated per model version
Tools	Can be called by accident	Refused by the protocol
Latency	Seconds	A few hundred milliseconds
Cost per decision	Tokens for the state and the prose	Tokens for the state

And the reverse is just as true. jev cannot write the reply to the customer, explain an outage or summarise a brief. That is what gemini-3.8-flash is for. The pipeline needs both, and senro is what puts each call in its own step, on its own pod, with its own log, retry policy and secret.

What we are building

A support-ticket pipeline. A raw email comes in; a reply goes out, or a brief goes to a human, or both.

Two runs, because of one rule in senro: a When condition is evaluated once, at run start, against facts the run already has, never against something a step produces. So the first run understands the ticket and decides, and the decision becomes the run parameters of the second run, where senro.When(senro.ParamIs(...)) prunes every branch the verdict did not pick before anything starts. Pruned steps settle as skipped_condition and the run stays green.

Run	Step	Where	Model	Writes
`triage`	`seed`	coordinator	none	puts the raw email into the run's workspace
`triage`	`ingest`	pod	Gemini	`ticket.json`, a structured ticket
`triage`	`triage`	pod	jev	`decision.json`: department, urgency, frustration, needs a human
`triage`	`collect`	coordinator	none	copies both files out of the run
`resolve`	`<dept>/draft`	pod	Gemini	`reply.md`, following that department's playbook
`resolve`	`<dept>/verify`	pod	jev	`verdict.json`; a rejection fails the step
`resolve`	`<dept>/send`	coordinator	none	the reply, only if `verify` passed
`resolve`	`escalate/brief`	pod	Gemini	`brief.md` for a human, when jev said so

Inside a branch, jev is a guardrail: verify reads the draft against the playbook, and if it rejects the draft, the step fails and senro skips send. The second decision also determines the next step, this time without leaving the run.

What you need

Go 1.27 or newer, Docker, kind and kubectl
GEMINI_API_KEY and TYPESAFE_API_KEY in your environment
the senro CLI if you want the terminal UI (optional)

go get github.com/firebase/genkit/go@v1.14.0
go get github.com/xavidop/senro@v1.4.5
go get github.com/xavidop/mamori

mamori is the library senro uses to resolve secrets. It is how the two API keys reach the pods without ever being in a pod spec.

Step 1: a cluster senro can reach

senro's Kubernetes executor never reads KUBECONFIG, ~/.kube/config or your current context. A kubeconfig typically holds a dozen contexts, most of them production, with whichever one you last used selected; a pipeline that inherited that would eventually deploy into the wrong cluster. Instead, the cluster comes from three environment variables, read at run start:

Variable	What it is
`SENRO_K8S_SERVER`	The apiserver URL
`SENRO_K8S_CA_FILE`	The cluster CA, as a PEM file. senro will not skip verification
`SENRO_K8S_TOKEN_FILE`	A bearer token in a file (or a client certificate and key)

So a local cluster needs four things: kind, a namespace, an identity with the permissions the executor documents, and those three files. The repo's scripts/cluster-up.sh does all of it; this is what it runs.

kind create cluster --name senro-ai --kubeconfig .kind/kubeconfig
export KUBECONFIG=$PWD/.kind/kubeconfig
kubectl apply -f k8s/rbac.yaml

The RBAC is the exact set senro needs and nothing more: in the namespace, create, get and delete on pods, get on pods/log, create and get on pods/exec, and create, patch, delete on secrets. At cluster scope, list and get on nodes, which is how senro reads the platform it has to compile for.

The get on pods/exec is the one to remember: senro uses the WebSocket exec protocol, which is a GET on that subresource, and a Role with only create fails every step that carries a workspace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: senro-steps
  namespace: support
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "get", "delete"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create", "get"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "patch", "delete"]

Then the three files, pulled out of the kubeconfig kind wrote and a short-lived ServiceAccount token:

kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.server}' > .kind/server
kubectl config view --minify --raw \
  -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > .kind/ca.crt
kubectl -n support create token senro --duration=4h > .kind/token

export SENRO_K8S_SERVER=$(cat .kind/server)
export SENRO_K8S_CA_FILE=$PWD/.kind/ca.crt
export SENRO_K8S_TOKEN_FILE=$PWD/.kind/token

A file rather than SENRO_K8S_TOKEN inline, because an environment variable is visible in /proc/<pid>/environ and inherited by every child process the coordinator spawns.

Step 2: the flows

Five flows, in one file. Nothing here knows about senro or Kubernetes.

Gemini turns an email into a ticket

type Ticket struct {
    Customer string   `json:"customer"`
    Company  string   `json:"company,omitempty"`
    Summary  string   `json:"summary"`
    Ask      string   `json:"ask"`
    Facts    []string `json:"facts"`
}

f.Ingest = genkit.DefineFlow(g, "ingestTicket", func(ctx context.Context, raw string) (Ticket, error) {
    t, _, err := genkit.GenerateData[Ticket](ctx, g,
        ai.WithModelName("googleai/gemini-3.8-flash"),
        ai.WithSystem("You turn raw support emails into a structured ticket. "+
            "Facts are identifiers, amounts, dates and error codes quoted from the email, nothing inferred."),
        ai.WithPrompt("Email:\n\n%s", raw))
    if err != nil {
        return Ticket{}, err
    }
    if t == nil {
        return Ticket{}, fmt.Errorf("ingest: the model returned no ticket")
    }
    return *t, nil
})

jev decides what the ticket is

With the System One plugin, the output type is the question set. Each field is one question, its jsonschema_description tag is the instruction, and the options, levels and criteria belong to Go types, so the same question means the same thing wherever it is asked.

import systemonex "github.com/firebase/genkit/go/plugins/systemone/exp"

type Dept string

func (Dept) Criteria() map[Dept]string {
    return map[Dept]string{
        "billing":   "Charges, invoices, refunds, subscriptions, plan changes",
        "technical": "Bugs, errors, outages, latency, integrations, webhooks, API",
        "account":   "Login, password, permissions, data export or deletion, privacy",
        "other":     "None of the above",
    }
}

type Frustration int

func (Frustration) Levels() []string { return []string{"Calm", "Annoyed", "Angry", "Furious"} }

type Urgent struct{}

func (Urgent) Criteria() (yes, no string) {
    return "Names a deadline, says now or today, or describes an outage in progress",
        "No time pressure is expressed, or the customer says there is no rush"
}

type Triage struct {
    Department  systemonex.Choice[Dept]       `json:"department" jsonschema_description:"Which team should own this ticket?"`
    IsUrgent    systemonex.NoulOf[Urgent]     `json:"is_urgent" jsonschema_description:"Does the ticket communicate time pressure?"`
    Frustration systemonex.Score[Frustration] `json:"frustration" jsonschema_description:"How frustrated is the customer?"`
    NeedsHuman  systemonex.Noul               `json:"needs_human" jsonschema_description:"Does resolving this need a human: a legal threat, a contract or SLA dispute, data loss, or a refund outside the playbook?"`
}

Four question kinds cover the protocol. Choice[T] picks one of T.Criteria() and gives a probability per option plus a Confidence. Score[L] rates on L.Levels() and answers Level() and Label(). Noul is "is this true?" as a probability, where 0.5 means the model could not tell. NoulOf[C] is the same with C.Criteria() saying what yes and no mean.

The flow sends the ticket as a data part, so the state is a JSON object and a question can refer to a field by name. It then turns the four answers into a flat verdict, with thresholds that belong to the caller:

type Decision struct {
    Department  string  `json:"department"`
    Confidence  float64 `json:"confidence"`
    Urgent      bool    `json:"urgent"`
    Frustration string  `json:"frustration"`
    Escalate    bool    `json:"escalate"`
    Model       string  `json:"model"`
}

f.Triage = genkit.DefineFlow(g, "triageTicket", func(ctx context.Context, t Ticket) (Decision, error) {
    out, resp, err := genkit.GenerateData[Triage](ctx, g,
        ai.WithModelName("typesafe/jev-latest"),
        ai.WithPromptParts(ai.NewDataPart(t)))
    if err != nil {
        return Decision{}, err
    }
    d := Decision{
        Department:  string(out.Department.Choice),
        Confidence:  out.Department.Confidence,
        Urgent:      out.IsUrgent.Probability >= 0.5,
        Frustration: out.Frustration.Label(),
        Model:       systemonex.ResponseInfo(resp).Model,
    }
    // A low-confidence routing decision is not a routing decision.
    if d.Confidence < 0.5 {
        d.Department = "other"
    }
    d.Escalate = d.Department == "other" ||
        out.NeedsHuman.Probability >= 0.6 ||
        out.Frustration.Level() >= 3
    return d, nil
})

ResponseInfo(resp).Model records which jev version answered. Confidence thresholds are calibrated per version, so a verdict that does not say who produced it is a threshold waiting to drift.

Gemini drafts, jev reviews

The draft flow is an ordinary GenerateData[Reply] call with the playbook and the ticket in the prompt. One difference from the triage flow: the ticket goes to Gemini as JSON text. A data part is how a decision model takes an object state; a chat model takes a prompt.

f.Draft = genkit.DefineFlow(g, "draftReply", func(ctx context.Context, in DraftInput) (Reply, error) {
    r, _, err := genkit.GenerateData[Reply](ctx, g,
        ai.WithModelName("googleai/gemini-3.8-flash"),
        ai.WithSystem("You are a support agent for Acme Cloud. Write the reply the customer will receive. "+
            "Follow the playbook strictly. Never state amounts, dates or identifiers that are not in the ticket. "+
            "Plain text, under 180 words, sign as 'Acme Cloud Support'."),
        ai.WithPrompt("Playbook:\n%s\n\n%s\n\nTicket:\n%s", in.Playbook, urgency, asJSON(in.Ticket)))
    // ...
})

The review flow is the second decision, and it asks different questions of a different state: the ticket, the draft and the playbook, together.

type Overpromise struct{}

func (Overpromise) Criteria() (yes, no string) {
    return "Commits to a refund amount, a credit, a release date, a deadline, or a deletion the playbook does not allow",
        "Every commitment in the reply is one the playbook permits"
}

type Tone int

func (Tone) Levels() []string { return []string{"Cold or dismissive", "Neutral", "Warm and specific"} }

type Review struct {
    Addresses    systemonex.Noul                `json:"addresses" jsonschema_description:"Does the reply address what the customer actually asked for?"`
    Overpromises systemonex.NoulOf[Overpromise] `json:"overpromises" jsonschema_description:"Does the reply promise something the playbook forbids?"`
    Invents      systemonex.Noul                `json:"invents" jsonschema_description:"Does the reply state amounts, dates, identifiers or causes that appear in neither the ticket nor the playbook?"`
    Tone         systemonex.Score[Tone]         `json:"tone" jsonschema_description:"How does the reply read to the customer?"`
}

f.Verify = genkit.DefineFlow(g, "verifyReply", func(ctx context.Context, in VerifyInput) (Verdict, error) {
    out, _, err := genkit.GenerateData[Review](ctx, g,
        ai.WithModelName("typesafe/jev-latest"),
        ai.WithSystem("The state has the customer's ticket, the reply an agent drafted, and the playbook the agent had to follow."),
        ai.WithPromptParts(ai.NewDataPart(in)))
    if err != nil {
        return Verdict{}, err
    }
    v := Verdict{Addresses: out.Addresses.Probability, Overpromises: out.Overpromises.Probability,
        Invents: out.Invents.Probability, Tone: out.Tone.Label()}
    if v.Addresses < 0.6 {
        v.Reasons = append(v.Reasons, "the reply does not address the customer's ask")
    }
    if v.Overpromises >= 0.5 {
        v.Reasons = append(v.Reasons, "the reply promises something the playbook forbids")
    }
    if v.Invents >= 0.5 {
        v.Reasons = append(v.Reasons, "the reply invents facts")
    }
    if out.Tone.Level() == 0 {
        v.Reasons = append(v.Reasons, "the tone is cold")
    }
    v.Approved = len(v.Reasons) == 0
    return v, nil
})

A system message is never part of the state with this plugin. It goes in front of every question's instructions, which is the right place to say what the state is.

Step 3: flows as steps that can run in a pod

A senro func step is a registered Go function. In the local-only article the flow lived in a package variable that main filled in. In a pod that does not work, because senro runs a func step by sending your pipeline binary into the pod and re-entering it as a fresh process; main never gets to the line that initialised Genkit. So Genkit is initialised lazily, inside the step, from the secrets senro delivered to that pod:

var (
    engineOnce sync.Once
    engine     *Flows
    engineErr  error
)

func flows(ctx senro.Ctx) (*Flows, error) {
    engineOnce.Do(func() {
        var plugins []api.Plugin
        gemini, err := readSecret(ctx, "Gemini")
        if err != nil {
            engineErr = err
            return
        }
        if gemini != "" {
            plugins = append(plugins, &googlegenai.GoogleAI{APIKey: gemini})
        }
        typesafe, err := readSecret(ctx, "TypeSafe")
        if err != nil {
            engineErr = err
            return
        }
        if typesafe != "" {
            ts := systemonex.TypeSafe()
            ts.APIKey = typesafe
            plugins = append(plugins, ts)
        }
        g := genkit.Init(ctx, genkit.WithPlugins(plugins...))
        engine = defineFlows(g)
    })
    return engine, engineErr
}

readSecret reads the file that ctx.Secret(name) names, and that is the point: in the pod it is a 0400 file under /run/senro/secrets, projected from a short-lived Kubernetes Secret owned by the pod. The key is never in the pod spec, never in env, never in the plan, never in a log. That is also why the plugin fields are set explicitly rather than read from the environment. And senro.Ctx embeds context.Context, so it goes straight into genkit.Init and flow.Run.

The steps themselves are short. Each one reads from the case workspace, runs a flow, and writes back:

const workspace = "case"

func init() {
    senro.RegisterFunc("ai/ingest", Ingest)
    senro.RegisterFunc("ai/triage", TriageStep)
    senro.RegisterFunc("ai/draft", Draft)
    senro.RegisterFunc("ai/verify", Verify)
    senro.RegisterFunc("ai/brief", Brief)
    senro.RegisterFunc("ai/collect", Collect)
    senro.RegisterFunc("ai/seed", Seed)
    senro.RegisterFunc("ai/send", Send)
}

func TriageStep(ctx senro.Ctx, _ NoParams) error {
    f, err := flows(ctx)
    if err != nil {
        return err
    }
    var t Ticket
    if err := readJSON(ctx, "ticket.json", &t); err != nil {
        return err
    }
    d, err := f.Triage.Run(ctx, t)
    if err != nil {
        return err
    }
    fmt.Fprintf(ctx.Stdout(), "%s decided: department=%s (confidence %.2f) urgent=%v frustration=%q escalate=%v\n",
        d.Model, d.Department, d.Confidence, d.Urgent, d.Frustration, d.Escalate)
    return writeJSON(ctx, "decision.json", d)
}

func Verify(ctx senro.Ctx, p PlaybookParams) error {
    // ...read ticket.json and reply.json, run f.Verify...
    if err := writeJSON(ctx, "verdict.json", v); err != nil {
        return err
    }
    if !v.Approved {
        return fmt.Errorf("reply rejected: %v", v.Reasons)
    }
    return nil
}

Verify returning an error is the guardrail: a func step that returns an error is a failed step, and every dependent of a failed step settles as skipped_upstream_failed.

One more line in main, before flag parsing, so that senro's re-entry into the binary does not hit your flags:

func main() {
    ctx := context.Background()
    if handled, err := senro.StepChild(ctx); handled {
        if err != nil {
            log.Fatal(err)
        }
        return
    }
    ticketPath := flag.String("ticket", "tickets/double-charge.txt", "raw ticket to process")
    // ...
}

Step 4: the Kubernetes target

One target, used by every AI workflow:

import "github.com/xavidop/senro/executor/k8s"

const stepImage = "docker.io/library/alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8"

func pods() senro.ExecutorTarget {
    return k8s.Pod(stepImage,
        k8s.Namespace("support"),
        k8s.Resources(
            map[string]string{"cpu": "250m", "memory": "128Mi"},
            map[string]string{"cpu": "1", "memory": "512Mi"},
        ))
}

Three rules of this executor show up in those six lines:

The image must be pinned to a digest and the namespace must be stated, or Build() refuses the pipeline. There is no fallback to default.
The image needs sh and tar, and nothing else. Alpine is 3 MB. Your pipeline binary is not in the image; senro cross-compiles it with CGO_ENABLED=0, sends it in as a tar over the apiserver's exec subresource, and re-enters it. No registry, no Dockerfile, no build step for your code.
Requests and limits apply to the step's container only. senro adds two containers of its own when a step mounts a workspace: an init container that fills the emptyDir, and an idle container that keeps the pod alive so the result can be read back out.

Step 5: run one, the triage

func triagePipeline(ticketPath, outDir string) *senro.Pipeline {
    p := senro.New("triage")
    ws := senro.Workspace(workspace)

    // The raw email enters the workspace from the coordinator, so the ingest
    // step can declare it as an input and be cached on it.
    intake := p.Workflow("intake", senro.On(senro.Local()))
    intake.Step("seed", senro.Func("ai/seed", DirParams{
        Dir:   filepath.Dir(ticketPath),
        Files: []string{filepath.Base(ticketPath)},
    })).Mount(ws.At("/case", senro.RW))

    understand := p.Workflow("understand", senro.Needs("intake"), senro.On(pods()))

    understand.Step("ingest", senro.Func("ai/ingest", IngestParams{File: filepath.Base(ticketPath)})).
        Mount(ws.At("/case", senro.RW)).
        SecretEnv("GEMINI_API_KEY", "Gemini").
        Timeout(stepTimeout).
        Retry(3, retry.OnInfra()).
        Pure().
        Inputs(artifact.File(filepath.Base(ticketPath))).
        Outputs(artifact.File("ticket.json"))

    understand.Step("triage", senro.Func("ai/triage", NoParams{})).
        Needs("ingest").
        Mount(ws.At("/case", senro.RW)).
        SecretEnv("TYPESAFE_API_KEY", "TypeSafe").
        Timeout(stepTimeout).
        Retry(3, retry.OnInfra()).
        Pure().
        Inputs(artifact.File("ticket.json")).
        Outputs(artifact.File("decision.json"))

    handoff := p.Workflow("handoff", senro.Needs("understand"), senro.On(senro.Local()))
    handoff.Step("collect", senro.Func("ai/collect", DirParams{
        Dir:   outDir,
        Files: []string{"ticket.json", "decision.json"},
    })).Mount(ws.At("/case", senro.RO))

    return p
}

Each step names only the secret it needs: the Gemini step never sees the TypeSafe key and the jev step never sees the Gemini key. Timeout matters on a remote func step, because the function has a process of its own in the pod and the deadline is what ends it if the coordinator dies. The two pod steps are Pure() with explicit Inputs and Outputs, so a repeated ticket is served from the cache. seed and collect run on the coordinator, and collect copies the two files to disk, which is how the verdict crosses into the second run.

The keys are declared once, as a struct mamori resolves from the environment:

type Secrets struct {
    Gemini   secret.String `source:"env:GEMINI_API_KEY"`
    TypeSafe secret.String `source:"env:TYPESAFE_API_KEY"`
}

secrets, err := mamori.Load[Secrets](ctx)

run := func(p *senro.Pipeline, extra ...senro.Option) error {
    att, err := attach.Listen(ctx, attach.Options{Bind: attach.AutoUnixSocket, Pipeline: p.Name()})
    if err != nil {
        return err
    }
    defer func() { _ = att.Close() }()
    opts := append([]senro.Option{
        senro.WithAttach(att),
        senro.WithSecrets(secrets),
        senro.WithFuncBuild("."),
    }, extra...)
    return senro.Run(ctx, p, opts...)
}

err = run(triagePipeline(ticket, out))

WithFuncBuild(".") names the package senro cross-compiles for the pods, since the pod is Linux and the coordinator may not be. WithAttach lets senro run and senro attach watch the run.

Run it against the first sample ticket, a double charge with a deadline. On a TTY you get senro's terminal UI; with --ui=plain you get the same events as lines:

senro run . --ui=plain -- -ticket tickets/double-charge.txt

seed started
seed succeeded
ingest started
ingest stdout | ticket from Marta Ruiz: Card charged twice for invoice INV-2291
ingest succeeded
triage started
triage stdout | jev-1.13.0 decided: department=billing (confidence 1.00) urgent=true frustration="Annoyed" escalate=false
triage succeeded
collect started
collect stdout | collected ticket.json
collect stdout | collected decision.json
collect succeeded
run succeeded

A Gemini call and a jev call, each in its own pod, each with its own log under runs/<id>/. The whole run took eleven seconds. jev-latest resolved to jev-1.13.0, and the decision says so. The verdict lands in out/decision.json:

{
  "department": "billing",
  "confidence": 1,
  "urgent": true,
  "frustration": "Annoyed",
  "escalate": false,
  "model": "jev-1.13.0"
}

And out/ticket.json is what Gemini extracted, facts quoted rather than inferred:

{
  "customer": "Marta Ruiz",
  "company": "Lumen Logistics",
  "summary": "Customer was charged twice for invoice INV-2291 on March 3rd",
  "ask": "Refund the duplicate charge today",
  "facts": ["INV-2291", "59.00 EUR", "March 3rd"]
}

Step 6: let the verdict choose the next steps

The second pipeline declares every branch. The parameters decide which ones run.

func resolvePipeline(outDir string, reckless bool) *senro.Pipeline {
    p := senro.New("resolve")
    ws := senro.Workspace(workspace)

    intake := p.Workflow("intake", senro.On(senro.Local()))
    intake.Step("seed", senro.Func("ai/seed", DirParams{
        Dir:   outDir,
        Files: []string{"ticket.json", "decision.json"},
    })).Mount(ws.At("/case", senro.RW))

    for _, d := range departments {
        draftPlaybook := d.Playbook
        if reckless {
            draftPlaybook = recklessPlaybook // see Step 7
        }
        w := p.Workflow(d.Name,
            senro.Needs("intake"),
            senro.On(pods()),
            senro.When(senro.ParamIs("department", d.Name)))

        w.Step(d.Name+"/draft", senro.Func("ai/draft", PlaybookParams{Playbook: draftPlaybook})).
            Mount(ws.At("/case", senro.RW)).
            SecretEnv("GEMINI_API_KEY", "Gemini").
            Timeout(stepTimeout).
            Retry(3, retry.OnInfra())

        w.Step(d.Name+"/verify", senro.Func("ai/verify", PlaybookParams{Playbook: d.Playbook})).
            Needs(d.Name + "/draft").
            Mount(ws.At("/case", senro.RW)).
            SecretEnv("TYPESAFE_API_KEY", "TypeSafe").
            Timeout(stepTimeout).
            Retry(3, retry.OnInfra())

        send := p.Workflow(d.Name+"-send",
            senro.Needs(d.Name),
            senro.On(senro.Local()),
            senro.When(senro.ParamIs("department", d.Name)))
        send.Step(d.Name+"/send", senro.Func("ai/send", DirParams{Dir: outDir})).
            Mount(ws.At("/case", senro.RO))
    }

    esc := p.Workflow("escalate",
        senro.Needs("intake"),
        senro.On(pods()),
        senro.When(senro.ParamIs("escalate", "true")))
    esc.Step("escalate/brief", senro.Func("ai/brief", NoParams{})).
        Mount(ws.At("/case", senro.RW)).
        SecretEnv("GEMINI_API_KEY", "Gemini").
        Timeout(stepTimeout).
        Retry(3, retry.OnInfra())

    // ...and an escalate-notify workflow on the coordinator that collects brief.md
    return p
}

The pipeline is a Go program, so the three departments are a for loop over a slice of playbooks, and each playbook is a step parameter. A func step's cache key includes the parameter digest, so editing the billing playbook changes the identity of the billing steps and nothing else.

main glues the two runs together with the verdict as parameters:

var decision Decision
b, _ := os.ReadFile(filepath.Join(out, "decision.json"))
json.Unmarshal(b, &decision)

err = run(resolvePipeline(out, *reckless), senro.WithParams(decision.Params()))

func (d Decision) Params() senro.Params {
    return senro.Params{
        "department": d.Department,
        "urgent":     fmt.Sprint(d.Urgent),
        "escalate":   fmt.Sprint(d.Escalate),
    }
}

Parameters are never recorded in an event or a cache key; they exist to be read by conditions. And this is what the second run looks like for the double-charge ticket, read back from disk with senro attach --run <id> --ui=plain:

seed succeeded
billing/draft stdout | drafted: Duplicate charge for invoice INV-2291
billing/draft succeeded
billing/verify stdout | review: addresses=0.81 overpromises=0.44 invents=0.25 tone="Warm and specific"
billing/verify stdout | approved
billing/verify succeeded
billing/send stdout | sent to customer:
billing/send succeeded
technical/draft skipped_condition
technical/verify skipped_condition
technical/send skipped_condition
account/draft skipped_condition
account/verify skipped_condition
account/send skipped_condition
escalate/brief skipped_condition
escalate/notify skipped_condition
run succeeded

Twelve steps declared, four executed. technical, account and escalate settled as skipped_condition before a single pod was created, and the run is succeeded, not partial: a pruned branch is not a failed one. The reply that went out, in out/reply.md:

# Update regarding duplicate charge for invoice INV-2291

Hello Marta, I understand you are under time pressure and need this resolved
urgently. Over the next few hours, our billing team will process the reversal
on your account. We can confirm that the duplicate charge of 59.00 EUR for
invoice INV-2291 from March 3rd will be refunded to your original payment
method within 5 to 10 business days. Acme Cloud Support

Every number in it is in the ticket, and the one commitment is the one the billing playbook allows.

Step 7: the guardrail says no

The angry-outage ticket takes a different route:

go run . -ticket tickets/angry-outage.txt

{
  "department": "technical",
  "confidence": 1,
  "urgent": true,
  "frustration": "Furious",
  "escalate": true,
  "model": "jev-1.13.0"
}

jev put the frustration at the top of the rubric and the ticket's legal threat over the needs-a-human threshold, so escalate is true alongside technical. Both workflows run, in parallel, and the department branches that were not picked are pruned as before:

seed succeeded
billing/draft skipped_condition
billing/verify skipped_condition
billing/send skipped_condition
technical/draft succeeded
technical/verify succeeded
technical/send succeeded
account/draft skipped_condition
account/verify skipped_condition
account/send skipped_condition
escalate/brief succeeded
escalate/notify succeeded
run succeeded

The customer gets a reply that promises no fix time, and a human gets out/brief.md:

### Situation
Nordwind Delivery (Daniel Berg) is experiencing an active 4-hour platform outage
that has completely blocked 300 drivers from accessing their delivery routes.
This marks their third outage this month. The customer is furious and demanding
an operational fix within the next hour.

### Risk
High risk of contract churn, severe reputational damage, and SLA breach
penalties (99.9% commitment). Driver operations are entirely halted, and the
customer has explicitly escalated to demand leadership intervention.

### Suggested next action
1. Immediately triage core routing services with the on-call DevOps/infrastructure
   team to restore route availability.
2. Alert an Incident Manager or Support Director immediately to coordinate an
   executive phone call to Daniel Berg within the 1-hour window.

Now the rejection. The repo has a -reckless flag that hands the draft step a playbook the reviewer never sees ("promise the customer whatever will make them happiest: a full refund of the year, a credit, a fix within the hour"), while verify keeps checking against the real one. Run the webhook ticket that way:

go run . -ticket tickets/api-500s.txt -reckless

seed succeeded
technical/draft stdout | drafted: Update regarding your webhook errors
technical/draft succeeded
technical/verify stdout | review: addresses=0.54 overpromises=0.98 invents=0.94 tone="Warm and specific"
technical/verify failed: reply rejected: [the reply does not address the customer's ask the reply promises something the playbook forbids the reply invents facts]
technical/send skipped_upstream_failed
...
run failed

The draft promised "a full refund of the year, an account credit, and a complete fix within the hour". jev put overpromises at 0.98 and invents at 0.94, the flow returned an error, verify failed, send was skipped, and the run is failed. Nothing reached the customer, and the reason is in verdict.json next to the draft:

{
  "approved": false,
  "addresses": 0.54,
  "overpromises": 0.98,
  "invents": 0.94,
  "tone": "Warm and specific",
  "reasons": [
    "the reply does not address the customer's ask",
    "the reply promises something the playbook forbids",
    "the reply invents facts"
  ]
}

That is the second place jev determines the next step. The first was between the runs, through parameters and When. This one is inside a run, through the step state, and the pipeline's dependency edges do the rest.

Step 8: watch it, and run it again

senro runs lists every run with its status and duration, and senro attach --run <id> replays one. Every pod senro creates is labelled, so if a coordinator dies mid-run you can still find and clean up what it left:

kubectl -n support get pods,secrets -l senro.dev/managed=true

And the two triage steps are Pure(), so a second run of the same ticket does not call either model:

seed succeeded
ingest cached
triage cached
collect stdout | collected ticket.json
collect stdout | collected decision.json
collect succeeded
run succeeded

cached is a step state, not a label: no pod, no model call, no tokens.

What the combination gives you

Concern	Who handles it
Reading and writing	Gemini, through `genkit.GenerateData` and `GenerateText`
Routing, scoring, guardrails	jev, through the same `GenerateData` call with a decision type
Where a step runs	`senro.On(k8s.Pod(...))`, one pod per step, platform read from the cluster
Which steps run	`senro.When(senro.ParamIs(...))`, fed by jev's verdict
What a rejection does	The step fails, dependents are `skipped_upstream_failed`
Credentials	mamori resolves, senro delivers a file per pod, per step, per attempt
Retries	`retry.OnInfra()`: an image pull or an evicted pod, never a model answer you did not like
Repeat work	`Pure()` steps with `Inputs` and `Outputs` are replayed, not re-run
Evidence	One `runs/<id>/` per run, with the plan, the events, and one log per step

Conclusion

The pattern fits in four lines:

Define the flows with genkit.DefineFlow. Make the routing and review flows return a decision type from the System One plugin.
Register each flow as a senro.Func step that initialises Genkit lazily from ctx.Secret(...).
Point the workflows at k8s.Pod(image@digest, k8s.Namespace(...)), and give every remote step a Timeout.
Turn the decision into run parameters, gate workflows with senro.When, and let a rejected guardrail fail its step.

A decision model is cheap enough and fast enough to sit at every switch point of a pipeline, and typed enough that the switch is a Go field, not a parsed string. senro makes the switch points visible: a branch jev did not pick is a skipped_condition step in the event stream, with the reason one senro cache explain away.

The code: genkit-k8s-flows-jev
Decision models in Genkit Go and Flows
senro's Kubernetes executor and Func steps off the coordinator
The previous article: Genkit flows as senro steps

Dash: An Open-Source Home Assistant Built on the Gemini 3.8 Live API

Xavier Portilla Edo — Mon, 19 Oct 2026 11:32:16 +0000

Introduction

I wanted a Nest Hub that I could actually program. Something on the kitchen wall that shows the time, the weather, my teams, what's cooking and what's playing, and that answers when I talk to it. That is Dash: an open-source ambient display for any tablet or screen in kiosk mode, one config.json, six languages, free data sources, MIT.

Most of Dash needs no account at all. The part that does is the one I want to talk about here: the "Hey Dash" voice assistant, which runs on Google's realtime voice model, Gemini 3.8 Live, through the Gemini Live API. If you have ever wanted to build a voice assistant without stitching speech-to-text, an LLM and text-to-speech together yourself, this is the article. You will see how the Live API works at the protocol level, how Dash keeps the API key off the device, how tool calls turn speech into shopping list items and timers, and what a question costs.

Everything below is the real code from the repository, as of October 2026, against gemini-3.8-live.

What Dash Is

Dash is an Express server plus a Vite and vanilla TypeScript front end, built to be left alone on a wall for weeks. Panels rotate every minute: Home with a "Today" strip (weather warnings, holidays, your teams playing today, reminders, bin day, the cheapest electricity hour in Spain, the moon), Agenda from any iCal link, news, markets with sparklines, Spotify with synced lyrics, a kitchen that suggests six recipes a day and walks you through them one step at a time, and live sports boards that take over the screen when one of your teams plays, with a celebration when they score.

Everything on screen also has an HTTP endpoint, so an iOS Shortcut or Home Assistant can add to the shopping list or start a timer. And then there is the voice layer, which is where Gemini comes in.

Why the Live API and Not a Pipeline

The classic way to build a voice assistant is a pipeline: a wake word engine, a speech-to-text service, an LLM call, a text-to-speech service, and a lot of glue to make the whole thing feel responsive. You can build Dash that way. You will spend most of your time on the glue, and it will still feel like a walkie-talkie.

The Gemini Live API collapses that pipeline into one stateful WebSocket. You stream raw microphone audio in, and the model streams spoken audio back, with transcripts of both sides, tool calls in the middle, and Google Search when it needs a current fact. This is what that buys you:

Concern	Pipeline (STT + LLM + TTS)	Gemini Live API
Latency	Three sequential network calls per turn	One bidirectional stream, audio starts before the sentence is finished
Interruptions	Hand-rolled: detect speech, cancel TTS, re-prompt	Built in: barge-in, `interrupted` event, playback stops
Language	Configure STT and TTS per language, detect yourself	Reply in whatever language the user spoke, 70+ languages
Transcripts	Two separate services, two formats	`inputTranscription` and `outputTranscription` on the same socket
Tools	Parse LLM output, dispatch, re-prompt, re-synthesize	`toolCall` and `toolResponse` messages inside the session
Current facts	Your own search integration	`googleSearch` tool with grounding
Voice	Pick a TTS vendor and voice	`voiceName` in the session config (Dash ships with `Kore`)

The trade-off is control: you do not get to inspect the text between STT and TTS, because there is no text in between. For a kitchen assistant that is the right deal.

How "Hey Dash" Works

Here is the whole shape of it. Three parties: the browser running the display, the Dash server on the same machine, and the Gemini Live API.

The two rules that drive the design:

The API key never leaves the server. The browser never sees it. For each conversation the server mints a single-use ephemeral token with the entire session setup locked in (model, voice, system instructions, tools), and the browser connects to Gemini with that token.
Audio never touches the server. The browser talks to Gemini directly over WebSocket: 16 kHz PCM up, 24 kHz PCM down. The server's only job during a conversation is to run the tools Gemini asks for.

Dash is on the kitchen wall, so a third rule follows from the first two: Gemini is only on the line while you are talking to it. Waiting for the wake word is free and runs in the browser.

What You Need

Node.js 22+ (Dash uses node --test, native fetch, and the WebSocket global)
A Gemini API key from Google AI Studio, in a .env file next to config.json
A browser with the Web Speech API for the wake word (Edge or Chrome; Firefox has no speech recognition, so there you tap the microphone button)
A tablet or screen in kiosk mode, with microphone access allowed for http://localhost:3000

git clone https://github.com/xavidop/dash && cd dash
npm install
cp config.example.json config.json
echo "GEMINI_API_KEY=your-key" > .env
npm run build && npm start      # http://localhost:3000

Then switch on Settings → General → Voice assistant (or set features.assistant to true in config.json). The rest of this article walks through what happens from the moment you say "Dash".

Step 1: Wait for the Wake Word Without Paying for It

A Live session is billed per minute of audio, so you do not want it open all day. Dash uses the browser's own speech recognition to wait for the wake word, and only opens a Gemini session once it hears it. This is the relevant part of web/src/assistant.ts:

function startWake() {
  if (!cfg() || wakeBlocked || phase !== 'off' || wakeOn) return;
  const SR = (window as any).SpeechRecognition || (window as any).webkitSpeechRecognition;
  if (!SR) return;
  startMic(); // keeps the pre-roll buffer filling
  rec = new SR() as Recognition;
  rec.lang = getState().config?.locale || defaultLocale();
  rec.continuous = true;
  rec.interimResults = true;
  const re = wakeRegex();
  rec.onresult = (e: any) => {
    for (let i = e.resultIndex; i < e.results.length; i++) {
      // Cooking: short commands ("next", "siguiente paso") act at once, without Gemini.
      if (e.results[i].isFinal && cookingCommand(norm(e.results[i][0].transcript))) return;
      if (re.test(` ${norm(e.results[i][0].transcript)} `)) {
        restartDelay = 300;
        open(true);
        return;
      }
    }
  };
  // ...
}

Two details matter here.

The wake regex is forgiving. A recognizer set to Spanish or German hears "Dash" as "das", "dach" or "dasch". Exact wake words match anywhere; sound-alikes only count after "hey", "oye", "ok" and friends, because "das" is just a word in German:

function wakeRegex() {
  const words = (cfg()?.wakeWords || ['dash']).map(norm).map((w) => w.trim()).filter(Boolean);
  const alike = words.flatMap((w) => (w === 'dash' ? ['dasch', 'dach', 'dache', 'dass', 'dahs', ...(getState().config?.language === 'de' ? [] : ['das'])] : []));
  const exact = words.map(esc).join('|');
  const loose = alike.map(esc).join('|');
  return new RegExp(`(?:^|\\s)(?:${exact})(?:\\s|$)${loose ? `|(?:^|\\s)(?:hey|hei|ei|ey|oye|ok|okay|hola|hallo|ehi|eh)\\s+(?:${loose})(?:\\s|$)` : ''}`);
}

The microphone keeps a 2.5-second ring buffer in memory. People say "Hey Dash, what's the weather?" in one breath. By the time the recognizer fires, the question is already half spoken. An AudioWorklet converts the mic to 16-bit PCM at 16 kHz in 100 ms chunks, and the last 25 chunks are kept so the whole sentence reaches Gemini when the session opens:

const IN_RATE = 16000;
const CHUNK = 1600;         // 100 ms
const PREROLL_CHUNKS = 25;  // 2.5 s of audio before the wake word fired

node.port.onmessage = (e) => {
  ring.push(e.data);
  if (ring.length > PREROLL_CHUNKS) ring.shift();
  onChunk?.(e.data);
};

Nothing in that buffer leaves the device before the wake word. What the browser's own speech service hears while waiting is a different matter, and Dash is explicit about it in docs/voice-assistant.md: in Edge and Chrome, recognition runs on the browser vendor's online service, not on Gemini, and Dash stores none of it.

Cooking mode adds a shortcut: while a recipe is open, "next", "back", "repeat", "start the timer" and "stop cooking" (in each of the six languages) are matched locally and act instantly, without opening a Gemini session at all.

Step 2: Mint an Ephemeral Token on the Server

This is the part of the Live API that most examples skip, and the one that makes a client-side connection safe. Instead of handing the browser your API key, your backend calls the token endpoint and gets back a short-lived credential that is locked to a full session setup. The browser can connect with it, but it cannot change the model, the voice, the instructions or the tools.

server/services/assistant.js:

const API = 'https://generativelanguage.googleapis.com/v1beta';
export const LIVE_WS = 'wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContentConstrained';

/** Mints a single-use token valid for 30 min (a new session must start within 1 min). */
async session() {
  if (!this.enabled) throw Object.assign(new Error('assistant disabled'), { status: 404 });
  if (Date.now() - this.lastSession < 2000) throw Object.assign(new Error('too many sessions'), { status: 429 });
  this.lastSession = Date.now();
  const iso = (min) => new Date(Date.now() + min * 60e3).toISOString();
  const res = await fetch(`${API}/auth_tokens`, {
    method: 'POST',
    headers: { 'x-goog-api-key': this.key, 'content-type': 'application/json' },
    body: JSON.stringify({ uses: 1, expireTime: iso(30), newSessionExpireTime: iso(1), bidiGenerateContentSetup: this.setup() }),
    signal: AbortSignal.timeout(10000),
  });
  const d = await res.json().catch(() => ({}));
  if (!res.ok || !d.name) throw Object.assign(new Error(d.error?.message || `token request failed (HTTP ${res.status})`), { status: 502 });
  return { url: `${LIVE_WS}?access_token=${encodeURIComponent(d.name)}`, idleSeconds: this.config.assistant.idleSeconds };
}

The fields, one by one:

uses: 1: the token starts exactly one session.
expireTime: how long messages can flow over a connection opened with it (30 minutes here; the API's default is also 30).
newSessionExpireTime: how long you have to open that session (1 minute; again the default). The browser requests a token and connects immediately, so a minute is plenty.
bidiGenerateContentSetup: the whole setup message the client would normally send on the WebSocket, baked into the token. In the SDKs this is called liveConnectConstraints; on the REST endpoint it is bidiGenerateContentSetup.

The response is tiny, and it is all the browser ever gets from the server:

{
  "name": "auth_tokens/<redacted>"
}

Two things follow from this. First, the browser connects to the constrained WebSocket endpoint, BidiGenerateContentConstrained, with the token as access_token. That endpoint ignores any setup the client sends, so the browser cannot repurpose the token for something else. Second, the token endpoint is the only place that sees the API key, and Dash puts a guard in front of it: only the display itself (requests from localhost) may ask for a token unless assistant.allowRemote is set, so other devices on your Wi-Fi cannot spend your quota.

The setup locked into the token is where the assistant gets its personality:

setup() {
  const a = this.config.assistant;
  return {
    model: `models/${a.model}`,                                  // gemini-3.8-live
    generationConfig: {
      responseModalities: ['AUDIO'],
      speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: a.voice } } },   // Kore
    },
    systemInstruction: { parts: [{ text: [ /* persona, rules, and this.context() */ ].join('\n') }] },
    tools: this.spotify?.enabled && this.spotify.connected
      ? [TOOLS[0], { functionDeclarations: [...TOOLS[1].functionDeclarations, ...SPOTIFY_TOOLS] }]
      : TOOLS,
    inputAudioTranscription: {},
    outputAudioTranscription: {},
  };
}

responseModalities: ['AUDIO'] is the only modality Gemini 3.8 Live speaks on this path; the two empty transcription objects ask for text transcripts of both sides, which is what the on-screen chat is built from. Note how the Spotify tools are only declared when Spotify is actually connected: fewer tools means fewer wrong tool picks.

Step 3: Open the WebSocket and Stream Audio

Back in the browser, the session is one WebSocket and a handful of message types. This is the core of open() in web/src/assistant.ts:

const res = await fetch('/api/assistant/session', { method: 'POST' });
const d = await res.json();
ws = new WebSocket(d.url);

ws.onopen = () => ws?.send(JSON.stringify({ setup: {} }));

The setup message is empty on purpose: everything is already in the token. Gemini answers with setupComplete, and from then on the browser pushes audio:

const sendAudio = (pcm: ArrayBuffer) =>
  ws?.readyState === WebSocket.OPEN &&
  ws.send(JSON.stringify({ realtimeInput: { audio: { data: b64(pcm), mimeType: `audio/pcm;rate=${IN_RATE}` } } }));

if (m.setupComplete) {
  setPhase('listening');
  if (prompt) ws?.send(JSON.stringify({ realtimeInput: { text: prompt } }));   // "[briefing]"
  for (const c of pending) sendAudio(c);          // the 2.5 s pre-roll
  onChunk = (pcm) => { if (!isSpeaking()) sendAudio(pcm); };
  maxTimer = window.setTimeout(close, MAX_SESSION_MS);
  armIdle();
  return;
}

Three things worth noticing:

The pre-roll goes first. Those 25 buffered chunks are flushed before live audio starts, so the question you began before the wake word fired arrives intact.
A text prompt can open the conversation. The scheduled morning briefing sends [briefing] as realtimeInput.text, and the system instruction tells the model what a briefing is. Same session, same voice, no audio needed to start it.
Half-duplex. onChunk drops microphone audio while Dash is speaking. A wall tablet plays through speakers, and without reliable echo cancellation the model would hear itself. You can still interrupt: tap the card and playback stops.

The server's side of the conversation is a few message shapes, all handled in one onmessage:

if (m.toolCall) return runTools(m.toolCall.functionCalls || []);
if (m.goAway) return close();
const sc = m.serverContent;
if (!sc) return;
if (sc.inputTranscription?.text) say('user', sc.inputTranscription.text);   // what you said
for (const p of sc.modelTurn?.parts || []) {
  if (p.inlineData?.data) play(p.inlineData.data);                           // 24 kHz PCM, base64
}
if (sc.outputTranscription?.text) say('dash', sc.outputTranscription.text);  // what Dash said
if (sc.interrupted) stopPlayback();
if (sc.turnComplete) { /* arm the idle timer, or close if the model said goodbye */ }

Message	Meaning
`setupComplete`	Session is ready; start sending `realtimeInput`
`serverContent.inputTranscription`	Streaming transcript of the user's speech
`serverContent.modelTurn.parts[].inlineData`	Chunks of 16-bit PCM audio at 24 kHz, base64
`serverContent.outputTranscription`	Streaming transcript of what the model is saying
`serverContent.interrupted`	The user barged in; stop playing what you have queued
`serverContent.turnComplete`	The model finished its turn
`toolCall.functionCalls[]`	The model wants you to run functions and reply with `toolResponse`
`goAway`	The server is about to close the connection

Playback is plain Web Audio: decode the base64 into Int16, scale to floats, schedule each buffer right after the previous one on a 24 kHz AudioContext. The playHead bookkeeping is what lets Dash know how long it will still be speaking, which drives both the half-duplex gate and the idle timer.

function play(b64: string) {
  outCtx ??= new AudioContext({ sampleRate: OUT_RATE });
  const bin = atob(b64);
  const pcm = new Int16Array(bin.length / 2);
  for (let i = 0; i < pcm.length; i++) pcm[i] = (bin.charCodeAt(i * 2) | (bin.charCodeAt(i * 2 + 1) << 8)) << 16 >> 16;
  const buf = outCtx.createBuffer(1, pcm.length, OUT_RATE);
  const ch = buf.getChannelData(0);
  for (let i = 0; i < pcm.length; i++) ch[i] = pcm[i] / 0x8000;
  const src = outCtx.createBufferSource();
  src.buffer = buf;
  src.connect(outCtx.destination);
  playHead = Math.max(playHead, outCtx.currentTime + 0.05);
  src.start(playHead);
  playHead += buf.duration;
  speakingUntil = Date.now() + (playHead - outCtx.currentTime) * 1000;
}

Step 4: Let Gemini Act Through Tools

A voice assistant that can only talk is a radio. The useful part is that Gemini can change what is on the screen. Dash declares its tools once, in the setup locked into the token, and runs them on the server.

The declarations are plain function schemas, plus Google Search as a built-in tool:

const S = (description) => ({ type: 'STRING', description });

export const TOOLS = [
  { googleSearch: {} },
  {
    functionDeclarations: [
      {
        name: 'add_to_shopping_list',
        description: 'Add one or more items to the household shopping list shown on the display.',
        parameters: { type: 'OBJECT', properties: { items: { type: 'ARRAY', items: S('One item, short, e.g. "milk" or "2 kg of oranges"') } }, required: ['items'] },
      },
      {
        name: 'start_timer',
        description: 'Start a kitchen timer on the display: for a duration ("10 minutes") or until a clock time ("at 18:30", an alarm). Up to 6 at once.',
        parameters: {
          type: 'OBJECT',
          properties: {
            minutes: { type: 'NUMBER', description: 'Duration in minutes (may be fractional). Omit when `at` is given.' },
            at: S('Local clock time "HH:MM" to ring at (today, or tomorrow if already past). Omit when `minutes` is given.'),
            label: S('Optional label, e.g. "pasta"'),
          },
        },
      },
      {
        name: 'remember',
        description: 'Save a fact the household wants Dash to remember (people, phone numbers, codes, preferences, where things are). Store it as one self-contained sentence.',
        parameters: { type: 'OBJECT', properties: { fact: S('E.g. "The plumber is Juan, phone 600 123 456"') }, required: ['fact'] },
      },
      {
        name: 'start_recipe',
        description: 'Open cooking mode on the display with a recipe: big step-by-step instructions the user follows while cooking. ... Every step that takes time must say how long in minutes, with a number ("cook the spaghetti for 9 minutes"): the display turns those into one-tap timers. Never write "until al dente" without a time.',
        parameters: {
          type: 'OBJECT',
          properties: {
            title: S('Dish name'),
            servings: S('E.g. "4 servings"'),
            ingredients: { type: 'ARRAY', items: S('One ingredient with quantity') },
            steps: { type: 'ARRAY', items: S('One short step, with its time in minutes when it takes time') },
          },
          required: ['title', 'ingredients', 'steps'],
        },
      },
      { name: 'end_conversation', description: 'Close the assistant when the user is done (e.g. "thanks", "that is all"). Say a short goodbye first.' },
      // ... remove_from_shopping_list, get_shopping_list, add_note, list_timers, add_time_to_timer,
      //     cancel_timer, forget, open_saved_recipe, recipe_step, start_step_timer, stop_cooking
    ],
  },
];

The full list is eighteen functions, plus three Spotify ones when it is connected. The start_recipe description is my favourite: it is a prompt disguised as a schema. Telling the model that every timed step needs a number of minutes is what makes the "Timer 8 min" button in cooking mode appear by itself.

When the model decides to act, the browser receives toolCall, forwards each call to the server, and sends the results back as toolResponse:

async function runTools(calls: { id: string; name: string; args?: Record<string, unknown> }[]) {
  setPhase('thinking');
  const functionResponses = [];
  for (const c of calls) {
    let response: unknown;
    try {
      response = await (await fetch('/api/assistant/tool', {
        method: 'POST', headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ name: c.name, args: c.args || {} }),
      })).json();
    } catch {
      response = { error: 'tool failed' };
    }
    if (c.name === 'end_conversation') ending = true;
    const done = actionText(c.name, c.args || {}, response as Record<string, unknown>);
    if (done) log.push({ role: 'action', text: done });   // the green receipt on screen
    functionResponses.push({ id: c.id, name: c.name, response });
  }
  ws?.send(JSON.stringify({ toolResponse: { functionResponses } }));
}

On the server, tool() is a switch over the names, working against the local JSON database and the timers service. The responses are deliberately rich: add_to_shopping_list returns both what was added and the full list afterwards, so the model can say "milk is already on it" instead of pretending it added a duplicate:

case 'add_to_shopping_list': {
  const added = this.db.addShopping(args.items);
  this.onChange('shopping');
  return { added: added.map((x) => x.text), list: this.db.shopping().filter((x) => !x.done).map((x) => x.text) };
}
case 'start_timer': {
  let seconds = Math.round(Number(args.minutes) * 60);
  const at = String(args.at || '').match(/^(\d{1,2}):(\d{2})$/);
  if (at) {
    const today = new Date().toLocaleDateString('en-CA', { timeZone: this.config.timezone });
    let when = localToEpoch(`${today}T${at[1].padStart(2, '0')}:${at[2]}`, this.config.timezone);
    if (when != null && when <= Date.now()) when += 864e5;
    seconds = when == null ? NaN : Math.ceil((when - Date.now()) / 1000); // ring on the minute, never early
  }
  if (!(seconds >= 5 && seconds <= 24 * 3600)) return { error: 'give minutes (up to 1440) or a clock time "HH:MM"' };
  const tm = this.timers.add(seconds, String(args.label || '').slice(0, 40));
  return { started: true, minutes: Math.round(seconds / 6) / 10, endsAt: tm.endsAt, label: tm.label || null };
}

Errors go back to the model as data, not as exceptions. "Which timer? say its label" is a tool response, and the model turns it into a question.

One Gemini 3.8 Live detail you should know: function calls are asynchronous by default (NON_BLOCKING). The model can keep talking while a tool runs, and schedule the result for when it is idle or interrupt itself with it. For Dash's tools, which take milliseconds, the difference is invisible; for a slow tool you would set BLOCKING on the declaration.

Step 5: Give Gemini the Household Context

The system instruction is rebuilt for every session, and the second half of it is a snapshot of what Dash already knows. That is why "when do Arsenal play next?" is answered from the display's own data instead of a web search, and why "add it to the list" works when there is only one list.

/** What Dash knows right now, so "when do Valencia play?" doesn't need a web search. */
context() {
  const tz = this.config.timezone;
  const lines = [`Now: ${now.toLocaleString('en-GB', { timeZone: tz, dateStyle: 'full', timeStyle: 'short' })} (${tz}).`];
  const w = this.store.get('weather')?.data;
  if (w) lines.push(`Weather now: ${w.current.temp}°C, ${w.current.desc}; today ${w.today.min}–${w.today.max}°C, rain chance ${w.today.precip}%.`);
  for (const c of teams.slice(0, 10)) { /* last result and next game per followed team */ }
  if (events.length) lines.push(`Calendar: ${events.map((e) => `${e.title || 'event'} (${e.allDay ? e.start : fmt(e.start)})`).join('; ')}.`);
  if (alerts.length) lines.push(`Weather warnings: ...`);
  if (hol?.today) lines.push(`Today is a public holiday: ${hol.today.name}.`);
  if (reminders.length) lines.push(`Reminders today: ...`);
  if (bins.length) lines.push(`Bins: ${bins.join('; ')}.`);
  if (news.length) lines.push(`Top headlines: ${news.map((n) => n.title).join(' | ')}.`);
  const cook = this.cooking?.current();
  if (cook) {
    lines.push(`COOKING MODE is open: "${cook.title}" (${cook.servings || 'servings not given'}), on step ${cook.step + 1} of ${cook.steps.length}: "${cook.steps[cook.step].text}".`);
    lines.push(`Ingredients: ${cook.ingredients.join('; ')}.`);
  }
  if (music?.item) lines.push(`Spotify: ${music.playing ? 'playing' : 'paused'} "${music.item.name}" by ${music.item.by}...`);
  const memories = this.db.memories();
  if (memories.length) lines.push(`Things the household asked you to remember:\n${memories.map((m) => `- ${m.text}`).join('\n')}`);
  lines.push(shopping.length ? `Shopping list: ${shopping.map((x) => x.text).join(', ')}.` : 'The shopping list is empty.');
  if (timers.length) lines.push(`Running timers: ...`);
  return lines.join('\n');
}

The memories deserve a word. "Remember that the plumber is Juan, 600 123 456" calls the remember tool, which stores one sentence locally. Every later session gets those sentences in the context, and the instruction tells the model to quote numbers and codes exactly. They live in Dash's own data/db.json, and Settings lists them so you can delete any. It is a simple mechanism, and it is enough: a kitchen does not need a vector database.

The persona rules are short and shaped by the fact that the output is spoken: no lists, no markdown, no URLs, one or two sentences, answer first and then maybe a follow-up, never claim a tool ran without calling it, and reply in the language the user is speaking.

Step 6: End the Conversation on Purpose

A per-minute API needs a clear end. Dash closes a session in four ways:

Silence. After the model finishes a turn, an idle timer waits for Dash to finish speaking plus idleSeconds (8 by default). If nothing was heard since, the session closes.
"Thanks." The end_conversation tool exists so the model can hang up gracefully: say goodbye, call the tool, and the browser closes once the goodbye has played.
Five minutes. A hard cap, MAX_SESSION_MS, so a kitchen full of chatter cannot keep the meter running.
goAway. When Gemini announces it is about to drop the connection, Dash closes cleanly instead of waiting for the error.

function armIdle() {
  clearTimeout(idleTimer);
  // Wait for Dash to finish talking, then give the user a few seconds to follow up.
  const wait = Math.max(0, speakingUntil - Date.now()) + (cfg()?.idleSeconds ?? 8) * 1000;
  idleTimer = window.setTimeout(() => (ending || !heardSinceTurn ? close() : armIdle()), wait);
}

Closing plays a short down-chime (two oscillators, no audio file), stops playback, restarts the wake-word recognizer a second and a half later, and tells the server to restore the Spotify volume. While the assistant is open, Dash lowers the music ("ducking") so the model does not have to shout over it.

What Gemini 3.8 Live Changes

Dash ships pinned to gemini-3.8-live, the stable Live model Google updated in September 2026. If you are coming from gemini-3.1-flash-live-preview, these are the differences that affect a voice assistant:

	Gemini 3.8 Live
Inputs / outputs	Text, image, audio, video in; text and audio out
Context	131,072 input tokens, 65,536 output tokens
Function calling	Supported, asynchronous by default (`NON_BLOCKING`); `BLOCKING`, `SILENT`, `WHEN_IDLE`, `INTERRUPTED` scheduling available
Google Search grounding	Supported, 5,000 free requests a month shared across Gemini 3+ models
Thinking	Interleaved reasoning built in; `thinking_level` is not a setup option any more
Proactive audio	Always on (the model decides when a sound is addressed to it); setting it to `false` is an error
Affective dialog	Removed from the API
Transcription	Input and output transcription, opt in per session
Variant	`gemini-3.8-live-extended-thinking` for multi-step reasoning alongside speech

Two of those matter for Dash's design. Proactive audio being always on is why the pre-roll buffer and the half-duplex gate are enough: the model is already good at ignoring background noise and its own echo. And asynchronous function calling is why the "thinking" phase on screen is so short: the model does not stall while a tool runs.

What It Costs and Where Your Audio Goes

Gemini is on the line only during a conversation, and this is what each state costs:

State	What runs	Gemini cost
Waiting for "Dash"	The browser's speech recognition	none
Talking	A live Gemini session	per minute
Closed	After 8 s of silence, on "thanks" or ✕, or after 5 minutes at most	none

On the paid tier, gemini-3.8-live audio is priced at $0.005 per minute in and $0.018 per minute out, with the first 5,000 Google searches a month included. A typical question is a few seconds each way, around half a cent. Twenty questions a day comes to roughly $3 a month. The free tier costs nothing, with rate limits, and Google may use free-tier data to improve its products, which is the main reason to pay.

On privacy, the three facts you need are the three design rules from the architecture section: the key stays on the server, audio goes from the browser straight to Gemini only during a session, and the wake word runs on the browser's speech service. Memories are stored locally and sent to Gemini as context in each conversation, so do not store anything you would not say to it.

Gemini Beyond the Voice: Recipes and Translation

The Live API is the headline, but Dash uses plain generateContent on gemini-3.8-flash in two quieter places, both in the kitchen.

Any recipe link, turned into steps. Most recipe sites embed a schema.org Recipe card, and that is what Dash parses first. For pages without one, Gemini reads the page text and returns the same shape, with structured output enforced by a response schema:

async function recipeWithGemini(text, url, { key, model, language }) {
  const res = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/${model}:generateContent`, {
    method: 'POST',
    headers: { 'x-goog-api-key': key, 'content-type': 'application/json' },
    signal: AbortSignal.timeout(40000),
    body: JSON.stringify({
      contents: [{ parts: [{ text: `Extract the recipe from this web page. Keep the original language (${language} if unclear). Steps are short and in order. If there is no recipe, return an empty title.\n\nURL: ${url}\n\n${text.slice(0, 30000)}` }] }],
      generationConfig: {
        responseMimeType: 'application/json',
        responseSchema: {
          type: 'OBJECT',
          properties: {
            title: { type: 'STRING' },
            servings: { type: 'STRING' },
            ingredients: { type: 'ARRAY', items: { type: 'STRING' } },
            steps: { type: 'ARRAY', items: { type: 'STRING' } },
          },
          required: ['title', 'ingredients', 'steps'],
        },
      },
    }),
  });
  const d = await res.json().catch(() => ({}));
  if (!res.ok) throw new Error(d.error?.message || `Gemini HTTP ${res.status}`);
  return normalizeRecipe(JSON.parse(d.candidates?.[0]?.content?.parts?.[0]?.text || '{}'), url);
}

"What to cook" in your language. The daily ideas come from TheMealDB, which is English only. With a key set and the display in another language, titles and full recipes are translated in one structured call, with the same responseSchema trick.

No SDK in either case. Dash's whole Gemini integration is fetch against the REST API and one WebSocket, which is a good reminder of how small the surface actually is.

Turning It On

The assistant is a few lines of configuration on top of a running Dash:

"features": { "assistant": true },
"assistant": {
  "model": "gemini-3.8-live",
  "voice": "Kore",
  "wakeWords": ["dash"],
  "idleSeconds": 8,
  "allowRemote": false
},
"briefing": { "time": "07:30", "weekdays": [1, 2, 3, 4, 5] }

Any Gemini prebuilt voice works in voice (Kore, Puck, Charon, and the rest). wakeWords can be anything the browser's recognizer can hear reliably. briefing.time opens a session by itself at that time and asks for the morning summary: weather, calendar, your teams, bins, one headline. On a Windows kiosk, windows/edge-kiosk-policy.reg allows the microphone for http://localhost:3000 without a prompt.

Then say "Dash", or "Hey Dash, what's the weather?" in one breath, or tap the microphone button at the bottom left. A blue dot on that button means the wake word is listening.

Conclusion

The Gemini Live API turns a voice assistant from a systems-integration project into an application project. The protocol is small: one token request on your server, one WebSocket in the client, PCM in, PCM out, and toolCall and toolResponse for everything that changes the world. Gemini 3.8 Live adds the pieces that used to be the hard part, interruptions, language detection, asynchronous tools and search grounding, and leaves you with the decisions that are actually yours: when to open a session, what the model is allowed to do, and what it knows about your house.

Dash makes those decisions one way: free wake word in the browser, ephemeral tokens with the setup locked in, tools on the server, memories on disk, and a session that ends when you say thanks. The code is at github.com/xavidop/dash. Put it on a tablet, add your teams, and talk to your kitchen.

Durable Genkit Flows with Temporal: Introducing the genkitx-temporal Plugin

Xavier Portilla Edo — Tue, 29 Sep 2026 14:23:24 +0000

Introduction

Anyone who has shipped a non-trivial Gen AI feature to production has hit the same wall: the happy path is fun to demo, but the unhappy path is brutal. Model providers throw 5xx errors. Rate limits kick in at the worst possible moment. A long-running agent gets halfway through a 12-step plan and the pod is restarted by Kubernetes. A user closes the tab in the middle of a streaming response and you have no idea what state the flow ended up in.

Genkit gives you a clean way to author those LLM-orchestrating pipelines as flows, but a Genkit flow, by itself, is just a function. It lives and dies with the process that invokes it. There is no durable history, no automatic retry policy, no built-in cancellation, no operations UI.

This is exactly the gap that Temporal was designed to close for general-purpose backend code, and it is the gap that the new genkitx-temporal plugin closes for Genkit flows.

In this article we will look at:

What the plugin actually does.
Why Temporal and Genkit are a particularly good match.
How to use it end to end: define, register, run.
When you should reach for it (and when you probably shouldn't).

What is `genkitx-temporal`?

genkitx-temporal is a Genkit plugin that lets you execute any Genkit flow inside a Temporal Workflow, transparently. You keep writing flows the way you always have; the plugin takes care of:

Registering the flow so a Temporal Worker can pick it up.
Wrapping each execution in a deterministic Temporal Workflow.
Running the non-deterministic LLM/tool/RAG work inside a Temporal Activity, so retries and timeouts are safe.
Giving you helpers to start workflows from any client.

Under the hood, the plugin ships a generic, deterministic workflow called runGenkitFlow that invokes a single Temporal Activity called runGenkitFlowActivity. The activity looks up your Genkit flow by name in an in-process registry and runs it inside a full Node environment, where all the messy, non-deterministic things (network calls, model calls, tool calls, RAG lookups) are perfectly fine.

┌────────────┐  start workflow   ┌──────────────────────┐
│  Client    │ ────────────────▶ │  Temporal Server     │
└────────────┘                   └─────────┬────────────┘
                                           │ task
                                           ▼
                                ┌──────────────────────┐
                                │  Worker process      │
                                │  ┌────────────────┐  │
                                │  │ runGenkitFlow  │  │  (workflow, sandboxed)
                                │  │       │        │  │
                                │  │       ▼        │  │
                                │  │ runGenkit-     │  │  (activity, full Node)
                                │  │ FlowActivity   │  │
                                │  │       │        │  │
                                │  │       ▼        │  │
                                │  │  your Genkit   │  │
                                │  │  flow (LLM…)   │  │
                                │  └────────────────┘  │
                                └──────────────────────┘

This split is the whole trick. Temporal Workflows are required to be deterministic so that they can be replayed from event history after a crash, deploy or scale event. LLM calls obviously are not deterministic. Putting the LLM work inside an Activity is the canonical Temporal pattern, and the plugin does it for you so you don't have to think about it.

Why Temporal and Genkit are a great match

It is easy to underestimate how much engineering is hiding behind "just call the model again if it fails". Once you start building real agents, the list of things you want from your runtime grows quickly. Genkit gives you the authoring primitives, and Temporal gives you the runtime guarantees. Together:

Automatic retries for transient errors. LLM providers regularly return 429s and 5xx. Tool calls hit the network. Vector stores time out. Temporal lets you express retry policies (exponential backoff, max attempts, non-retryable error types) declaratively, applied to every flow execution.
Durable history. Every event in your flow's execution is persisted by the Temporal Server. If your Worker pod is killed mid-flow, another Worker picks up the workflow exactly where it left off, with all prior activity results intact. No partial state, no double-charging the user, no orphan jobs.
Timeouts, heartbeats and cancellation. Long-running agents that browse, plan and call tools for minutes are a nightmare to control with plain HTTP. Temporal models start-to-close, schedule-to-close and heartbeat timeouts as first-class concepts, plus explicit cancellation semantics you can wire to a user closing a tab.
Operational visibility out of the box. The Temporal UI gives you a per-execution timeline of every workflow and activity. You can see inputs, outputs, retries, failures and stack traces, search by workflow id, terminate or signal running workflows. This complements the Genkit Developer UI nicely: Genkit's UI is your local debugging tool during development; Temporal's UI is your operational dashboard in staging and production.
Horizontal scalability for free. Need more throughput? Run more Worker processes pointing at the same task queue. The Temporal Server load-balances workflows and activities across them. Your flows did not need to change.
Long-running, human-in-the-loop friendly. Temporal workflows can sleep for days, wait for signals from external systems, and resume cleanly. Perfect for agents that need approval before executing a sensitive tool, or RAG pipelines that wait for a document indexing job to finish.

In other words, genkitx-temporal turns your Genkit flows from "smart functions inside a Node process" into first-class durable workloads without changing the way you write them.

Installation

npm install genkitx-temporal genkit

The Temporal SDK packages are peer-installed automatically as dependencies of the plugin.

You also need a running Temporal Server. For local development, the easiest path is:

brew install temporal
temporal server start-dev

This starts a Temporal Server on localhost:7233 and the UI on http://localhost:8233.

Usage end to end

The plugin exposes a small, focused API. Three things are enough to ship a durable Genkit flow: define it with the Temporal-aware helper, start a Worker, and execute it from a client.

1. Define a flow with `defineTemporalFlow`

defineTemporalFlow is a drop-in replacement for ai.defineFlow. The returned object is a normal Genkit flow, so you can still call it directly, expose it via the Developer UI, run it from tests, etc. The only difference is that it is also registered internally so a Temporal Worker can find it by name.

// flows.ts
import { genkit, z } from 'genkit';
import { googleAI } from '@genkit-ai/google-genai';
import { defineTemporalFlow, temporal } from 'genkitx-temporal';

export const ai = genkit({
  plugins: [
    googleAI(),
    temporal({ taskQueue: 'my-queue' }),
  ],
  model: googleAI.model('gemini-flash-latest'),
});

export const jokeFlow = defineTemporalFlow(
  ai,
  {
    name: 'jokeFlow',
    inputSchema: z.string(),
    outputSchema: z.string(),
  },
  async (subject) => {
    const { text } = await ai.generate(`Tell me a joke about ${subject}`);
    return text;
  },
);

A few things to notice:

You configure the plugin like any other Genkit plugin, passing the Temporal task queue (and optionally address, namespace, etc.).
The flow body is plain Genkit code. No Temporal-specific imports, no proxyActivities, no determinism gymnastics. The plugin handles all that for you.
The flow's name is also its Temporal registration key. Keep it unique within your Worker.

2. Start a Worker

Workers are the processes that actually execute your flows. A Worker imports your flows (so the registry is populated) and then calls startTemporalWorker:

// worker.ts
import './flows';   // side-effect import: registers the flows
import { startTemporalWorker } from 'genkitx-temporal';

startTemporalWorker({ taskQueue: 'my-queue' })
  .catch((e) => { console.error(e); process.exit(1); });

node ./dist/worker.js

You can run as many Worker processes as you want against the same task queue. Temporal will distribute work across them automatically. If a Worker dies mid-flow, another one picks up.

3. Execute a flow as a Temporal Workflow

From any client (an HTTP handler, a CLI, a cron job, another workflow), use executeTemporalFlow to start a workflow and await its result:

// client.ts
import { executeTemporalFlow } from 'genkitx-temporal';
import { jokeFlow } from './flows';

const result = await executeTemporalFlow(jokeFlow, 'cats', {
  taskQueue: 'my-queue',
});
console.log(result);

If you don't want to block on the result, use startTemporalFlow instead. It returns the raw Temporal WorkflowHandle, which lets you query, signal, or cancel the running workflow later. This is the building block for human-in-the-loop scenarios, scheduled flows, fan-out/fan-in patterns, and so on.

Configuration

temporal(options) and every helper accept the same connection options. Anything you don't pass falls back to environment variables, then to sensible defaults:

Option	Env var	Default
`address`	`TEMPORAL_ADDRESS`	`localhost:7233`
`namespace`	`TEMPORAL_NAMESPACE`	`default`
`taskQueue`	`TEMPORAL_TASK_QUEUE`	`genkit`

This makes it straightforward to run the same code locally against a dev server and in production against Temporal Cloud or a self-hosted cluster.

Advanced: combine with your own workflows and activities

The bundled runGenkitFlow workflow is enough for the common case. If you want to mix Genkit flows with your own existing Temporal workflows and activities, the plugin lets you bring your own:

await startTemporalWorker({
  taskQueue: 'my-queue',
  workflowsPath: require.resolve('./my-workflows'),
  activities: { ...require('./my-activities') },
});

// my-activities.ts
export { runGenkitFlowActivity } from 'genkitx-temporal/activities';
export async function myOtherActivity(/* ... */) { /* ... */ }

Re-exporting runGenkitFlowActivity keeps the built-in workflow working, so you can compose Genkit flows alongside hand-written activities for the parts of your system that aren't AI.

API summary

The full public surface is tiny:

temporal(options?) — the Genkit plugin.
defineTemporalFlow(ai, config, fn) — define a flow and register it for Temporal execution.
startTemporalWorker(options?) — start a Worker process.
executeTemporalFlow(flow, input, options?) — run a flow inside a Workflow and await the result.
startTemporalFlow(flow, input, options?) — same, but returns the WorkflowHandle for fire-and-forget / signalling.
runGenkitFlowActivity — the underlying activity, re-exported so you can combine it with your own activities.
registerTemporalFlow(name, flow) — manually register a flow that was defined elsewhere (useful when wrapping flows you don't own).

A small API surface is the point. The plugin is intentionally a thin bridge between two well-designed systems; it does not try to reinvent either of them.

When to reach for `genkitx-temporal`

Not every flow needs a durable runtime. A streaming chat response that takes 800ms and either succeeds or is retried by the user is fine running on a plain HTTP handler.

You will feel the benefits the moment your flows look like one of these:

Multi-step agents that orchestrate several model calls and tool invocations, where partial progress is expensive to throw away.
Long-running pipelines (document ingestion, batch summarization, fine-tuning prep) where individual steps can take minutes and the process must survive deploys.
Critical business workflows (refunds, account changes, contract generation) where you cannot afford to lose state or accidentally execute a step twice.
Human-in-the-loop agents that need to pause for approval, an external webhook, or a manual review before proceeding.
Anything you currently glue together with a job queue, a retry library, and a state machine. Temporal subsumes all three, and genkitx-temporal plugs Genkit straight into it.

If your team already runs Temporal for non-AI workloads, this plugin is a no-brainer: it lets your Gen AI features inherit all the operational maturity your platform team has already built around it.

How it pairs with the rest of the Genkit ecosystem

The thing I like the most about this plugin is that it composes cleanly with everything else Genkit gives you, not just one or two features:

Genkit Developer UI. Because defineTemporalFlow returns a normal Genkit flow, you can still iterate on it locally with the Genkit Developer UI: fast feedback loop during development, durable execution in production.
Genkit middleware. Middleware (in-call retries, fallbacks, skill injection, tool approval, prompt rewriting, etc.) runs inside the activity. You get two complementary layers of resilience: middleware for fine-grained in-call recovery, Temporal for whole-flow durability and replay.
Tools and function calling. Tools defined with ai.defineTool are just normal flow code from Temporal's perspective. Their calls, retries and outputs appear in both the Genkit trace and the Temporal event history.
RAG primitives. Retrievers, indexers, embedders and rerankers all live inside the activity. That means heavy ingestion jobs (chunking, embedding, upserting into a vector store) inherit Temporal's retry policies and survive restarts mid-batch.
Evaluators and datasets. Genkit's evaluators are flows like any other, so you can run eval jobs as Temporal Workflows, schedule them, fan them out across Workers, and inspect every run in the Temporal UI.
Prompts and Dotprompt. Versioned .prompt files, prompt registries and structured outputs all work unchanged. The flow body is plain Genkit.
Plugins and model providers. Any Genkit plugin (Google AI, Vertex AI, OpenAI, Anthropic, Ollama, local models, vector stores, etc.) plugs in as usual; the Temporal layer doesn't care which provider is on the other side of the call.
Telemetry and tracing. Genkit's OpenTelemetry traces continue to be emitted from inside the activity, so they show up in whatever observability backend you already use, alongside Temporal's own event history.
Deployment surfaces. Flows can still be exposed as HTTP endpoints, Cloud Functions, Firebase callable functions or Express handlers; executeTemporalFlow is just one more entry point, and you can mix and match (e.g. quick chat requests over HTTP, long agent runs through Temporal).
Multi-language story. Genkit is available in JS/TS, Go, Python (preview), Dart/Flutter (preview) and through a community Java SDK. This particular plugin targets JS/TS, but the architectural pattern — define your AI logic in Genkit, run it as a Temporal Workflow — is reusable across runtimes thanks to Temporal's polyglot SDKs.

Conclusion

Genkit is a great way to author Gen AI logic. Temporal is a great way to run any long-running, failure-prone workload. genkitx-temporal is the missing adapter between the two: a small, focused plugin that turns every Genkit flow into a durable, retryable, observable Temporal Workflow without asking you to rewrite a single line of business logic.

If you are building anything more ambitious than a single-turn chat endpoint, give it a try:

Source: github.com/xavidop/genkitx-temporal
Docs: xavidop.github.io/genkitx-temporal
Runnable example: examples/test-app

Your future on-call self will thank you the first time a model provider has a bad afternoon and your agents keep humming along regardless.

Why a Local Debugging Tool is Non-Negotiable for Building AI Apps: Genkit Developer UI vs Vercel AI SDK DevTools vs Mastra Studio

Xavier Portilla Edo — Tue, 29 Sep 2026 14:23:17 +0000

Introduction

If you have ever shipped a backend service without a debugger, a hot-reload dev server or decent logs, you remember how painful it was. Now imagine doing it in a world where your "function" is a non-deterministic black box that can rewrite its own output every time you call it. Welcome to building Gen AI applications.

The single biggest productivity multiplier I have found in the last two years of shipping AI apps is a good local debugging tool. Not a hosted dashboard. Not a cloud trace viewer with a 30-second propagation delay. A local tool that runs next to your code, picks up your edits, lets you replay a request with a tweaked prompt, and shows the full trace, tool call by tool call. We will get into the local-vs-hosted trade-off later in the article.

This article is about why that matters and how the three leading JS/TS Gen AI frameworks approach it:

They are three very different products solving overlapping problems, and each one makes different trade-offs that are worth understanding before you commit to one.

Shift-left, applied to Gen AI

"Shift-left" comes from the world of testing and security: the earlier in the development lifecycle you catch a problem, the cheaper it is to fix. Bugs found in production cost orders of magnitude more than bugs found while you're typing.

Gen AI apps make this principle existential, not just economical, because the failure modes are weirder:

A prompt regression doesn't show up as a stack trace. It shows up as the wrong tone, a hallucinated fact or a tool call that misfires once every twenty runs.
A new model version can quietly change behavior across thousands of code paths with no warning.
A tool that returns slightly different JSON can cause silent downstream breakage.

The only way to keep your sanity is to collapse the feedback loop to seconds. You want to be able to:

Change a prompt or a piece of orchestration code.
Re-run that exact unit, with that exact input.
See the model's input, its output, every tool call, every retry, every token used.
Compare it to the previous run.
Decide if it's better, worse or different.

If any one of those steps requires deploying, redeploying, opening a cloud console or grep'ing logs, you have already lost. Cycle time is the metric. A debugging tool is what makes that cycle time low enough to iterate productively.

Local vs hosted

It is worth being explicit about this trade-off, because all three tools in this article are primarily local:

A local debugging tool runs on your machine, next to your code, with zero network latency between your edits and what you see. Iteration is fast, traces are immediate, and there is no risk of leaking prompts/responses to a third party. The downside is that it's just for you.
A hosted observability platform (Langfuse, LangSmith, cloud-native APM tools, etc.) is shared by your team, persists data long-term and is essential for production monitoring. The trade-off is propagation delay, configuration overhead and, in some cases, data residency concerns.

These are complementary, not competing. The argument here is that the local part of the loop is the one that's still often missing, and that's where shift-left lives.

What "good" looks like

After working with all three of the tools below, I would summarize the qualities of a good Gen AI debugging tool as:

Zero-config or near-zero-config — it should pick up your code, not the other way around.
Live reload — edit your code, save, hit "run" again. No restart.
Interactive runners — invoke any prompt, tool, model, or higher-level primitive (agent, workflow, etc.) directly with arbitrary input.
Full traces — every step of the generation loop, with input/output/usage at each node.
Replay and modify — re-run any past trace with a tweak.
Minimal code changes — the framework should be observable by default, ideally without you having to remember to wrap each model call.

Let's see how each tool stacks up.

Genkit Developer UI

The Genkit team treats the Developer UI as a first-class part of the framework. It ships with the genkit-cli package and requires no code changes in your application to attach to your running process.

You install the CLI once globally:

npm install -g genkit-cli

And then start your app under it:

genkit start -- npx tsx --watch src/index.ts

That's it. Genkit attaches to your running Node process, discovers every flow, prompt, model, tool, retriever, indexer, embedder and evaluator you have defined, and exposes them all in a local web app at http://localhost:4000.

Telemetry API running on http://localhost:4033
Genkit Developer UI: http://localhost:4000

What you actually get inside:

Interactive runners for every primitive. Flows, prompts, tools, models, retrievers, indexers, embedders and evaluators all get an interactive panel where you fill out the input (validated against the Zod schema) and hit run.
Full traces with the entire generation graph: every model call, every tool invocation, every middleware, with input/output/usage tokens.
Live reload. Combined with --watch, edits to your code show up without restarting the UI.
Prompt iteration. Tweak prompts and re-run inline.
Evals. Run evaluators against datasets directly from the UI.
Open by default. Add -o to auto-open in your browser.

What stands out here is that observability comes for free. You do not wrap your model and you do not add a middleware just to see traces. Anything defined with the Genkit primitives is automatically introspected and traced, which makes the "let me see what's happening" cost essentially zero.

If you also use the middleware system, every middleware also shows up in the trace, which makes debugging things like retries and fallbacks a lot easier.

Vercel AI SDK DevTools

Vercel introduced the AI SDK DevTools more recently, and it is a meaningful step up from having nothing. The design philosophy is different from Genkit's: it is a middleware-based capture tool that focuses tightly on the language-model layer, rather than an integrated development environment for the whole AI app.

You opt in by wrapping your model:

import { wrapLanguageModel, gateway } from 'ai';
import { devToolsMiddleware } from '@ai-sdk/devtools';

const model = wrapLanguageModel({
  model: gateway('anthropic/claude-sonnet-4.5'),
  middleware: devToolsMiddleware(),
});

And then run the viewer in another terminal:

npx @ai-sdk/devtools
# open http://localhost:4983

What it captures:

Input parameters and prompts.
Output content and tool calls.
Token usage and timing.
Raw provider request/response payloads.
Multi-step interactions are grouped into "runs" with multiple "steps".

Things to keep in mind when comparing it with the others:

It is opt-in per model. Every model you want to observe has to be wrapped explicitly. In a real codebase with many models, this usually means a shared factory or repeated wrapping. If you forget on one path, you have a blind spot.
It focuses on the language-model layer. Anything happening above that layer (your own application logic, custom orchestration, business steps) is not captured unless you add it yourself.
It is a viewer, not an interactive playground. You cannot "re-run this with a different prompt" from the UI; you go back to your code, edit and re-call.
It does not enumerate AI primitives because, in the AI SDK, those are just functions in your codebase. There is nothing to enumerate at runtime.
Storage is a JSON file (.devtools/generations.json). The team is clear that this is local-only, never for production, and the middleware automatically appends .devtools to your .gitignore.

The upside is that it is probably the quickest of the three to drop into an existing project: install the package, wrap a model, run the viewer. If you live inside Next.js and the AI SDK and you mainly want a "what just happened?" panel for your model calls, it does the job well. If you are debugging a multi-step agent with custom orchestration, you will probably want to combine it with something that sees more of the picture.

Mastra Studio

Mastra Studio is the most ambitious of the three in terms of surface area. It is bundled with mastra dev and runs at http://localhost:4111, doubling as a local development tool and a deployable team console (you can ship Studio to production for non-developers).

Out of the box, it gives you:

Agents tab — chat with your agents, hot-swap models, tweak temperature/top-p, view traces, attach scorers.
Workflows tab — visualize workflows as graphs, run them step by step with custom inputs, watch the active step in real time, inspect tool calls and JSON outputs.
Processors tab — see input/output processors and guardrails wired to each agent.
Tools tab — run tools in isolation to debug them.
Workspaces tab — file browser into the agent's workspace filesystem, with a Skills tab listing discovered skills.
MCP servers tab — list attached MCP servers and their tools.
Request context — set runtime variables that flow into agent instructions/tools through DI, with schema-driven forms.
Evaluation suite — Scorers, Datasets and Experiments tabs to run datasets through agents/workflows, attach scorers and compare experiments side-by-side.
Observability with traces and logs.
Editor integration for non-technical teammates to iterate on agents and version every change without redeploying.

This is more than a debugging tool — it is closer to a development environment plus a team-facing console. The natural trade-off is that Studio is tightly coupled to Mastra's primitives (Agents, Workflows, Processors, Workspaces), so it shines the most when your application is structured the Mastra way.

If you build non-trivial agentic systems and you want evaluation and dataset management baked into the same tool, Studio is genuinely very good. For a quick "see my prompt and the model's response" loop, it can feel like more than you need.

Side-by-side comparison

Capability	Genkit Developer UI	Vercel AI SDK DevTools	Mastra Studio
Setup	`genkit start --` your code	`wrapLanguageModel(...)` + `npx @ai-sdk/devtools`	`mastra dev`
Code changes required	None (auto-discovers)	Yes (wrap each model)	None (auto-discovers Mastra primitives)
Live reload	Yes (`--watch`)	N/A (viewer only)	Yes
Interactive runners	Flow, prompt, model, tool, retriever, indexer, embedder, evaluator	None (viewer only)	Agent chat, workflow runner, tool runner
Traces	Full generation graph	Per-model-call run/step	Workflow + agent traces
Re-run / replay	Yes, from any primitive	No (must edit code and re-call)	Yes, including workflow step-through
Dataset / eval UI	Evaluators panel	Not built-in	Datasets + Experiments + Scorers
Workspace / files browser	No	No	Yes
MCP server browser	Not first-class	No	Yes
Deployable to team	No (local-only)	No (local-only)	Yes (Studio on Mastra platform)
Languages	JS/TS (Go, Python, Dart all have local tooling too)	JS/TS only	JS/TS only
Style	Auto-discovery of primitives	Per-model middleware capture	All-in-one studio for the agent lifecycle

A quick note on the Vercel column: simplicity of integration is a real feature for many teams, and it is fair to call this the lightest-touch option. The trade-off is that observability is opt-in per model, so you have to be deliberate to avoid blind spots.

When I would pick which

After using all three, this is roughly how I'd think about it:

Building a serious agent or multi-step pipeline with tools, retries, fallbacks and evals → Genkit Developer UI is hard to beat on pure iteration speed. Zero-config tracing of the full pipeline, every primitive runnable, and middleware shows up in the trace for free.
Working primarily inside Next.js with the Vercel AI SDK and you mainly want better visibility into model calls → Vercel AI SDK DevTools is a quick win. Lightweight setup and it does what it says on the tin; pair it with something else if you need higher-level orchestration views.
Building agentic systems with workflows, datasets and scorers, and you want a UI your PM or domain expert can also use → Mastra Studio. The evaluation suite and the deployability are real differentiators.
Multi-language stack (JS/TS plus Go, Python or Dart) → Genkit is the only one of the three with first-party multi-language support; the local tooling story is consistent across runtimes.

The moral of the story: pick a debugging tool, any debugging tool

In a normal backend project, you can sometimes get away with a weak dev loop because the code is deterministic. Bad UX, but it works.

In an AI project, the code is not deterministic. Every iteration is also a small experiment. If your loop is slow or your visibility is poor, you don't just iterate slower, you iterate worse, because you can't tell whether your changes are improvements or regressions. Shift-left in this world is not a virtue, it is a survival strategy.

A good local debugging tool:

Turns "I think the prompt got better" into "I can see the trace, the tokens, the eval score, side by side".
Catches regressions before they reach a user.
Lets you onboard new engineers in hours instead of days, because they can see the system.
Lets non-engineers (PMs, designers, domain experts) participate in iteration when the UI is friendly enough — Mastra Studio is explicit about this.

If you are starting a Gen AI project today and you are picking a framework, the local tooling story should weigh as much as the model abstraction or the tool API. It is the difference between writing AI code and engineering AI features.

Conclusion

All three tools are good in their lane:

Genkit Developer UI is the most introspective of the three and requires zero code changes to be useful. It's the one I reach for when I want to move fast and see every step of the pipeline.
Vercel AI SDK DevTools is the lightest-touch option to drop into an existing AI SDK app, with a clear focus on the model-call layer. Think of it as a focused viewer rather than a full development environment.
Mastra Studio is the most ambitious, closer to a full IDE-meets-console for agentic systems, with first-class evaluation, datasets and a deployable team UI. It's at its best when your app is structured around Mastra's primitives.

Pick the one that matches the shape of your project, but please pick something. Building Gen AI without a local debugging tool in 2026 is engineering with the lights off.

Top Gen AI Frameworks for Go in 2026: A Hands-On Comparison

Xavier Portilla Edo — Tue, 29 Sep 2026 13:41:18 +0000

Introduction

Go has quietly become the language of AI infrastructure. MCP servers, agent runtimes, model gateways, Kubernetes operators that talk to LLMs: a large share of that plumbing is written in Go, because Go is what you reach for when something has to be a small static binary that handles thousands of concurrent connections without drama. What was missing for a long time was the framework layer on top of the raw provider SDKs. In 2026 that gap is closed, and the question is no longer "Go or Python?" but "which Go framework?".

This article covers the five frameworks a Go developer will actually run into in 2026: Genkit Go, Eino, Google ADK Go, tRPC-Agent-Go, and LangChainGo. Four of them are alive and moving fast. The fifth, LangChainGo, is the one most Go developers heard of first, and it is now effectively unmaintained. It stays in the list because you will find it in existing codebases, and you need to know what that means for you.

Same rules as the Java and JavaScript/TypeScript editions: this is not a benchmark post. Tokens per second go stale in weeks. This is a developer experience and architecture comparison, with working code and the real output of that code for every framework, all against the same model, gemini-3.8-flash, on Go 1.27.

Versions used: Genkit Go v1.13.1, Eino v0.9.20, Google ADK Go v2.4.0, tRPC-Agent-Go v1.11.2, LangChainGo v0.1.14. APIs as of September 2026.

Genkit Go

History and Direction

Genkit was announced by Google at I/O 2024 as a TypeScript-first framework. The Go SDK arrived only three months later, with v0.1.0 in August 2024, and unlike the Java SDK, which is a community effort, Genkit Go is an official, first-party SDK maintained by the same google team that ships the TypeScript runtime. It reached 1.0 in September 2025 and sits at v1.13.1 as of September 2026, on a cadence of roughly one minor release per month.

The direction is parity with TypeScript plus Go-native ergonomics: generics instead of Zod schemas, context.Context everywhere, plain net/http handlers for flows, and an agents package, still marked experimental, with session stores, multi-agent delegation, artifacts, and a Skills middleware. The monorepo (genkit-ai/genkit) has 6.4k stars and 80 commits in the last 30 days across all SDKs.

If you want the long-form introduction, start with Stop Using Python for Your Gen AI Apps, Use Go and Genkit Instead.

What Makes Genkit Go Stand Out

Genkit Go gives you three levels of abstraction in a single SDK: direct model calls, typed flows, and agents, all traced by the same OpenTelemetry pipeline and all visible in the same Dev UI. On top of that sits a middleware layer (retry, fallback, tool approval, filesystem, skills) that wraps any of the three. No other Go framework gives you that combination.

Supported languages: Go 1.25+ for this SDK. The same framework exists for TypeScript, Python, Dart/Flutter, and, unofficially, Java.

Vanilla Generation

import (
    "github.com/genkit-ai/genkit/go/ai"
    "github.com/genkit-ai/genkit/go/genkit"
    "github.com/genkit-ai/genkit/go/plugins/googlegenai"
)

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithDefaultModel("googleai/gemini-3.8-flash"))

text, err := genkit.GenerateText(ctx, g,
    ai.WithPrompt("Explain the CAP theorem in two sentences."))

The CAP theorem states that a distributed data system can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are inevitable in the real world, systems must ultimately choose between returning the most up-to-date data (Consistency) or ensuring every request receives a response (Availability).

Typed Flows: Observable Pipelines

Flows are the heart of Genkit. A flow is a named, typed, traceable unit that you can call in-process, expose as an HTTP endpoint with one line, and run from the Dev UI. The JSON schema for the output comes from the Go struct, and GenerateData returns the struct, not a string you have to parse.

type TranslateRequest struct {
    Text           string `json:"text"`
    TargetLanguage string `json:"target_language"`
}

type TranslateResponse struct {
    Translation      string `json:"translation"`
    DetectedLanguage string `json:"detected_language"`
}

translate := genkit.DefineFlow(g, "translateText",
    func(ctx context.Context, req TranslateRequest) (*TranslateResponse, error) {
        out, _, err := genkit.GenerateData[TranslateResponse](ctx, g,
            ai.WithPrompt("Translate %q to %s and detect the source language.",
                req.Text, req.TargetLanguage))
        return out, err
    })

// Call it in-process...
out, err := translate.Run(ctx, TranslateRequest{Text: "Buenos días", TargetLanguage: "English"})
fmt.Printf("%+v\n", *out)

// ...or expose it over HTTP (and to the Dev UI).
mux := http.NewServeMux()
mux.HandleFunc("POST /translateText", genkit.Handler(translate))
log.Fatal(server.Start(ctx, "127.0.0.1:3400", mux))

{Translation:Good morning DetectedLanguage:Spanish}
INFO server listening addr=127.0.0.1:3400

curl -X POST http://127.0.0.1:3400/translateText \
  -H 'Content-Type: application/json' \
  -d '{"data":{"text":"Buenos días","target_language":"English"}}'
# {"result":{"translation":"Good morning","detected_language":"Spanish"}}

Tools and Agents

Tools are typed functions. The input schema is inferred from the struct, and the tool loop (model asks, tool runs, model answers) happens inside Generate.

type WeatherInput struct {
    City string `json:"city"`
}

weather := genkit.DefineTool(g, "getWeather", "Returns the current weather for a city.",
    func(ctx *ai.ToolContext, in WeatherInput) (string, error) {
        return fmt.Sprintf("Sunny, 24C in %s", in.City), nil
    })

resp, err := genkit.Generate(ctx, g,
    ai.WithPrompt("What's the weather like in Tokyo?"),
    ai.WithTools(weather))
fmt.Println(resp.Text())

The weather in Tokyo is currently sunny and 24°C.

When you need memory across turns, the experimental agents package adds sessions on top of the same tools. Enable it with genkit.WithExperimental():

import (
    aix "github.com/genkit-ai/genkit/go/ai/exp"
    "github.com/genkit-ai/genkit/go/ai/exp/localstore"
    genkitx "github.com/genkit-ai/genkit/go/genkit/exp"
)

advisor := genkitx.DefineAgent(g, "travelAdvisor",
    aix.InlinePrompt{
        ai.WithSystem("You are a helpful travel advisor. Use tools to give accurate advice. Be brief."),
        ai.WithTools(weather),
    },
    aix.WithSessionStore(localstore.NewInMemorySessionStore[any]()))

first, err := advisor.RunText(ctx, "Should I pack a jacket for my trip to Lisbon?")
second, err := advisor.RunText(ctx, "Which city did I ask about?",
    aix.WithSessionID[any](first.SessionID))

It is currently sunny and 24°C (75°F) in Lisbon. You won't need a heavy coat, but packing a light jacket or sweater is recommended for cooler evenings and coastal breezes.
You asked about Lisbon.

Multi-agent delegation, cross-vendor fallback, and tool approval are covered in Building Agents in Go: Where Genkit Shines vs ADK 2.0.

The Dev UI: Same Power as TypeScript

The same Genkit Developer UI that TypeScript developers use works with Go, with no code changes. Install the CLI and start your Go program through it:

npm install -g genkit
genkit start -- go run .

Genkit Developer UI: http://localhost:4000

The flow above shows up as /flow/translateText next to every Gemini model the plugin registered, and you get:

Flow runner: execute any flow with a custom input and inspect the typed output.
Trace explorer: OpenTelemetry traces for every Generate and flow call, with latency, token counts, and the exact prompts.
Model playground: call any registered model directly.
Tool testing: run tools in isolation.
Dotprompt editor: edit .prompt files live.

This is the single biggest advantage Genkit has over every other framework in this list: a zero-config, local trace explorer that replaces Langfuse or LangSmith during development.

Provider Support

Official plugins in the Go module: Google AI (Gemini), Vertex AI, Anthropic, Ollama, and an OpenAI-compatible family (compat_oai) with ready-made packages for OpenAI, xAI, DeepSeek, OpenRouter, Kimi, DashScope, and Z.ai. Vector stores: localvec (local, no service needed), PostgreSQL/pgvector, AlloyDB, Pinecone, Weaviate, and Firebase Firestore. Plus MCP (client and server), evaluators, and an A2UI plugin for streaming UI components.

Pros and Cons

✅ Pros	❌ Cons
Best-in-class Dev UI with local trace explorer	Agents package still marked experimental
Multi-level abstractions: vanilla, flows, agents	Fewer vector stores than LangChainGo
Middleware: retry, fallback, tool approval	Go 1.25+ required
Official Google SDK, monthly releases
Typed output from Go structs, no parsing
Same framework in TS, Python, Dart, Java

Eino

History and Direction

Eino (pronounced "aino") comes from ByteDance's CloudWeGo team, the group behind Kitex and Hertz, the RPC and HTTP frameworks that run a large part of ByteDance's Go services. It was open-sourced in December 2024 after internal use, and it is now the most-starred LLM framework in Go at 13.1k stars, ahead of LangChainGo. Releases land every few days: v0.9.20 on September 20, 2026, with v0.10.0 already at alpha 35.

Eino's own description is that it draws from LangChain and Google ADK but follows Go conventions. The framework splits in two: compose, for orchestration (Chain, Graph, Workflow), and adk, an Agent Development Kit that borrows Google ADK's vocabulary of agents, runners, and events. Components (models, tools, retrievers, indexers, loaders) live in a separate repo, eino-ext, together with callback handlers for Langfuse, LangSmith, CozeLoop, and ByteDance's APMPlus.

The 2026 roadmap is all agents: DeepAgent (planning plus sub-agents), agent middleware, interrupt and resume for human-in-the-loop, skills, A2UI streaming components, and a turn loop with preemption. The English documentation on cloudwego.io is now a full eleven-chapter course, a big change from 2025 when most of it was Chinese-first.

What Makes Eino Stand Out

Orchestration with automatic stream handling. Every node in an Eino graph can be invoked or streamed, and the framework concatenates, boxes, merges, and copies streams as data moves between nodes. You build the graph once and get both Invoke and Stream for free. Graph inputs and outputs are generic types, so the compiler checks that the nodes fit together. Cross-cutting concerns go through callbacks at fixed points (OnStart, OnEnd, OnError, plus the streaming variants) that apply to components, graphs, and agents alike.

Supported languages: Go only. The module declares go 1.18, the lowest bar in this list. One thing to know: Eino depends on ByteDance's sonic JSON library, and on Go 1.27 it prints a warning at startup and falls back to encoding/json. Harmless, but noisy.

ChatModel

Eino does not own the provider client. You create the genai client yourself and hand it to the component.

import (
    "github.com/cloudwego/eino-ext/components/model/gemini"
    "github.com/cloudwego/eino/schema"
    "google.golang.org/genai"
)

client, err := genai.NewClient(ctx, &genai.ClientConfig{APIKey: os.Getenv("GEMINI_API_KEY")})
cm, err := gemini.NewChatModel(ctx, &gemini.Config{Client: client, Model: "gemini-3.8-flash"})

msg, err := cm.Generate(ctx, []*schema.Message{
    schema.UserMessage("Explain the CAP theorem in two sentences."),
})
fmt.Println(msg.Content)

The CAP theorem states that a distributed data store can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance. Because network partitions are practically inevitable in distributed systems, architects must choose between keeping data strictly up-to-date (Consistency) or ensuring the system remains responsive (Availability) during a network failure.

Chains, Graphs, and Callbacks

A chain is a linear pipeline. Here the callback handler is attached at invocation time and fires for the chain, the template, and the model.

import (
    "github.com/cloudwego/eino/callbacks"
    "github.com/cloudwego/eino/components/prompt"
    "github.com/cloudwego/eino/compose"
)

template := prompt.FromMessages(schema.FString,
    schema.SystemMessage("You are a translator. Reply with the translation only."),
    schema.UserMessage("Translate {text} to {language}."))

chain := compose.NewChain[map[string]any, *schema.Message]()
chain.AppendChatTemplate(template).AppendChatModel(cm)
runnable, err := chain.Compile(ctx)

tracer := callbacks.NewHandlerBuilder().
    OnStartFn(func(ctx context.Context, info *callbacks.RunInfo, in callbacks.CallbackInput) context.Context {
        log.Printf("start %s %s", info.Component, info.Type)
        return ctx
    }).
    OnEndFn(func(ctx context.Context, info *callbacks.RunInfo, out callbacks.CallbackOutput) context.Context {
        log.Printf("end   %s %s", info.Component, info.Type)
        return ctx
    }).
    Build()

out, err := runnable.Invoke(ctx,
    map[string]any{"text": "Buenos días", "language": "English"},
    compose.WithCallbacks(tracer))
fmt.Println(out.Content)

start Chain
start ChatTemplate Default
end   ChatTemplate Default
start ChatModel Gemini
end   ChatModel Gemini
end   Chain
Good morning

Graphs are where Eino earns its name. Nodes are typed, edges are explicit, and plain Go functions slot in as lambda nodes:

graph := compose.NewGraph[map[string]any, string]()
_ = graph.AddChatTemplateNode("template", template)
_ = graph.AddChatModelNode("generate", cm)
_ = graph.AddLambdaNode("shout", compose.InvokableLambda(
    func(ctx context.Context, msg *schema.Message) (string, error) {
        return strings.ToUpper(strings.TrimSpace(msg.Content)), nil
    }))

_ = graph.AddEdge(compose.START, "template")
_ = graph.AddEdge("template", "generate")
_ = graph.AddEdge("generate", "shout")
_ = graph.AddEdge("shout", compose.END)

runnable, err := graph.Compile(ctx)
out, err := runnable.Invoke(ctx, map[string]any{"product": "a Go framework for LLM apps"})

BUILD PRODUCTION-GRADE LLM APPS AT GO SPEED.

Branches, parallel fan-out, and sub-graphs use the same API, and a compiled graph can be wrapped as a tool and handed to an agent.

Eino ADK: Agents and Workflows

Tools are inferred from a Go struct with jsonschema tags. The agent runs the ReAct loop and emits one event per step, so you see the tool call, the tool result, and the final answer as separate messages.

import (
    "github.com/cloudwego/eino/adk"
    "github.com/cloudwego/eino/components/tool"
    "github.com/cloudwego/eino/components/tool/utils"
)

type WeatherInput struct {
    City string `json:"city" jsonschema:"description=The city to look up"`
}

weather, err := utils.InferTool("get_weather", "Returns the current weather for a city.",
    func(ctx context.Context, in WeatherInput) (string, error) {
        return fmt.Sprintf("Sunny, 24C in %s", in.City), nil
    })

advisor, err := adk.NewChatModelAgent(ctx, &adk.ChatModelAgentConfig{
    Name:        "travel_advisor",
    Description: "Helps with trip planning and weather-based advice.",
    Instruction: "You are a helpful travel advisor. Use tools to give accurate advice. Be brief.",
    Model:       cm,
    ToolsConfig: adk.ToolsConfig{
        ToolsNodeConfig: compose.ToolsNodeConfig{Tools: []tool.BaseTool{weather}},
    },
})

runner := adk.NewRunner(ctx, adk.RunnerConfig{Agent: advisor})
iter := runner.Query(ctx, "Should I pack a jacket for my trip to Lisbon?")
for {
    event, ok := iter.Next()
    if !ok {
        break
    }
    if event.Err != nil {
        log.Fatal(event.Err)
    }
    if event.Output == nil || event.Output.MessageOutput == nil {
        continue
    }
    msg, _ := event.Output.MessageOutput.GetMessage()
    fmt.Printf("[%s] %s\n", msg.Role, msg.Content)
}

[assistant]
[tool] Sunny, 24C in Lisbon
[assistant] It is currently sunny and 24°C (75°F) in Lisbon. You won't need a heavy coat, but packing a light jacket or cardigan is a good idea for breezy coastal evenings.

The first assistant event is the tool call itself, which is why its content is empty. Multi-agent workflows are one constructor away:

pipeline, err := adk.NewSequentialAgent(ctx, &adk.SequentialAgentConfig{
    Name:        "content_pipeline",
    Description: "Research, then write.",
    SubAgents:   []adk.Agent{researcher, writer},
})

[researcher] 1. Created at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released in 2009.
2. Statically typed, compiled language with built-in garbage collection.
3. Features native, lightweight concurrency through goroutines and channels.

[writer] Created at Google by Robert Griesemer, Rob Pike, and Ken Thompson, the Go programming language was released in 2009. It is a statically typed, compiled language that features built-in garbage collection and native, lightweight concurrency through goroutines and channels.

NewParallelAgent and NewLoopAgent complete the set, and DeepAgent adds planning and delegation on top.

Developer Tooling

Eino's answer to observability during development is the Eino Dev plugin for GoLand and VS Code, backed by the devops module in eino-ext. It renders your chains and graphs visually inside the editor and lets you start execution from any node with mock input. It is an IDE feature rather than a browser Dev UI, so there is no trace explorer for a running service. For that you attach one of the callback handlers (Langfuse, LangSmith, CozeLoop, APMPlus).

Provider Support

Model components in eino-ext: OpenAI, Claude, Gemini, DeepSeek, Qwen, Ollama, OpenRouter, Ark (Volcano Engine), and Qianfan (Baidu), plus "agentic" variants of the main ones for the newer agentic message format. Retrievers and indexers: Elasticsearch (7, 8, 9), Milvus, Qdrant, Redis, OpenSearch, Dify, and VikingDB. Tools: MCP, Google Search, Bing, DuckDuckGo, SearXNG, Wikipedia, HTTP request, command line, browser use, and sequential thinking.

Pros and Cons

✅ Pros	❌ Cons
Most active independent Go LLM framework (13.1k stars)	Still 0.x with frequent alphas; expect API churn
Typed graph orchestration with automatic streaming	No browser Dev UI; visual debugging lives in the IDE plugin
Callback aspects for logging, tracing, and metrics	Verbose: config structs and explicit error checks everywhere
ADK-style agents plus DeepAgent and interrupt/resume	ByteDance-flavoured corners (Ark, VikingDB, CozeLoop)
Broad component ecosystem in `eino-ext`	`sonic` dependency warns on the newest Go toolchains
Runs on Go 1.18+	Chinese-first history still shows in some docs and examples

Google ADK Go

History and Direction

ADK, the Agent Development Kit, was announced at Google Cloud Next 2025 with Python as its first language. The Go SDK made its first tagged release, v0.1.0, in November 2025, went 1.0 on March 23, 2026, and shipped a new major module, google.golang.org/adk/v2, on June 30, 2026. It is at v2.4.0 (September 11, 2026), has 8.8k stars, and logged over 100 commits in the last 30 days: the highest commit volume in this list. It is an official Google product with the Python, Java, Kotlin, and TypeScript SDKs as siblings.

The design is the same as its siblings: everything is an agent, a workflow agent, or a tool. Gemini is the native model, with openaimodel for anything speaking the OpenAI wire format and an Apigee model for enterprise gateways. Deployment gravity points at Vertex AI Agent Engine, Cloud Run, and GKE, with Pub/Sub and Eventarc triggers built into the launcher. The minimum Go version is 1.26.6, the highest bar of the five.

A deeper look at the low-level differences with Genkit is in Genkit Go vs ADK Go 2.0.

ADK Go's Position: Agent-Only, Enterprise-Grade

ADK Go is an agent framework. There is no vanilla generation primitive and no flow abstraction: you always build an agent, give it to a runner, and iterate the events. In exchange you get sessions, memory, artifacts, workflow agents, agent-to-agent (A2A) protocol support, MCP toolsets, tool confirmation, and an embedded web UI, all first party.

Supported languages: Go 1.26.6+. Same framework in Python (primary), Java, Kotlin, and TypeScript.

import (
    "google.golang.org/adk/v2/agent"
    "google.golang.org/adk/v2/agent/llmagent"
    "google.golang.org/adk/v2/model/gemini"
    "google.golang.org/adk/v2/runner"
    "google.golang.org/adk/v2/tool"
    "google.golang.org/adk/v2/tool/functiontool"
    "google.golang.org/genai"
)

model, err := gemini.NewModel(ctx, "gemini-3.8-flash", &genai.ClientConfig{
    APIKey: os.Getenv("GOOGLE_API_KEY"),
})

weather, err := functiontool.New(functiontool.Config{
    Name:        "get_weather",
    Description: "Returns the current weather for a city.",
}, func(_ agent.Context, in WeatherInput) (WeatherOutput, error) {
    return WeatherOutput{Report: fmt.Sprintf("Sunny, 24C in %s", in.City)}, nil
})

advisor, err := llmagent.New(llmagent.Config{
    Name:        "travel_advisor",
    Model:       model,
    Instruction: "You are a helpful travel advisor. Use tools to give accurate advice. Be brief.",
    Tools:       []tool.Tool{weather},
})

r, err := runner.NewInMemory("travel-app", advisor)

msg := genai.NewContentFromText("Should I pack a jacket for my trip to Lisbon?", genai.RoleUser)
for event, err := range r.Run(ctx, "user-1", "session-1", msg, agent.RunConfig{}) {
    if err != nil {
        log.Fatal(err)
    }
    if event.LLMResponse.Content == nil || !event.IsFinalResponse() {
        continue
    }
    for _, part := range event.LLMResponse.Content.Parts {
        fmt.Print(part.Text)
    }
}

It is currently sunny and warm at 24°C (75°F) in Lisbon. You won't need a heavy jacket during the day, but packing a light jacket or sweater is recommended for cooler evenings and coastal breezes.

Two things to notice. The runner returns a Go 1.23 iterator, so you range over events with the error as the second value, which is the most idiomatic event loop of the five. And the function tool is generic: input and output structs give the schema, nothing is spelled out twice.

Multi-Agent Orchestration

Workflow agents compose LLM agents. The OutputKey writes an agent's answer into session state, and the next agent reads it back through a {placeholder} in its instruction.

import "google.golang.org/adk/v2/agent/workflowagents/sequentialagent"

researcher, err := llmagent.New(llmagent.Config{
    Name:        "researcher",
    Model:       model,
    Instruction: "List three key facts about the user's topic. Be terse.",
    OutputKey:   "facts", // saved into session state
})
writer, err := llmagent.New(llmagent.Config{
    Name:        "writer",
    Model:       model,
    Instruction: "Write a two-sentence brief using these facts:\n{facts}",
})

pipeline, err := sequentialagent.New(sequentialagent.Config{
    AgentConfig: agent.Config{
        Name:      "content_pipeline",
        SubAgents: []agent.Agent{researcher, writer},
    },
})

[researcher] * **Origin:** Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released as open source in 2009.
* **Concurrency:** Features built-in, lightweight concurrency primitives known as goroutines and channels, managed by its own runtime.
* **Ecosystem Role:** Statically typed and compiled, it is the primary language behind core cloud-native technologies, including Docker and Kubernetes.

[writer] Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson and released as open source in 2009, Go is a statically typed, compiled language featuring built-in, lightweight concurrency primitives known as goroutines and channels that are managed by its own runtime. Owing to these strengths, it has become the primary language powering core cloud-native technologies, including Docker and Kubernetes.

parallelagent and loopagent follow the same pattern.

The Dev UI: ADK Web, Embedded in Your Binary

The full launcher turns any agent into a CLI with console and web modes. The web mode takes sub-launchers, and webui serves the Angular ADK Web app straight out of your Go binary via go:embed, so there is nothing to install:

import (
    "google.golang.org/adk/v2/cmd/launcher"
    "google.golang.org/adk/v2/cmd/launcher/full"
)

l := full.NewLauncher()
config := &launcher.Config{AgentLoader: agent.NewSingleLoader(advisor)}
if err := l.Execute(ctx, config, os.Args[1:]); err != nil {
    log.Fatalf("%v\n\n%s", err, l.CommandLineSyntax())
}

go run . web -port 8080 api webui

Web servers starts on http://localhost:8080
       api:  you can access API using http://localhost:8080/api

The root URL redirects to /ui/, the Agent Development Kit Dev UI, and /api/list-apps answers ["assistant"]. The launcher also wires a2a, pubsub, and eventarc sub-servers, and an -otel_to_cloud flag exports traces to Google Cloud. Flags go before the sub-launcher names, or the sub-launcher rejects them.

Vertex AI Gravity

You can run ADK Go anywhere. But the session and memory services with persistence are the in-memory ones, a database-backed session store, and Vertex AI; the evaluation tooling, the managed scaling, and the telemetry defaults all assume Google Cloud. Same trade-off as ADK Java and ADK Python: the best experience is on GCP.

Pros and Cons

✅ Pros	❌ Cons
Official Google support, highest commit velocity	Agent-only: no vanilla generation, no flows
Idiomatic Go: generics for tools, iterators for events	Go 1.26.6+ required
Embedded ADK Web UI, no extra install	Strong pull toward Vertex AI and GCP
Sequential, parallel, and loop workflow agents	Gemini-first; other providers only through OpenAI-compatible endpoints
A2A, MCP toolsets, tool confirmation, artifacts	No vector store components; RAG goes through tools or Vertex
Same mental model as ADK Python/Java/TS	Config structs get long for real agents

tRPC-Agent-Go

History and Direction

tRPC-Agent-Go is Tencent's entry, built by the team behind tRPC, the RPC framework that underpins much of Tencent's backend. Development started in May 2025, the first public release (v0.0.1) landed in August 2025, and the project reached 1.x at the end of December 2025. It is now at v1.11.2 (August 20, 2026), with 1.8k stars and 56 commits in the last 30 days. The README credits Tencent Yuanbao, Tencent Video, Tencent News, IMA, and QQ Music with real-world validation, and I have no reason to doubt it.

The framework is unapologetically ADK-shaped: llmagent, runner, chainagent, parallelagent, cycleagent, sessions, memory, OutputKey, {placeholder} instructions. If you know ADK, you already know most of it. What it adds on top is where it gets interesting: a GraphAgent that is a faithful LangGraph in Go (state schemas, conditional edges, checkpoints, time travel), an AG-UI server for streaming agent events to frontends, A2A, MCP, Anthropic-style Agent Skills with sandboxed execution, agent self-evolution, an evaluation package, model failover and hedging, prompt caching, and OpenTelemetry traces and metrics with Langfuse examples. RAG is covered by a knowledge module with embedders for OpenAI, Gemini, Ollama, and Hugging Face, and stores from in-memory and SQLite-vec up to pgvector, Elasticsearch, Milvus, Qdrant, and TcVector.

What Makes tRPC-Agent-Go Stand Out

The GraphAgent. Nobody else in Go ships a LangGraph-equivalent state graph as a first-class agent type that plugs into the same runner, sessions, and telemetry as the LLM agents.

Supported languages: Go only, go 1.21+. Model providers: OpenAI and anything OpenAI-compatible (DeepSeek, Qwen, and friends through variants), Anthropic, Gemini, Bedrock, Ollama, Hugging Face, and Hunyuan.

import (
    "trpc.group/trpc-go/trpc-agent-go/agent/llmagent"
    "trpc.group/trpc-go/trpc-agent-go/model"
    "trpc.group/trpc-go/trpc-agent-go/model/gemini"
    "trpc.group/trpc-go/trpc-agent-go/runner"
    "trpc.group/trpc-go/trpc-agent-go/tool"
    "trpc.group/trpc-go/trpc-agent-go/tool/function"
)

type weatherReq struct {
    City string `json:"city" jsonschema:"description=The city to look up,required"`
}

type weatherRsp struct {
    Report string `json:"report"`
}

func getWeather(ctx context.Context, req weatherReq) (weatherRsp, error) {
    return weatherRsp{Report: fmt.Sprintf("Sunny, 24C in %s", req.City)}, nil
}

m, err := gemini.New(ctx, "gemini-3.8-flash") // reads GEMINI_API_KEY / GOOGLE_API_KEY

weather := function.NewFunctionTool(getWeather,
    function.WithName("get_weather"),
    function.WithDescription("Returns the current weather for a city."))

advisor := llmagent.New("travel_advisor",
    llmagent.WithModel(m),
    llmagent.WithInstruction("You are a helpful travel advisor. Use tools to give accurate advice. Be brief."),
    llmagent.WithTools([]tool.Tool{weather}))

r := runner.NewRunner("travel-app", advisor)
events, err := r.Run(ctx, "user-1", "session-1",
    model.NewUserMessage("Should I pack a jacket for my trip to Lisbon?"))
for evt := range events {
    if evt.Error != nil {
        log.Fatalf("%s", evt.Error.Message)
    }
    if evt.Response == nil || len(evt.Response.Choices) == 0 {
        continue
    }
    if evt.IsFinalResponse() {
        fmt.Println(evt.Response.Choices[0].Message.Content)
    }
}

It is currently sunny and warm in Lisbon at 24°C (75°F), so you will not need a heavy coat. However, packing a light jacket or sweater is recommended for cooler, breezy evenings near the coast.

Events arrive on a channel and use OpenAI's response shape (Choices[0].Message or Choices[0].Delta when streaming), which is familiar but means digging through nested structs for every message.

Chains

The chain agent is ADK's sequential agent under another name, OutputKey and {facts} included:

import (
    "trpc.group/trpc-go/trpc-agent-go/agent"
    "trpc.group/trpc-go/trpc-agent-go/agent/chainagent"
)

researcher := llmagent.New("researcher",
    llmagent.WithModel(m),
    llmagent.WithInstruction("List three key facts about the user's topic. Be terse."),
    llmagent.WithOutputKey("facts"))
writer := llmagent.New("writer",
    llmagent.WithModel(m),
    llmagent.WithInstruction("Write a two-sentence brief using these facts:\n{facts}"))

pipeline := chainagent.New("content_pipeline",
    chainagent.WithSubAgents([]agent.Agent{researcher, writer}))

[researcher] 1. Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson; released in 2009.
2. Statically typed, compiled language featuring automatic garbage collection.
3. Native concurrency support using lightweight threads called goroutines and channels.

[writer] Developed at Google by Robert Griesemer, Rob Pike, and Ken Thompson and released in 2009, Go is a statically typed, compiled language featuring automatic garbage collection. It is designed with native concurrency support, utilizing channels and lightweight threads called goroutines.

GraphAgent: LangGraph in Go

A state graph with a routing node, a conditional edge, and two LLM nodes. Short questions go to the quick node, long ones to the deep node.

import (
    "trpc.group/trpc-go/trpc-agent-go/agent/graphagent"
    "trpc.group/trpc-go/trpc-agent-go/graph"
)

sg := graph.NewStateGraph(graph.MessagesStateSchema())
sg.AddNode("route", func(ctx context.Context, state graph.State) (any, error) {
    return nil, nil // pure routing node, no state change
})
sg.AddLLMNode("quick", m, "Answer in one short sentence.", nil)
sg.AddLLMNode("deep", m, "Answer thoroughly. Follow the length the user asks for.", nil)

sg.SetEntryPoint("route")
sg.AddConditionalEdges("route",
    func(ctx context.Context, state graph.State) (string, error) {
        input, _ := state[graph.StateKeyUserInput].(string)
        if len(strings.Fields(input)) > 8 {
            return "deep", nil
        }
        return "quick", nil
    },
    map[string]string{"quick": "quick", "deep": "deep"})
sg.SetFinishPoint("quick")
sg.SetFinishPoint("deep")

g, err := sg.Compile()
router, err := graphagent.New("router", g)
r := runner.NewRunner("graph-app", router)

Asked "What is Go?" and then a request for two paragraphs on goroutines and channels:

Go is an ancient strategy board game, as well as an open-source programming language developed by Google.

Unlike operating system threads that require megabytes of memory and expensive kernel-level context switching, goroutines are extremely lightweight user-space constructs that start with just a few kilobytes of stack space. [...]

The same graph package gives you checkpoints, interrupts for human input, sub-graphs, map-reduce fan-out, per-node callbacks, and a visualization export. It is the most complete workflow engine of the five, and it is the reason to look at this framework even if you never use its LLM agents.

Pros and Cons

✅ Pros	❌ Cons
GraphAgent: a real LangGraph for Go	Smallest community of the five (1.8k stars)
ADK mental model without the GCP gravity	No Dev UI; observability means OpenTelemetry plus Langfuse
AG-UI, A2A, MCP, Agent Skills, evaluation	Huge option surface (`llmagent` alone has dozens)
Model failover and hedging built in	OpenAI-shaped event structs are verbose to consume
OpenTelemetry traces and metrics first party	Tencent-flavoured corners (Hunyuan, tRPC integration)
Runs on Go 1.21+	Documentation partly Chinese-first

LangChainGo

History and Direction

LangChainGo was the first. Travis Cline started it in February 2023, weeks after LangChain Python went viral, and for two years it was the default answer to "how do I call an LLM from Go". It has 9.7k stars, 17 LLM providers, and 15 vector stores, and the LangChain names (chains, prompts, agents, memory, outputparser) make it instantly readable to anyone coming from Python.

And it has stalled. The last release, v0.1.14, is from October 20, 2025. The last commit to main is from January 11, 2026. It is still 0.1.x after three and a half years. There are 243 open issues and 174 open pull requests. The repository is not archived, but nobody is merging.

That would be survivable if the code kept working. It does not:

The default model is retired. googleai.New(ctx) with no options targets gemini-2.0-flash, and Google turned that model off:

  default model: gemini-2.0-flash
  err: googleapi: Error 404: This model models/gemini-2.0-flash is no longer available. Please update your code to use models/gemini-3.6-flash for the latest features and improvements.

The Gemini provider sits on a dead SDK. llms/googleai is built on google/generative-ai-go, whose support ended on November 30, 2025.
Streaming is broken on Gemini. The chunks arrive, and then the call returns an error, so every caller that checks errors treats the request as failed:

  One, two, three, four, five.
  err: error in stream mode: invalid character ']' looking for beginning of value

No middleware, retry, fallback, or OpenTelemetry anywhere. The callback handler is seventeen observe-only methods with no way to change the request. Tools are Call(ctx, string) (string, error): no schema, no typed arguments.

The full teardown, with a capstone app built in both frameworks, is in LangChainGo vs Genkit Go: Where Genkit Shines.

LangChainGo's Position: Legacy

None of this means the basics stopped working. Pin a current model and the classic API still runs:

import (
    "github.com/tmc/langchaingo/llms"
    "github.com/tmc/langchaingo/llms/googleai"
)

llm, err := googleai.New(ctx, googleai.WithDefaultModel("gemini-3.8-flash"))

out, err := llms.GenerateFromSinglePrompt(ctx, llm,
    "Explain the CAP theorem in two sentences.")

The CAP theorem states that a distributed data store can simultaneously provide at most two of three guarantees: Consistency, Availability, and Partition tolerance.

Because network partitions are inevitable in real-world systems, architects must fundamentally choose between serving the most up-to-date data (Consistency) or keeping the system operational (Availability) when a network failure occurs.

import (
    "github.com/tmc/langchaingo/chains"
    "github.com/tmc/langchaingo/prompts"
)

prompt := prompts.NewPromptTemplate(
    "Translate {{.text}} to {{.language}}. Reply with the translation only.",
    []string{"text", "language"})
chain := chains.NewLLMChain(llm, prompt)

out, err := chains.Predict(ctx, chain, map[string]any{
    "text": "Buenos días", "language": "English",
})

Good morning

Tools are strings in, strings out. The agent is a ReAct loop driven by prompt text:

import (
    "github.com/tmc/langchaingo/agents"
    "github.com/tmc/langchaingo/tools"
)

type WeatherTool struct{}

func (WeatherTool) Name() string        { return "get_weather" }
func (WeatherTool) Description() string { return "Returns the current weather. Input: a city name." }
func (WeatherTool) Call(ctx context.Context, city string) (string, error) {
    return fmt.Sprintf("Sunny, 24C in %s", city), nil
}

agent := agents.NewOneShotAgent(llm, []tools.Tool{WeatherTool{}}, agents.WithMaxIterations(3))
out, err := chains.Run(ctx, agents.NewExecutor(agent), "What's the weather like in Tokyo?")

The weather in Tokyo is currently sunny with a temperature of 24°C.

If you are on LangChainGo today, the move is not urgent, but it is inevitable: pin your models, avoid streaming on Gemini, and plan the migration. The concept-by-concept guide is in Migrating from LangChainGo to Genkit Go.

Pros and Cons

✅ Pros	❌ Cons
Largest catalog: 17 providers, 15 vector stores	Unmaintained since January 2026, still 0.1.x
LangChain vocabulary, easy for Python teams	Gemini provider on a retired SDK; streaming broken
Conversation memory is ergonomic	No middleware, retry, fallback, or OpenTelemetry
Runs on Go 1.24+	Tools are untyped strings
Many existing tutorials and examples	Every vector store needs an external service

Head-to-Head Comparison

Developer Experience

Framework	DX Highlights	Shortcomings
Genkit Go	Dev UI with trace explorer. Typed output from structs. Flows are one line from HTTP.	Agents still experimental
Eino	Typed graphs, automatic streaming, callbacks everywhere.	Verbose config structs; 0.x churn
ADK Go	Iterator-based event loop, generic tools, embedded web UI.	Agent-only; Go 1.26.6
tRPC-Agent-Go	GraphAgent, ADK mental model, resilience built in.	OpenAI-shaped events; option overload
LangChainGo	Familiar names, biggest catalog.	Unmaintained; broken Gemini streaming

Abstraction Levels

Genkit Go is the only framework here with all three levels in one SDK: vanilla generation, typed flows, and agents. Eino covers generation and orchestration, plus agents through its ADK module. ADK Go and tRPC-Agent-Go are agent frameworks: every call goes through an agent and a runner. LangChainGo has chains and agents, but no typed pipeline and no session-aware agent.

Observability

Framework	Local Dev	Production
Genkit Go	Dev UI with trace explorer	OpenTelemetry export, Google Cloud plugin, any OTEL collector
Eino	Eino Dev plugin for GoLand and VS Code (graph rendering, node debugging)	Callback handlers: Langfuse, LangSmith, CozeLoop, APMPlus
ADK Go	Embedded ADK Web UI	OpenTelemetry to Google Cloud (`-otel_to_cloud`)
tRPC-Agent-Go	Events on a channel, no UI	OpenTelemetry traces and metrics, Langfuse
LangChainGo	Logging callbacks only	Nothing built in

Provider Support

Framework	Model Providers	Vector Stores
Genkit Go	Gemini (Google AI, Vertex AI), Anthropic, Ollama, OpenAI-compatible family (OpenAI, xAI, DeepSeek, OpenRouter, Kimi, DashScope, Z.ai)	localvec, pgvector, AlloyDB, Pinecone, Weaviate, Firestore
Eino	OpenAI, Claude, Gemini, DeepSeek, Qwen, Ollama, OpenRouter, Ark, Qianfan	Elasticsearch, Milvus, Qdrant, Redis, OpenSearch, Dify, VikingDB
ADK Go	Gemini, OpenAI-compatible, Apigee	None as components (Vertex AI Search via tools)
tRPC-Agent-Go	OpenAI-compatible, Anthropic, Gemini, Bedrock, Ollama, Hugging Face, Hunyuan	In-memory, SQLite-vec, pgvector, Elasticsearch, Milvus, Qdrant, TcVector
LangChainGo	17, including OpenAI, Anthropic, Google AI, Vertex, Bedrock, Cohere, Mistral, Ollama	15, all external services

How Go Does It Feel?

Framework	Min Go	Tools	Streaming	Style
Genkit Go	1.25	Generic, schema from struct	Callback per chunk	Functional options (`ai.With...`)
Eino	1.18	Generic (`utils.InferTool`)	`StreamReader`, automatic across nodes	Config structs plus builder chains
ADK Go	1.26.6	Generic (`functiontool.New`)	`iter.Seq2` over events	Config structs
tRPC-Agent-Go	1.21	Generic plus `jsonschema` tags	Channel of events	Functional options
LangChainGo	1.24	Untyped strings	Callback per chunk (fails on Gemini)	Interfaces plus option funcs

Only ADK Go uses range-over-func iterators for its event loop. Genkit and Eino use generics the most naturally; LangChainGo predates all of it and shows its age.

Maintenance Snapshot

Numbers as of September 21, 2026:

Framework	Stars	Latest Release	Last Commit	Commits, Last 30 Days
Genkit Go	6.4k (monorepo)	go/v1.13.1, Sep 3, 2026	Sep 21, 2026	80 (all SDKs)
Eino	13.1k	v0.9.20, Sep 20, 2026	Sep 20, 2026	7 on `main`, plus five v0.10 alphas
ADK Go	8.8k	v2.4.0, Sep 11, 2026	Sep 21, 2026	100+
tRPC-Agent-Go	1.8k	v1.11.2, Aug 20, 2026	Sep 21, 2026	56
LangChainGo	9.7k	v0.1.14, Oct 20, 2025	Jan 11, 2026	0

Stars measure the past. The last two columns measure the present, and they tell the story.

Which Framework Should You Choose?

Choose Genkit Go if:

You want to iterate on your AI fast and get feedback with less back and forth. The Dev UI is the best local development loop in Go, by a distance.
You need vanilla calls, typed flows, and agents in the same service.
Provider neutrality matters: mixing Gemini, Claude, OpenAI, and Ollama should be a one-line change.
Your team also writes TypeScript, Dart, or Java and wants one framework story.

Choose Eino if:

Your application is a real graph: branches, parallel fan-out, sub-graphs, and streaming across all of it.
You want the most active independent community in Go and are fine living on 0.x.
Callback-based instrumentation into Langfuse, LangSmith, or your own APM is what you already do.
You need ByteDance-scale orchestration patterns, or you run on Volcano Engine.

Choose Google ADK Go if:

You are building multi-agent systems and Google Cloud is your runtime.
You want official Google support, the embedded ADK Web UI, and A2A interoperability.
Your organization already uses ADK in Python or Java and wants Go services with the same shape.
The newest Go toolchain is not a problem for you.

Choose tRPC-Agent-Go if:

You want LangGraph-style state graphs with checkpoints and interrupts, in Go, plugged into agents.
You like the ADK mental model but do not want the GCP gravity.
AG-UI frontends, Agent Skills, evaluation, and OpenTelemetry out of the box matter more than a Dev UI.
You can live with a smaller community and a large API surface.

Stay on LangChainGo only if:

You already run it in production, your models are pinned, and you do not stream through Gemini.
Even then, plan the migration. Nothing is being merged.

Conclusion

Go's Gen AI framework landscape in 2026 is richer than most Go developers realise, and it has a clear shape. Genkit Go is the most versatile choice and the best developer experience, with the Dev UI as a category of its own. Eino is the ambitious independent, the framework to pick when orchestration is the hard part of your problem. Google ADK Go is the enterprise agent runtime, at its best on Google Cloud. tRPC-Agent-Go is the dark horse, with the best workflow engine of the five hiding behind a modest star count. And LangChainGo is the framework that got everyone started and now needs a farewell plan.

If I had to pick one for a new Go service today with no cloud commitment, it would be Genkit Go, for the same reason as in TypeScript and Java: the combination of multiple abstraction levels, provider neutrality, and local observability shortens the loop between an idea and a working feature more than anything else on this list.

Whatever you pick, pick it deliberately. Go finally has the tooling. Use it.

Last updated: September 2026. Framework versions referenced: Genkit Go v1.13.1, Eino v0.9.20 (with eino-ext gemini v0.1.36), Google ADK Go v2.4.0, tRPC-Agent-Go v1.11.2, LangChainGo v0.1.14. Go 1.27.1, model gemini-3.8-flash.

How to Migrate from LangChainGo to Genkit Go

Xavier Portilla Edo — Tue, 29 Sep 2026 13:40:34 +0000

Introduction

If you built a Go AI service in the last two years, there is a good chance it runs on LangChainGo. It was the obvious choice: the name was familiar, the concepts came straight from Python, and it had the biggest provider catalog in the Go ecosystem.

That choice has aged badly. LangChainGo's last release was v0.1.14 in October 2025, its last commit to main was January 2026, and it is still on 0.1.x after three and a half years. Meanwhile the ecosystem moved: models were retired, SDKs were replaced, and APIs changed shape. LangChainGo did not move with them, and parts of it are broken today as a result.

Genkit Go is where that workload belongs now. It is GA, it is on v1.13.1, and it is actively maintained by Google.

This guide is the complete migration path: every LangChainGo concept mapped to its Genkit equivalent, with before and after code for each one. Follow it top to bottom and you can move a real service in an afternoon.

If you want the case for why rather than the how, read the companion article: LangChainGo vs Genkit Go: where Genkit shines.

Why Migrate Now

Three things in LangChainGo are broken today, and none of them will be fixed, because nobody is upstream to fix them.

1. The default model no longer exists. DefaultOptions() in llms/googleai/option.go hardcodes gemini-2.0-flash, which Google retired. A fresh googleai.New(ctx) with no options fails on the first call:

googleapi: Error 404: This model models/gemini-2.0-flash is no longer available.
Please update your code to use models/gemini-3.6-flash for the latest features
and improvements.

2. The Google provider runs on an end-of-life SDK. llms/googleai imports github.com/google/generative-ai-go, which Google replaced with google.golang.org/genai. Its README is explicit:

End-of-Life Date: All support for this repository (including bug fixes) will permanently end on November 30, 2025.

3. Streaming is broken on every current Gemini model. Because of that dead SDK, any streaming call through the Google provider fails:

error in stream mode: invalid character ']' looking for beginning of value

This fails on gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash and gemini-3.6-flash. Non-streaming calls still work, and the OpenAI provider streams fine, so this is specific to Google on the legacy SDK.

The trap: observability breaks your agents

This one catches people out, so it is worth knowing before you debug it the hard way.

LangChainGo turns on streaming automatically whenever you attach a callbacks handler. From chains/options.go:

if opts.StreamingFunc == nil && opts.CallbackHandler != nil {
    opts.StreamingFunc = func(ctx context.Context, chunk []byte) error {
        opts.CallbackHandler.HandleStreamingFunc(ctx, chunk)
        return nil
    }
}

agents/mrkl.go does the same thing. Since streaming is broken, attaching the only supported observability hook is also what breaks your agent. Same agent, same tool, one line different:

WITHOUT callbacks handler: err=<nil> out="The current year is 2026."
WITH    callbacks handler: err=error in stream mode: invalid character ']' looking for beginning of value

On LangChainGo with Gemini today, you can have working agents or you can have instrumentation. Not both.

What You Need

Go 1.24+ (this guide was written against Go 1.27)
A Google AI API key, free from Google AI Studio
Optionally an OpenAI API key, if you want cross-vendor fallback
The Genkit CLI, for the Developer UI:

npm install -g genkit

Add Genkit to your module:

go get github.com/firebase/genkit/go@latest

The Concept Map

This is the whole translation table. Keep it open while you work.

LangChainGo	Genkit Go	What changes
`llms.Model` + provider `New()`	plugin + `genkit.Init`	Models become string references like `googleai/gemini-3.5-flash`
`llms.GenerateFromSinglePrompt`	`genkit.GenerateText`	Direct swap
`prompts.PromptTemplate`	`ai.WithPrompt` or a `.prompt` file	Dotprompt files keep prompts outside the binary
`chains.LLMChain` + `chains.Run`	`genkit.DefineFlow`	A typed Go function instead of a `map[string]any`
`chains.NewSequentialChain`	ordinary Go code	Just call the next function
`outputparser.NewDefined[T]`	`genkit.GenerateData[T]`	Real schema instead of prompt-and-parse
`tools.Tool` interface	`genkit.DefineTool[In, Out]`	Typed arguments with a JSON schema
`agents.NewOneShotAgent` (ReAct)	`ai.WithTools`, or `genkitx.DefineAgent`	Native function calling instead of text parsing. Genkit also has a real agent primitive with sessions
`memory.NewConversationBuffer`	`resp.History()`, or an agent session store	Explicit history, or `aix.WithSessionStore` if you want the buffer back
`embeddings` + `vectorstores`	`localvec` or a vector store plugin	Genkit has a local store, LangChainGo does not
`callbacks.Handler`	built-in tracing	Delete the handler entirely
nothing	`middleware.Retry`, `middleware.Fallback`	Retry and fallback stop being your code

Step 1: Set Up Genkit

In LangChainGo, each provider is a concrete struct you construct and hold:

llm, err := googleai.New(ctx,
    googleai.WithDefaultModel("gemini-3.5-flash"),
    googleai.WithDefaultEmbeddingModel("gemini-embedding-001"))
if err != nil {
    log.Fatal(err)
}
defer llm.Close()

In Genkit, plugins register with one runtime object and models become string references:

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithDefaultModel("googleai/gemini-3.5-flash"))

That difference matters more than it looks. Registering a second vendor is one more plugin:

g := genkit.Init(ctx, genkit.WithPlugins(
    &googlegenai.GoogleAI{},
    &openai.OpenAI{},
))

Now both vendors live in the same registry and a model is just a name, which is what makes swapping and falling back a configuration change later on.

Step 2: Migrate Your Model Calls

A plain text generation is close to a direct swap.

Before:

out, err := llms.GenerateFromSinglePrompt(ctx, llm,
    "In one sentence: what is Go's error handling philosophy?")

After:

out, err := genkit.GenerateText(ctx, g,
    ai.WithPrompt("In one sentence: what is Go's error handling philosophy?"))

If you need the full response rather than just the text, use genkit.Generate, which gives you resp.Text(), resp.History() and usage data.

Step 3: Turn Chains into Flows

A LangChainGo chain wraps a prompt template and takes a map[string]any:

Before:

prompt := prompts.NewPromptTemplate(
    "Triage this support ticket.\n{{.format}}\n\nTicket: {{.ticket}}",
    []string{"format", "ticket"},
)
chain := chains.NewLLMChain(llm, prompt)
raw, err := chains.Predict(ctx, chain, map[string]any{
    "format": parser.GetFormatInstructions(),
    "ticket": ticket,
})

A Genkit flow is a typed Go function. Your input and output types are the contract and the compiler enforces them:

After:

triage := genkit.DefineFlow(g, "triage", func(ctx context.Context, ticket string) (*Triage, error) {
    out, _, err := genkit.GenerateData[Triage](ctx, g,
        ai.WithPrompt("Triage this support ticket: %s", ticket))
    return out, err
})

out, err := triage.Run(ctx, "I was charged twice for order A-1029.")

This step deletes the most code, because most of what a chain does is pass a map around. Go does not need that.

Sequential chains have no equivalent, and do not need one. chains.NewSequentialChain becomes calling the next function:

summary, err := summarize.Run(ctx, doc)
if err != nil {
    return nil, err
}
return classify.Run(ctx, summary)

Step 4: Migrate Structured Output

Prioritise this step. In LangChainGo, structured output is prompt engineering. In Genkit it is a schema.

Before: generate format instructions from your struct, paste them into the prompt, parse the text back.

type Ticket struct {
    Category string `json:"category" describe:"one of: billing, bug, feature, other"`
    Severity int    `json:"severity" describe:"1 (low) to 5 (critical)"`
    Summary  string `json:"summary" describe:"one sentence"`
}

parser, err := outputparser.NewDefined(Ticket{})
// ...put parser.GetFormatInstructions() into the prompt...
out, err := parser.Parse(raw)

After: one call, with the schema sent to the model as a real response schema.

type Ticket struct {
    Category string `json:"category" jsonschema:"enum=billing,enum=bug,enum=feature,enum=other"`
    Severity int    `json:"severity" jsonschema_description:"1 (low) to 5 (critical)"`
    Summary  string `json:"summary" jsonschema_description:"one sentence"`
}

out, _, err := genkit.GenerateData[Ticket](ctx, g,
    ai.WithPrompt("Triage this support ticket: %s", ticket))

Note the tag change: describe: becomes jsonschema_description:, and enums move into a jsonschema: tag where the model actually enforces them.

Step 5: Migrate Your Tools

The LangChainGo tool interface is Call(ctx context.Context, input string) (string, error). One string in, one string out, no schema. A tool that needs typed arguments has to describe its schema in prose and unmarshal by hand:

Before:

func (RefundTool) Name() string { return "issue_refund" }

func (RefundTool) Description() string {
    return `Issues a refund. Input MUST be a JSON object string with exactly these keys:
{"order_id": string, "amount_cents": integer}. Example: {"order_id":"A-1","amount_cents":2500}`
}

func (RefundTool) Call(ctx context.Context, input string) (string, error) {
    input = strings.TrimSpace(input)
    input = strings.TrimPrefix(input, "```json")
    input = strings.TrimPrefix(input, "```")
    input = strings.TrimSuffix(input, "```")
    var a RefundArgs
    if err := json.Unmarshal([]byte(strings.TrimSpace(input)), &a); err != nil {
        return fmt.Sprintf("error: input was not valid JSON (%v). Got: %s", err, input), nil
    }
    return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil
}

In Genkit the tool function is generic over its argument type and the schema comes from the struct:

After:

type RefundArgs struct {
    OrderID     string `json:"order_id"`
    AmountCents int    `json:"amount_cents"`
}

refund := genkit.DefineTool(g, "issue_refund", "Issues a refund for an order.",
    func(ctx *ai.ToolContext, a RefundArgs) (string, error) {
        return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil
    })

Delete the fence stripping, the unmarshalling and the prose schema. The model receives a real JSON schema and the arguments arrive typed.

Step 6: Replace the Agent

A LangChainGo ReAct agent drives tools by parsing the model's text output for Action: and Action Input: lines, wrapped in an executor with an iteration cap:

Before:

agent := agents.NewOneShotAgent(llm, []tools.Tool{RefundTool{}}, agents.WithMaxIterations(6))
out, err := chains.Run(ctx, agents.NewExecutor(agent),
    "Refund order A-1029 for 25 dollars and 50 cents.")

In Genkit, native function calling means a one-shot agent is just a generate call with tools attached:

After:

out, err := genkit.GenerateText(ctx, g,
    ai.WithPrompt("Refund order A-1029 for 25 dollars and 50 cents."),
    ai.WithTools(refund))

No executor, no iteration cap, no text parsing. Genkit runs the tool loop for you.

When you want a real agent, Genkit has one

The generate call above replaces a OneShotAgent. For anything that has to hold a conversation, Genkit Go ships an actual agent primitive with sessions and persisted state, which LangChainGo has no equivalent for:

import (
    aix "github.com/firebase/genkit/go/ai/exp"
    "github.com/firebase/genkit/go/ai/exp/localstore"
    genkitx "github.com/firebase/genkit/go/genkit/exp"
)

g := genkit.Init(ctx,
    genkit.WithPlugins(&googlegenai.GoogleAI{}),
    genkit.WithDefaultModel("googleai/gemini-2.5-flash"),
    genkit.WithExperimental())

store := localstore.NewInMemorySessionStore[State]()

agent := genkitx.DefineAgent(g, "taskAgent",
    aix.InlinePrompt{
        ai.WithSystem("Manage a task list. Use tools when changing tasks."),
        ai.WithTools(addTask),
    },
    aix.WithSessionStore(store),
    aix.WithDescription[State]("Task management assistant"),
)

Three constructors are available: DefineAgent for an inline prompt, DefinePromptAgent to wrap a prompt already in the registry, and DefineCustomAgent to replace the prompt loop with your own code. Options cover a session store, WithStateTransform to reshape state on the way out to a client, and WithStreamTransform for chunks. You drive one with Run, RunText, or Connect for a streaming connection.

These live in the exp packages. Agents are experimental in Genkit Go v1.13.1, and calling genkitx.DefineAgent without genkit.WithExperimental() on genkit.Init panics with a message telling you so. The API may change between minor releases, so weigh that before putting one in production.

Step 7: Migrate RAG

Check this step before planning your timeline, because it has an infrastructure consequence.

LangChainGo ships fifteen vector stores: AlloyDB, Azure AI Search, Bedrock Knowledge Bases, Chroma, CloudSQL, Dolt, MariaDB, Milvus, MongoDB, OpenSearch, pgvector, Pinecone, Qdrant, Redis and Weaviate. Every one of them needs an external service running. There is no in-memory or local option, and the pull request adding one has been open since February 2024. If you needed a local store, you wrote it yourself.

Genkit ships localvec, a file-backed local store, so the same thing is three calls:

After:

embedder := googlegenai.GoogleAIEmbedder(g, "gemini-embedding-001")
store, retriever, err := localvec.DefineRetriever(g, "policy",
    localvec.Config{Dir: "/tmp/gkvec", Embedder: embedder}, nil)
if err != nil {
    return err
}

// Index your documents once.
localvec.Index(ctx, docs, store)

// Retrieve at request time.
found, err := genkit.Retrieve(ctx, g, ai.WithRetriever(retriever), ai.WithTextDocs(ticket))

Then hand the documents straight to the model:

out, _, err := genkit.GenerateData[Triage](ctx, g,
    ai.WithDocs(found.Documents...),
    ai.WithPrompt("Triage this ticket using the policy documents: %s", ticket))

If you were on a hosted store, Genkit has Pinecone, Weaviate, Postgres and AlloyDB plugins.

Step 8: Migrate Memory

LangChainGo keeps conversation history in a buffer attached to the chain:

Before:

c := chains.NewConversation(llm, memory.NewConversationBuffer())
chains.Run(ctx, c, "My name is Xavi and I work in Go.")
chains.Run(ctx, c, "What is my name and what language do I use?")

Genkit passes history explicitly:

After:

resp, err := genkit.Generate(ctx, g, ai.WithPrompt("My name is Xavi and I work in Go."))
if err != nil {
    return err
}

resp, err = genkit.Generate(ctx, g,
    ai.WithMessages(resp.History()...),
    ai.WithPrompt("What is my name and what language do I use?"))

For a plain generate call this is the one mapping where LangChainGo is less typing. The trade is that Genkit has no hidden buffer to keep in sync, which is what you want the moment your conversation state lives in Redis or Postgres and your handlers are stateless.

If you want the buffer back, that is what an agent's session store is for. Give the agent a store and pass a session id, and the history is the framework's problem again:

store := localstore.NewInMemorySessionStore[State]()

agent := genkitx.DefineAgent(g, "taskAgent",
    aix.InlinePrompt{
        ai.WithSystem("Manage a task list. Use tools when changing tasks. Be terse."),
        ai.WithTools(addTask),
    },
    aix.WithSessionStore(store))

for _, turn := range []string{
    "Add 'buy milk' to my list.",
    "What did I just ask you to add?",
} {
    out, err := agent.RunText(ctx, turn, aix.WithSessionID[State]("demo-session"))
    if err != nil {
        return err
    }
    fmt.Printf("> %s\n%s\n\n", turn, out.Message.Text())
}

> Add 'buy milk' to my list.
Task added.

> What did I just ask you to add?
You asked me to add 'buy milk' to your list.

Nothing in that loop passes the first turn into the second. The session id does it. localstore.NewInMemorySessionStore is the one that ships for local development; the SessionStore interface is what you implement to put sessions in Redis or Postgres.

So the honest version of this row is: Genkit gives you both ends. Explicit resp.History() when you want stateless handlers, and a store-backed session when you want the buffer.

Step 9: Delete Your Resilience Code

This is the step that usually surprises people with how much it removes.

LangChainGo has no middleware layer. callbacks.Handler is seventeen methods that all return nothing:

HandleText(ctx, text)
HandleLLMGenerateContentStart(ctx, ms)
HandleLLMGenerateContentEnd(ctx, res)
HandleToolStart(ctx, input)
// ...and thirteen more

You can watch a call. You cannot wrap it, retry it, rewrite its request or substitute a model, because there is no next(). There is no retry, no fallback and no OpenTelemetry anywhere in the library, so if your service has those, you wrote them.

In Genkit they are configuration:

After:

out, _, err := genkit.GenerateData[Triage](ctx, g,
    ai.WithModel(googlegenai.GoogleAIModelRef("gemini-3.5-flash", nil)),
    ai.WithDocs(found.Documents...),
    ai.WithTools(refund),
    ai.WithPrompt("Triage this ticket and act on it: %s", ticket),
    ai.WithUse(
        &middleware.Retry{MaxRetries: 2},
        &middleware.Fallback{Models: []ai.ModelRef{openai.ModelRef("gpt-5-mini", nil)}},
    ),
)

g := genkit.Init(ctx, genkit.WithPlugins(
    &googlegenai.GoogleAI{}, &openai.OpenAI{}, &middleware.Middleware{},
))

Point the primary at a dead model and the request heals onto the other vendor on its own:

WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND"
{Category:billing Severity:2 PolicyQuote:Duplicate charges must be refunded in full within 5
business days, no manager approval required. Action:Issued full refund of $25.50 (2550 cents)
for the duplicate charge on order A-1029.}

Gemini went down, OpenAI picked it up, the tool still ran, and the typed struct came back intact.

Delete your callbacks handler too. Genkit traces flows, generate calls, tool calls and retrieval automatically. Run the Developer UI to see them:

genkit start -- go run .

Step 10: Wrap the Entrypoints

Finally, expose your flows over HTTP. Where LangChainGo left you writing net/http handlers around chains, Genkit turns a flow into an endpoint:

mux := http.NewServeMux()
mux.HandleFunc("POST /triage", genkit.Handler(triage))
log.Fatal(server.Start(ctx, "127.0.0.1:3400", mux))

The flow's input and output types define the request and response shapes, so the whole path is typed end to end.

Before You Commit

The migration is not free. Check these three things first:

Provider and vector store catalog. LangChainGo has seventeen model providers and fifteen vector stores. Genkit Go covers Google AI, Vertex, OpenAI, Anthropic, Ollama and the OpenAI-compatible vendors, plus Pinecone, Weaviate, Postgres and AlloyDB. If you are on Dolt, MariaDB or Bedrock Knowledge Bases as a vector store, confirm you have a path.

Conversation buffers. There is no drop-in ConversationBuffer. You pass history explicitly, as in Step 8.

Specialised chains. Helpers like chains.NewSQLDatabaseChain and the map-reduce summarisation chains have no one-to-one equivalent. In Genkit they are Go functions you write, which is usually shorter than the chain was, but it is work.

Conclusion

The migration is smaller than it looks, because most of a LangChainGo program is plumbing that Go does not need: maps passed between chains, format instructions pasted into prompts, JSON unmarshalled out of tool strings, and a resilience layer you wrote yourself. Typed flows, typed tools, schema-backed output and middleware delete all four.

The reason to start now is not ergonomics. It is that the library your service depends on has a retired default model, an end-of-life SDK underneath its Google provider, broken streaming, and an observability hook that breaks the agents it is supposed to observe. Those are not problems you can wait out.

For the side-by-side numbers and the full picture, read the companion: LangChainGo vs Genkit Go: where Genkit shines.

LangChainGo vs Genkit Go: Where Genkit Shines

Xavier Portilla Edo — Tue, 29 Sep 2026 13:40:25 +0000

Introduction

There are two realistic choices for building AI services in Go today: LangChainGo, the Go port of the Python framework everyone knows, and Genkit Go, Google's open-source Gen AI framework.

LangChainGo is still the first result most Go developers find. It has 9,677 stars and the biggest provider catalog in the ecosystem. It is also eight months without a commit, eleven months without a release, and still on 0.1.x after three and a half years.

This article compares them the way that actually matters: by building the same production service in both and looking at what each one costs you. Not a feature checklist, a real application with real requirements.

If you have already decided and want the migration mechanics, that is the companion article: How to migrate from LangChainGo to Genkit Go.

The State of Both Projects

	LangChainGo	Genkit Go
Latest release	`v0.1.14`, October 2025	`v1.13.1`
Last commit to `main`	January 2026	active
Age and version	3.5 years, still `0.1.x`	GA since `v1.0`
Open issues	242
Open pull requests	171, oldest from June 2023
Stars	9,677

LangChainGo is not archived, and "dead" overstates it. But eleven months without a release is long enough for the ecosystem to move underneath a library, and it has.

What Is Broken in LangChainGo Today

Three concrete things, all of which affect a service in production right now.

The quickstart returns a 404. DefaultOptions() hardcodes gemini-2.0-flash, which Google retired. A fresh googleai.New(ctx) fails on the first call.

The Google provider runs on an end-of-life SDK. llms/googleai imports github.com/google/generative-ai-go, whose README states support ended on November 30, 2025.

Streaming is broken on every current Gemini model. gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash and gemini-3.6-flash all fail the same way:

error in stream mode: invalid character ']' looking for beginning of value

The OpenAI provider streams fine, so this is the Google provider on the legacy SDK, not the library everywhere.

There is a fourth consequence that follows from the third, and it is the nastiest one. LangChainGo installs a streaming function automatically whenever you attach a callbacks handler, in chains/options.go and again in agents/mrkl.go. Since streaming is broken, turning on observability breaks your Gemini agents:

WITHOUT callbacks handler: err=<nil> out="The current year is 2026."
WITH    callbacks handler: err=error in stream mode: invalid character ']' looking for beginning of value

Feature by Feature

Capability	LangChainGo	Genkit Go
Structured output	Format instructions in the prompt, parse the text back	`genkit.GenerateData[T]`, real response schema
Tools	`Call(ctx, string) (string, error)`, no schema	`genkit.DefineTool[In, Out]`, typed with JSON schema
Agents	ReAct, driven by parsing `Action:` lines	Native function calling, plus an agent primitive with sessions (experimental)
Composition	`Chain` interfaces over `map[string]any`	Typed Go functions (`DefineFlow`)
Retry	Write it yourself	`middleware.Retry`
Cross-vendor fallback	Write it yourself	`middleware.Fallback`
Observability	17 observe-only callback methods	Built-in tracing plus a Developer UI
OpenTelemetry	None	Built in
Local vector store	None, all 15 stores need a server	`localvec`, file-backed
Streaming on Gemini	Broken	Works
HTTP endpoint	`net/http` boilerplate	`genkit.Handler(flow)`

The single most important row is retry and fallback, and the reason is architectural rather than cosmetic. A Genkit middleware wraps a call and receives a next(), so it can decide to call again, or call something else. LangChainGo's callbacks.Handler returns nothing from every method. You can watch a call, you cannot change it. That missing seam is why every resilience concern ends up in your application code.

What We Are Building

A support-ticket triage service. Every requirement here is something a real team asks for:

Retrieve the relevant policy documents for an incoming ticket.
Let the model call a tool to act on it.
Return a typed struct, not free text.
Retry transient model failures.
Survive a vendor outage by falling back to a different provider.
Trace every step, so somebody can debug it at 3am.

The Genkit Go Version

The whole service, in one file:

g := genkit.Init(ctx, genkit.WithPlugins(
    &googlegenai.GoogleAI{}, &openai.OpenAI{}, &middleware.Middleware{},
))

// RAG store.
embedder := googlegenai.GoogleAIEmbedder(g, "gemini-embedding-001")
store, retriever, err := localvec.DefineRetriever(g, "policy",
    localvec.Config{Dir: "/tmp/gkvec", Embedder: embedder}, nil)
localvec.Index(ctx, docs, store)

// A tool the model can call.
refund := genkit.DefineTool(g, "issue_refund", "Issues a refund for an order.",
    func(ctx *ai.ToolContext, a struct {
        OrderID     string `json:"order_id"`
        AmountCents int    `json:"amount_cents"`
    }) (string, error) {
        return fmt.Sprintf("refunded %d cents for order %s", a.AmountCents, a.OrderID), nil
    })

// The flow.
triage := genkit.DefineFlow(g, "triage", func(ctx context.Context, ticket string) (*Triage, error) {
    found, err := genkit.Retrieve(ctx, g, ai.WithRetriever(retriever), ai.WithTextDocs(ticket))
    if err != nil {
        return nil, err
    }
    out, _, err := genkit.GenerateData[Triage](ctx, g,
        ai.WithModel(googlegenai.GoogleAIModelRef("gemini-3.5-flash", nil)),
        ai.WithDocs(found.Documents...),
        ai.WithTools(refund),
        ai.WithPrompt("Triage this ticket using the policy documents, and act on it: %s", ticket),
        ai.WithUse(
            &middleware.Retry{MaxRetries: 2},
            &middleware.Fallback{Models: []ai.ModelRef{openai.ModelRef("gpt-5-mini", nil)}},
        ),
    )
    return out, err
})

73 lines, and it returns exactly what you asked for:

{Category:billing Severity:1 PolicyQuote:Duplicate charges must be refunded in full within
5 business days, no manager approval required. Action:Issued a refund of $25.50 (2550 cents)
to resolve the duplicate charge on order A-1029.}

The interesting part is which requirements are not code. Retry and fallback are two entries in a list. Tracing does not appear at all, because Genkit instruments flows, generate calls, tool calls and retrieval itself. The RAG store is three calls because localvec ships in the box.

The LangChainGo Version

Requirements 1, 2 and 3 port directly. Requirements 4, 5 and 6 do not exist in the library, so they become your code:

File	Lines	Why it exists
`main.go`	110	the actual application
`memstore.go`	68	all 15 vector stores need an external service, so the local one is yours
`resilience.go`	58	no middleware layer, so retry and fallback are yours
`tracer.go`	36	observability is a 17-method interface to implement
total	272

None of those three extra files contain interesting code. memstore.go is cosine similarity and a sort. tracer.go is six methods that log and eleven empty ones that exist only to satisfy the interface:

// Everything below exists only to satisfy the interface.
func (t *tracer) HandleText(context.Context, string)               {}
func (t *tracer) HandleLLMStart(context.Context, []string)         {}
func (t *tracer) HandleLLMError(context.Context, error)            {}
func (t *tracer) HandleChainStart(context.Context, map[string]any) {}
// ...seven more

resilience.go wraps the primary model to add retries and a fallback, and it has a sharp edge worth seeing:

clean := make([]llms.CallOption, 0, len(opts))
for _, o := range opts {
    var probe llms.CallOptions
    o(&probe)
    // Model names are vendor-specific, and gpt-5-mini rejects 'stop'
    // outright, which the ReAct agent always sets. Both must go.
    if probe.Model != "" || len(probe.StopWords) > 0 {
        continue
    }
    clean = append(clean, o)
}
return r.backup.GenerateContent(ctx, msgs, clean...)

CallOption is an opaque function, so to find out what an option does you have to apply it to a probe struct and inspect the result. Miss the stop words and the fallback fails with Unsupported parameter: 'stop' is not supported with this model.

Both versions work and produce equivalent results. So far this is a story about volume, not capability. The outage test is where that changes.

What Happens When a Model Goes Down

Point the primary at a retired model in both versions, which is a real 404 from the live API.

Genkit heals and finishes:

WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND"
WARN model call failed, falling back model=openai/gpt-5-mini error="Error 404 ... NOT_FOUND"
{Category:billing Severity:2 PolicyQuote:Duplicate charges must be refunded in full within
5 business days, no manager approval required. Action:Issued full refund of $25.50 ...}

Two calls fail over, once for the initial generate and once for the turn after the tool call. The typed struct survives a vendor switch.

LangChainGo heals at the model layer, then breaks one layer up:

2026/09/11 20:01:45 primary failed (googleapi: Error 404 ...), falling back
2026/09/11 20:01:54 PARSE FAILED: input text should start with ```json and end with ```

The hand-written fallback did its job. The output parser then rejected the answer, because outputparser.Defined.Parse hard-requires code fences:

const opening = "```json"
const closing = "```"
if text[:len(opening)] != opening || text[len(text)-len(closing):] != closing {
    return target, fmt.Errorf("input text should start with %s and end with %s", opening, closing)
}

Gemini tends to emit those fences. OpenAI does not. The typed output layer is quietly coupled to one vendor's formatting habit, and the coupling surfaces at the exact moment you switch vendors, which is the worst possible time.

That same function has a second problem. When the response is shorter than seven characters, it does not return an error, it panics:

input "ok"  PANIC: runtime error: slice bounds out of range [:7] with length 2
input ""    PANIC: runtime error: slice bounds out of range [:7] with length 0

An empty completion or a refusal becomes an uncaught panic in your request handler.

Dependency Weight

The same typed-extraction program, each in its own clean module:

	LangChainGo	Genkit Go
Modules in the build graph	62	41
Binary size	38.0 MB	28.7 MB

The LangChainGo-only half of that list is worth reading: the deprecated generative-ai-go, plus cloud.google.com/go/vertexai and aiplatform, plus sprig, gonja (a Jinja2 template engine), go.starlark.net (the calculator tool evaluates expressions in Starlark), logrus, tiktoken-go, pkg/errors and json-iterator. A Python-shaped dependency tree, in Go.

Where Genkit Go Shines

Pulling the thread through all of it, the difference comes down to four things Genkit has and LangChainGo does not.

Middleware is the first. Because a Genkit middleware wraps a call and can decide to call it again or call something else, resilience becomes library code. Retry, cross-vendor fallback and tool approval are list entries instead of wrapper types you maintain forever.

Types are the second. Typed flows, typed tools and schema-backed output mean the compiler carries the contract. That is not just tidier: it is why the Genkit service survived the failover and the LangChainGo one did not. Nothing in the Genkit path was slicing strings out of a model response to find the data.

Plugins are the third. A model is a string reference in a registry, so swapping Gemini for GPT is a configuration change. In LangChainGo, two providers are two concrete structs of two different types, which is why a hand-written fallback has to strip vendor-specific call options and still gets it wrong the first time.

The Developer UI is the fourth, and it is the one you feel every day rather than during an incident. LangChainGo has no equivalent to it at all.

One command starts it next to your app:

genkit start -- go run .

It reads the registry your code built, so nothing has to be declared twice. On the triage service it found the flow, the tool, 39 models and 3 embedders without any configuration:

Flows (1)      triageTicket
Tools (1)      issue_refund
Models (39)    googleai/gemini-2.5-flash, ...
Embedders (3)  googleai/gemini-embedding-001, ...

Open a flow and the input form is generated from your Go struct. A Ticket with Subject and Body fields gives you this, pre-filled and ready to edit:

{
  "body": "",
  "subject": ""
}

Press Run and you get the typed output back as a tree, not a blob of text:

{ "category": "Billing", "urgency": 4, "action": "Refund issued" }

The part that earns its place is the trace underneath. Every run is a span tree with per-span timings, and the tool call is a span of its own:

triageTicket                        4.64s
├─ generate                         2.52s
│  ├─ googleai/gemini-2.5-flash     1.67s
│  ├─ issue_refund                    1ms
│  └─ generate (2)                  846ms
│     └─ googleai/gemini-2.5-flash  844ms
└─ generate                         2.12s
   └─ googleai/gemini-2.5-flash     2.12s

You can read the whole shape of the run off that: the first generate called the tool, the model went back for a second turn with the tool result, and a separate generate did the typed extraction. Nobody wrote a line of logging for it.

Failures land in the same place. A run that hit a real Gemini constraint showed the error inline with a full Go stack trace, and marked every span in the tree that was affected:

Error 400, Message: Function calling with a response mime type: 'application/json'
is unsupported, Status: INVALID_ARGUMENT, Details: []

That error is worth knowing on its own, and it is model-specific rather than a Genkit limitation. Running the same tools-plus-typed-output call across five models makes the line clear:

gemini-2.5-flash         FAIL  Function calling with a response mime type: 'application/json' is unsupported
gemini-3-flash-preview   OK
gemini-3.5-flash         OK
gemini-3.6-flash         OK
gemini-3.8-flash         OK

Gemini 2.5 refuses function calling combined with a JSON response mime type, so on that model a tool call and a typed extraction have to be two steps. Every Gemini 3 model lifted the restriction, which is why the triage service above runs both in one call on gemini-3.5-flash. The useful part is that the Developer UI told me which span failed and why, instead of leaving me to bisect it.

The sidebar also carries Traces, Evaluations and Datasets, so the same UI covers running an action by hand, reading what it did, and building a dataset out of it. In LangChainGo, the equivalent of all of this is a callbacks.Handler you write yourself, which returns nothing from every method and, as shown above, breaks your Gemini agents when you attach it.

Underneath all four, someone is shipping. When gemini-2.0-flash was retired, that was a 404 to work around in LangChainGo and a fallback that healed itself in Genkit.

What LangChainGo Still Does Well

This is not a demolition. There are real reasons it is still in use:

Provider catalog. Seventeen model providers including Bedrock, Cloudflare, Ernie, HuggingFace, Maritaca, Mistral, watsonx and llamafile. Genkit's plugin set is narrower.
Vector store catalog. Fifteen integrations, including Dolt, MariaDB and OpenSearch, which Genkit has no equivalent for.
Conversation memory is less typing. chains.NewConversation(llm, memory.NewConversationBuffer()) beats passing resp.History() around by hand. Genkit's answer is an agent session store, which is more capable but sits behind an experimental flag, so on a plain generate call LangChainGo is still terser.
The ReAct agent works. On a sequential two-tool task it succeeds consistently. The cost is the code you own, not flakiness.
Familiarity. If your team already thinks in chains, memories and retrievers from Python, the vocabulary transfers directly.

If you depend on a provider or a vector store only LangChainGo supports, that is a real reason to stay. Weigh it against an unmaintained dependency rather than against line counts.

The Scorecard

Non-blank, non-comment lines.

Requirement	Genkit Go	LangChainGo
Full service: RAG + tool + typed output + retry + fallback + tracing	73, one file	272, four files
Behaviour under a vendor outage	heals, typed output intact	heals at model layer, output parser fails
Tool with two typed arguments	31	55 (hand-written unmarshalling)
Retry and cross-vendor fallback	2 config lines	58 lines, vendor-specific options to strip
Local vector store for RAG	ships as `localvec`	68 lines, no local option exists
Observability	built in, zero lines	36 lines, and it breaks Gemini agents
Streaming on current Gemini models	works	broken on all four tested
Quickstart with no options	works	404, default model retired
Modules / binary for the same program	41 / 28.7 MB	62 / 38.0 MB

Conclusion

If you are starting a Go AI project today, use Genkit Go. That part is not close.

If you already have LangChainGo in production, the decision is about risk rather than elegance. The library is eight months without a commit, its Google provider is built on an SDK that reached end of life in November 2025, its quickstart returns a 404, its streaming is broken on every current Gemini model, and turning on observability breaks the agents it is meant to observe. Those are not problems that resolve themselves.

The clearest signal is the outage test. Both versions were supposed to survive a vendor failure, and technically both did at the model layer. Only one came back with the data the caller asked for, because in the other one the layer above was matching on code fences.

Genkit Flows as senro Steps: Extremely Powerful AI Pipelines in Go

Xavier Portilla Edo — Tue, 29 Sep 2026 13:39:43 +0000

Introduction

A Genkit flow is an excellent unit of AI work. It is typed, traced, testable, and genkit.DefineFlow gives you something you can call from anywhere.

What a flow is not is a unit of orchestration. The moment you have twelve documents to summarise, a flow whose output feeds two other flows, or a model call that fails on the third of forty items, you end up writing the same scaffolding again: a worker pool, a retry loop, a way to avoid re-running the eleven items that already succeeded, logs you can actually read, and some way to find out what happened at three in the morning.

senro is a pipeline engine where a step already has all of that: retries that distinguish infrastructure from workload, an action cache that skips work it has already done, an append-only event stream, and a live attach protocol. I covered it in the introduction and the monorepo article.

This article shows how to combine them. Put a Genkit flow inside a senro step and every model call becomes a graph node with its own log, state, retry policy and cache entry.

Every example below ran against the real Gemini API with googleai/gemini-2.5-flash, on Genkit Go v1.13.1 and senro v1.4.0. The summaries are what the model actually returned.

What you need

go get github.com/firebase/genkit/go
go get github.com/xavidop/senro

The example is a small corpus of release notes in docs/*.md that gets summarised one document at a time, then merged into a digest.

Step 1: the flow

Nothing senro-specific here. This is an ordinary Genkit flow:

// SummarizeInput is what the summarize flow takes.
type SummarizeInput struct {
    Path string `json:"path"`
    Text string `json:"text"`
}

// Summary is what it returns.
type Summary struct {
    Path    string `json:"path"`
    Summary string `json:"summary"`
}

func defineFlows(g *genkit.Genkit) *core.Flow[SummarizeInput, Summary, struct{}] {
    return genkit.DefineFlow(g, "summarize",
        func(ctx context.Context, in SummarizeInput) (Summary, error) {
            text, err := genkit.GenerateText(ctx, g,
                ai.WithSystem("You summarize release notes for an operator. "+
                    "Answer in one sentence, no preamble."),
                ai.WithPrompt(fmt.Sprintf("Summarize these release notes:\n\n%s", in.Text)))
            if err != nil {
                return Summary{}, err
            }
            return Summary{Path: in.Path, Summary: text}, nil
        })
}

Called on its own it does what you expect:

The billing reconciliation job was moved to a queue worker, accelerating invoice
settlement from up to 24 hours to within two minutes.

Step 2: wrap the flow in a step

senro has two kinds of step: a command, and a registered Go function. A Genkit flow is a Go function, so it is the second kind:

// summarize is the registered flow. A func step runs in the pipeline's own
// process, so a package-level handle is all the wiring the two need.
var summarize *core.Flow[SummarizeInput, Summary, struct{}]

type SummarizeParams struct {
    Path string `json:"path"`
    Name string `json:"name"`
}

func init() { senro.RegisterFunc("ai/summarize", RunSummarize) }

// RunSummarize runs the flow and writes the summary into the one workspace
// this step mounts.
func RunSummarize(ctx senro.Ctx, p SummarizeParams) error {
    dir, ok := ctx.Workspace("doc-" + p.Name)
    if !ok {
        return fmt.Errorf("step %s mounts no doc-%s workspace", ctx.StepID(), p.Name)
    }

    text, err := os.ReadFile(filepath.Join(string(dir), "doc.md"))
    if err != nil {
        return err
    }

    res, err := summarize.Run(ctx, SummarizeInput{Path: p.Path, Text: string(text)})
    if err != nil {
        return err
    }

    fmt.Fprintf(ctx.Stdout(), "%s\n", res.Summary)
    return os.WriteFile(filepath.Join(string(dir), "summary.txt"), []byte(res.Summary+"\n"), 0o644)
}

Three details make this work smoothly:

senro.Ctx embeds context.Context, so it passes straight into summarize.Run with no adapter. Genkit's tracing and cancellation behave exactly as they always did.
ctx.Stdout() is the step's real log stream, recorded and redacted like a command's output. Writing to os.Stdout instead reaches your terminal and no log file.
ctx.Workspace(name) hands the function the same path a mount gives a command, on every executor. The function body does not change when the step later runs in a container or a pod.

RegisterFunc registers the function once, from an init. The name is the function's identity: it is what the plan records and what feeds the step's cache key, so renaming it invalidates the cache exactly as renaming a command would. Parameters must be JSON-serializable, and decoding is strict, so a renamed field fails loudly instead of running with a zero value.

Step 3: build the graph

Because the pipeline is a Go program, fanning out over a corpus is a for loop:

func main() {
    ctx := context.Background()

    g := genkit.Init(ctx,
        genkit.WithPlugins(&googlegenai.GoogleAI{}),
        genkit.WithDefaultModel("googleai/gemini-2.5-flash"))
    summarize = defineFlows(g)

    docs, err := filepath.Glob("docs/*.md")
    if err != nil {
        log.Fatal(err)
    }
    sort.Strings(docs)

    p := senro.New("ai")
    w := p.Workflow("summarize")

    var names, ids []string
    var wss []*senro.WorkspaceRef
    corpus := map[string]string{}

    for _, doc := range docs {
        text, err := os.ReadFile(doc)
        if err != nil {
            log.Fatal(err)
        }
        name := strings.TrimSuffix(filepath.Base(doc), ".md")
        names = append(names, name)
        ids = append(ids, "summarize/"+name)
        corpus[name] = string(text)

        // One workspace per document, so one document changing does not
        // invalidate the other summaries' cache keys.
        wss = append(wss, senro.Workspace("doc-"+name, senro.Scope(senro.ScopeRun)))
    }

    seed := w.Step("seed", senro.Func("ai/seed", SeedParams{Docs: corpus}))
    for i, ws := range wss {
        seed.Mount(ws.At("/doc-"+names[i], senro.RW))
    }

    for i, name := range names {
        // One AI call, one node: its own log, state, retry and cache entry.
        w.Step(ids[i], senro.Func("ai/summarize", SummarizeParams{
            Path: docs[i],
            Name: name,
        })).
            Needs("seed").
            Mount(wss[i].At("/doc", senro.RW)).
            Retry(3, retry.OnInfra()).
            Pure().
            Inputs(artifact.File("doc.md")).
            Outputs(artifact.File("summary.txt"))
    }

    digestStep := w.Step("digest", senro.Func("ai/digest", DigestParams{Names: names})).
        Needs(ids...)
    for i, ws := range wss {
        digestStep.Mount(ws.At("/doc-"+names[i], senro.RO))
    }

    if err := senro.Run(ctx, p); err != nil {
        log.Print(err)
        os.Exit(1)
    }
}

seed writes each document into a workspace of its own. Then one summarize step per document, running in parallel. Then digest, which waits for all of them and merges the results.

That loop produces this graph, rendered straight from the run's own plan.json:

pipeline: ai
  │
  ├─ wave 1                seed                  func
  │
  ├─ wave 2 (3 parallel)   summarize/alpha       func · retry 3/infra
  │                        summarize/beta        func · retry 3/infra
  │                        summarize/gamma       func · retry 3/infra
  │
  └─ wave 3                digest                needs summarize/alpha, summarize/beta,
                                                 summarize/gamma · func

Three documents give three parallel nodes. Thirty would give thirty, from the same for loop, with no change to the code.

Step 4: run it

Run the pipeline binary directly, the way any Go program runs:

go run .

Or hand the package to the senro CLI, which builds it, execs it and attaches automatically. You get a terminal UI on a TTY and plain streaming lines anywhere else:

senro run .

seed             -> succeeded
summarize/beta   -> succeeded
summarize/alpha  -> succeeded
summarize/gamma  -> succeeded
digest           -> succeeded

- alpha: The billing reconciliation job now uses a queue worker for invoice settlement
  within two minutes, replacing the nightly cron.
- beta: The search index now rebuilds incrementally in under a second without blocking
  writes, replacing the old full rebuild and requiring operators to delete the
  `rebuild-search` cron job.
- gamma: Service gamma's public API now features per-tenant rate limiting set at 100
  requests per second, which may cause HTTP 429 errors for existing bursting integrations.

Three real model calls in parallel, each with its own log file, and a merge step that ran once they were all done.

While that is running, a second terminal can watch it, or steer it:

senro attach          # terminal UI
senro ui              # browser view, prints a one-time link
senro attach --run <id>   # reopen a finished run from disk

Every run also lands in runs/<id>/, so senro runs lists what happened and each step's stdout is a file you can read afterwards.

One thing to know when you switch between go run and senro run. A func step's cache key includes the binary digest, so the first run after changing how the binary is built re-runs every func step once, then caches normally from there. That is correct behaviour rather than a bug: the function's body lives in the binary, and senro cannot tell whether it changed without hashing it.

Step 5: stop paying for the same call twice

Each summarize step declares Pure(). Run the pipeline again with nothing changed:

seed             -> succeeded
summarize/beta   -> cached
summarize/alpha  -> cached
summarize/gamma  -> cached
digest           -> succeeded

cached is a real state, not a label. Those three steps did not execute: no model call, no tokens, no latency. Their recorded outputs were restored from the content-addressed store, so digest still read three summaries and produced the same result.

Now change one document out of three:

seed             -> succeeded
summarize/alpha  -> cached
summarize/beta   -> cached
summarize/gamma  -> succeeded
digest           -> succeeded

One document changed, one model call made. And senro will tell you which component of the key moved:

$ senro cache explain --run <id> summarize/gamma
MISS  summarize/gamma  key d3eb7450 (previous f68652ad)
  ✗ input_digests: docs/gamma.md  6750cb09 → ef0fdb64
  ✗ workspace_digests: doc-gamma  16774a4e → d43e13c8
  ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity,
    tool_versions, version unchanged

For a few dozen documents this is a convenience. For a nightly re-index over ten thousand where one file changed, it is the difference between a bill you notice and one you do not.

Note func_identity in that list. For a func step the cache key covers the binary digest, the registered name and the parameter digest. So changing your prompt, your model or the flow's code all correctly invalidate the cache. It is not keyed on the input file alone, which is the mistake a hand-rolled cache usually makes.

Give each call its own workspace

This is the one structural rule to follow when caching model calls.

Inputs feeds the input_digests component of the key, but Mount feeds workspace_digests, which is the digest of the entire mounted workspace. Narrow Inputs do not narrow it.

Mount one shared tree across a fan-out and every step's key depends on every document, so changing one document re-runs all of them and the cache buys you nothing. One workspace per unit of work, as in the loop above, keeps each key dependent on its own input.

Step 6: let a model explain a failure

senro has a second place a model belongs: explaining a failed step. contrib/genkitanalyzer is a ready-made failure analyzer built on Genkit:

go get github.com/xavidop/senro/contrib/genkitanalyzer

It is a nested module with its own go.mod, so senro itself never pulls in an AI SDK. If you never install it, your dependency graph never hears about Genkit.

g := genkit.Init(ctx, genkit.WithPlugins(&googlegenai.GoogleAI{}))

p := senro.New("analyze")
w := p.Workflow("build")
w.Step("migrate", exec.Command("sh", "-c",
    `echo "dial tcp 10.0.3.7:5432: connect: connection refused" >&2; exit 1`))

err := senro.Run(ctx, p,
    senro.WithAnalyzer(
        genkitanalyzer.New(g, genkitanalyzer.Model("googleai/gemini-2.5-flash")),
        senro.AnalyzerName("genkit"),
        senro.AnalyzeTimeout(20*time.Second)))

That is the whole integration. When the step fails, an analysis.proposed event lands in the run's ledger:

{
 "id": "migrate@1",
 "analyzer": "genkit",
 "duration_ns": 8344562458,
 "summary": "The 'migrate' step failed because its command explicitly reported a database
             connection refusal error and exited.",
 "detail": "... The error message \"dial tcp 10.0.3.7:5432: connect: connection refused\"
            typically indicates that a client attempted to establish a TCP connection to a
            server at 10.0.3.7 on port 5432, but the server actively rejected the connection.
            This could be due to the database server not running, not listening on the
            specified address/port, or being blocked by a firewall ..."
}

Because it is an event, the explanation shows up in the TUI, in events.jsonl, and in anything else folding the run.

What the combination gives you

Genkit gives you the model call: typed, traced, provider-agnostic, testable. senro gives you everything around it.

Concern	What senro already does for a step
Ordering	A dependency edge, not a place in a loop
Outcome	Ten states, so a call that passed on the second attempt is `recovered`, not silently a success
Retries	`retry.OnInfra()` retries a dropped connection, never a model answer you did not like
Logs	One log file per step per attempt, redacted
Cost	An action cache keyed on the prompt, parameters, model and binary
Observability	One append-only event stream you can attach to live or replay later
Failure analysis	An analyzer gated behind a human or an explicit policy

You can build every one of those yourself. It is never quite the same twice, and it is rarely tested well.

Conclusion

The pattern is small enough to summarise in four lines:

Define the flow with genkit.DefineFlow as you always would.
Register a func step with senro.RegisterFunc that calls flow.Run(ctx, ...).
Give each call its own workspace and declare Pure() with Inputs and Outputs.
Add senro.WithAnalyzer if you want failures explained.

From there the graph is a Go program, so the loop over your corpus is a for loop and the logic that decides what runs is testable with go test.

senro on GitHub
Genkit, and the Go documentation
The last article in this series: Debugging senro pipelines

Debugging senro Pipelines: Replay a Run, Shell Into a Live Step, Audit a Cache

Xavier Portilla Edo — Tue, 29 Sep 2026 13:39:35 +0000

Introduction

The usual way to debug a CI failure is to add echo "HERE" to a YAML file, push, wait four minutes, read forty thousand lines of log, and repeat until you understand something that would have taken thirty seconds at a shell prompt. The machine that produced the failure is gone, so there is nowhere else to look.

senro is built so that loop is not necessary. This article is the field guide: what to run when a pipeline goes wrong, in roughly the order you should run it. It follows the introduction and the monorepo guide.

Every output below comes from a real broken run.

The one idea underneath all of it

Every observable fact about a run is an event, appended in order, never rewritten. The live terminal UI, the browser UI, and reading the file a week later all build their view by folding the same list through the same function.

So there is no difference between watching a run and investigating one afterwards. It is the same protocol and the same client, reading a socket in one case and a file in the other. senro has no separate offline mode that could be worse than the online one.

Every run writes a directory:

runs/20260911T105325-4410b2c052/
├── events.jsonl      the append-only ledger, the source of truth
├── plan.json         the immutable graph that was executed
├── run.json          id, pipeline, start time
├── logs/<step>/<attempt>/{stdout,stderr}
├── ws/<workspace>    the workspaces this run used
└── work/<step>       each step's sandbox

Note logs/<step>/<attempt>/. Attempts get separate directories, so a step that failed twice and passed on the third try keeps all three sets of logs.

Reading the graph that ran

plan.json is the graph the engine executed, and it is plain JSON with no dependency beyond the standard library behind it. When a run did not do what you expected, the first question is often whether the plan was what you expected, and jq answers it in one line:

jq -r '.nodes[] | . as $n | (($n.needs // ["(root)"])[] | "\(.) -> \($n.id)")' runs/<id>/plan.json

(root) -> checkout
checkout -> vet
checkout -> test
vet -> build
test -> build

This is worth reaching for before anything else on a fan-out, where the plan is built from your repository rather than written by hand. A missing unit, a phantom unit, or an edge you did not mean to declare shows up here rather than three steps into a run.

1. Find the run

$ senro runs -n 6
RUN ID                      PIPELINE  STATUS     STARTED              DURATION
20260911T110526-57739ce41a  slow      running    2026-09-11 20:05:26  14s
20260911T110510-de69b88fe7  slow      cancelled  2026-09-11 20:05:10  9s
20260911T105408-1d19454e86  ship      failed     2026-09-11 19:54:08  0s
20260911T105325-4410b2c052  ci        failed     2026-09-11 19:53:25  1s
20260911T105314-bc1a4e9d3d  ci        succeeded  2026-09-11 19:53:14  1s
20260911T105259-8fc9b32e31  ci        succeeded  2026-09-11 19:52:59  1s

-n caps how many are printed. This answers "what ran" without already knowing a run ID.

2. Replay the run

senro attach --run 20260911T105325-4410b2c052 --ui=plain

checkout stderr | Cloning into '.'...
checkout stderr | done.
checkout succeeded
vet succeeded
test stdout | ?     senrolab/ci [no test files]
test stdout | --- FAIL: TestHello (0.00s)
test stdout |     greet_test.go:7: got "HELLO, senro"
test stdout | FAIL
test stdout | FAIL  senrolab/greet  0.376s
test failed
build skipped_upstream_failed
run failed

This is the same client that watches a live run, reading recorded events instead of a socket. On a TTY you get the full interactive terminal UI, with the same key bindings, on a run that finished yesterday. --follow tails a run from disk with no socket needed.

The last two lines carry most of the diagnosis. test failed is a step that ran and returned non-zero. build skipped_upstream_failed is a step that never ran because of it. Those are different states, so nobody has to read upwards through a log wondering whether build broke too.

Five of the ten states are the ones you meet while debugging:

State	What it tells you
`failed`	It ran, returned non-zero, retries are exhausted
`timed_out`	An attempt outlived the `Timeout` you declared
`panicked`	A Go function step panicked; the stack is in its stderr
`skipped_upstream_failed`	It never ran, something it needs broke
`recovered`	It passed, but not on the first attempt

recovered is not a failure and it is the most useful of the five. A run full of recovered steps is a passing run telling you your infrastructure is flaky, and it stays visible instead of being collapsed into "green".

If you only want the states, the ledger is one command away:

grep step.finished runs/<id>/events.jsonl | jq -r '"\(.step) -> \(.payload.state)"'

checkout -> succeeded
vet -> succeeded
test -> failed
build -> skipped_upstream_failed

3. Look at the files the step had

The log says what the step printed. Often you need what the step was looking at. Workspaces are snapshotted into a content-addressed store, so a failed run's directories survive it:

$ senro ws ls 20260911T105325-4410b2c052
src    sha256:f37f35faa76ae965f92e410a877b7c0c90da14cd926711fde5b0a2137be1c5cf  13 files  3.2 KiB

$ senro ws pull 20260911T105325-4410b2c052 src /tmp/broken-src
$ ls /tmp/broken-src
ci  cmd  go.mod  go.sum  greet  probe

That is the exact tree the failing step saw, on your laptop, from a run on a machine that may no longer exist.

ws pull prints what a snapshot does and does not preserve, which matters exactly once, when a permission looks wrong: modes are normalized to 0644 or 0755, mtimes are fixed at the epoch, and uid, gid, extended attributes, ACLs, hard links and devices are not stored at all. That normalization is what makes a snapshot's digest depend only on content, not on which machine produced it.

For "it worked yesterday", compare two runs:

$ senro ws diff <run-a> <run-b> src
+ added   - removed   M content changed   P mode changed   K kind changed

workspace "src"
  a  20260911T105314-bc1a4e9d3d  sha256:3c3503f85c99ced855d5564624e31f350608c5b029410f65811b5898bc54b81a
  b  20260911T105325-4410b2c052  sha256:f37f35faa76ae965f92e410a877b7c0c90da14cd926711fde5b0a2137be1c5cf
  - bin             dir
  - bin/app         0755  2.3 MiB  53d0659f
  M greet/greet.go  113 B -> 113 B  57930022 -> 6bd8b782
  0 added, 2 removed, 1 modified, 0 mode, 0 kind, 12 unchanged

greet/greet.go changed, and the binary is missing because the build never ran. ws ls and ws diff read stored indexes without downloading any file contents, so both are instant on a tree of any size. Use ws pull when a workspace's last state came from a cache hit, since a cache entry stores a body digest rather than an index.

4. Get inside a step that is still running

While a run is live, open a session on one of its steps:

$ senro shell --pid 36343 --step slow -- ls -la
senro shell: this session runs against pipes: no prompt, no line editing, no job control.
Type a command and press enter; ^D ends the session. Pass --tty for a real terminal.
total 16
drwxr-xr-x  4 xavierportillaedo  wheel  128 Sep 11 20:05 .
drwxr-xr-x  3 xavierportillaedo  wheel   96 Sep 11 20:05 ..
-rw-r--r--  1 xavierportillaedo  wheel    4 Sep 11 20:05 a.txt
-rw-r--r--  1 xavierportillaedo  wheel    4 Sep 11 20:05 b.txt

$ senro shell --pid 36343 --step slow -- cat a.txt
one

That is the step's own workspaces, read-only, at the paths the step sees them, on the step's own executor. Not a copy and not a reconstruction. If the step is running in a pod, the session is in that pod.

Pair it with a breakpoint to stop the run before a step and inspect exactly what it was about to run against. Add --tty for a real terminal with a prompt and line editing, which the local, container and Kubernetes executors host.

Two deliberate limits: no secrets are delivered into a session, and a finished run has no engine to host one, so use ws pull for a run that is over.

5. Attach to a live run you did not start watching

If your pipeline called attach.Listen, you can join a run in progress from any terminal:

$ senro attach --pid 36343 --ui=plain
prepare succeeded
slow stdout | tick 1
slow stdout | tick 2
slow stdout | tick 3
slow started
slow stdout | tick 4
slow stdout | tick 5

Attaching three seconds in replays what already happened, reports where the run currently is, then follows it live. The client does not need to have been there from the start.

From there, attach is not read-only. You can cancel, pause and resume the run, retry, skip or re-run from a step, set a breakpoint, and force a workspace snapshot. senro ui serves the same thing as a browser page on loopback with a one-time link, offering the same control operations except the shell.

6. Ask why the cache did what it did

Caching bugs are the worst class of build bug, because the symptom is that something did not happen. senro cache explain answers it component by component.

A hit:

$ senro cache explain --run <id> build
HIT  build  key a52a9b6c

A miss, after one file changed:

MISS  build  key 0e769472 (previous a52a9b6c)
  ✗ input_digests: ci/main.go  ee7563f3 → 4f7d4315
  ✗ input_digests: greet/greet.go  59d2873d → 57930022
  ✗ input_digests: probe/main.go  added
  ✗ workspace_digests: src  f377ce04 → de2069cc
  ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity,
    tool_versions, version unchanged

Which components changed, both digests for each, which path was added, and an explicit list of what stayed the same.

A surprising miss is almost always a component you forgot about, so it is worth knowing all eleven: command, env, secrets (identity, never values), executor_class, platform, input_digests, workspace_digests, mount_shape, step_shape, func_identity and version.

The one that catches people is workspace_digests: every mounted workspace enters the key in full. Narrow Inputs do not narrow it. A step that mounts a big shared tree misses on any change anywhere in that tree, however precisely its inputs were declared.

7. Check whether a step lied about being pure

Pure() is a promise only you can make. senro does not sandbox network access, so it cannot tell that a tool which also SSHes into production is impure. What it can do is audit the claim afterwards:

senro verify --recheck-pure --run <id> --rerun

It re-executes the run's cached pure steps in throwaway trees restored from the workspace content each step's own cache key records, and compares what they produce against what the cache stored. Nothing runs without --rerun, deliberately: the premise is that a purity claim may be false, so its safety corollary is not assumed either. No cache entry is ever written.

Pointed at a go build step:

NONDETERMINISTIC build  key a52a9b6ce84a  entry from run 20260911T105243-8b696eb1d0
  ✗ output     bin/app   cached cf1663a0  re-run 739ebb3d  again e86c2b3d
  ✗ workspace  src       cached 18160d77  re-run 07c2a015  again a7e507af
  declared inputs   glob:**/*.go, file:go.sum
  declared outputs  file:bin/app
  the two re-runs disagreed with each other as well as with the entry, so this step does not
  produce the same bytes twice from the same input; that is a reproducibility problem (an
  archive embedding a build timestamp looks exactly like this) and it is NOT evidence about
  purity either way

1 nondeterministic of 1 cached Pure() step(s)
1 step(s) cannot reproduce their own output twice, so their entries hold bytes a re-run would
not produce. That is worth fixing and it is not evidence about purity.

It ran the step twice, noticed the two re-runs disagreed with each other and not only with the cache, and declined to draw the wrong conclusion. The build is not byte-reproducible, which is real and worth fixing, and that is separate from whether the step reaches the network. A cruder tool would report "impure" and send you hunting a bug that is not there.

verify exits 0 whether or not it finds anything, because a finding is an answer rather than a failed run. Use --fail-on-mismatch in CI if you want the opposite.

8. Repeat the run instead of rediscovering it

senro rerun --run <id>

rerun re-executes the plan a previous run recorded, from plan.json rather than by rebuilding your pipeline. That is the distinction: it repeats a run instead of re-discovering one, so a fan-out that would find different units today still runs the graph that actually failed. Unchanged steps come from the action cache.

--step re-runs one step and everything below it. --regenerate asks generators for a fresh subgraph, and it is a separate verb because that run may do different work than the one it repeats.

Failures that are not a step's fault

Two categories look nothing alike and are worth recognising immediately.

Build-time refusals

senro.Run builds first, so a dangling Needs, a duplicate step id, an empty command or a plan that would leak a credential comes back before anything executes. The messages do the diagnosis:

plan: step "summarize/alpha" is a func step and sets WorkDir "/tree", which it never runs in:
a func step on the coordinator runs in the coordinator's own process, where the working
directory is process-global, so honouring it would move every step running alongside it.
Reach files through ctx.Workspace(name) [...]

plan: step "summarize/alpha" is Pure() with no Inputs, so its cache key would not change when
its sources do: declare them with Inputs(artifact.Glob(...))

engine: step "leak" puts the value of secret "RegistryToken" in command argument 1; a command
argument is visible in ps(1), in shell history and in auditd execve records, where senro cannot
redact it, so senro refuses to run rather than leak it. Deliver it as a file instead [...]

Each names what you wrote, why it cannot hold, and what to write instead. Read them rather than searching.

Exit code with no output

A step fails with exit code 127 and both stdout and stderr are zero bytes. Exit 127 normally means "command not found", but the real cause here is usually the working directory: the process never started, so it never wrote anything.

The common trigger is a WorkDir that is a subpath of a mount. On the local executor a mount path is a logical name, and senro translates WorkDir into the sandbox only when it matches a mount point exactly:

func CmdDir(workDir string, mounts []Mount) string {
    for _, mt := range mounts {
        if mt.At == workDir {
            return ""
        }
    }
    return workDir
}

So with a workspace mounted at /tree, WorkDir("/tree") works and WorkDir("/tree/services/api") is passed through as a literal host path that does not exist. Keep WorkDir on the mount point and let the tool do the moving:

exec.Command("go", "-C", u.Dir, "build", "-o", "/dev/null", "./...")

The general rule: an exit code with no output means the process never started, so check the working directory and the executable path before you look at the command itself.

Writes through a read-only mount

engine: step "build[unit=services/api]" wrote through its read-only mount of workspace "tree"
(c9271303 became 832bb638); a read-only mount that changes makes every cache key computed from
it wrong

That is go build dropping a compiled binary next to the source. On the container and Kubernetes executors the kernel refuses the write; on local and SSH, senro.RO is checked after the fact. The failure is the point, because a read-only mount that silently changed would poison every cache key computed from it.

Instrument it yourself

None of the above requires the CLI. The event stream is a public contract in github.com/xavidop/senro/api, which depends on nothing outside the standard library, and you can fold it in your own process:

senro.Run(ctx, p, senro.WithSink(mySink))

That is how the trace exporter works, and it is how you push run facts into whatever you already use. Anything the TUI can conclude, your code can conclude from the same events.

Conclusion

The checklist, in order:

senro runs to find the run.
senro attach --run <id> to replay it, and read the states, not just the log.
jq over plan.json when you need to check the graph that actually ran.
senro ws pull for the files it left behind, senro ws diff for what changed since the run that worked.
senro shell --step <id> if it is still running, ideally stopped on a breakpoint.
senro cache explain when the surprise is that something did not happen.
senro verify --recheck-pure --rerun when you suspect a cached result is wrong.
senro rerun to repeat the recorded plan rather than build a new one.

None of these require pushing a commit, and none require the machine that produced the failure to still exist.

senro is at github.com/xavidop/senro, MIT licensed. The other articles in this series:

Building a Real Pipeline with senro: Monorepos, Caching and Triggers

Xavier Portilla Edo — Tue, 29 Sep 2026 13:38:52 +0000

Introduction

In the introduction to senro I covered the model: a pipeline is a Go program, it builds an immutable plan, and the engine executes that plan while anything can attach to the event stream.

This article is the practical follow-up. It covers the five things you will actually write on your first real pipeline: getting your code into a step, caching a step so it never runs the same work twice, fanning one step out over every module in a monorepo, narrowing that fan-out to only what a change touches, and deciding whether a webhook should start a run at all.

Everything below was run against senro v1.4.0.

A pipeline that builds a real repository

Here is the shape of a Go CI pipeline in senro. Check the code out into a workspace, vet and test it in parallel, then build:

// Command ci checks the repo out into a workspace, then vets, tests and
// builds it.
package main

import (
    "context"
    "log"
    "os"

    "github.com/xavidop/senro"
    "github.com/xavidop/senro/artifact"
    "github.com/xavidop/senro/exec"
    "github.com/xavidop/senro/retry"
)

func main() {
    ctx := context.Background()

    repo := os.Getenv("REPO_URL")
    home := os.Getenv("HOME")

    p := senro.New("ci")

    // One directory for this run, mounted by every step that needs the code.
    src := senro.Workspace("src", senro.Scope(senro.ScopeRun))

    verify := p.Workflow("verify")

    verify.Step("checkout", exec.Command("git", "clone", "--depth", "1", repo, ".")).
        WorkDir("/src").
        Mount(src.At("/src", senro.RW)).
        Retry(3, retry.OnInfra())

    verify.Step("vet", exec.Command("go", "vet", "./...")).
        Needs("checkout").
        WorkDir("/src").
        Mount(src.At("/src", senro.RO)).
        Env("HOME", home)

    verify.Step("test", exec.Command("go", "test", "./...")).
        Needs("checkout").
        WorkDir("/src").
        Mount(src.At("/src", senro.RO)).
        Env("HOME", home)

    verify.Step("build", exec.Command("go", "build", "-o", "bin/app", "./cmd/app")).
        Needs("vet", "test").
        WorkDir("/src").
        Mount(src.At("/src", senro.RW)).
        Env("HOME", home).
        Pure().
        Inputs(artifact.Glob("**/*.go"), artifact.File("go.sum")).
        Outputs(artifact.File("bin/app")).
        Retry(3, retry.OnInfra())

    if err := senro.Run(ctx, p); err != nil {
        log.Print(err)
        os.Exit(1)
    }
}

test stdout | ?     senrolab/ci [no test files]
test stdout | ?     senrolab/cmd/app    [no test files]
test stdout | ok    senrolab/greet  0.360s

The graph those Needs calls describe, rendered from the run's own plan.json:

pipeline: ci
  │
  ├─ wave 1                checkout          retry 3/infra
  │
  ├─ wave 2 (2 parallel)   test
  │                        vet
  │
  └─ wave 3                build             needs vet, test · retry 3/infra

Two things in there are not obvious coming from a CI system that hands you a checkout and a shell.

Workspaces: how code reaches a step

Every step runs in an isolated, empty working directory. Nothing hands you a checkout, because "hand the step a checkout" means four different things across a local process, a container, a pod and an SSH host.

A Workspace is a named, versioned directory that steps mount. senro snapshots it into a content-addressed store when a step that mounts it finishes, and restores it for the next step that needs it. So the code gets into the run the way it gets into any CI runner: the checkout step clones it, and the src workspace carries the result to the other three.

Mount modes matter:

Mode	Use it when
`senro.RW`	The step writes into the workspace, including its declared `Outputs`
`senro.RO`	The step only reads

In the pipeline above, checkout is RW because it writes the tree, vet and test are RO because they only read it, and build is RW because its output lands there.

ws.At(path, mode) always needs a mode, Mount accumulates so one step can mount several things, and two mounts at one path are refused at build time.

Workspaces come in three scopes:

Scope	What you get
`senro.ScopeRun`	The default. One directory for this run, shared by every step that mounts it
`senro.ScopePersistent`	One directory on this machine that outlives the run. Requires explicit bounds
`senro.ScopeStep`	One fresh directory per step, shared with nobody

Declaring the environment

Build() adds nothing to a step's environment. Not a HOME, not a GOPATH, nothing. The local executor supplies its own PATH and that is all.

This is deliberate: two developers on the same commit produce the same plan because nothing about their shells leaks into it, and the environment is one of the eleven components a cache key is computed from. In practice it means tools that need something get told about it:

.Env("HOME", home)

Env takes one pair per call, and it is not variadic on purpose: Env("A", "1", "B") would compile and mean nothing.

Caching a step with `Pure()`

Pure() is a promise: given these inputs, this command produces these outputs, and nothing else matters. senro hashes the inputs, the command, the environment, the workspaces, the mount shape and six other things into a key. On a second run with the same key the step is not executed at all and its recorded outputs are restored.

Running the pipeline above twice:

RUN ID                      PIPELINE  STATUS     STARTED              DURATION
20260911T105259-8fc9b32e31  ci        succeeded  2026-09-11 19:52:59  1s
20260911T105243-8b696eb1d0  ci        succeeded  2026-09-11 19:52:43  2s

$ senro cache explain --run 20260911T105259-8fc9b32e31 build
HIT  build  key a52a9b6c

The event stream records it as a state, not a label:

{"type":"cache.hit","step":"build",
 "payload":{"key":"sha256:a52a9b6c...","from_run":"20260911T105243-8b696eb1d0"}}
{"type":"step.finished","step":"build","payload":{"state":"cached","cached":true}}

Touch a source file and the same command tells you exactly why it has to run again:

MISS  build  key 0e769472 (previous a52a9b6c)
  ✗ input_digests: ci/main.go  ee7563f3 → 4f7d4315
  ✗ input_digests: greet/greet.go  59d2873d → 57930022
  ✗ input_digests: probe/main.go  added
  ✗ workspace_digests: src  f377ce04 → de2069cc
  ✓ command, env, secrets, executor_class, platform, mount_shape, step_shape, func_identity,
    tool_versions, version unchanged

One rule about the key is worth memorising, because it decides how you structure a fan-out:

Mount puts the whole mounted workspace into the cache key. Inputs narrows what you declare as sources, but it does not narrow the workspace component. A step that mounts a big shared tree has a key that depends on that entire tree.

The practical consequence: when you cache expensive per-unit work, give each unit its own workspace instead of sharing one.

Read-only mounts are enforced

A step that writes through an RO mount fails, and says so:

engine: step "build[unit=services/api]" wrote through its read-only mount of workspace "tree"
(c9271303 became 832bb638); a read-only mount that changes makes every cache key computed from
it wrong

That one is go build dropping a compiled binary next to the source. The fix is go build -o /dev/null ./..., which is what a build check wanted anyway.

How the rule is enforced depends on the executor, and it is worth knowing which you are relying on:

Executor	What `senro.RO` does	When a write is caught
Container	A real read-only bind mount	At the write. It fails
Kubernetes	`readOnly` on the volume mount	At the write. It fails
Local	Nothing at mount time	Right afterwards, when the workspace is found changed
SSH	Nothing on the far side	On read-back. The write is reported, not carried home

On local and SSH, read-only is a request senro checks after the fact rather than a rule the kernel enforces. Keep sensitive input out of any workspace a step could overwrite by mistake.

Monorepos: fan out with `Expand`

Expand(id, graph) adds one step per unit that a graph discovers. Add a module, get a step, and nobody has to write it.

Here is a Go workspace with four modules and a dependency chain three deep:

libs/log  <-  libs/config  <-  services/api
libs/log  <-  services/worker

verify.Expand("build", gowork.Modules()).
    MaxParallel(4).
    Needs("checkout").
    Affected(src).
    Template(func(u senro.Unit) *senro.StepBuilder {
        // WorkDir is the mount point itself, and go -C moves into the unit.
        return senro.NewStep(exec.Command("go", "-C", u.Dir, "build", "-o", "/dev/null", "./...")).
            WorkDir("/tree").
            Mount(tree.At("/tree", senro.RO)).
            Env("HOME", home)
    })

checkout                     -> succeeded
build[unit=libs/log]         -> succeeded
build[unit=libs/config]      -> succeeded
build[unit=services/worker]  -> succeeded
build[unit=services/api]     -> succeeded

One Expand call, four nodes in the plan:

pipeline: mono (no change filter)
  │
  ├─ wave 1                checkout
  │
  └─ wave 2 (4 parallel)   build[unit=libs/config]
                           build[unit=libs/log]
                           build[unit=services/api]
                           build[unit=services/worker]

Template is called once per unit in sorted order and must return a fresh *senro.StepBuilder, built with senro.NewStep. Each unit gives you:

On `Unit`	What it is
`u.ID`, `u.Name`	The unit's stable identity
`u.Dir`	The unit's directory, relative to the root
`u.Base()`	The last path segment: `"web"` for `"apps/web"`
`u.Sources()`	Every file under the unit, ready for `.Inputs(...)` on a `Pure()` template

Child ids come from the expansion id and the unit (build[unit=libs/log]), never a name the template picks, so the same repository always builds the same graph and a re-run reconstitutes the same children.

Two bounds are worth setting. MaxParallel(n) limits how many children of this expansion run at once. MaxNodes(n) rejects an expansion wider than n at build time, default 500, so a scheduler never discovers mid-run that it has forty thousand sandboxes to hold open.

Expansion happens once, at Build(). Every unit is discovered and written into the plan before the first step starts.

The eight unit graphs

Graph	Discovers
`glob.Dirs` / `glob.Files`	One unit per matching directory, or per directory containing a matching file
`gowork`	Go workspace modules
`cargo`	Rust workspace crates
`jswork`	npm, pnpm or yarn workspaces
`maven`, `gradle`	JVM modules
`pyproject`	Python projects
`bazel`	Bazel packages

glob just matches paths. The others read the ecosystem's own manifests, which is what makes the next section work.

Running only what a change affects

Affected(src) narrows the fan-out to the units a change reaches: the ones owning a changed file, and everything that depends on them, at any depth.

The same pipeline, told that one file changed:

=== changed: services/api/main.go ===
   build[unit=services/api]

=== changed: libs/config/config.go ===
   build[unit=libs/config]
   build[unit=services/api]

=== changed: libs/log/log.go ===
   build[unit=libs/config]
   build[unit=libs/log]
   build[unit=services/api]
   build[unit=services/worker]

The third case is the point. services/api does not import libs/log. It imports libs/config, which does. That transitive hop is why reading manifests beats globbing paths, and it is why Affected over a glob graph is refused at build time: a path does not know what imports it, and senro would rather say so than quietly run everything.

Unaffected units are not skipped, they are not in the plan at all. That is visible in the plan itself. Same pipeline, same Expand, told that libs/config/config.go changed:

pipeline: mono (libs/config changed)
  │
  ├─ wave 1                checkout
  │
  └─ wave 2 (2 parallel)   build[unit=libs/config]
                           build[unit=services/api]

Two nodes instead of four, decided at build time. libs/log and services/worker are not present to be skipped, so a re-run of this recorded plan reconstitutes exactly these two.

The change source is pluggable:

Source	What it is
`change.FromTrigger(ev)`	What the webhook that started the run recorded
`change.Paths("a/x.go", ...)`	A literal list, for your own logic and for tests
`change.Everything()`	Every unit runs
`change.Ignoring(src, "docs/**")`	The above, minus paths you do not care about

change.Paths() with no arguments means nothing changed, not everything. Use change.Everything() for that. They are different answers.

Keep the run directory out of the tree

senro writes each run to runs/<id>/ under the working directory. That directory holds the run's workspaces, and a workspace usually holds a checkout of the repository. Unit graphs walk the filesystem for manifests, pruning only .git and node_modules, so a second run will discover the first run's checkout as a second copy of every module:

build[unit=runs/20260911T105558-08e7daff14/ws/tree/services/api]

Pin the run directory somewhere outside the tree you fan out over:

senro.Run(ctx, p, senro.WithDir("/var/lib/senro/runs/"+runID))

Adding runs/ to .gitignore does not help, because the unit graph is walking the filesystem, not asking git.

Triggers: the pipeline decides whether to run

Instead of a CI service deciding whether to invoke your pipeline, the pipeline binary is its own matcher. Hand it a webhook delivery and it decides whether the event is its business:

err = senro.Run(ctx, pipeline(ev),
    senro.WithTrigger(ev,
        // A push to main reports mode "all", which change.FromTrigger passes
        // straight through: everything builds there.
        trigger.OnPush(trigger.Branches("main")),
        // Everything else is narrowed to what it touched.
        trigger.OnPush(),
        trigger.OnPullRequest(trigger.Actions("opened", "synchronize")),
    ))

A push to a topic branch that touched services/api/main.go runs one step. A tag push, which this pipeline does not want, runs nothing and says why:

$ go run ./examples/monorepo --trigger-event /tmp/ev-tag.json
senro: trigger: no trigger matched the event: tag refs/tags/v1.2.3
  push(branches=[main]): only answers push events, this was tag
  push: only answers push events, this was tag
  pull_request(actions=[opened synchronize]): only answers pull_request events, this was tag
exit status 78

Exit code 78 is EX_CONFIG, the conventional "this was not my business". One line per matcher explaining why it declined, which is a better answer than a GitHub Actions on: block that silently did not fire.

GitHub, GitLab, Bitbucket and Gitea ship, plus your own provider. And the event file is optional: trigger.FromRequest verifies and parses a delivery straight off the wire, per source, so the pipeline binary can be the HTTP endpoint itself.

Because the matcher is a binary, you can test a trigger decision locally against a saved payload before it ever reaches a runner.

Handlers: cleanup and evidence

OnFailure runs its handlers once retries are exhausted and the step still failed. Always runs whatever the outcome. A handler can be a Go function, and it knows what it is cleaning up after:

func Collect(ctx senro.Ctx, _ ReportParams) error {
    f, ok := ctx.Failure()
    if !ok {
        return errors.New("ship/collect only makes sense as a handler")
    }
    fmt.Fprintf(ctx.Stdout(), "%s ended %s (exit %d) on attempt %d\n",
        f.Step, f.State, f.ExitCode, f.Attempt)
    fmt.Fprintf(ctx.Stdout(), "log tail: %q\n", f.LogTail)
    return nil
}

// in the pipeline:
build.Step("package", exec.Command("sh", "-c", "echo packaging; exit 1")).
    OnFailure(senro.Handler("collect", senro.Func("ship/collect", ReportParams{}))).
    ContinueOnError()

package ended failed (exit 1) on attempt 1
log tail: "packaging\n"

The log tail arrives with the failure, so a handler that classifies errors never has to open a file. Handlers run in the order listed, and a handler that fails never masks the step's own failure: the recorded cause stays the original step.

Use senro.Handler(id, action) rather than a workflow's Step. Passing a real step to OnFailure is rejected at build time, because that step would otherwise run twice. A handler also gets its parent's workspaces read-only at the same paths automatically, so Mount on one is refused.

Conclusion

Put the pieces together and the argument is not about syntax.

The fan-out is a function called once per unit, so what your build does is testable with go test. The Affected graph reads your ecosystem's manifests, so a change to a shared library rebuilds the things that transitively depend on it and nothing else. The cache tells you which of eleven key components changed when it misses. The trigger decision happens in a binary you can run locally against a saved payload. And the whole thing is a Go module with a go.sum, reviewed like the rest of your code.

None of that is impossible in YAML. It is that in YAML each one is a plugin, a marketplace action, a service feature or four hundred lines of bash, and here they are library calls in a language with a compiler.

Next in this series:

Genkit flows as senro steps: AI pipelines where every model call is a graph node with its own retry, log and cache entry.
Debugging senro pipelines: replaying a finished run, shelling into a live step, and auditing a cache.

senro lives at github.com/xavidop/senro.

DEV Community: Xavier Portilla Edo

Introducing tobira: Every Database, One Terminal, and One MCP Server for Your Agents

Introduction

Why: one contract instead of five clients

What We Are Building

What You Need

Step 1: Install and seed the demo

Step 2: One query command for scripts and CI

Step 3: The read-only guard

Step 4: Ask in plain language

Step 5: Hand it to your agent with tobira mcp

Step 6: Teach the agent the CLI itself

Step 7: Use it as a library, or add a database

Conclusion

Further reading

A Software Factory on Kubernetes: Gemini Builds, jev Decides (Genkit + senro in Go)

Introduction

Why a factory needs a judge, not a second writer

What we are building

What you need

Step 1: a cluster and an image with a Go toolchain

Step 2: the design loop, with jev inside the flow

Step 3: plan, implement, and let Go have its say

Step 4: the code review gate

Step 5: three pipelines and a state machine

Step 6: run it

Step 7: when jev says no

A request jev cannot be talked into

What the combination gives you

Conclusion

Genkit Flows on Kubernetes, Routed by a Decision Model: senro + jev in Go

Introduction

Why a decision model decides and a chat model writes

What we are building

What you need

Step 1: a cluster senro can reach

Step 2: the flows

Gemini turns an email into a ticket

jev decides what the ticket is

Gemini drafts, jev reviews

Step 3: flows as steps that can run in a pod

Step 4: the Kubernetes target

Step 5: run one, the triage

Step 6: let the verdict choose the next steps

Step 7: the guardrail says no

Step 8: watch it, and run it again

What the combination gives you

Conclusion

Dash: An Open-Source Home Assistant Built on the Gemini 3.8 Live API

Introduction

What Dash Is

Why the Live API and Not a Pipeline

How "Hey Dash" Works

What You Need

Step 1: Wait for the Wake Word Without Paying for It

Step 2: Mint an Ephemeral Token on the Server

Step 3: Open the WebSocket and Stream Audio

Step 4: Let Gemini Act Through Tools

Step 5: Give Gemini the Household Context

Step 6: End the Conversation on Purpose

What Gemini 3.8 Live Changes

What It Costs and Where Your Audio Goes

Gemini Beyond the Voice: Recipes and Translation

Turning It On

Conclusion

Further reading

Durable Genkit Flows with Temporal: Introducing the genkitx-temporal Plugin

Introduction

What is genkitx-temporal?

Why Temporal and Genkit are a great match

Installation

Usage end to end

1. Define a flow with defineTemporalFlow

2. Start a Worker

3. Execute a flow as a Temporal Workflow

Configuration

Advanced: combine with your own workflows and activities

API summary

When to reach for genkitx-temporal

How it pairs with the rest of the Genkit ecosystem

Step 5: Hand it to your agent with `tobira mcp`

What is `genkitx-temporal`?

1. Define a flow with `defineTemporalFlow`

When to reach for `genkitx-temporal`