DEV Community: Journal A.I.

"🧪 RAG Without CI Metrics Is Just a Chatbot"

Journal A.I. — Thu, 29 Oct 2026 18:41:41 +0000

Staging “looks fine” is how RAG regressions ship.

Someone updates the corpus, bumps an embedding model, or tweaks chunk size. Answers still sound confident. A week later support hears “it cites the wrong policy page.”

Put metrics on the pull request

You need a frozen question set with expected doc ids or answer keys. Run it in CI when retrieval code or corpus fingerprints change.

Useful offline signals:

Recall@k — did the right chunk appear in the top k?
MRR — how high was the first relevant hit?
Faithfulness / citation hit — does the answer point at retrieved ids?
Empty retrieval rate — sudden spikes mean a broken index build

You do not need a perfect LLM judge on day one. Start with retrieval metrics. Add answer judges once retrieval is stable.

# evals/rag_ci.py
from statistics import mean

def recall_at_k(hits, relevant, k=5):
    top = {h.doc_id for h in hits[:k]}
    return len(top & set(relevant)) / max(1, len(relevant))

def evaluate(cases, retrieve):
    scores = []
    for case in cases:
        hits = retrieve(case["question"], k=5)
        scores.append(recall_at_k(hits, case["relevant_ids"], k=5))
    return mean(scores)

def main():
    score = evaluate(load_cases("evals/rag_cases.json"), build_retriever())
    threshold = 0.75
    print(f"recall@5={score:.3f} threshold={threshold}")
    if score < threshold:
        raise SystemExit(1)

if __name__ == "__main__":
    main()

When CI should run

Trigger on:

changes under rag/, embeddings/, chunking/
corpus fingerprint file changes
scheduled nightly for full rebuild drift

Skip on unrelated frontend PRs so people do not learn to hate the job.

Honest trade-off

A fixed case set goes stale. Budget an hour every two weeks to add failures from production. If you never refresh cases, CI protects an old product.

LLM-as-judge is convenient and noisy. Keep it advisory until you measure agreement with humans on a sample.

Wire it like any other test

- run: python -m evals.rag_ci
- uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: rag-eval
    path: evals/last-run.json

Make the check required when retrieval is on the critical path for customers. If RAG is still experimental, keep the job visible but non-blocking, and write a date to flip it to required.

Monday plan

Export 30 real questions from support or search logs.
Label relevant doc ids by hand (painful once, valuable forever).
Fail the PR under a recall threshold you can defend.

Do you know last week’s recall@5, or only that the chatbot “felt better”?

"🔌 MCP Servers Are Spreading. Treat Them Like APIs"

Journal A.I. — Thu, 29 Oct 2026 18:41:05 +0000

MCP crossed from “interesting RFC” into “paste this server into your agent.”

GitHub READMEs, Discord drops, and one-click installs now ship tools that can read files, hit HTTP, and sometimes run shell. That is power. It is also the same surface that ate API keys in every other plugin ecosystem.

What is actually shipping

A Model Context Protocol server exposes tools and resources to a host (Claude Code, IDE agents, custom runtimes). The host decides which servers attach. The model decides which tools to call. Your job is everything between those two sentences: allowlists, auth, and blast radius.

Community servers move fast. Pin a commit or a release tag. “Latest” is how supply-chain surprises arrive.

Minimum hardening before attach

Run the server as a separate process with the least filesystem and network it needs.
Allowlist tools — if the server offers run_shell, leave it off until you have a reason.
Inject secrets at the edge — the model should not see raw keys; the host or a proxy injects them into approved calls.
Log every tool call with args redacted. You will need the trail.

// sketch: only expose a subset of tools from an MCP server
const allow = new Set(["search_docs", "get_issue"]);

for (const tool of server.listTools()) {
  if (!allow.has(tool.name)) continue;
  host.registerTool(wrapWithAudit(tool));
}

MCP vs a custom tool adapter

	MCP server	In-repo tool adapter
Speed to try	Fast	Slower
Versioning	External	Yours
Review	Often thin	PR-reviewed
Fit when	Shared protocol across hosts	Critical side effects

Use MCP to explore and to standardize. Keep money movement, production deploys, and secret rotation in first-party adapters until the server has the same review bar as your services.

Honest trade-off

Blocking community servers slows experimentation. Opening them without a sandbox is how you buy someone else’s bug. The middle path is a staging host with fake credentials and a recorded allowlist, then promote servers the same way you promote libraries.

Practical checklist

[ ] Pinned version / commit
[ ] Network egress limited
[ ] Tool allowlist reviewed in a PR
[ ] Secrets never in model context
[ ] Audit log on every invocation

Which MCP servers in your stack could delete data today if the model asked nicely?

"⚡ Kafka Is a Better Agent Bus Than Your Chat Loop"

Journal A.I. — Thu, 29 Oct 2026 18:40:29 +0000

Most agent runtimes are a while-loop with hope.

Call a model. Call a tool. Call the model again. If the tool is slow, the loop waits. If the tool fails halfway, you retry and double-charge the side effect. If ten users arrive together, you discover that “async” meant “threads until OOM.”

What Kafka gives the agent

Treat tool work and model steps as messages, not nested awaits:

Backpressure — consumers lag instead of melting the process
Retries with isolation — poison messages go to a DLQ, not into an infinite model retry
Fan-out — one “task created” event can wake planner, tool worker, and auditor
Audit — the log is the trace

This is the same reason we put payment capture and email sends on a bus. Agents are just another worker with expensive CPU and sticky side effects.

A minimal shape

user.request        → agent.tasks
agent.tasks         → agent.model.calls
agent.model.calls   → agent.tool.requests
agent.tool.results  → agent.tasks   (continue or finish)
agent.deadletters   ← anything that exceeded retries

Each message carries task_id, attempt, idempotency_key, and a hard budget (tokens + wall clock).

Idempotency is not optional

Tool calls that create tickets, charge cards, or send mail must key on idempotency_key. Kafka at-least-once delivery will replay. If your tool layer is not idempotent, the agent will look “smart” while it double-books the world.

public CompletionStage<ToolResult> run(ToolRequest req) {
    return idempotencyStore
        .begin(req.idempotencyKey())
        .thenCompose(token -> {
            if (token.alreadyDone()) {
                return CompletableFuture.completedFuture(token.result());
            }
            return execute(req).whenComplete((r, err) ->
                idempotencyStore.complete(token, r, err));
        });
}

Backpressure beats bigger boxes

When model latency spikes, lag grows on agent.model.calls. That is visible. A chat loop hides the same overload as rising latency and silent queueing inside one JVM. Prefer lag alerts over “the agent feels slow.”

Tune max.poll.records, pause partitions when budgets trip, and keep tool workers separate from model workers so a stuck HTTP tool does not block planning.

Honest trade-off

Kafka is operational weight. For a single-user CLI agent, a loop is fine. The bus starts to pay rent when you have concurrent users, shared tools with side effects, or a need to replay incidents.

Do not put prompt text in cleartext topics if the cluster is a shared compliance boundary. Encrypt or store payloads by reference.

When to reach for it

Multiple agent workers must share one task stream
Tools have side effects that need idempotent retries
You already run Kafka for the product and know the ops cost

Would you debug last week’s agent incident from a topic, or from a screenshot of a chat transcript?

"⛔ Stop Merging Agents Without a Required Check"

Journal A.I. — Thu, 29 Oct 2026 18:39:39 +0000

Optional checks are how bad agents reach main.

Teams add agent-evals to Actions, celebrate, then leave the check non-required. The red X becomes a suggestion. The merge button stays green if someone is in a hurry.

Required means required

In GitHub: branch protection → require status checks → select the eval job name exactly. If the name drifts after a workflow rename, protection silently stops enforcing. Pin the check name in the same PR that renames the workflow.

Also fail closed on infra errors. A missing API key in CI secrets is not “skip evals.” It is a red build.

- name: agent evals
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  run: |
    if [ -z "$OPENAI_API_KEY" ]; then
      echo "OPENAI_API_KEY missing"
      exit 1
    fi
    python -m evals.ci_gate --suite merge

Split suites so merges stay fast

One fat suite that needs twenty minutes will get bypassed. Split:

Suite	When	Goal
`merge`	every PR	5–15 cases, under 3 minutes
`nightly`	cron	broader set, live models, richer traces
`release`	tag	full regression before a version bump

The merge suite must be boring and mostly offline. Nightly can spend money. Release can be slow.

Flake is a product bug

If the same case fails one in five runs, do not mute it. Fix the case:

pin model + temperature 0 when the vendor allows it
assert on tool names and structured fields, not full free text
isolate network with recorded HTTP fixtures

A flaky required check trains the team to ignore CI. That is worse than no check.

Trace every failure

Store the run artifact: prompt, tool calls, tokens, final output. Without traces, “evals failed” becomes a Slack shrug.

python -m evals.ci_gate --suite merge --out artifacts/eval-run.json

Upload artifacts/ on failure. The next engineer should replay the case without guessing.

Honest trade-off

Required checks slow the first weeks. People will ask to “just merge this one.” Say no once, then the habit sticks. The cost is friction. The alternative is production loops that look like “the model was weird today.”

Checklist

[ ] Eval job name matches branch protection
[ ] Secrets missing → fail, not skip
[ ] Merge suite under a few minutes
[ ] Flaky cases owned, not silenced
[ ] Traces uploaded on failure

Is your agent check required on main, or is it still a polite warning?

"🚫 Your Agent Demo Is Not a Release Gate"

Journal A.I. — Thu, 29 Oct 2026 18:39:36 +0000

The demo looked green. The merge still broke a customer workflow.

That is the whole story for most agent work I see in 2025–2026. A notebook shows “pass rate 92%.” A slide says “evals.” Nobody can point to the GitHub check that would have blocked the bad prompt.

What broke

We shipped a tool-calling agent behind a feature flag. The notebook suite used a frozen fixture set from two weeks earlier. Production traffic hit a new tool schema the same day. The agent retried, burned tokens, and wrote partial side effects.

The notebook never saw that schema. CI never asked.

The smallest CI gate that would have caught it

You do not need a research harness on day one. You need three checks that fail the PR:

Contract check — tool JSON schemas in the repo must match what the agent loads at runtime (hash or OpenAPI snapshot).
Golden path — 5–20 fixed user tasks with expected tool names and final answers. Deterministic fixtures only.
Budget check — max tokens / max steps per task. A loop that “eventually works” is a fail.

Wire them as one job:

# .github/workflows/agent-evals.yml
name: agent-evals
on:
  pull_request:
  push:
    branches: [main]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-evals.txt
      - run: python -m evals.ci_gate --suite golden --max-tokens 4000

Keep the suite offline when you can. If a live model is required, pin the model id and store traces as artifacts.

How the gate looks in code

# evals/ci_gate.py
import json
import sys
from pathlib import Path

def load_suite(path):
    return json.loads(Path(path).read_text())

def run_case(case, agent):
    result = agent.run(case["input"], max_steps=case.get("max_steps", 8))
    ok_tools = result.tool_names == case["expect_tools"]
    ok_answer = case["expect_substr"] in (result.final or "")
    ok_budget = result.tokens <= case["max_tokens"]
    return ok_tools and ok_answer and ok_budget, result

def main():
    suite = load_suite("evals/golden.json")
    failed = []
    for case in suite["cases"]:
        ok, result = run_case(case, build_agent())
        if not ok:
            failed.append((case["id"], result.summary()))
    if failed:
        for cid, summary in failed:
            print(f"FAIL {cid}: {summary}")
        sys.exit(1)
    print(f"PASS {len(suite['cases'])} cases")

if __name__ == "__main__":
    main()

golden.json is boring on purpose: ids, inputs, expected tool sequence, substring checks, token caps. Boring is what merge gates need.

Honest trade-off

A CI gate will reject useful prompt experiments that still need exploration. That is the point. Exploration stays on a branch or a notebook. Mainline gets the harness.

You will also spend time babysitting fixtures when tools change. That ownership cost is real. Paying it in CI is cheaper than paying it in incident reviews.

What to do Monday

Pick 10 real failures from the last month. Turn them into golden cases.
Add schema hash check for every tool the agent can call.
Make the workflow required on the PR.

If the check is optional, it is still a demo.

Has your agent stack blocked a merge yet, or is “evals” still a slide?