DEV Community: TribeROI

Stop chasing fads. Operate Claude like a startup founder.

Mark Birch — Sat, 22 Aug 2026 19:07:41 +0000

I am running an AI-native startup. Early on though, I would spend so much time staying ahead of every new AI hot take that I lost focus on what I needed to do to move the company forward. I didn't need a hundred skills, fifty MCP servers, or Ralph Wiggum loops sending my API bill to the moon. I needed answers to plain questions. What is this model costing me? Is it writing code I keep? Is that tip I saw on X helping, or just making me feel busy?

Those are operating questions. Using Claude is the easy part. You send a prompt, it writes code, you ship. Operating is what turns prompts into something you can trust at speed. That isn't about model quality. It's about what you build around the model.

What I didn't realize is that everything I needed, Claude had already shipped. There are three layers Claude provides, depending on the task:

Extend Claude Code itself with configuration,
Call the API directly for one small task,
Embed Claude as an agent inside your own code.

The skill is knowing which of the three to reach for, and when. Here are five ways I used these three layers to build an AI-coding configuration I trust.

1. See what it's costing you

Beyond the monthly subscription and the API line item, do you know the numbers behind the numbers? They are sitting on your laptop. Claude Code writes a JSONL log of every session to disk. A free tool called ccusage reads those logs and reconstructs your spend and cache-hit rate across your entire history instantly:

npx ccusage@latest daily

There's a number that matters even more, whether the model's edits are getting accepted or thrown away. You won't find that one in the log. That answer comes from Claude Code's OpenTelemetry export. You turn it on in settings and point it at a dashboard:

// .claude/settings.json
{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_METRICS_EXPORTER": "otlp"
  }
}

The accept/reject rate is the closest thing you have to knowing whether the model's output is worth keeping. A high reject rate means you're paying to generate edits you then delete.

There was no need to write code or make API calls. All it required was simple configuration. That's the first layer, extending Claude Code.

2. Cache the part that doesn't change

Prompt caching is your biggest single cost lever. A cached read costs roughly 10% of the normal input price. A stable prefix you reuse on every call, like your system prompt, CLAUDE.md, or tool definitions, should be almost free after the first call. Anthropic's caching runs on an explicit cache_control marker, and the savings on a long stable prefix are around 90%.

The issue is that caching matches on an exact prefix. If anything at the top of your context changes between calls, like a timestamp, a session ID, or reordered tool definitions, every byte downstream of the change is a cache miss, meaning you pay full write price. One volatile token at the top invalidates everything under it, and it's the most common reason a cache-hit rate sits far lower than it should.

You can see it working by reading the API response yourself. This is your first direct call to the Messages API, the Call layer, firing the same request twice to print the cache fields:

// cache-probe.ts: a direct Messages API request, twice
import Anthropic from "@anthropic-ai/sdk";
import { readFileSync } from "node:fs";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY
const stablePrefix = readFileSync("sample-context.md", "utf8"); // a big, stable body

async function call(label: string) {
  const res = await client.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 128,
    system: [{
      type: "text",
      text: stablePrefix,
      cache_control: { type: "ephemeral", ttl: "1h" }, // the lever
    }],
    messages: [{ role: "user", content: "One-line summary." }],
  });
  console.log(label, {
    write: res.usage.cache_creation_input_tokens, // tokens written to cache
    read:  res.usage.cache_read_input_tokens,     // tokens served from cache
  });
}

await call("cold"); // write > 0, read = 0  -> you paid to fill the cache
await call("warm"); // write = 0, read > 0  -> served from cache, ~90% cheaper

The cold call shows a write and no read. The warm call flips, with no write and a read served back at a tenth of the price. If both reads come back zero, your prefix is either below the cache minimum (about 4,096 tokens for Haiku) or it's changing between calls, which is the bug you're searching for. The usage object tells you exactly what got cached.

One detail that's easy to miss is that the effective cache TTL defaults to five minutes. Pause to read a diff for ten minutes, and the cache can expire and re-warm at full write price on your next call. Setting ttl: "1h" on a slow interactive loop is worth it. The five-minute default is tuned for an agent running flat out, not an engineer thinking through a problem.

3. Put your guardrails in code, not in a prompt

In July 2025, an AI agent at Replit deleted a production database during a code freeze. A worse version went viral in 2026 when a founder watched an agent wipe both the production database and its backups, even though the project had explicit written rules telling it not to.

The lesson most people took was "the AI did it." The actual lesson is that the safety rule lived in a config file and was treated like a prompt. The model can reason its way around it or just lose the thread once the context window fills. If a rule must stick, it has to run as code.

Claude Code gives you that through hooks. A hook is a deterministic check that runs before a tool call and can block it outright, with no model judgment in the loop. Here's a PreToolUse hook that inspects every shell command for a destructive operation against anything related to production, and blocks it unless it sees a typed confirmation:

// a PreToolUse hook: deterministic, runs before the tool, can refuse
const command = toolInput.command ?? "";
const destructive = /\b(drop\s+table|truncate|rm\s+-rf|reset-db)\b/i.test(command);
const looksProd  = /(prod|production)/i.test(command);

if (destructive && looksProd) {
  return {
    decision: "block",
    reason: "Destructive op against a production target. Requires typed confirmation.",
  };
}

You can use the same method to guard against hallucinated packages. Models will confidently recommend a package that doesn't exist. Attackers register those exact names and wait, an exploit called slopsquatting. A second hook checks that an install target actually exists, and clears a minimum age and download count, before it runs. The key point is that the check runs in code, where the model can't ignore it.

One warning, though. Once a hook is scanning every command for trigger words, it will block your own commands too, including a harmless echo that happens to mention one. When you need a trigger word in a command, pipe it in from a text file so it never appears in the command line the hook scans.

This is another example of extending Claude Code. The key pattern to follow is that anything that has to be reliable belongs in a hook, never left to a model's judgment.

4. Catch the code that looks right and isn't

The top frustration with AI coding tools isn't the model failing outright. It's output that's almost right but not quite. It is the most common complaint by a wide margin, named by 66% of developers in the Stack Overflow 2025 survey. Almost-right is worse than plainly wrong. Wrong gets caught immediately. Almost-right looks shippable, passes review, and fails many weeks later because of an edge case that was never checked.

A normal CI gate won't help. Lint, types, and happy-path tests all go green on code that's subtly off. Take this simple function:

// lib/stats.ts
export function averageEventsPerMember(members: Member[], events: Event[]): number {
  return events.length / members.length; // looks fine. returns NaN when members is empty.
}

The happy-path test passes. CI is green. The bug only appears when the members list is empty, and then a user sees NaN on the screen.

Two simple tools catch most of this, and neither needs a full agent. The first is a scope check, the Call layer doing one small job. After a change, a single stateless API call scores the diff for whether it stayed inside the task you asked for:

// scope-check.ts: a disposable scorer on a single API call
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();

export async function scopeScore(taskSpec: string, diff: string) {
  const res = await client.messages.create({
    model: "claude-haiku-4-5",          // fast and the right tool for a bounded job
    max_tokens: 200,
    system:
      "You score code diffs for scope creep. Reply ONLY with JSON: " +
      '{"in_scope": boolean, "out_of_scope": string[], "risk": "low"|"med"|"high"}.',
    messages: [{ role: "user", content: `TASK:\n${taskSpec}\n\nDIFF:\n${diff}` }],
  });
  const raw = res.content.find((b) => b.type === "text")?.text ?? "{}";
  const clean = raw.replace(/```
{% endraw %}
json|
{% raw %}
```/g, "").trim(); // strip fences the model adds
  try { return JSON.parse(clean); } catch { return { in_scope: true, out_of_scope: [], risk: "low" }; }
}

That call costs a fraction of a cent and catches the model editing three files when you asked it to touch one. The fence-stripping and the try/catch matter. Models often wrap JSON in markdown, which crashes a raw parse in production, so you strip the fences and catch the failure.

The second is a verifier subagent. After a change, it writes adversarial tests aimed at exactly what you changed and runs them before you do, so almost-right gets caught by a grader you built. That is the Extend layer again, a subagent you configure once and reuse. I wrote about the checks I run in an earlier post, Four checks I built to trust them.

5. Prove a config change helped instead of guessing

Every time you edit your CLAUDE.md, swap a model, or restructure your context, you're running an experiment, but no one treats it like one. You make the change and you keep it or revert it based on a hunch. ETH Zurich evaluated LLM-generated context files and found they decreased task success by a few percent while raising cost by more than 20%. Context the model wrote for itself was making the agent worse and more expensive.

The only fix is to measure against your own tasks. You build a small harness that runs your setup headless against a fixed set of tasks, under two configurations, and compares them. This is the third layer, where you embed Claude as an agent inside your own code, through the Agent SDK, which makes implementing the headless part simple:

// harness.ts: drive headless Claude agents from your own code
import { query } from "@anthropic-ai/claude-agent-sdk";

async function runTask(task: GoldenTask, configDir: string) {
  let cost = 0;
  for await (const msg of query({
    prompt: task.prompt,
    options: { cwd: task.repoPath, settingSources: [configDir] }, // config A or config B
  })) {
    if (msg.type === "result") cost = msg.total_cost_usd ?? 0;
  }
  const passed = await task.check(task.repoPath);
  return { passed, cost };
}

// Run the task set under config A and config B, several times each,
// then compare pass-rate and cost.

When I ran this on a bloated config against a lean one, the heavy one stuffed with every "best practice" I'd collected, the verdict was that it cost 1.26 times as much for the same pass rate. Based on vibes, I would have kept it, but the numbers told me otherwise.

One caveat is that agentic eval results drift with API latency by time of day, and a single A-versus-B run can hand you a difference that's pure noise. You have to run the comparison several times to give this eval real rigor.

Putting it to work

None of this is hard. Each part is less than an hour of work. Almost all of it ships with Claude, the telemetry, the cache controls, the hooks, the disposable API calls, and the headless SDK.

These five ideas run across the three layers Claude already provides. You extend it with hooks and telemetry. You call it for simple, bounded jobs. You embed it as an agent when you want it to run on its own. Operating well is knowing which of the three a task needs.

Practice	Layer
Telemetry & accept-rate	Extend
Prompt caching probe	Call
Hooks as guardrails	Extend
Scope check + verifier	Call + Extend
A/B eval harness	Embed

Small teams usually skip all of this as it feels tedious. Nothing forces the issue until the bill spikes or a database disappears. The hot takes will keep coming. Implementing these now is what lets you ignore them and operate Claude reliably so you can focus on shipping product.

I built all five into a workshop that walks you through writing each tool yourself, against a sample codebase, one part at a time. It's open source on GitHub, and you can run it solo this weekend or with your team. Start the Claude Founders Workshop.

startupmark / claude-founders-workshop

acme-community

Ask a founder what their last feature cost in tokens, or whether the model's output is worth keeping, and most can't answer. Not for lack of caring. Nothing on their screen shows them, so the question never gets asked.

That gap is the whole point of this workshop. The Stack Overflow 2025 survey found that the top frustration with AI coding tools isn't outright failure, it's output that's "almost right but not quite" (66% of developers), the kind that looks shippable until it breaks on an edge case. Many teams are using Claude. Very few are operating it: watching the spend, capping the blast radius, catching the near-misses, and proving a config change actually helped instead of guessing.

This repo is the reference build for the Claude Founders Workshop, a hands-on, five-part series that takes a small team from using Claude Code to operating it. acme-community is the…

View on GitHub

I'll go deeper on each lesson in the posts that follow, the caching mechanics, the hook patterns, and the eval harness. If you run it and something breaks, or you find it super valuable, please reach out and let me know in the comments.

I use AI agents to code. Four checks I built to trust them.

Mark Birch — Sat, 01 Aug 2026 01:36:05 +0000

I am a solo startup founder. In order to launch TribeROI in a reasonable amount of time, I use AI agents that read the codebase, write code, run tests, and open pull requests. My job is to review and merge the code.

What makes this process work is not the agents, but a set of checks I built so I could trust what they produce.

Everyone worries that AI writes bad code. Bad code was never my problem. The problem is that agents produce more than one person can properly review, and this is when defects surface. A few weeks ago, I hit twelve big defects in a single day, which all passed my tests. I only found them by reading the live system.

The failures fell into three patterns. First, artifacts that exist but nothing runs them. Second, systems that could not find an answer and substituted a plausible default. Third, work that was rebuilt without checking for already existing code that solved the same issue.

The second pattern burned me twice. When the agent cannot determine something, it will write a fallback. While this seems reasonable to the agent, the result is a duplicate resource in production.

Here are the four rules I have now built to give me higher confidence in the agent's code.

1. A rule is a test, or it doesn't exist

Every safety rule I keep has to name the specific automated check that fails when someone breaks it. If I can't name one, it is not a guarantee and it goes on a list titled "things I have to remember to check."

When I held my own rules to that standard, eight of the eleven items on my code review checklist enforced nothing whatsoever. They were boxes you tick that change no outcome. Three of the eight rules I'd labeled "hard requirements" were in the same shape, and a batch of alerts had done nothing since the day I wrote them. I'd been reading that checklist and thinking everything was fine.

An unenforced rule is worse than no rule at all, because you lean on it.

2. Never let "unknown" look like "fine"

This is the highest-value rule of the four and the cheapest to act on. Go find every place your code turns "I could not determine this" into a value, and make it stop instead.

It bites in application code. Mine met a permission role it didn't recognize, fell back to least privilege because that felt like the safe choice, and locked eight paying customers out of the admin screens in their own accounts. Nothing threw. The fallback was doing exactly what I'd told it to do.

The expensive version was in the ops scripts that create-or-update cloud resources:

CHANNEL=$(gcloud beta monitoring channels list --format="value(name)" | head -n1)

The pipe hands the exit status to head, which happily succeeds, so set -e never fires. A transient auth blip becomes an empty string, the empty string reads as "does not exist," and the script cheerfully creates a second copy of a resource that was already there. That was 7 July. I patched that one script, told myself I was done, and on 21 July the same shape turned up somewhere else and orphaned a notification channel that eleven alert policies were pointing at.

The second time taught me more than the first. A fact that every code path depends on belongs in one place they all call:

must_read() {
  if ! MR_OUT=$("$@" 2>"$MR_ERR"); then
    echo "!! READ FAILED, refusing to continue: $*" >&2
    echo "   A failed lookup is not proof the resource is absent." >&2
    exit 1
  fi
  printf '%s\n' "$MR_OUT" | head -n1
}

Written as VAR=$(must_read ...), it propagates the failure, the assignment fails, and set -e ends the run. Two outcomes survive: the real value, or a halt.

3. Test the wiring, not just the artifact

A file that exists reads like proof that it works, and almost nothing checks whether anything runs it. I shipped an alert policy as JSON, declared it in the manifest, and the apply script never called apply_policy on it. The dry-run hid the whole thing, because the validator scans the whole directory, while the applier names each policy by hand, one line per policy. One of them saw the file and the other didn't.

So there's now a test that greps the apply script for its own calls and checks both directions. Every declared policy gets applied, and every call points at something declared. It's ugly and I'm not proud of it, and it's the only thing that proves the file is reached.

The other assertion is the one worth copying:

def test_the_policy_pair_is_not_vacuous() -> None:
    assert ALERT_POLICIES, "no declared policies — the guard would be vacuous"
    assert _applied_policy_patterns(), "no apply_policy calls — same"

A guard that iterates an empty collection passes by looking at nothing. Rename the constant it reads and both sides go empty, so the check stays green while proving absolutely nothing. That's the same failure one level up. If you write guards, assume you already have one of these.

4. An agent's authority is a mechanism, not an instruction

I run a mode where an agent works unattended for hours, and it gets Bash(gcloud:*) outright, because a permission prompt nobody is awake to answer is just a hang with extra steps. That's only safe because the real boundary sits somewhere else: a hook where gcloud is default-deny, letting through reads and a named set of additive commands, and refusing everything else.

A config file can't do this job. In Claude Code, ask beats allow regardless of how specific the allow rule is, so a narrow allow sitting next to a broad ask(gcloud *) is dead code. Prefix matching is fragile too, and gcloud run with two spaces doesn't match Bash(gcloud run *). As a way to permit things, that's a hole. Inside a default-deny hook it's harmless, because anything unrecognized gets refused anyway.

The hook's catch block denies as well. Every other guard I've written fails open, which is the right call almost everywhere. This one is the only thing standing between a broad grant and production, so a crash has to refuse rather than guess.

What I'd build first

Take the second rule. Search your codebase for the fallbacks that fire when something couldn't be determined, and make each one stop instead. It is quick, needs no new tooling, and it is a failure that crops up surprisingly often.

The insight across all these rules is that systems confidently report success when it's clearly not the case. In AI development, agents make this an even higher priority, because they produce code faster than you can read it, so these checks have to scale.

I'd like to hear what other people are building to harden their agentic development workflows, especially if you've hit a class of failure I haven't. Let me know in the comments.