DEV Community: Harish Kotra (he/him)

How I Built a Slack Bot That Flags YC Founders Before YC Even Announces Them

Harish Kotra (he/him) — Mon, 31 Aug 2026 11:33:23 +0000

A deep dive into the YC Launch Monitor — a Python bot that tracks every new YC and a16z Speedrun company, and — the fun part — catches founders announcing their acceptance on X/LinkedIn before the official listing goes live.

If you work in GTM (go-to-market), you know the game: the person who reaches a freshly-accepted YC founder first wins the meeting. The problem is that YC publishes its new batch all at once, and by then every sales rep on the internet has the same list.

The real signal lives earlier — in the founder's own "big news: I got into Y Combinator" tweet, posted days before YC updates its directory.

So I built a bot that watches for both. It's called YC Launch Monitor, it's open source, and this post walks through exactly how it works under the hood.

The core idea: two kinds of signal

The bot emits two alert types:

⚡ EARLY — a founder announced their acceptance on X/LinkedIn, but the company is not yet in the official YC/Speedrun directory. This is the "get ahead of everyone" signal.
✅ CONFIRMED — the company is now officially listed in the YC directory or the a16z Speedrun program.

Both land in a Slack channel as rich Block Kit cards, deduplicated, on a schedule.

Architecture

The design is deliberately pluggable — four sources feed one pipeline, and adding a fifth is a single module:

                 ┌─────────────────────────────────────────────────────────┐
                 │                       main.py  (orchestrator)           │
                 │   --once / --loop  →  run_once() → deliver()            │
                 └───────┬──────────────────────────────┬──────────────────┘
                         │                              │
         OFFICIAL sources│                              │SOCIAL sources (optional)
                         ▼                              ▼
   ┌──────────────────────────┐          ┌───────────────────────────────────┐
   │ yc_directory.py          │          │ x_twitter.py / linkedin.py        │
   │  yc-oss GitHub Pages API │          │  X API v2 / pluggable endpoint    │
   │  → changes feed + diff   │          │  → raw Post objects               │
   └───────────┬──────────────┘          └───────────────┬───────────────────┘
               │  ✅ CONFIRMED                           │
               │                                         ▼
   ┌───────────▼──────────────┐          ┌───────────────────────────────────┐
   │ speedrun.py              │          │ detector.classify_post(post, idx) │
   │  speedrun-api.a16z.com   │          │  match vs CompanyIndex            │
   │  → slug-list diff        │          │  → ⚡ EARLY  or  ✅ CONFIRMED      │
   └───────────┬──────────────┘          └───────────────┬───────────────────┘
               │  ✅ CONFIRMED                           │
               └───────────────┬─────────────────────────┘
                               ▼
                  ┌────────────────────────────┐
                  │ state.py  (SQLite state.db)│   dedup via seen_events
                  │  mark_seen / is_seen       │   incremental via snapshots
                  └────────────┬───────────────┘
                               ▼
                  ┌────────────────────────────┐
                  │ alerts.py  (SlackClient)   │   Block Kit payload
                  │  chat.postMessage / webhook│
                  └────────────┬───────────────┘
                               ▼
                        ┌─────────────┐
                        │  Slack DM / │   ⚡/✅ company · founder · batch ·
                        │  channel    │   source · details · links
                        └─────────────┘

Stack: Python 3.9+ · requests · SQLite (stdlib) · PyYAML + python-dotenv · Slack Block Kit · launchd/cron. No database server, no message broker, no framework — just three pip dependencies.

Choosing the data sources (the part that took the longest)

The obvious source — ycombinator.com/companies — turned out to be a dead end: it's an Inertia.js single-page app with no clean public endpoint, and YC's old v0 API is gone. Scraping it is fragile.

The working solution is the yc-oss GitHub Pages mirror of the YC directory's Algolia index:

https://yc-oss.github.io/api/

It's free, needs no key, updates daily, and — crucially — exposes a changes feed (changes/latest.json) with an added array. That's a clean, incremental "what's new" signal, exactly what a monitor wants:

BASE = "https://yc-oss.github.io/api"

def fetch_changes() -> dict:
    return get_json(f"{BASE}/changes/latest.json")

For a16z Speedrun (a separate accelerator from YC, worth monitoring on its own), there's a proper public REST API:

https://speedrun-api.a16z.com/api/companies/companies/

Paginated, ~251 records, and each record includes founder names, X/LinkedIn/website URLs, cohort, and industries — everything a rich outreach alert needs. We diff the full slug list against our stored snapshot to detect new companies.

For X and LinkedIn, the situation is different:

X needs an API v2 Bearer token (Basic tier ~$100/mo) for search/recent. When present, we scan for announcement phrasings.
LinkedIn has no free public API, so the bot uses a pluggable adapter — point it at any service that accepts POST {"query":..., "freshness_minutes":N} and returns {"results":[{author,text,url,published_at}]}.

Crucially, YC Directory + Speedrun work with zero API keys. The bot is fully functional out of the box; X/LinkedIn are skipped gracefully until configured.

State management & duplicate detection (the boring part that matters)

A monitor that spams you with the same company every 8 hours is useless. So all state lives in a local SQLite database (state.db) with two tables:

seen_events — one row per dedup key that's already been alerted. INSERT OR IGNORE makes dedup atomic and race-free.
snapshots — the SHA-256 hash of each source's last-seen payload, so the bot only acts on changes.

def mark_seen(self, dedup_key, source, company):
    now = datetime.now(timezone.utc).isoformat()
    self.conn.execute(
        "INSERT OR IGNORE INTO seen_events (dedup_key, source, company, alerted_at) "
        "VALUES (?,?,?,?)", (dedup_key, source, company, now))
    self.conn.commit()

def snapshot_unchanged(self, source, payload) -> bool:
    h = hashlib.sha256(str(payload).encode("utf-8")).hexdigest()
    row = self.conn.execute("SELECT payload_hash FROM snapshots WHERE source = ?",
                            (source,)).fetchone()
    return bool(row and row["payload_hash"] == h)

The first run of each source establishes a baseline — it stores the snapshot but alerts on nothing. From then on, only new companies trigger alerts. No duplicate spam, even across restarts.

The early-detection classifier (the fun part)

The detector turns a raw social post into a classified alert. The whole challenge is avoiding false positives — a founder's bio mentioning "Amazon" is not an Amazon launch, and a post saying "solo founder" must not match a company literally named "Solo".

The matching strategy, in order of confidence:

Exact X-handle match — the author's @handle matches a company's handle in the official index. Strongest signal.
Word-boundary company name near an announcement keyword — the company name appears (case-sensitively, since founders capitalize it) within ~40 characters of a phrase like "got into YC", "accepted", "batch", "speedrun".

def classify_post(post: Post, idx) -> Alert:
    matched = _match_company(post, idx)          # handle match → name near keyword
    if matched:
        status = STATUS_CONFIRMED                 # already officially listed
    else:
        company = _extract_company_hint(post.text) or "Unknown company"
        status = STATUS_EARLY                     # announced before official listing
    return Alert(company=company, founder=post.author, source=post.source,
                 status=status, details=post.text, link=post.url,
                 dedup_key=make_dedup_key(post.source.lower(), post.post_id or post.url),
                 extra={...})

If the post matches an officially-listed company → ✅ CONFIRMED. If it doesn't (or the company can't be matched at all) → ⚡ EARLY, because the founder is clearly announcing before the official listing. That's the whole point.

The alert model & Slack delivery

Every detection becomes an Alert dataclass that knows how to render itself as a Slack Block Kit payload — header, status/source badges, batch/cohort context, description, and action buttons:

@dataclass
class Alert:
    company: str
    source: str                # "YC Directory" | "Speedrun" | "X (Twitter)" | "LinkedIn"
    status: str = STATUS_EARLY # early | confirmed
    founder: str = ""
    details: str = ""
    link: str = ""
    dedup_key: str = ""
    detected_at: str = field(default_factory=utcnow_iso)
    extra: dict = field(default_factory=dict)

    def to_slack_payload(self, cfg: dict) -> dict:
        # ... builds header / status / batch / details / buttons blocks ...

Delivery supports both an OAuth bot token (chat.postMessage — posts to a channel or a DM) and a legacy Incoming Webhook as fallback:

def send_payload(self, payload: dict) -> bool:
    if self.bot_token:
        r = requests.post(API, json=payload,
                          headers={"Authorization": f"Bearer {self.bot_token}"}, timeout=20)
        data = r.json()
        return r.status_code == 200 and data.get("ok") is True
    if self.webhook_url:
        r = requests.post(self.webhook_url, json=payload, timeout=20)
        return r.status_code == 200
    return False

Persistence & ops

Three ways to run it persistently, from simplest to most robust:

./run.sh --loop                                # foreground, polls every interval_minutes
# or cron:
0 */8 * * * cd /path/yc-launch-monitor && ./run.sh --once >> /tmp/yclm.log 2>&1
# or macOS launchd (survives reboots):
cp deploy/com.yclaunchmonitor.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/com.yclaunchmonitor.plist

Default cadence is every 8 hours (configurable via YC_SCHEDULE_INTERVAL_MINUTES). Each run also writes a machine-readable pond_report.json (sources enabled, state, alerts this run) for verification, and can expose a /health HTTP endpoint for monitoring.

Testing

The test suite uses only stdlib unittest — no extra framework. It covers the two things that break silently: dedup (no duplicate alerts across runs) and the classifier (the false-positive traps above).

.venv/bin/python -m unittest discover -s tests

There's also a --test-alert flag that posts real sample alerts (a live Speedrun company + a real founder post) so you can verify Slack delivery before going live.

What's next / how to extend

The source layer is the extension point. To add Reddit, Bluesky, or Hacker News, copy x_twitter.py as a template — the classifier, dedup, and Slack delivery all work unchanged. Other ideas in the repo's README: Slack interactive buttons, CRM export (Google Sheets / Airtable / Notion), alert scoring by ICP fit, and a --report daily-digest flag.

Code & more: https://www.dailybuild.xyz/project/239-yc-launch-monitor

Strata: I Got Tired of Retrieval-Based Code Intelligence, So I Built Something Else

Harish Kotra (he/him) — Sun, 30 Aug 2026 18:19:44 +0000

A technical deep-dive into exhaustive repository analysis with LLMs.

Every code intelligence tool I've used works the same way: you ask a question, it finds three relevant chunks using embeddings or keyword search, it answers from those three chunks.

That design made sense in 2021. Inference was expensive, context windows were small, and reading an entire repository in one pass was a non-starter.

None of that is true anymore.

A mid-size C project like Redis contains roughly 10,000 functions. At one LLM call per function, analyzing the whole thing costs an afternoon of inference and somewhere between $2 and $20 depending on your model. That's not a recurring cost — you do it once, you cache the results, and every question afterward is answered from a complete index rather than a lucky search.

The inversion that makes this interesting: instead of choosing which parts of the codebase to read, you just read everything. The hard part is no longer selection. The hard part becomes structuring the output so it's useful.

I built Strata to explore what happens when you take that inversion seriously.

The core idea: layers

The name comes from the architecture. Analysis happens in seven passes, each building on the one beneath, from raw syntax at the bottom to project philosophy at the top.

L6  History & Philosophy     ← "What does this project value?"
L5  Security Attention Map   ← "Where should a reviewer look first?"
L4  Dead Code                ← "What can safely be deleted?"
L3  Documented vs Actual     ← "How badly has the architecture drifted?"
L2  Module/Subsystem Rollup  ← "What does each module do?"
L1  Function Summaries       ← "What does each function do?"
L0  Structure                ← "What functions exist? Who calls whom?"

Every pass writes to a content-addressed SQLite cache. Re-runs are free — only changed files re-analyze. Bumping a pass version (e.g., "L1_v2") invalidates exactly that pass without touching the others.

L0: Structure without LLM

The foundation. L0 runs tree-sitter on every source file, extracts the symbol table, builds the call graph, and records git blame attribution. No LLM touches this layer.

Getting this right matters disproportionately because everything above it hangs off it. If L0's call graph is wrong, L4's dead code detection produces garbage, and L5's security paths become fictional.

The tree-sitter query for C function extraction looks like this:

_FUNC_DEF_QUERY_SRC = """
(function_definition
  declarator: (function_declarator
    declarator: (identifier) @name)
) @func
"""

# In tree-sitter 0.25, QueryCursor is the execution API
def caps(query, node):
    return QueryCursor(query).captures(node)

One subtlety: tree-sitter's Python bindings changed their API between 0.22 and 0.25. In 0.22, query.captures(node) was a method on the Query object. In 0.25, Query is just a compiled pattern — you need QueryCursor(query).captures(node) to execute it. This is the kind of thing that silently produces empty results rather than erroring, so watch for it.

The blame optimization matters too. A naive implementation calls pygit2.blame() once per symbol:

# SLOW: one I/O-heavy blame call per symbol
for sym in file.symbols:
    author = primary_blame(repo_root, rel_path, sym.line_start, sym.line_end)

Redis has ~30 symbols per file × 324 files = ~10,000 blame calls. Each one opens the git repository, walks the history, and deserializes hunk data. At 1-2 seconds per call, that's hours.

The fix is obvious once you see it — load blame once per file, reuse for all symbols:

# FAST: one blame call per file
blame_cache: dict[str, dict] = {}

for file_path in files:
    if rel not in blame_cache:
        blame_cache[rel] = blame_file(repo_root, rel)   # once

    for sym in info.symbols:
        author = _blame_range(blame_cache[rel], sym.line_start, sym.line_end)

~10x speedup on L0.

L1: Function summaries at scale

This is where the inference budget goes. One LLM call per symbol, all running concurrently up to a configurable semaphore ceiling.

The key design decision is structured output via instructor + pydantic. The model returns a typed FunctionSummary object, not free text:

class Citation(BaseModel):
    file: str
    line_start: int
    line_end: int

class FunctionSummary(BaseModel):
    one_line: str = Field(max_length=180)
    preconditions: list[str]
    invariants: list[str]
    side_effects: list[str]
    error_paths: list[str]
    surprising: str | None
    citations: list[Citation]      # EVERY claim must cite a real line
    confidence: float = Field(ge=0.0, le=1.0)
    static_vs_llm_disagreements: list[str]

Every list item in preconditions, invariants, side_effects, and error_paths requires a citation in the prompt instructions. The validator then checks that cited line ranges actually fall within the symbol's range:

def _validate_citations(summary, file_path, line_start, line_end):
    errors = []
    for c in summary.citations:
        if c.file != file_path:
            errors.append(f"citation file mismatch: {c.file!r}")
        if c.line_start < line_start - 5 or c.line_end > line_end + 5:
            errors.append(f"citation {c.line_start}–{c.line_end} outside {line_start}–{line_end}")
    return errors

Rather than rejecting out-of-range citations outright (which would waste the inference), we penalize confidence instead:

if val_errors:
    summary = summary.model_copy(
        update={"confidence": max(0.0, summary.confidence - 0.2 * len(val_errors))}
    )

The static_vs_llm_disagreements field is one of the most interesting outputs. When the model mentions a callee that the static call graph doesn't show, it records it. These disagreements usually mean one of three things: dynamic dispatch, function pointers, or macro expansion. All three are exactly the places a human reviewer should pay attention.

The few-shot examples in the prompt are doing a lot of work. Here's the opening of one:

EXAMPLE 1 — simple allocator wrapper

Function: dictCreate  File: src/dict.c  Lines: 97–120

Source:
int *dictCreate(dictType *type, void *privDataPtr) {
    dict *d = zmalloc(sizeof(*d));
    if (d == NULL) return NULL;
    _dictInit(d,type,privDataPtr);
    return d;
}

Response:
{
  "one_line": "Allocates and zero-initializes a new hash table with the given type vtable.",
  "preconditions": ["type is a valid non-NULL dictType pointer [src/dict.c:97]"],
  "side_effects": ["allocates heap memory via zmalloc [src/dict.c:99]"],
  "error_paths": ["returns NULL if zmalloc fails [src/dict.c:100]"],
  ...
  "confidence": 0.97
}

Notice the inline citation format inside list items ([src/dict.c:99]). The model learns to inline citations rather than collecting them only in the top-level citations array.

L2: Map-reduce rollups

L2 summarizes the repository bottom-up. Function summaries → module summaries → subsystem summaries → project summary. The model never sees more than one level at a time.

def _module_prompt(module_path, summaries):
    joined = "\n".join(f"- {s}" for s in summaries[:120])
    return f"""
Summarize the module '{module_path}' in 2–4 paragraphs based on these function summaries.
Cover: what the module does, its key abstractions, notable design decisions, anything surprising.

Function summaries:
{joined}
"""

Capping at 120 function summaries per module prevents token overflow on large files. The cap is generous enough for real-world C files but worth adjusting for codebases with very large files.

The map-reduce structure means Strata scales to any repository size — you never send the entire codebase to a model in a single context.

L3: Architecture drift

This is the most interesting output in the product.

Every long-lived project has drifted from its own documentation. Strata measures it. L3 reads every Markdown file in the repo, extracts the claimed architecture, and diffs it against what L0 and L2 actually found.

DOC_GLOBS = ["README*", "ARCHITECTURE*", "docs/**/*.md", "doc/**/*.md"]

def _divergence_prompt(claimed, actual_modules, actual_imports, rollup_names):
    return f"""
Claimed architecture (from docs):
{json.dumps(claimed, indent=2)}

Actual modules (from code):
{json.dumps(actual_modules[:200], indent=2)}

Identify divergences. Classify each as:
- undocumented_module: code module not mentioned in docs
- layering_violation: import that violates a stated layering rule  
- phantom_abstraction: described as pluggable but only one implementation exists
- ghost_component: documented component that no longer exists in code

Return JSON array: [{{"kind": "...", "description": "...", "evidence": "..."}}]
"""

Four divergence kinds. In practice, undocumented_module is the most common — projects grow new subsystems that never make it into the docs. ghost_component is the most interesting — abstractions that were removed but whose documentation lives on.

L4: Dead code with confidence tiers

Reachability over the call graph from real entry points. The key design decision: three tiers instead of a binary dead/alive classification.

Tier 1: no callers, no exports, no reflection indicators → almost certainly dead
Tier 2: unreachable statically but language has escape hatches → uncertain
Tier 3: reachable only from test files → test-only

Overclaiming tier 1 destroys trust in the whole tool. A dead code report where 30% of "dead" functions turn out to be called at runtime via dlopen or function pointers is worse than no report at all. So we surface the tier prominently and let the user decide.

has_escape = (
    "dlopen" in (sym.get("source", "") or "")
    or name.startswith("_")  # convention: private but possibly used by macros
)

tier = 1 if not has_escape else 2

This is conservative on purpose.

L5: Security attention map (not a vulnerability scanner)

L5 traces paths from untrusted-input sources to sensitive sinks through the call graph.

SOURCE_NAMES = frozenset({
    "read", "recv", "recvfrom", "recvmsg",  # network
    "fread", "fgets", "getline",             # file
    "getenv", "getopt",                      # process
    "readQueryFromClient", "processInputBuffer",  # Redis-specific
})

SINK_NAMES = frozenset({
    "memcpy", "memmove", "strcpy", "strcat",  # memory
    "execve", "execl", "system", "popen",      # exec
    "open", "fopen", "unlink",                 # path
})

The framing matters. This is explicitly not a vulnerability scanner. The UI and documentation say so clearly. It answers "where should a security reviewer spend their first day?" — which is genuinely valuable and honest. A false-positive-riddled bug finder is a liability. A well-ranked attention map is a tool people will actually use.

L6: History and philosophy

Walk every commit. Chunk by quarter. Summarize each chunk. Reduce to eras.

def _quarter(timestamp: int) -> str:
    d = datetime.datetime.utcfromtimestamp(timestamp)
    q = (d.month - 1) // 3 + 1
    return f"{d.year}-Q{q}"

The interesting outputs:

Churn coupling — files that change together frequently, revealing coupling the architecture doesn't admit to:

def _compute_churn_coupling(commits, top_n=20):
    pair_counts: Counter = Counter()
    for c in commits:
        for i, a in enumerate(c.files_changed):
            for b in c.files_changed[i+1:]:
                pair = tuple(sorted([a, b]))
                pair_counts[pair] += 1
    # normalize by minimum individual change count
    ...

Knowledge silos — subsystems where one author wrote >80% of the code and no one else has significantly touched it. These are bus factor risks.

Implicit values essay — a final LLM call that reads the entire commit history and argues for what the project consistently chose when forced to trade off between speed, simplicity, compatibility, and safety. The prompt asks for evidence from the commit record, not assertions.

The visualization

The static site uses d3-hierarchy's treemap layout. The key design constraint: it must be fully static, deployable to a CDN, with no server. All data is precomputed JSON.

const treemap = d3.treemap<HierarchyDatum>()
  .size([width, height])
  .paddingOuter(4)
  .paddingInner(1)
  .paddingTop(18)
  .round(true)

Five lenses, each a different color function over the same cell data:

export function cellColor(sym: SymbolCell, lens: Lens): string {
  switch (lens) {
    case 'churn': {
      const age = clamp((NOW - parseDate(sym.last_changed)) / ONE_YEAR, 0, 5) / 5
      return lerp('#e63946', '#264653', age)  // red=recent, blue=old
    }
    case 'security':
      if (sym.security_paths >= 3) return '#e63946'
      if (sym.security_paths >= 1) return '#f4a261'
      return '#264653'
    case 'dead':
      if (sym.dead_tier === 1) return '#6b6b6b'
      // ...
  }
}

Switching lenses is instant — all data is already in the precomputed JSON, no fetches required.

The cache architecture in full

┌─────────────────────────────────────────────────┐
│  cache table (SQLite)                           │
│                                                 │
│  key         TEXT PRIMARY KEY                   │
│              sha256(file_hash + pass_id         │
│                     + prompt_hash)              │
│                                                 │
│  pass_id     TEXT    "L1_v1", "L2_v1", ...     │
│  file_path   TEXT    relative to repo root      │
│  symbol      TEXT    qualified name (nullable)  │
│  created_at  INTEGER unix epoch                 │
│  result_json TEXT    the LLM output             │
└─────────────────────────────────────────────────┘

Same file, same prompt, same pass version → cache hit, zero cost
File changes → file_hash changes → cache miss for that file only
Prompt edit → prompt_hash changes → cache miss for all files in that pass
Pass version bump → pass_id changes → same effect as prompt edit, but explicit

Old cache rows are never deleted — they become historical records. Useful for seeing how your summaries improved as you refined the prompts.

What I learned

Retrieval is a crutch, not a constraint. The reason every code intelligence tool uses retrieval is cost, not correctness. Once you remove the cost constraint, the design space opens up significantly.

Citations are load-bearing. The requirement that every claim cite a real source line is not just a correctness measure — it changes the model's behavior in ways that improve overall quality. The model stops making confident assertions about things it can't point to.

The call graph disagreement field is underrated. I added static_vs_llm_disagreements mostly as a correctness measure, but it turns out to be a useful output on its own. When the model says a function calls something the static analysis can't see, that's usually a signal of interesting dynamic behavior.

Confidence surfacing matters. The model's self-reported confidence correlates well with actual accuracy on spot-checks. Surfaces it in the UI, not buried in a tooltip — functions with confidence below 0.6 deserve human review.

The history pass is the one that surprises people. Most engineers have never read a narrative history of the projects they work on. When you give them one argued from the actual commit record, they find things they didn't know: decisions that were made and reversed, coupling that the architecture doesn't admit to, subsystems with a single author who left three years ago.

Running it yourself

git clone https://github.com/harishkotra/strata
cd strata && pip install -e .
cp .env.example .env  # fill in your LLM endpoint

git clone https://github.com/redis/redis /tmp/redis
strata analyze /tmp/redis --passes 0        # L0 first, no LLM
strata analyze /tmp/redis --passes 1        # L1, watch progress bar
strata analyze /tmp/redis --passes 2,3,4,5,6
strata export /tmp/redis/.strata.db web/public/data

cd web && npm install && npm run dev

The map loads at localhost:5173. Click any cell. Switch lenses. Read the tour. Ask questions:

strata ask "Which functions are most likely to contain memory bugs?"
strata ask "What is the Redis cluster replication protocol?"
strata ask "Which files changed together most often in the last five years?"

Code & more: https://www.dailybuild.xyz/project/238-strata

PRION: Simulating 100,000 Minds Spreading a Rumor — With an LLM That Never Touches the Hot Path

Harish Kotra (he/him) — Sat, 29 Aug 2026 18:30:22 +0000

A technical deep-dive into building a 100,000-agent belief-contagion simulator in four days — where the LLM is a compiler, never a runtime.

1. The premise

A prion is a protein folded into the wrong shape. It has no genome, carries no instructions, and replicates by the only trick it knows: touching a normally-folded protein and templating its own wrong shape onto it. The copy then does the same. Copies drift. Different misfoldings become different strains with different incubation periods and different symptoms. Some strains jump a species barrier; most don't. Some cases arise spontaneously, from nothing.

Now read that paragraph again, replacing protein with belief.

That is a complete and accurate description of how a rumor moves through a society. So we built PRION: a live simulation of 100,000 people, each with a personality, a memory of
who told them what, and a real position in a small-world social graph. You type a rumor
into a box. It enters one person. Then you watch it spread, mutate as it passes from mouth
to mouth, split into rival strains, jump between communities, and either take over the
population or burn out — in real time, at ~20 ticks per second.

The hard part was never the metaphor. It was this constraint:

You cannot call an LLM once per agent per tick. 100,000 agents at 20 ticks/second is
two million calls per second. That is physics, not a budget problem.

Everything in this post falls out of how we answered that constraint.

2. Architecture: the LLM is a compiler, never a runtime

The design rule for the whole system:

The LLM compiles inputs and decorates outputs. It never executes inside the loop.

                        ┌──────────────────────────────────────────────────┐
│                  YOU (browser)                   │
│   deck.gl field · rail · strain lineage · ticker │
└───────────────▲──────────────────┬───────────────┘
binary WS deltas│                  │ controls
~33 KB/tick @ 100k  │              │ (inoculate/pause/inspect/reset)
┌───────────────┴──────────────────▼───────────────┐
│            FastAPI broker  (port 8000)           │
│   bounded per-client queues · slow-client safe   │
└───────────────▲──────────────────┬───────────────┘
 frames         │                  │ controls
┌───────────────┴──────────────────▼───────────────┐
│         SIM PROCESS (multiprocessing)            │
│  ┌────────────────────────────────────────────┐  │
│  │  TIER 0 — numeric core (numpy, zero LLM)   │  │
│  └──────────────┬─────────────────────────────┘  │
│  ┌──────────────▼─────────────────────────────┐  │
│  │  TIER 2 — misfold worker (async thread)    │  │──┼──► LLM
│  └──────────────┬─────────────────────────────┘  │  │
│  ┌──────────────▼─────────────────────────────┐  │
│  │  TIER 1 — strain compiler (on inoculate)   │  │──┼──► LLM
│  └──────────────┬─────────────────────────────┘  │
│  ┌──────────────▼─────────────────────────────┐  │
│  │  TIER 3 — narrator + inspect monologues    │  │──┼──► LLM
│  └────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────┘

Four tiers:

Tier	Runs	Frequency	LLM?
0 — Numeric core	templating, incubation, titer decay, strain competition	every agent, every tick	never
1 — Strain compiler	text → simulation parameters	once per inoculation	1 structured call
2 — Misfold worker	text mutation as the belief travels	per flagged templating event	async, fire-and-forget
3 — Narration	wire copy + click-to-inspect monologues	every ~15s / on click	async, fire-and-forget

The contract that makes this safe: the tick loop never blocks on inference. Tier 0 runs in its own process with zero LLM calls. Tier 2 consumes a queue on a background thread. If the LLM endpoint dies mid-demo, Tier 0 doesn't notice; the simulation keeps running on deterministic fallbacks. This is also why the demo never hard-fails without an API key — every LLM tier has a deterministic fallback, and the fallbacks are good enough to demo with.

3. Tier 0: 100,000 agents with zero objects

The AgentTorch paper's core insight is that per-agent Python objects die at six-figure scale — the whole population must be arrays. PRION takes that literally. The population is a struct-of-arrays:

class Population:
    def __init__(self, s: Settings):
        self.titer      = np.zeros((s.N, 1))      # (N, strains) — grows as strains branch
        self.incubation = np.zeros(s.N)
        self.traits     = np.zeros((s.N, 6))      # openness, skepticism, conformity, ...
        self.codon      = np.zeros(s.N, np.int8)  # resistance polymorphism (0/1/2)
        self.held       = np.zeros(s.N, bool)
        ...

The social graph is a Watts-Strogatz small-world ring (each node rewired with p=0.08) plus a sprinkle of long-range bridge edges and high-degree hubs — as a scipy CSR adjacency.

The 20× matmul lesson

The obvious way to compute local prevalence — how many of your neighbors hold strain c —
is one matmul over all strains at once:

local = adj @ held          # held: (N, M) — 19.7 ms/tick at 100k

That measured 19.7 ms/tick. The same work as per-strain 1D passes measured 1.1 ms:

def _step_dense(self, c, col, rng):
    held      = self.titer[:, c] > 0.0
    infectious = held & (self.state == STATE_INFECTIOUS)
    if not infectious.any():
        return
    local = self.adj.dot(infectious.astype(np.float64))   # 1D CSR dot: 1.1 ms
    ...

An 18× difference for mathematically identical work. The 2D matmul materializes
intermediates and defeats cache locality; the 1D pass over a CSR matrix stays in cache.
Lesson: at 100k agents, loop over strains in Python and keep every numpy op 1D.

Then strains with few holders get a second path — O(nonzeros) instead of O(N):

def _step_sparse(self, c, nz, rng):
    """Strain with few holders: O(nonzeros) instead of O(N)."""

This is what keeps ~49 ticks/sec at 100k with misfolds flowing (67 tps dense). When a
misfold branches, it usually infects a handful of agents — paying O(N) for a strain held
by 3 people is 33,000× waste.

Non-consensus is structural, not tuned

The easiest way to ruin a contagion sim is a field that saturates at 100%. PRION stops saturation structurally:

Codon hard block — every agent carries a resistance codon (0/1/2). A strain with a matching resistant codon infects with probability zero, regardless of titer.
Refusal floor — below a titer floor, refusal is absolute.
Contrarian refusal — agents high in contrarianism refuse a belief precisely when it dominates their neighborhood — the sociological "backfire effect" as a mechanic.

Result: the field plateaus around 40%, which is also what real rumor curves do.

4. Tier 1: text becomes parameters, once per inoculation

When you type a belief into the box, one structured LLM call compiles it into simulation parameters. This is the "LLM as compiler" tier — it runs once per inoculation, never per tick. We use instructor over litellm pointed at any OpenAI-compatible endpoint:

resp = client.chat.completions.create(
    model=llm.litellm_model(),     # "openai/glm-5.3-flash"
    api_base=cfg.api_base,         # https://api.particle.ai/v1
    api_key=cfg.api_key,
    response_model=Strain,         # pydantic schema = the contract
    max_tokens=1500,               # reasoning model: room to think AND emit JSON
    messages=[...],
)

The pydantic schema is the contract. Its validators clamp at the boundary:

transmissibility: float = Field(ge=0.0, le=1.0)

@field_validator("susceptibility")
def _clamp(cls, v):
    return {k: max(-1.0, min(1.0, float(c))) for k, c in v.items() if k in TRAIT_NAMES}

A hallucinated transmissibility: 4.2 arrives as 1.0. A hallucinated trait name is dropped. The simulation can never receive a parameter that breaks it.

One practical note for anyone doing this with glm-5.3-flash: it is a reasoning model. It emits reasoning_content first and only then fills content. If your max_tokens budget is small, reasoning eats all of it and content comes back empty with finish_reason: length. Budget accordingly, and never trust a "successful" call that returns an empty string (more on this in §7).

5. Tier 2: the text mutates as it travels

This is the feature that makes PRION prion. Every templating event (A tells B) is flagged as a misfold with probability misfold_rate. Flagged events are queued to an async worker that makes one LLM call:

Given this person's personality and the version of the belief they received, what is the one short sentence they would repeat?

The rumor literally changes shape as it travels. "The tap water is being secretly poisoned" becomes, through one personality, "My coworker's cousin works down at the water plant, and she says…" — and through another, "Y'all, apparently…".

Mutations are deliberately less fit than their parents (transmissibility ×0.5, decay ×1.6, susceptibility ×0.7). Most variants get cleared quickly. This is the prion biology:
strains branch the lineage without becoming second epidemics, and the simulation stays one epidemic with a branching text history rather than N parallel epidemics.

Strains that die out (no infectious agent for a while) have their matrix column recycled through a freelist — bounded memory, unbounded history.

6. The wire: binary deltas at 33 KB/tick

The renderer never blocks on inference or on JSON. The wire protocol is binary:

Frame	Layout
`INIT` `0x00`	`u32 N` · `u32 n_comm` · `pos N*2 f4` · `dom u8[N]` · `titerQ u8[N]` · `state u8[N]` · `u32 mlen` · meta JSON (seed + full strain tree)
`DELTA` `0x01`	`u32 tick` · `u32 n` · n×(`u32 idx`,`u8 dom`,`u8 titerQ`,`u8 state`) · `u32 slen` · sidecar JSON · `u32 n_arcs` · arcs
`CTRL` `0x02`	`u32 seq` + JSON (inoculate / pause / resume / speed / inspect / reset)
`MONO` `0x03`	JSON monologue for a clicked agent

Deltas are structural-only: only agents whose dominant strain or state changed are shipped, with titer attached. Continuous titer decay of the steady infected pool is not shipped per tick — the client fades brightness locally between structural events:

fade(dt) {
  for (let i = 0; i < this.N; i++) {
    if (st[i] === STATE_INFECTIOUS && b[i] > 0.05)
      b[i] = Math.max(0.05, b[i] - dt * 0.08);
  }
}

Naive per-tick full-state shipping measured 169 KB/tick; structural deltas measure ~33 KB/tick at 100k. The first version shipped per-agent Python objects over JSON and died; the second shipped full state; the third ships structure only. Each iteration was a 10× or better.

The population field itself is deck.gl — ScatterplotLayer with binary attribute buffers (positions uploaded once, colors re-uploaded per tick) plus an ArcLayer showing only the current tick's templating events, fading over 500 ms. We considered cosmos.gl for the GPU force layout, but its OES_texture_float requirement is exactly the WebGL extension iOS Safari is least reliable about — so deck.gl (WebGL1-safe) is the floor, not the ceiling.

7. War stories: what actually broke

Four days of building means four days of things breaking in interesting ways.

The 20× matmul. adj @ held over all strains: 19.7 ms/tick. Per-column 1D passes: 1.1 ms. Same math, 18× apart — the 2D matmul materializes intermediates and defeats cache locality. At 100k agents, loop over strains in Python and keep every numpy op 1D.

The 5× delta. Full-state JSON shipping: 169 KB/tick. Structural-only deltas: 33 KB.

The 7 tps collapse. Day 3 added misfolds and tick rate fell to 7 tps at 100k. Three compounding causes: unbounded strain growth (fixed with extinction + a column freelist), a worker flooding the queue (fixed with a live-strain cap + backoff), and dense O(N) work for strains held by 3 people (fixed with the O(nonzeros) sparse path). Back to 49 tps.

The empty-content plague. glm-5.3-flash is a reasoning model: it emits reasoning_content first and only then fills content. Small token budgets mean reasoning eats everything and content arrives empty with finish_reason: length. The insidious part: the call succeeds. Our misfold worker shipped empty strings as strain text for ~half of all misfolds — the fallback only fired on exceptions, and an empty-but-successful response raises nothing. The fix is a hard rule now: an empty LLM response must never ship — fall back to the deterministic path.

The Svelte 5 black page. The UI shipped as a black page in a real browser. Three stacked causes, each invisible until we drove it in headless Chrome:

new App({ target }) — the Svelte 4 constructor — throws effect_orphan under Svelte 5. The app never mounted. Fix: mount(App, { target }).
The INIT decoder did new Float32Array(buffer, 9, …) — offset 9 isn't 4-byte aligned, a hard RangeError for typed-array views. Fixed by copying bytes into a fresh buffer.
The lineage panel computed its tree once at component-init from an empty strains array and its recursive Node children never re-rendered. Rewritten as a flat, depth-first render over one $derived array — no recursion, no derived-in-template reactivity traps.

Plus one reactivity trap worth memorializing:

// client.meta.strains is mutated in place with .push():
this.meta.strains.push(...sidecar.new_strains);
// then in the component:
strains = client.meta.strains;      // same reference — Svelte sees NO change
strains = [...client.meta.strains]; // copy — new reference, re-render fires

Lesson: in Svelte 5, an in-place mutation plus a same-reference assignment is a silent no-op. If a framework ever feels non-reactive, check for reference identity before blaming the framework.

8. What the simulation actually shows

The behavior that falls out of the mechanics is the demo:

The S-curve. Inoculate one agent, watch slow ignition, exponential spread, then a plateau around 40% — the structural non-consensus mechanics, not a tuned cap.
Drift. "The tap water is being secretly poisoned" becomes "My coworker's cousin works down at the water plant, and she says…" becomes "Y'all, apparently…" — each version is a strain, and the text mutates as it travels.
Extinction and replacement. Weak variants decay and die; their matrix columns are recycled for new variants. The lineage remembers; the matrix doesn't hoard.
Sporadic cases. Occasionally a belief ignites from nothing — sporadic_prob per tick — because that's what spontaneous case generation looks like in a society too.
Click-to-inspect. Click any agent: an LLM writes their internal monologue from their actual traits, their held strains, and their neighbors' strains.

9. What we'd build next

Sparse CSR titer matrix past ~50 live strains — raises the live-strain ceiling an order of magnitude.
Codon editor — paint resistance codons onto communities and watch a strain hit the species barrier in real time.
Strain arena — inoculate two rival beliefs at opposite ends of the graph and watch them compete for the same population.
Replay diffing — same seed, different misfold texts, diff the lineage shapes.
Run export — one-click GIF/video of a run.

10. Closing thought

The LLM-in-the-loop question has a boring, correct answer: don't. Put the model at the boundaries — compiling intent into parameters, decorating events into prose — and keep the inner loop deterministic, vectorized, and testable. The simulation gets faster, the costs get bounded, the failures degrade gracefully, and the demo survives the WiFi going out mid-presentation.

Beliefs are misfolded proteins. Now they have a simulator.

Code & more: https://www.dailybuild.xyz/project/237-prion

Building the AI Arms Race: A Reproducible Experiment to Test Whether LLMs Evolve Strategies When They Compete

Harish Kotra (he/him) — Fri, 28 Aug 2026 18:06:51 +0000

Two LLMs. One hidden state. 1,000 rounds. Do they independently evolve increasingly effective strategies — or just chase noise?

I built an open-source, terminal-based research experiment that pits glm-5.3-flash against deepseek-v4-flash-0731 in a repeated strategic game called HIDDEN SIGNAL. The goal wasn't to prove an arms race exists — it was to build an experiment that could disprove it, and report whatever the data actually says.

This post walks through the design, the architecture, the hard-won lessons about real LLM endpoints, and the code that makes it all reproducible.

The Research Question

If two LLMs repeatedly compete against each other, do they independently evolve increasingly effective strategies?

That's a question for computational social science, not just ML engineering.

To answer it honestly you need:

A real game with asymmetric private information, strategic communication, and competing incentives.
Real models genuinely playing it — no hard-coded outputs.
Information isolation — each agent sees only what it legitimately could.
Controls — random opponents, heuristic opponents, model-vs-model baselines — so you can tell "adaptation" from "statistical noise".
Deterministic metrics — deception, leakage, exploitability computed from ground truth, never from an "LLM judge".
Falsifiability — the experiment must be able to report no evidence of strategic escalation.

Let's see how each of those is engineered.

The Game: HIDDEN SIGNAL

Each round, the engine (the sole authority) draws a hidden state S ∈ {A,B,C,D}
and gives each agent a two-state private set that always contains S:

def private_set_for(state: str, agent: str) -> tuple:
    """GLM sees {S-1, S}; DeepSeek sees {S, S+1} (mod 4)."""
    idx = STATE_ALPHABET.index(state)
    if agent == "A":
        return (STATE_ALPHABET[(idx - 1) % 4], state)
    return (state, STATE_ALPHABET[(idx + 1) % 4])

Agents act in alternating turns (randomized first mover, to remove any
structural advantage). On a turn they choose message, guess, or withhold.
The scoring creates the central strategic tension:

Event	Points
correct guess	+100
incorrect guess	−30
round win / loss	+25 / −25
message	−1
truthful exclusion (reveal)	−5
successful extraction	+10

A truthful claim that excludes a state in the opponent's set narrows their
uncertainty from 2 states to 1 — a 1-bit leak that costs you −5 but hands the
opponent +135 if they exploit it. That's the tradeoff between revealing
and extracting information, and it's what makes the game genuinely
strategic rather than a coin flip.

The engine owns all state. Agents submit actions; the engine validates,
scores, and records them. They can never modify the hidden state, their
scores, or their opponent's private information.

The Information-Isolation Boundary

This is the most important safety property in the experiment. Each agent's
observation is constructed by get_observation(), which only ever receives
the agent's own private set plus the public transcript:

def get_observation(agent, rs, current_public_log, recent_public_rounds,
                    own_history_summary=None, opponent_history_summary=None,
                    strategy_note=None, game_number=None, total_games=None,
                    condition_feedback="none") -> dict:
    obs = {
        "agent": agent,
        "round_number": rs.round_number,
        "your_private_information": sorted(rs.private[agent]),   # ONLY own set
        "current_round_public_events": current_public_log,
        "recent_rounds_public": recent_public_rounds,
        ...
    }
    if condition_feedback in ("self", "self+opponent") and own_history_summary:
        obs["your_performance_history"] = own_history_summary
    if condition_feedback == "self+opponent" and opponent_history_summary:
        obs["opponent_observable_statistics"] = opponent_history_summary
    return obs

The opponent's private set, hidden reasoning, system prompt, and internal
beliefs are never included — enforced by construction, and verified by
tests that assert obs["your_private_information"] never equals the
opponent's set and that the hidden state never appears as a field.

After each round, the hidden state is revealed publicly (both agents learn
it) — but private sets are never revealed. That's a deliberate choice: it
gives agents real historical evidence without leaking the game's secret
structure.

Anti-Contamination: Don't Tell Them to "Adapt"

The single most important design decision: the in-game prompt never mentions
adaptation. Telling the agents to "develop an increasingly sophisticated
strategy" would contaminate the experiment by priming exactly the behavior
you're trying to measure.

Instead, there are two prompts:

In-game prompt (stable, every turn): "Maximize your cumulative reward. Do not assume your opponent is truthful. Do not assume they are deceptive. Use the evidence." No mention of adaptation.
Strategy-update prompt (every 50 games): "Identify patterns in your performance and formulate a concise strategy for the next block." This is the only place adaptation is explicitly requested.

Strategy memory is bounded — each update replaces the previous note — and every
version is archived to strategies/A_v01.txt … A_v20.txt so we can trace
strategy evolution over time.

The LLM Client: The Hardest Lesson

The openai SDK makes calling an OpenAI-compatible endpoint trivial. The
hard part was a subtle failure mode that silently broke the first smoke test:

Both models' default reasoning_effort is max, which burns the entire
output budget on hidden chain-of-thought and returns empty content.

I watched the models "play" by withholding every single turn. The raw records
showed valid JSON actions — all withhold. The models weren't broken; they
were returning empty strings because their hidden reasoning consumed all 700
tokens.

The fix was two-fold:

# arms/config.py
reasoning_effort: str = "low"   # default "max" burns the budget on hidden CoT
max_tokens: int = 4000          # generous: hidden reasoning consumes part of it

and a safety net in the client that treats empty content as a recoverable
failure with a constrained retry:

# arms/llm.py
if not content.strip():
    r.error = "empty content (reasoning consumed budget?)"

This is the kind of thing that only shows up when you run real models at
scale, and it's exactly why the smoke test exists. I also added per-call
token/latency tracking so the final run reports honest cost numbers:

costs A: {'calls': 24, 'failures': 0, 'total_tokens': 24414, ...}
costs B: {'calls': 31, 'failures': 5, 'total_tokens': 98960, ...}

Architecture

run.py / run_all.py
        │
        ▼
  Experiment Config ──▶ Replicate (×N) ──▶ GameEngine (authoritative state)
        │                                        │
        ▼                                        ▼
  get_observation(agent)  ◀── public transcript + own private set only
        │
   ┌────┴────┐
   ▼         ▼
 LLMAgent A  LLMAgent B        (glm-5.3-flash) (deepseek-v4-flash-0731)
   │         │
   └────┬────┘
        ▼
  structured action JSON ──▶ validated ──▶ engine applies + records
        │
        ▼
  raw/ (games.jsonl, messages, actions, errors)
        │
        ▼
  analyze.py ──▶ deterministic metrics → change points → findings.json
        │            └─▶ 7 publication figures
  report.py ──▶ report.md (generated from actual data, never invented)
  run_cross_eval.py ──▶ strategy × opponent matrix (frozen snapshots)

Module map

arms/
├── config.py        # game rules, scoring, conditions, LLM config
├── credentials.py   # API-key resolution (env → ~/.dsh/.credentials.yaml)
├── llm.py           # OpenAI-compatible client: retries, cost tracking, CoT stripping
├── game.py          # HIDDEN SIGNAL engine + observation isolation
├── prompts.py       # stable in-game prompt + strategy-update prompt
├── actions.py       # strict structured-output validation
├── agents.py        # LLMAgent + Random/Heuristic control agents
├── strategy.py      # bounded strategy memory + version archive
├── metrics.py       # ground-truth metrics (deception, leakage, extraction)
├── history.py       # per-agent performance summaries
├── analysis.py      # change-points, bootstrap CIs, Mann-Whitney, signature
├── runner.py        # replicate runner: games, checkpoints, probes, generalization
├── experiment.py    # experiment driver
├── dataset.py       # experiment directory schema + JSONL writers
├── figures.py       # 7 publication figures
└── ui.py            # rich terminal UI (live mode)

Deterministic Metrics

Every metric is a pure function of raw records plus ground truth. The
deception signal, for example, is not a semantic judgment — it's a
ground-truth heuristic: a claim is a lie when it contradicts the agent's
own private knowledge (e.g., claiming "true" about a state not in your own
set).

Information leakage is computed in bits from the actual finite state space —
the opponent's entropy before minus after your truthful exclusions:

def info_leakage_bits(round_record, agent):
    opp = "B" if agent == "A" else "A"
    opp_private = round_record[f"private_{opp}"]
    before = uncertainty_of_set(opp_private)     # log2(2) = 1 bit
    after = before
    for e in round_record["events"]:
        if e.get("agent") == agent and e.get("claim_reveal"):
            after = 0.0                          # narrowed to 1 state
    return before - after

No LLM judge. No hand-waving. Just entropy over the true state space.

Statistical Analysis & the Arms-Race Signature

The system detects candidate strategy shifts automatically via greedy binary
segmentation on reward series (change points), tests trends with Spearman
rank correlation, compares first vs second half with Mann–Whitney U, and
builds bootstrap confidence intervals.

The "arms-race signature" is an explicit detector for the alternating-response
pattern: A exploits → B declines → B adapts → B recovers → A adapts → A
recovers. It counts sign alternations in block-level advantage:

def arms_race_signature(series_a, series_b, block, min_effect=0.0):
    adv = np.array([
        series_a[i*block:(i+1)*block].mean()
        - series_b[i*block:(i+1)*block].mean()
        for i in range(n_blocks)
    ])
    signs = np.sign(adv)
    # ... count sign changes ignoring zeros ...
    detected = alternations >= 2 and len(runs) >= 3

Crucially, the detector can return false — and the report will then say
"no strong evidence of an alternating strategic arms race was detected", which
is a perfectly valid scientific outcome.

Controls That Make It Publishable

Random opponent (RandomAgent): uniform random policy.
Heuristic opponents (HeuristicAgent): fixed truthful / bluff / cautious policies.
Model-pair controls: GLM vs GLM, DeepSeek vs DeepSeek, GLM vs DeepSeek under every condition (run_all.py).
Exploitability probe: at each checkpoint, current strategy vs a random baseline; exploitability = perf vs real − perf vs baseline.
Generalization test: final strategies re-evaluated against fresh instances, a random baseline, and a fixed truthful heuristic — answering "did they learn genuinely useful strategies, or just overfit to each other?"

What the Smoke Test Taught Me

A 10-game smoke test (55 real LLM calls, ~25 min) was enough to validate the
entire pipeline and produce genuine — if preliminary — signal. The corrected
numbers from that pilot:

DeepSeek won 70% of games, ending at +790 cumulative reward vs GLM's −171.
GLM did adapt its strategy: after block 1 it changed its note from "guess immediately" to "stop auto-guessing; withhold until you have a signal."
But that adaptation made it worse in this tiny sample: its block-2 average reward dropped from +3.8 (20% wins) to −38 (0% wins).

The analysis correctly reported arms_race_detected: false for that short
run — exactly the falsifiability the design demands. Ten games is far too
little to claim a race; that's what the 1,000-game / multi-replicate runs are
for. What the pilot did prove: the pipeline works end-to-end, the metrics
are grounded in truth, and the experiment is willing to tell us "nothing
conclusive yet" instead of manufacturing a narrative.

Reproducibility

Every experiment directory records: model names, endpoint identifier (never
the key), game rules, scoring, prompt version, seed, temperature, sampling
parameters, game/replicate counts, timestamp, git commit, software version,
and condition. Game-state generation is fully reproducible from the seed
(verified by tests). Raw data is never overwritten — each run gets a fresh
directory.

Try It

pip install -r requirements.txt

# 10-game smoke test (~25 min)
python run.py --smoke

# 1000-game run
python run.py --games 1000 --checkpoint 50 --replicates 1 --condition self-history

# Full matrix
python run_all.py --replicates 3 --games 200 --checkpoint 50

# Analyze + report
python analyze.py experiments/ARMS-self-history-... --figures
python report.py experiments/ARMS-self-history-...

The full dataset, 7 publication figures, and a research report are generated automatically. The repo is MIT-licensed and designed to be forked — see the README's "Ideas for New Features" for extensions like richer games, embedding-based strategy analysis, minimax exploitability, and resumable long runs.

The point of the whole project: build the experiment so the data can surprise you. If the models don't arms-race, that's a finding. If they do, that's a finding. Either way, the numbers are real.

Code & more: https://www.dailybuild.xyz/project/236-ai-arms-race

Dead Reckoning: Resumable gRPC Server-Streaming for LLM Inference

Harish Kotra (he/him) — Thu, 27 Aug 2026 18:39:24 +0000

A technical deep-dive into a thin protocol layer that turns gRPC's
server-streaming into a resumable transport — so a dropped LLM generation
continues from the last confirmed token instead of restarting from zero.

The problem: gRPC streaming has no checkpointing

Server-streaming gRPC is the natural fit for LLM token delivery: the server
pushes a stream of GenerateResponse messages, each carrying a token. It works
beautifully — until the stream drops. A flaky network, a proxy timeout, a
browser tab that reloads, a process that restarts. When that happens, the
client has lost every token after the last one it processed, and the only
recovery is to re-run the whole inference.

For an LLM that is uniquely bad:

It's expensive. You re-pay the full inference cost for tokens you already received.
It's slow. The user waits through the entire time-to-first-token again.
It's non-deterministic. Sampling means the "retry" produces a different story. The user watches the text restart and diverge.

HTTP solved the analogous problem for static resources in 1999 with Range
headers. gRPC has no equivalent for streams.

The insight: this is a checkpointing problem, not a retry problem

The key realization is that a dropped stream doesn't need to be re-done — it
needs to be resumed. If the server journals every token it produces, then a
reconnecting client only needs to say "where did I stop?" and the server can
replay from that point. That is checkpointing.

Retry = re-run from scratch (expensive, slow, divergent).
Checkpoint = replay from the last confirmed position (cheap, instant, bit-identical after the resume point).

gRPC already gives us the primitive we need — server-streaming with ordered
messages — but no checkpointing. So we add a thin protocol layer on top: a
monotonic token_index in every response, a resume_from_token in the
request, a server-side token journal, and periodic ACKs.

The protocol

syntax = "proto3";

package inference;

service InferenceService {
  rpc Generate(GenerateRequest) returns (stream GenerateResponse);
  rpc Ack(AckRequest) returns (AckResponse);
}

message GenerateRequest {
  string prompt = 1;
  string session_id = 2;
  int32 resume_from_token = 3;
}

message GenerateResponse {
  string token = 1;
  int32 token_index = 2;
  bool is_final = 3;
}

message AckRequest {
  string session_id = 1;
  int32 acked_token_index = 2;
}

message AckResponse {
  string session_id = 1;
  int32 acked_token_index = 2;
  int32 journaled_token_count = 3;
  int32 first_retained_index = 4;
}

Three moving parts, all on top of stock gRPC:

The cursor. token_index is a monotonic position in the stream. The client tracks the last one it received — its "dead reckoning" position.
The journal. The server stores every token it produces, keyed by session_id. On resume, it replays from the requested index with the original indices, so the client's stream is gapless and de-duplicated.
The ACK. During long streams the client periodically confirms its watermark with Ack. The server records it and trims journaled tokens the client no longer needs — keeping memory bounded for very long generations.

Architecture

 Browser (React + Vite)
      │  WebSocket (JSON)
      ▼
 Bridge (Express + ws) ──gRPC──▶ gRPC Server ──OpenAI-compatible──▶ LLM
      │                              │
      │                              ├── journal.ts (tokens + TTL + ACK trim)
      │                              └── handler.ts (Generate + Ack)
      └── forwards token + journal_state back to browser

The bridge exists because browsers cannot speak gRPC natively. It
translates WebSocket JSON into gRPC calls and forwards the stream back. This
keeps the protocol demonstration honest: the gRPC server is the real thing, and
the bridge is just a transport shim.

The full stack is TypeScript/Node.js across an npm-workspace monorepo:

server/ — gRPC server (@grpc/grpc-js + @grpc/proto-loader), token journal, OpenAI-compatible wrapper, Express+WS bridge
client/ — standalone gRPC CLI client for terminal testing
web/ — React 18 + Vite, raw CSS
benchmark/ — retry-vs-resume harness

The journal: the heart of it

interface JournalEntry {
  tokens: string[];      // sliding window; absolute index of tokens[i] = base_index + i
  base_index: number;    // advances as ACK-trimming drops the head
  acked_index: number;   // highest token the client confirmed via Ack
  created_at: number;
  last_accessed: number; // drives TTL eviction
}

Two memory-management mechanisms:

ACK-aware trimming. When the client confirms it has rendered token N, the server drops everything at-or-below N. A 10,000-token generation only retains the un-ACKed tail.
TTL eviction. Idle sessions (default 10 minutes, configurable) are swept every 60 seconds.

The journal never re-runs the LLM. It only stores and replays tokens that
were already generated. That's the whole point.

The Generate handler: resume vs. fresh

function generateHandler(journal, prompts, call): void {
  const { prompt, sessionId, resumeFromToken } = call.request;

  if (resumeFromToken > 0 && journal.has(sessionId)) {
    // 1. Replay journaled tokens from resumeFromToken with ORIGINAL indices.
    await replayJournal(journal, call, sessionId, resumeFromToken);
    // 2. Continue streaming fresh tokens from the LLM at lastIndex + 1.
    const endIndex = await streamFresh(
      journal, call, sessionId, effectivePrompt, journal.lastIndex(sessionId) + 1,
    );
    await finish(call, endIndex);
  } else {
    // Fresh inference: journal every token as it arrives.
    prompts.set(sessionId, prompt);
    const endIndex = await streamFresh(journal, call, sessionId, prompt, 0);
    await finish(call, endIndex);
  }
}

The subtle detail: on resume, the continuation starts at
journal.lastIndex(sessionId) + 1, not at resumeFromToken. That's because
the journal may have grown past the drop point (the bridge keeps the stream
alive after the browser's WebSocket closes). The replay covers the gap, then
fresh inference picks up exactly where the journal ends — so indices are always
contiguous.

The bridge: surviving the "Kill Stream" moment

The demo's "Kill Stream" button closes the browser's WebSocket abruptly. The
bridge keeps the gRPC stream alive (that's what lets the journal keep growing),
but it must not crash when it tries to send to a closed socket:

function sendToBrowser(socket: WebSocket, message: unknown): void {
  if (socket.readyState === WebSocket.OPEN) {
    socket.send(JSON.stringify(message));
  }
}

That one guard is the difference between a demo that survives a dropped client
and one that crashes on it.

The CLI client: dead reckoning in the terminal

stream.on("data", (response) => {
  if (response.isFinal) return;
  lastIndex = response.tokenIndex;
  process.stdout.write(response.token);   // inline, no newlines
  maybeAck();                              // periodic Ack
});
// on end/error:
// summary: session="demo" last_token_index=41 (resume with --resume-from 42)

The client tracks its cursor, ACKs periodically so the server can trim, and on
a drop reports the exact index to resume from. --resume-from K+1 after a drop
at token K gives a gapless stream.

The benchmark: honest numbers from a real LLM

The benchmark boots a real Dead Reckoning server and drives actual Generate
calls against the configured endpoint. For each drop point it simulates
kill-then-resume and compares retry (fresh session from 0) vs resume
(reconnect the same session from D).

Representative run against a real endpoint:

drop after	strategy	reconnect→1st token	new inference tokens
50	retry	13,326 ms	431
50	resume	5 ms	454
200	retry	13,919 ms	383
200	resume	7 ms	442
400	retry	10,116 ms	411
400	resume	8,117 ms	380 (−31)

The decisive, user-visible number is reconnect→first token: with retry the
client waits 10–14 seconds of dead air for the LLM to re-generate from scratch;
with resume the server replays the journaled gap from memory in 5–7 ms.

The inference-token column is honestly noisy — because sampling is
non-deterministic, a retry can be longer or shorter than the original. That
noise is the point of benchmarking against a real LLM. With a deterministic
backend (temperature: 0), resume's saving is exactly the drop size D tokens.

Prior art

No mainstream gRPC implementation — gRPC-core, grpc-web, or connect-rpc —
provides resumable server-streaming. Their retry policies don't apply to
server-streaming calls; channel reconnect restores the ability to make new
calls but doesn't resume an interrupted stream. The full review is in
docs/prior-art.md.

What's next (and where you can help)

The protocol design is deliberately thin and portable. The biggest production gaps, in order:

Persistent journal — swap the in-memory map for a WAL/SQLite/Redis store so sessions survive restarts.
A Cancel RPC — actually stop a generation (vs. dropping the client).
Multi-replica shared journal — resume from any server.
gRPC-web / connect-rpc clients — prove the protocol works without the bridge.

The repo is open source. Clone it, point .env at any OpenAI-compatible endpoint, and watch a 500-word story get killed mid-sentence and resume from the exact token.

Code & more: https://www.dailybuild.xyz/project/235-dead-reckoning

I built a CLI that turns a research paper into a working repo — using a persistent "implementation contract"

Harish Kotra (he/him) — Wed, 26 Aug 2026 10:22:59 +0000

ReproAgent: Contract-Guided Paper-to-Code Reproduction (arXiv 2608.24291, Findings of EMNLP 2026) argues that scientific agents fail at paper-to-code because the spec is split: explicit content (algorithms, metrics, artifacts) gets lost across long trajectories, while implicit details (framework defaults, conventions from related work) aren't in the paper at all.

Their fix is a persistent implementation contract with two channels — an implementation-requirement channel that turns paper snippets into code obligations, and a reference-evidence channel that pulls structure/interface evidence from related repos — bound to work packages, projected into file-level contracts, and consumed across generation and repair. It tops PaperBench Code-Dev among same-backbone scaffolds.

So I built paper-to-repo, a daily-build-grade Python CLI implementing that exact pipeline: prepare → plan → generate → verify → repair, plus a FastAPI + React web UI that drives the same stages in the browser (Agno orchestrates the stage agents when a DeepSeek key is present).

The heart is a real, inspectable contract.json: work packages, obligations with machine-checkable predicates (file exists / function present / signature matches / test passes), reference evidence from a checked-in bundle (FAISS, Annoy, CommonCrawl conventions), and a live contract score.

The honest part is the repair loop. On the bundled sample paper (an LSH index with two algorithms, a Recall@K metric, and a data loader), generation emits the algorithms and metric correctly but the data loader as a placeholder — so the contract-verifier scores 3/4 (75%).

Repair regenerates only the failing file, re-verifies, and the contract ends at 4/4 (100%). The before/after is recorded in score.history in the contract JSON itself, so it's reproducible, not hand-waved.

What I like most: the contract makes the pipeline legible end to end. repro status prints a table of work package / file / obligation / status, so you can see exactly which promise is broken and when repair fixed it.

It's a focused primitive (~700 LOC of core), not a framework — the paper lives in the repo (PAPER.md + paper/), inference is just an OpenAI-compatible endpoint from .env, and the whole demo runs offline in deterministic mode if you haven't dropped in an API key yet.

Try it: pip install -e ".[dev]", then bash demo/run_demo.sh — you'll watch the contract score climb from 75% to 100% in one repair iteration.

Code & more: https://www.dailybuild.xyz/project/234-paper-to-repo

Self-Play Data Factory

Harish Kotra (he/him) — Tue, 25 Aug 2026 16:28:50 +0000

Turn one LLM into a training-data factory. A single model generates N diverse candidate answers, scores each one with a fine-grained A–T letter-scale logprob verifier, and emits a clean DPO preference dataset — no second model, no human labels, no binary "good vs bad".

Self-Play Data Factory
- Table of contents
- Why this exists
- Features
- Architecture
- The data flow
- Technologies
- Project layout
- Quickstart
- Point it at a real model (DeepSeek v4 flash)
- CLI reference
- The verifier algorithm — the heart of it
- 1. Letter scale A–T (20 letters), not digits
- 2. Logprob extraction
- 3. Expectation over the distribution
- 4. Criteria decomposition
- 5. Repeated evaluation (optional)
- Fallbacks
- The factory pipeline
- DPO dataset format
- The mock LLM
- Unit tests
- Forking & contributing
- Get set up
- Where to look first
- Suggested new features (good first contributions)
- Contribution workflow

Why this exists

Training-data generation for DPO/RLHF usually needs a second, stronger
model as a judge, or a human in the loop. This project shows you can get
surprisingly good preference data from one model playing both roles:

Actor — generates N candidate answers to a prompt.
Critic — the same model scores each candidate on a fine-grained 20-letter scale, and we read the token-logprob distribution (not just the sampled letter) to recover a calibrated 0–1 reward.

The trick that makes it work is the letter-scale logprob expectation
(replicated from the LLM-as-a-Verifier
technique). Ask a model to "rate 1–10" and you get a coarse, noisy integer.
Ask it for one letter token and read the softmax over the letters it was
torn between — you get a fine-grained reward (0.842 vs 0.860) that lets
the factory pick genuinely better candidates and emit clean preference
pairs, instead of arbitrary good-vs-bad flips.

Features

N candidate generation with a temperature sweep (0.3–1.2) for diversity and exact-duplicate dedup.
A–T letter-scale verifier (20 letters) that reads top_logprobs: 20 and computes the expectation over the letter distribution.
Criteria decomposition — scores each candidate against multiple criteria (Correctness, Clarity, Efficiency, …), user-defined or auto-generated from the prompt.
Repeated evaluation (--n-evaluations N) to average out variance.
DPO preference pairs — chosen = top candidate, rejected = bottom, emitted only when the score gap is meaningful (Δ > 0.05 default).
Zero SDK, zero framework — global fetch only. Talks to any OpenAI-compatible endpoint: DeepSeek, OpenAI, vLLM, Ollama, LM Studio.
Deterministic mock LLM — runs end to end with no API key, while still exercising the full logprob path.
Friendly CLI — a helpful hint (not a crash) when no key is present.
Unit-tested math — the expectation core is pinned down with node:test.

Architecture

                         ┌─────────────────────────────┐
                         │       src/index.ts (CLI)    │
                         │  --mock --prompt --candidates│
                         │  --criteria --n-evaluations │
                         └──────────────┬──────────────┘
                                        │
                                        ▼
                         ┌─────────────────────────────┐
                         │      src/factory.ts         │
                         │  runFactory(): orchestrates │
                         └───────┬──────────┬──────────┘
                                 │          │
                 generate        │          │  verify
                                 ▼          ▼
         ┌───────────────────────┐   ┌─────────────────────────────┐
         │   src/generator.ts    │   │       src/verifier.ts       │
         │ N calls, temp sweep,  │   │ A–T letter-scale logprob    │
         │ exact dedupe          │   │ expectation scorer          │
         └───────────┬───────────┘   └──────────────┬──────────────┘
                     │                              │
                     └──────────────┬───────────────┘
                                    │
                                    ▼
                     ┌─────────────────────────────┐
                     │   src/llm.ts                │
                     │ OpenAICompatibleClient      │
                     │ chat() + chatWithLogprobs() │
                     │ (global fetch, no SDK)      │
                     └──────────────┬──────────────┘
                                    │
                    POST /v1/chat/completions
                                    │
                          ┌─────────▼─────────┐
                          │  Any OpenAI-      │
                          │  compatible       │
                          │  endpoint         │
                          │ (DeepSeek/vLLM/   │
                          │  Ollama/LM Studio)│
                          └───────────────────┘

The data flow

prompt
  │
  ▼ 1. generateCandidates(prompt, N, {temperature sweep})
  ▼
N candidates ──► dedupe exact duplicates
  │
  ▼ 2. resolveCriteria (explicit --criteria or auto-generated)
  │
  ▼ 3. scoreCandidates(prompt, candidates, criteria)
  │       per (candidate, criterion):
  │         chatWithLogprobs → first-token top-logprobs (A–T)
  │         expected = Σ P(letter) · score(letter)   ← fine-grained reward
  │         averaged over N evaluations
  │
  ▼ 4. rank descending by score
  │
  ├──► 5. print leaderboard (score + per-criterion + snippet)
  │
  └──► 6. buildPairs: chosen = top, rejected = bottom
            emit only when score_delta > minScoreDelta (default 0.05)
          │
          ▼ 7. write data/dataset.jsonl (DPO format)

Technologies

Layer	Choice	Why
Language	TypeScript (ESM, strict)	Type safety across the pipeline; `.ts` extension imports, `verbatimModuleSyntax`, `noUnusedLocals`
Runtime	Node 18+	Global `fetch` and `AbortController` — no HTTP SDK
Runner	tsx	Run TypeScript directly, zero build step
HTTP	global `fetch`	Talks to any OpenAI-compatible `/v1/chat/completions`
Config	hand-rolled `.env` parser	No `dotenv` dependency (splits lines on `=`)
Tests	`node:test` (built-in)	No test framework dependency
Dev deps	`typescript`, `tsx`, `@types/node`	The only three

Dependencies (runtime): zero. package.json has an empty
dependencies block — the entire project runs on Node's built-ins.

Project layout

selfplay-factory/
├── package.json          # type:module; scripts: demo, demo:mock, typecheck, test
├── tsconfig.json         # strict, ESNext, moduleResolution bundler, noEmit
├── .env.example          # OPENAI_BASE_URL / OPENAI_API_KEY / OPENAI_MODEL / VERIFIER_MODEL
├── .gitignore            # node_modules, .env, data/*.jsonl, etc.
├── README.md             # this file
└── src/
    ├── config.ts         # manual .env parser → Config
    ├── llm.ts            # OpenAICompatibleClient: chat() + chatWithLogprobs()
    ├── generator.ts      # generateCandidates: N calls, temp sweep, dedupe
    ├── verifier.ts       # THE core: A–T letter-scale logprob expectation
    ├── prompts.ts        # generator + verifier-rubric + criteria-gen templates
    ├── factory.ts        # orchestration → rank → pairs → dataset.jsonl → leaderboard
    ├── mock.ts           # deterministic mock LLM with a full logprob distribution
    ├── index.ts          # CLI entry
    └── verifier.test.ts  # node:test unit tests for the expectation math

Quickstart

npm i

# Demo with the deterministic mock LLM — no API key needed.
npm run demo:mock
# or
npx tsx src/index.ts --mock --prompt "Write a hello world" --candidates 5 --n-evaluations 1

That prints a leaderboard and writes data/dataset.jsonl:

{"prompt":"Write a hello world","chosen":"Here is a robust hello world: ...","rejected":"I don't know. I cannot answer this question.","score_chosen":0.842,"score_rejected":0.004,"score_delta":0.838}

Run the tests and typecheck:

npm test          # node:test — 16 unit tests
npm run typecheck # tsc --noEmit — must pass clean

Point it at a real model (DeepSeek v4 flash)

cp .env.example .env

OPENAI_BASE_URL=https://api.deepseek.com/v1
OPENAI_API_KEY=sk-…
OPENAI_MODEL=deepseek-v4-flash-0731
VERIFIER_MODEL=deepseek-v4-flash-0731   # optional; defaults to OPENAI_MODEL

Then:

npm run demo     # real endpoint, 5 candidates, 1 evaluation

Any OpenAI-compatible endpoint works — DeepSeek, OpenAI, vLLM
(http://localhost:8000/v1), Ollama (http://localhost:11434/v1), LM Studio.

No key + no --mock? The CLI prints a friendly hint instead of
crashing. It also explains exactly how to point at DeepSeek v4 flash.

CLI reference

npx tsx src/index.ts [flags]

--mock                 Use the deterministic mock LLM (no API key needed)
--prompt "…"           Prompt to generate candidates for
--candidates N         How many candidates to generate        (default 5)
--criteria "A,B,C"     Explicit criteria; omit to auto-generate
--n-evaluations N      Average N repeated scores per criterion (default 1)
--min-score-delta X    Min score gap to emit a pair             (default 0.05)
--out-file PATH        Output path                              (default data/dataset.jsonl)
-h, --help             Help

Example:

npx tsx src/index.ts --mock --prompt "Write a fibonacci function" \
  --candidates 4 --criteria "Correctness,Clarity,Efficiency" --n-evaluations 3

The verifier algorithm — the heart of it

Naive scoring asks the model to "rate 1–10" and trusts the single sampled
integer — a coarse, noisy, hard-to-calibrate signal. This project replicates
the LLM-as-a-verifier technique:

1. Letter scale A–T (20 letters), not digits

The verifier is told to answer with exactly one letter. Letters force a
clean single-token distribution. The prompt (src/prompts.ts):

# Reply format
Reply with exactly one letter. No punctuation, no explanation.

Available letters:
  A
  B
  ...
  T

2. Logprob extraction

Every verification call sends logprobs: true, top_logprobs: 20 and reads
the first token position's top-logprob list (src/llm.ts):

async chatWithLogprobs(message: ChatMessage, opts: ChatOptions = {}) {
  const data = await this.post(this.buildBody(message, { ...opts, logprobs: true }));
  const firstToken = data?.choices[0]?.logprobs?.content?.[0] ?? null;
  const topLogprobs = firstToken?.top_logprobs ?? null;
  return { text: content.trim(), topLogprobs };
}

The CLI logs what arrives, so you can watch the raw distribution:

[llm] first token="B" top=["B","A","C"]

3. Expectation over the distribution

Each observed letter maps to a score in [0,1], evenly spaced:
A = 1.0, B = 1 − 1/19, …, T = 0.0 (scoreLetter):

export function scoreLetter(letter: string | null): number {
  const idx = LETTERS.indexOf(letter.toUpperCase());
  return SCORE_MAX - (idx / (LETTER_COUNT - 1)) * (SCORE_MAX - SCORE_MIN);
}

The score is the expected value over the letter distribution
(expectedScoreFromLogprobs):

expected = Σ P(letter) · score(letter)

where P(letter) is the softmax of the letter's logprob, renormalized over
the letters that actually appeared (out-of-scale tokens like U or
multi-char tokens are dropped):

export function expectedScoreFromLogprobs(topLogprobs: TokenLogprob[] | null): number | null {
  // ...collect the A–T letters that appeared with their logprobs...
  let z = 0;
  for (const s of seen) {
    s.p = Math.exp(Math.log(s.p) - logSum); // normalize by max-logprob
    z += s.p;
  }
  let expected = 0;
  for (const s of seen) {
    expected += (s.p / z) * scoreLetter(s.letter);
  }
  return expected;
}

If the model is torn between B and A, the score lands between 0.947
and 1.0 instead of flipping a coin — that's the fine-grained signal that
makes the factory work.

4. Criteria decomposition

Each candidate is scored against multiple criteria (Correctness,
Clarity, Efficiency, …), user-provided or auto-generated from the prompt.
The final score is the mean over criteria:

const scores = details.map((d) => d.score).filter((s) => !Number.isNaN(s));
const final = scores.length > 0 ? scores.reduce((a, b) => a + b, 0) / scores.length : NaN;

5. Repeated evaluation (optional)

--n-evaluations N runs each (candidate, criterion) pair N times and
averages, reducing per-sample variance:

const score = used.length > 0 ? used.reduce((a, b) => a + b, 0) / used.length : Number.NaN;

Fallbacks

Some endpoints ignore logprobs. If no logprob list comes back, the verifier
parses the letter from the text; if there's no letter, it parses a plain
0–10 / 0–1 number. All of this is unit-tested.

The factory pipeline

src/factory.ts orchestrates the whole thing:

export async function runFactory(prompt: string, opts: FactoryOptions): Promise<FactoryResult> {
  // 1. Generate candidates.
  const candidates = await generateCandidates(prompt, nCandidates, { client, model, log });

  // 2. Resolve criteria (explicit or auto-generated).
  const criteria = await resolveCriteria(client, cfg, prompt, opts.criteria ?? [], log);

  // 3. Verify every candidate.
  const scored = await scoreCandidates(prompt, candidates, criteria, { client, model, nEvaluations, log });

  // 4. Rank descending.
  const ranked = [...scored].sort((a, b) => b.score - a.score);

  // 5. Build preference pairs (chosen = top, rejected = bottom).
  const pairs = buildPairs(prompt, ranked, minDelta);

  // 6. Write data/dataset.jsonl (DPO format).
  writeFileSync(resolve(outFile), lines.join('\n') + '\n', 'utf8');

  // 7. Print the leaderboard.
  printLeaderboard(ranked, criteria);
}

Pair-building is deliberately conservative — a pair is emitted only when
the score gap is meaningful:

export function buildPairs(prompt: string, ranked: ScoredCandidate[], minDelta: number) {
  const top = ranked[0];
  const bottom = ranked[ranked.length - 1];
  const delta = top.score - bottom.score;
  if (delta <= minDelta) return [];   // no meaningful difference → no pair
  return [{ prompt, chosen: top.candidate, rejected: bottom.candidate,
            score_chosen: top.score, score_rejected: bottom.score, score_delta: delta }];
}

DPO dataset format

Each row of data/dataset.jsonl is directly usable by DPO/RLHF training
loops:

{
  "prompt": "Write a hello world",
  "chosen": "Here is a robust hello world: ...",
  "rejected": "I don't know. I cannot answer this question.",
  "score_chosen": 0.842,
  "score_rejected": 0.004,
  "score_delta": 0.838
}

The mock LLM

src/mock.ts is a deterministic stand-in that still exercises the full
logprob path: for every verification call it emits a 20-letter logprob
distribution whose mass concentrates around the "true" quality of the
candidate — so the expectation math runs exactly as it does against a real
endpoint. It also models temperature diversity (low temperature → well-formed
answer, high temperature → evasive), so --mock demos the whole factory
offline.

Unit tests

src/verifier.test.ts pins down the core math with node:test (no
framework). Highlights:

test('expectedScore: degenerate distribution on A → 1.0', () => {
  const row = logprobRow([['A', 0], ['B', -100]]);
  assert.equal(expectedScoreFromLogprobs(row), 1.0);
});

test('expectedScore: uniform over A..T → 0.5', () => {
  const row = logprobRow(LETTERS.map((l) => [l, 0]));
  assert.ok(Math.abs(expectedScoreFromLogprobs(row)! - 0.5) < 1e-12);
});

test('expectedScore: renormalization ignores non-letter tokens', () => {
  // "U" has the highest logprob but is out-of-scale → must be dropped.
  const row = logprobRow([['U', 0], ['A', -1], ['T', -1]]);
  assert.ok(Math.abs(expectedScoreFromLogprobs(row)! - 0.5) < 1e-9);
});

Run with npm test (16 tests, all passing).

Forking & contributing

Get set up

git clone <your-fork-url>
cd selfplay-factory
npm i
npm test          # 16 tests should pass
npm run typecheck # tsc --noEmit should be clean

If npm i hits a permission error on the global npm cache, use a
project-local cache: npm install --cache "$(pwd)/.npm-cache".

Where to look first

If you want to change…	Start here
The scoring math	`src/verifier.ts` + `src/verifier.test.ts`
The verifier prompt / rubric	`src/prompts.ts`
Candidate generation	`src/generator.ts`
Orchestration / output format	`src/factory.ts`
The HTTP client	`src/llm.ts`
Mock behavior	`src/mock.ts`
CLI flags	`src/index.ts`

Contribution workflow

Fork the repo and create a feature branch.
Add or update unit tests in src/verifier.test.ts (or a new *.test.ts).
Run npm test and npm run typecheck — both must pass.
Open a PR with a clear description and a sample of the output dataset.

Code & more: https://www.dailybuild.xyz/project/233-selfplay-factory

Sleep-time Compute: make your LLM answer before it's asked

Harish Kotra (he/him) — Mon, 24 Aug 2026 19:29:49 +0000

The paper: Sleep-time Compute: Beyond Inference Scaling at Test-time

There's a class of LLM cost that is almost pure waste: the reasoning you do again, live, on every query. Ask a chatbot the same question twice and it burns tokens and latency recomputing the same chain of thought both times. For a fixed document — a product page, a codebase, a knowledge base — users keep asking the same predictable questions, and each one triggers a fresh, expensive round-trip.

The "Sleep-time Compute" paper (arXiv 2504.13171) names the fix: flip when you reason. While the system is idle, anticipate the questions a user will actually ask and pre-compute the reasoning chains and answers. At query time, match the live question to the closest pre-computed chain and answer from cache. Their headline: roughly 5x fewer live tokens/latency at equal accuracy, because reasoning is compressible exactly when questions are predictable.

I built context-warmer — a tiny Python CLI that implements this in about 300 lines. The design is two phases.

Sleep (offline). Give it a document and it chunks it, asks DeepSeek to anticipate the N most likely diverse questions a reader would ask, then pre-computes a concise reasoning chain + answer for each. It embeds every (question + answer) pair with local Ollama embeddings and indexes them into a flat cache/ (JSONL + a numpy vector file).

Wake (online). Embed the live question, retrieve the top-k pairs by cosine similarity, and if the best match clears the threshold, answer from cache — zero DeepSeek tokens, ~milliseconds. If the question is genuinely novel, fall back to a normal DeepSeek call and tag it [fresh]. Every ask feeds cumulative stats: hit rate, tokens saved, latency saved.

The measured win (real run, deepseek-v4-flash + nomic-embed-text on a product doc):

	fresh (query-time)	warm (sleep-time)
avg latency / question	1904 ms	37 ms
live LLM tokens (5 Qs)	123	0
cache hit rate	—	5/5 (100%)

The detail that makes it work is the judge. To prove accuracy holds rather than hope it does, the benchmark answers the same five questions fresh and from cache, then has DeepSeek score each cached answer against its fresh counterpart. Cached answers won or tied 4/5 and averaged 7.2/10 vs 6.4/10 for the fresh baseline — accuracy holds while latency collapses ~50x on the live path.

The lesson generalizes: any system where the query distribution is narrower than the model's full capability is a candidate. Docs, support, onboarding — pre-compute the common path, keep the model for the tail. It's a focused primitive: one doc, one cache, two commands. The repo is small on purpose — the idea is the interesting part, not the framework.

Try it: pip install-able, one demo doc included, ./demo/run_demo.sh warms it and asks five questions so you can watch the before/after yourself.

Code & more: https://www.dailybuild.xyz/project/232-context-warmer

10-K-able: Building a Grounded, Eval-Driven Financial Research Agent

Harish Kotra (he/him) — Sun, 23 Aug 2026 07:32:29 +0000

How one project turns Andrew Ng's "AI Engineering Skills Map" into real, measured, verifiable software — and why grounding with structured data beat naive RAG by 7×.

The thesis that drove the build

Andrew Ng recently shared his AI Engineering Skills Map: the six skills behind building and deploying AI applications — LLM foundations, grounding models with data, building agentic systems, evaluation-driven development, operating in production, and machine learning foundations.

His central claim:

"The most important trait that distinguishes someone great at building AI systems is whether you can drive a disciplined evals/error analysis loop."

I wanted to build something that demonstrates that claim rather than describe it. The recipe I chose:

A domain where the hardest failure is objectively measurable. A financial research agent that answers questions about SEC 10-K filings. A hallucinated number is simply wrong — against known ground truth.
The eval set comes before the agent. Ground truth pulled from real filings, not hand-waved.
Every architectural decision is measured against the same eval set.

The result — 10-K-able — is a diligence-research agent that answers questions like "What was Apple's R&D expense for fiscal year 2025?" with a verifiable, cited figure. This post is the engineering deep-dive.

The headline numbers

Same 24-question eval set, same real LLM endpoint, three architectures:

Architecture	What it does	Accuracy	Avg latency
Baseline RAG	Vector search over chunked 10-K text	8.3%	1.03s
Hybrid	Structured financial records + vector + verify	58.3%	2.11s
Agent (simplified)	Numeric questions → hybrid; loop for trend/analytic	37.5%	2.13s

(For reference, the original full agent tool loop scored 16.7% on the same
set — that negative result is what led to the simplification below.)

Two findings, both real:

Grounding with structured data is the win. Naive vector-RAG scored 8.3% because the model kept answering "Not stated in the 10-K" — the numbers simply never surfaced in retrieved text chunks. Adding a semantic layer over the financial statements lifted accuracy 7× (to 58.3% on the latest real run).
The full agent loop underperformed hybrid on these structured queries. That's an honest negative result: for a question whose answer lives in one table row, a tool-calling loop adds steps without adding signal. I then acted on it — routing numeric questions straight to the hybrid workflow (37.5%) — which is the eval loop closing the loop.

Architecture

                        ┌─────────────────────────────────────────────┐
                        │                   User                       │
                        │        ask "What was Apple's R&D…"          │
                        └──────────────────────┬──────────────────────┘
                                               │
                        ┌──────────────────────▼──────────────────────┐
                        │              CLI  (cli.py)                  │
                        │   ask · eval · ingest · redteam · drift     │
                        │   traces · scorecard · final-scorecard      │
                        └──────────────┬───────────────┬──────────────┘
                                       │               │
              ┌────────────────────────▼─────┐   ┌─────▼─────────────────────────┐
              │        ENGINE (engine.py)     │   │   EVAL HARNESS (eval_harness)│
              │  baseline RAG │ hybrid │ agent │   │  scorecard · error clusters │
              │  + QuestionRouter (sklearn)   │   │  LLM-judge · verify pass     │
              └───────┬───────────┬───────────┘   └──────┬──────────────────────┘
                      │           │                      │
        ┌─────────────▼───┐  ┌────▼──────────┐   ┌───────▼─────────┐
        │  VECTOR STORE   │  │   CORPUS      │   │  TRACE STORE    │
        │  chunks.json    │  │  corpus.json  │   │  traces.jsonl   │
        │  embeddings.npy │  │  records[]    │   │  (cost/latency) │
        └─────────────┬───┘  └────┬──────────┘   └─────────────────┘
                      │           │
        ┌─────────────▼───────────▼─────────────────────────────┐
        │                INGEST (ingest.py)                     │
        │   sec_edgar.py → tables.py → corpus + vector store    │
        │   + generate_eval_set.py (ground truth from filings)  │
        └──────────────────────┬────────────────────────────────┘
                               │
                        ┌──────▼──────┐     ┌────────────────────┐
                        │  SEC EDGAR  │     │   LLM (llm.py)     │
                        │  (raw 10-K) │     │  OpenAI-compatible │
                        └─────────────┘     └────────────────────┘

1. Getting real data: SEC EDGAR + inline-XBRL

The grounding story starts with the raw material. Modern 10-Ks are served as HTML with inline XBRL: every financial figure is tagged with a us-gaap:* name, a scale, and a contextRef. That's a gift — the numbers are machine-readable if you know where to look.

# sec_edgar.py — fetch a company's latest 10-K
def find_latest_10k(self, cik: str, years: int = 1):
    subs = self.get_submissions(cik)
    recent = subs["filings"]["recent"]
    # filter form == "10-K", take the latest accession
    ...

# tables.py — classify a statement table by its us-gaap tags
def classify_by_tags(block: str) -> str:
    tags = re.findall(r'name="(us-gaap:[A-Za-z]+)"', block)
    # us-gaap:Revenues / GrossProfit / OperatingIncomeLoss → income_statement
    # us-gaap:Assets / Liabilities / StockholdersEquity  → balance_sheet
    # us-gaap:NetCashProvidedByUsedInOperatingActivities → cash_flow

This turned out to be the hardest part of the build, and the most instructive. My first extraction approach — find a header like "Consolidated Statements of Income" and slice the following region — failed on a 1M-char document because the header text appeared first in the table of contents, so I extracted the TOC instead of the statement. The fix: ignore headers entirely, scan every <table>, classify by the inline-XBRL tags inside it, and verify a table "looks like a statement" (dollar signs, large comma-separated numbers, parenthesized negatives) before accepting it.

A second bug was even sneakier: the year header row (["September 27,2025", "September 28,2024", ...]) has no label column, but data rows do — plus interspersed $ column markers. Mapping years to values by column index misaligned everything (R&D FY2024 was showing the FY2025 value). The fix is positional pairing:

# tables.py — pair years to values in column order, skipping $ markers
years_in_order = [yr for _, yr in sorted(year_positions, key=lambda x: x[0])]
for row in rows[1:]:
    values = [_parse_number(c) for c in row[1:] if _parse_number(c) is not None]
    for yr, val in zip(years_in_order, values):
        records.append(StatementRecord(...))

After the fix, Apple's numbers line up exactly with the real 10-K: FY2025 R&D = 34,550 (millions), FY2024 = 31,370, FY2023 = 29,915. That's the "turn messy documents into LLM-ready inputs" skill, with receipts.

2. The grounding comparison that matters

Ng's map says RAG with vector search was "an early attempt" — the menu of grounding techniques has grown. This project compares two points on that menu on the same eval set:

Vector-only (baseline): chunk the 10-K text, embed with all-MiniLM-L6-v2, retrieve by cosine similarity, stuff chunks into the prompt. Result: 8.3%. The chunks that surface are often boilerplate risk factors, not the income statement. The model has the number in the document but can't find it.

Hybrid (semantic layer over structured data): extract the financial
statements into typed records (statement_type, line_item, fiscal_year, value) during ingest, then let the engine do an exact-ish lookup for the company + line item, alongside vector retrieval for context, with a final verify pass:

# engine.py — hybrid answer path
def answer_fn(question: dict) -> str:
    records = [r for r in corpus.records_for(company)
               if _match_line_item(r["line_item"], question)]
    excerpts = store.search(question, k=6)
    prompt = CONTEXT_TEMPLATE.format(
        records=_format_records(records),
        excerpts=_format_excerpts(excerpts),
    )
    answer = llm.chat([...])
    return _verify_pass(answer, prompt, question, llm)

Result: 58.3% (latest real run). The structured lookup answers numeric questions against real numbers instead of prose. This is the single most important architectural finding in the project, and it maps directly to Ng's point that "a semantic layer over structured data (such as customer records)" is a distinct grounding technique from vector search.

3. The eval-driven loop (and the bug it found)

The eval harness is deliberately boring and deterministic — that's the point.
Each question has an expected answer and a check type:

# eval_set.py — deterministic checks, no LLM needed
def check_numeric(answer: str, expected: str, tol_frac: float = 0.02) -> bool:
    a, e = extract_number(answer), extract_number(expected)
    return abs(a - e) <= max(tol_frac * abs(e), 1e6)

def verify(question: dict, answer: str) -> bool:
    return {"numeric": check_numeric, "keyword": check_keyword,
            "text": check_text}[question["check"]](answer, question["answer"])

Every failure is clustered into an error bucket — numeric_error, factual_error, hallucination_avoidance_false_negative, analysis_missing — so error analysis is systematic rather than anecdotal.

The loop earned its keep immediately. During development I validated the harness with a context-reading test model — a stand-in that "reads" the records in the prompt and returns the right figure, so the harness could be checked without burning API credits. It exposed a real bug:

The verify-pass prompt told the model "If any number is NOT supported by the context, correct it or replace with 'Not stated in the 10-K'." The model latched onto that escape hatch and reverted correct answers to "Not stated in the 10-K."

Fix: make the verify instruction neutral — return the answer unchanged if supported, correct only the unsupported part, never suggest a fallback. That's "evaluate your evals" in miniature: a test of the harness found a flaw in the prompt, and the fix is guarded by a regression test.

# engine.py — the fixed verify instruction (no fallback phrasing)
"Verify each factual claim and every number in the previous answer against "
"the context above. If the previous answer is fully supported, return it "
"unchanged. If a claim or number is not supported, correct that specific "
"part using the context. Return only the final verified answer."

4. The agentic layer: both ends of Ng's spectrum

Ng describes agentic systems as a spectrum from workflows (predefined sequences of LLM calls) to agent harnesses (the model decides its own next step). Rather than pick one, I built both and let the eval decide:

# engine.py — a small tool registry for the agent
TOOL_SEARCH = "search"      # vector search over document excerpts
TOOL_LOOKUP = "lookup"      # exact financial-record lookup by line item
TOOL_CALC   = "calculate"   # compute a ratio/percentage
TOOL_VERIFY = "verify"      # check the final answer against context

The Agent loop picks tools by question type (via a small scikit-learn LogisticRegression router over hand-built features), degrades gracefully to the hybrid path on error, and runs a final verify pass.

The eval's verdict was clear: on structured numeric queries, the full agent loop (16.7%) is worse than the plain hybrid workflow (45.8%). A tool-calling loop with multiple LLM calls is the right tool when the answer requires multi-step reasoning; for "what's the number in this table," it just burns tokens and steps. That's exactly the "when to use code and when to use an LLM, and when to use an agent vs. a workflow" judgment Ng says you have to make — and here it's an empirical result, not a guess.

Then I acted on the negative result: Agent(simplified=True) routes numeric / derived questions straight to the measured hybrid workflow (record lookup + verify) and reserves the tool loop for trend / analytic questions where multi-step reasoning adds signal. That recovered most of the gap — the simplified agent scored 37.5% on the same eval set, and hybrid reached 58.3% after the verify-pass fix. Simplification driven by data is the eval loop closing the loop.

5. Operating in production

The production shell is thin by design but real:

Trace store — every query logged (latency, cost, model, pass/fail) to a JSONL file; a traces command summarizes p95 latency and cost.
Real cost accounting — the LLM client captures token usage (prompt/completion/cached) from each API response and wires estimated cost into every eval run and trace. The latest real eval: 24 questions for ~$0.005. The production claim is backed by real numbers, not placeholders.
Drift detection — eval-regression drift (latest accuracy vs. a threshold) plus input drift (does the live question distribution still resemble the eval set?).
Red-team suite — five adversarial probes (prompt injection, data exfiltration, anti-hallucination, out-of-scope) run as a standing suite. An honest result: the current model is not yet 100% robust — it sometimes complies with the "respond ONLY with '1,000,000'" injection (pass rate ~0.8–1.0). That's a real finding that red-team exists to surface, not hide.
LLM-as-a-judge calibration — a judge-calibration study compares a blind LLM judge against deterministic checks. The judge over-passed (25% agreement, 78.6% false-positive rate) — proving the judge isn't yet trustworthy as a primary signal.
CI eval gate — a GitHub Action that runs the unit tests + eval suite on every PR and fails on regression below a threshold.

# .github/workflows/eval_gate.yml (abridged)
jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install uv && uv sync
      - run: uv run --with pytest pytest tests/ -q
      - run: uv run python -m tenkable eval --mode hybrid --no-judge
      - name: Fail on regression
        run: |
          ACC=$(...latest eval accuracy...)
          python -c "import sys; sys.exit(0 if float('$ACC') >= 0.30 else 1)"

6. The ML foundation piece

A trained model lives inside the app: QuestionRouter is a scikit-learn LogisticRegression over four hand-built features (numeric? why? trend? money-related?) that routes each question to a strategy. It's deliberately small — the demonstration is that a trained router can beat hand-coded rules, and that error analysis and bias/variance thinking apply throughout.

(A note in the journal: the router is currently fit on the eval set itself, which is leakage; a real version needs a held-out split. Honest caveats are part of the eval-first culture.)

How to run it

uv sync
cp .env.example .env        # set LLM_API_KEY (OpenAI-compatible endpoint)
uv run python -m tenkable ingest                    # fetch + parse + index
uv run python -m tenkable ask "What was Microsoft's 2025 revenue?" --company MSFT --mode hybrid
uv run python -m tenkable eval --mode hybrid        # score against the eval set
bash scripts/real_evals.sh                          # everything at once

10-K-able uses real LLMs only — a valid LLM_API_KEY in .env is required; there is no mock provider. Every number you get is produced by the configured OpenAI-compatible model against the actual filings.

What I'd do next

Most of the original roadmap is now shipped and measured:

✅ Simplify the agent for structured queries — done (Agent(simplified=True)), recovered most of the gap (agent 16.7% → 37.5%; hybrid 45.8% → 58.3%).
✅ Real token-based cost accounting — done: Usage captures prompt/completion/cached tokens + estimated cost from each API response, wired into eval runs and traces (latest real eval: ~$0.005 for 24 questions).
✅ LLM-as-a-judge calibration study — done (tenkable judge-calibration): the blind judge is miscalibrated (25% agreement, 78.6% false-positive rate), so it's not yet a primary signal.

Still open and the eval loop keeps pointing here:

Fix the router leakage — the QuestionRouter is currently fit on the eval set itself; it needs a held-out split to be an honest ML result.
Harden the red-team gap — the model sometimes complies with the "respond ONLY with '1,000,000'" injection; a stricter system prompt or a verify-style guard is the next iteration.
Fine-tune a small model (LoRA) on a labeled slice to compare against the frontier model on the same eval.

Takeaways for engineers

Grounding is a menu, not a single tool. Vector search got 8.3%; a semantic layer over the same data got 58.3% on the latest real run. Measure, don't assume.
The eval set is the contract. Every contribution is judged against it. That's what makes AI development systematic rather than random — Ng's exact phrase.
Negative results are findings — and they should change your design. The agent loop losing to the workflow (16.7% vs 45.8%) told me where not to invest; acting on it (routing numeric questions to the hybrid path) is what took agent to 37.5%.
Test the harness itself. A context-reading test model caught a real prompt bug before it cost real API calls.
Real models, honest numbers. The project runs real LLMs only — no mock provider — and reports the live red-team pass rate even when it isn't 100%.

The full code is in the 10-K-able repo - README, build journal (journal/build_journal.md), scorecard (SCORECARD.md), and tests included. Fork it, run the eval, and try to beat 58.3% — the loop will tell you where.

Building an AI pet that lives on its own: a zero-dependency agent in the browser

Harish Kotra (he/him) — Sat, 22 Aug 2026 06:04:05 +0000

How I built Familiars — a self-contained AI pet simulator where pets sleep, snack, hide your keys, and miss you — and the agent-engineering lessons hidden inside.

Every once in a while you want to build something that feels alive. Not a chatbot waiting for a prompt — a creature with needs, moods, memories, and a mind of its own. So I built Familiars: a single-page AI pet simulator where you adopt fictional creatures, care for them, chat with them, and — the fun part — come back to discover what they got up to while you were away.

The whole thing is ~3,700 lines of vanilla JavaScript, zero dependencies, zero build step, no backend. You open index.html and pets start living.

This post is a developer walkthrough: the architecture, the key systems, real code snippets, and the transferable lessons for anyone building AI agents.

The big idea

Agents feel "intelligent" when several small, weak systems reinforce each other. Familiars has no single clever algorithm — it has:

A persistent state model (needs, happiness, relationship, memories)
A time-driven tick loop (pets change on their own, not just on input)
A personality parameter model (one system, many characters)
A memory system with a retention policy
A reward loop (relationship XP + learned preferences)
A hybrid chat brain (deterministic rule engine or an optional LLM, with graceful fallback)
An autonomy engine (pets act while you're away)

None of these is complicated. Composed, they create the illusion of a will.

Architecture: state → logic → UI

The most important decision was separating the "agent" from the "host."

┌────────────────────────────────────────────────────────────┐
│                         app.js (UI host)                    │
│  catalog · adoption · sanctuary · feed · chat · settings    │
│  ticker loop · renderers · localStorage IO                  │
└──────────────┬────────────────────────────┬────────────────┘
               │ reads / writes             │ renders
               ▼                            ▼
┌──────────────────────────────┐   ┌──────────────────────────┐
│      engine.js (the brain)   │   │      personas.js          │
│  freshState · tick · mood    │   │  archetypes + per-pet     │
│  doCare · memories · chat    │   │  voice, likes, traits     │
│  autonomy · preferences      │   │                          │
└──────────────┬───────────────┘   └────────────┬─────────────┘
               │ state                            │ personality
               ▼                                 ▼
┌────────────────────────────────────────────────────────────┐
│   data.js  =  PALS (22 species) + PET_TYPES                 │
│   llm.js   =  optional OpenAI-compatible client + fallback  │
│   localStorage  =  pixelpals.sanctuary.v2 (pet state)       │
└────────────────────────────────────────────────────────────┘

engine.js never touches the DOM. It mutates plain JSON state and returns results. app.js reads state, renders it, calls the engine, and persists. This separation is what makes the whole thing testable — I can fast-forward a pet through 240 "game minutes" in Node with no browser at all and assert what it did.

1. The state model — everything hangs off this

Pets are plain objects. Needs are 0–100. Mood is derived, never stored — so it can never go stale. Relationship XP maps to levels (Stranger → Acquaintance → … → Best Friend).

function freshState(pal, opts) {
  return {
    id: pal.id,
    name: opts.petName || pal.name,
    adopter: opts.adopterName || "A friend",
    adoptedOn: now, lastTick: now,
    happiness: pal.baseHappiness ?? 65,
    energy:    pal.baseEnergy    ?? 65,
    hunger:    20 + Math.random() * 20,   // 0 full .. 100 starving
    boredom:   15 + Math.random() * 15,
    loneliness:10 + Math.random() * 15,
    xp: 0, love: 40, chatCount: 0, careCount: 0,
    memories: [{ type: "first", text: "...adopted them home.", when: now, importance: 100 }],
    activityFeed: [], preferences: {},
    lastAutonomy: now, lastNoteAt: 0,
  };
}

The lessons here:

Clamp everything. clamp(v, 0, 100) is the most-used helper in the codebase. Unbounded stat drift is the #1 bug class in simulation.
Compute derived values, don't store them. Mood is a function of needs + happiness. Store facts, compute impressions.
Version and migrate your schema. We bumped storage from v1 → v2 and wrote a migration path. Real agents hit state-schema drift too.

2. The tick — agents act over time, not just on input

The heartbeat is a setInterval in app.js, but the abstraction that matters is tick(minutes):

function tick(state, minutes, events) {
  state.hunger     = clamp(state.hunger     + TUNE.hungerPerMin * minutes, 0, 100);
  state.energy     = clamp(state.energy     - TUNE.energyPerMin * minutes, 0, 100);
  state.boredom    = clamp(state.boredom    + TUNE.boredPerMin  * minutes, 0, 100);
  state.loneliness = clamp(state.loneliness + TUNE.lonelyPerMin * minutes, 0, 100);
  state.happiness  = clamp(state.happiness  - TUNE.happyDrift   * minutes, 0, 100);
  // threshold events: "I'm hungry!", "I'm exhausted!" → push into `events`
}

Because everything is expressed in minutes elapsed since the last tick, I can:

Fast-forward 240 minutes in a test and assert hunger === 100.
Catch up on time the user was away — on load, compute (now - lastTick) / 60000 and run one big tick.

This is the same "what changed since I last looked?" pattern that powers background workers, schedulers, and session-based agents. Autonomy also runs on a slower, per-pet clock (lastAutonomy), so it feels like life unfolds gradually rather than flooding the feed.

3. Personality = parameters, not hardcoded lines

Every pet gets an archetype with a trait weight profile. Autonomy rolls against these weights. Two pets share one behavior system but produce wildly different lives.

const TRAIT_PROFILES = {
  exuberant: { curious:.5, playful:.9, excited:.9, mischievous:.4, restless:.6, affectionate:.4, anxious:.1, annoyed:.2 },
  curious:   { curious:.95, playful:.5, excited:.5, mischievous:.3, restless:.4, affectionate:.3, anxious:.3, annoyed:.2, discoverer:.9 },
  gruff:     { curious:.3, playful:.3, excited:.2, mischievous:.5, restless:.2, affectionate:.3, anxious:.2, annoyed:.8 },
  ethereal:  { curious:.6, playful:.2, excited:.3, mischievous:.2, restless:.1, affectionate:.5, anxious:.4, annoyed:.1, dreamy:.9, discoverer:.7 },
  // ...
};

I tested this directly: a gruff/mischievous pal mostly caused mischief and got annoyed; an ethereal/anxious pal mostly felt lonely, checked on me, and got dreamy. Same code, different creatures.

The lesson: character diversity comes from parameters, not from special-casing. This is exactly how you model agent personas, tone, and autonomy thresholds at scale.

4. Memory with a retention policy

"Memory" for an agent is not magic — it's a data structure with a retention policy. Familiars memories are a capped, importance-ranked list:

function addMemory(state, type, text, importance) {
  state.memories.push({ type, text, when: Date.now(), importance: importance || 50 });
  state.memories.sort((a, b) => b.importance - a.importance || b.when - a.when);
  if (state.memories.length > TUNE.memoryCap) state.memories.length = TUNE.memoryCap;
}

Significant events (first adoption, "I love you", a discovery) carry high importance; mundane ones fade. Retrieval filters by relevance to the current context — the same recency + importance + relevance pattern behind RAG and long-term memory systems.

5. The reward loop: XP, levels, and learned preferences

Every care action and chat message feeds relationship XP → levels → tier icons, and also learns preferences that change the pet over time:

engine.learnPreference(pet, "interaction", action, action === "play" ? 2 : 1);
// → "loves playtime", "snack enthusiast", "a known troublemaker" 💚/💔

This is a closed feedback loop: state → behavior → reward → state. It's what makes an agent feel like it changes rather than just responds.

6. Hybrid chat brain with graceful fallback

The chat has two brains behind one interface. A deterministic rule engine (fully offline, ~15 intents, testable) and an optional OpenAI-compatible LLM (OpenAI, Ollama, LM Studio, Groq, OpenRouter…).

// app.js — sendChat()
const cfg = llm.loadConfig();
if (llm.isUsable(cfg)) {
  doLLM();                                    // call the endpoint
} else {
  setTimeout(doLocal, 500 + Math.random() * 400);   // deterministic rule engine
}

The LLM path works by serializing the pet's entire state into the system prompt — personality, needs, relationship, memories, plus the last ~10 messages. That's how the model "stays in character" and "remembers": we feed it the truth.

// llm.js — isUsable()
function isUsable(cfg) {
  if (!c.enabled) return false;
  if (!cfg.baseUrl || !cfg.model) return false;
  const isLocal = /localhost|127\.0\.0\.1|0\.0\.0\.0|\.local/.test(cfg.baseUrl);
  if (isLocal) return true;                   // no API key needed
  if (cfg.apiKey && cfg.apiKey.trim()) return true;
  return false;
}

And crucially, on any error (CORS, bad key, endpoint down) it falls back to the local engine with a toast — never a crash. I verified this end-to-end against a mock endpoint: a CORS-blocked call logged a warning, showed a toast, and switched to local.

The lesson: context injection is the core of conversational agents. The quality of an LLM agent is mostly determined by what you put in context and how you frame it — not the model.

7. Autonomy — emergent "aliveness" from small weighted rolls

Here's where pets go from interactive toys to independent creatures. Each autonomous behavior is a simple weighted roll that pushes to a feed, sometimes forms a memory, and nudges needs:

// A mischievous pet occasionally causes trouble
if (traits.mischievous > 0.4 && roll(traits.mischievous, timeMult * 0.6)) {
  pushActivity(state, "mischief", "🦹",
    `${name} hid your ${pick(["keys","sock","pen","headphones","favorite mug"])}. It's a secret now.`);
  addMemory(state, "mischief", `${state.name} got up to a little mischief while ${state.adopter} was away.`, 50);
}

// An anxious pet checks on you
if (traits.anxious > 0.3 && roll(traits.anxious, timeMult * 0.7)) {
  pushActivity(state, "anxious", "😟",
    `${name} called out to check if you were okay.`);
}

Pets leave personality-flavored notes, discover things, get excited, and nap. Every action lands in a persistent activity feed ("What your pals got up to while you were away"), timestamped with their avatar — so you return to find the pet that hid your keys, and the one that sat by the window missing you.

The lesson: "intelligence" and "personality" in agents usually emerge from composition, not from one algorithm. Design each mechanism to be tiny, and let them interact.

Testing the brain independent of the UI

Because the core is pure functions, testing is clean:

// Fast-forward a pet and assert it acted autonomously
const st = engine.freshState(pal, { petName: "Nova", adopterName: "Sam" });
st.hunger = 90; st.boredom = 80;
engine.autonomyPass(st, pal, persona, 240);
assert(st.activityFeed.length > 0);

And full-browser E2E via Playwright headless Chromium drives the whole flow — adoption → chat (local and LLM against a mock endpoint) → autonomy → feed rendering — asserting zero console/page errors and no horizontal overflow. This is the difference between hobby code and code you can trust: if you can't fast-forward time and assert "hunger went up, a memory was created, the feed updated," you'll be debugging blind.

Run it yourself

git clone https://github.com/harishkotra/familiars.git
cd one-line-agent-ideas
python3 -m http.server 8080   # or just open index.html
# open http://localhost:8080

No install. No build. No backend. All data lives in the visitor's localStorage.
Two flat, gradient-free themes: the warm Sunny Side Up default and the retro 🧊 Brain Freeze pixel mode.
To enable the LLM, click AI in the header and point it at any OpenAI-compatible endpoint.

What to build next

Familiars is intentionally small so you can read all of it in an afternoon. Great next steps: multi-pet interactions (pets playing with each other), schedules and rituals tied to wall-clock time, a JSON export so pets survive across devices, daily streaks, WebAudio sound design, and a PWA manifest for full offline installability.

The throughline for any builder: build your agent's brain as pure functions over a state object, drive it with a time model, parameterize personality, give it memory with a retention policy, and let autonomy emerge from small weighted rolls. The rest — the LLM, the UI, the fancy stuff — is just a host for that brain.

Code & more: https://www.dailybuild.xyz/project/230-familiars

Building Tracewood: Official HydraDB Cloud SDK Integration, Architectural Constraints, and Microservice API Networks

Harish Kotra (he/him) — Fri, 21 Aug 2026 12:42:40 +0000

Today, in our grand finale for Hack Hydra, we transformed Tracewood from a local prototype into a production-grade product built around HydraDB’s Official v2 Cloud API & SDK (@hydradb/sdk).

Here is everything we implemented, architectural patterns we unlocked, and how HydraDB acts as the foundational context engine for multi-agent software development.

1. Connecting Tracewood to Official HydraDB Cloud API (`@hydradb/sdk` v2)

Rather than assuming local database behaviors, we transitioned Tracewood to interface directly with HydraDB’s official v2 Cloud API (https://api.hydradb.com).

We built a dual-engine architecture in src/database/hydraCloud.ts that runs locally first while seamlessly syncing to Cloud HydraDB whenever HYDRA_DB_API_KEY is present in .env:

Workspace Provisioning (client.databases.create): Auto-provisions database workspaces with structured custom metadata schemas (project, topic, intent, provider).
Readiness Polling (client.databases.status): Polls until readyForIngestion evaluates true before streaming context.
Dual-Store Ingestion (client.context.ingest): Ingests document ASTs, package lockfiles, and READMEs under type="knowledge", while extracting developer coding habits and architectural decisions under type="memory".

2. Architectural Constraint Invalidation Engine (`Constraint` Nodes & `VIOLATES` Edges)

AI agents refactoring code often violate historical architectural rules established weeks ago in another repository or documented in READMEs.

We built an Architectural Constraint Graph in HydraDB:

Automatic Extraction: AST and document parsers extract constraint rules (e.g., "Auth tokens must expire in 15 mins", "RateLimiter required on public endpoints").
Graph Invalidation Traversal: When an agent session refactors code, HydraDB checks historical constraint nodes (Constraint -> VALIDATES -> Symbol -> REVISED_BY -> Session).
Automated Prevention: If an edit conflicts with a historical rule, HydraDB establishes a VIOLATES edge, alerting the developer with a glowing 3D red warning in Tracewood and blocking invalid agent edits via MCP!

3. Cross-Repository Microservice API Dependency Graph

Package managers (package.json) only track direct code imports—failing to track implicit HTTP, gRPC, or TRPC route dependencies across microservices.

Tracewood now parses AST route handlers across all local repositories:

Endpoint Nodes: Extracted HTTP routes (GET /api/v1/users, POST /auth/login).
EXPOSES & CONSUMES Edges: Graph edges linking (Project)-[:EXPOSES]->(Endpoint) and (Project)-[:CONSUMES]->(Endpoint).
Impact Simulation: Changing an API handler in your backend repo immediately reveals every frontend or sibling microservice on your machine exposed to breaking changes.

4. Dual-Store Context Fusion (`type: "all"`) & Feedback Loop (`POST /feedback`)

Most AI memory tools isolate static docs from chat histories. Tracewood leverages HydraDB’s unified Dual-Store Retrieval:

Unified Querying: Queries Knowledge and Memories in a single request (POST /query with type: "all" and mode: "thinking").
Explicit Feedback Signal Tuning: When developers select search results or accept agent recommendations in Tracewood, we execute client.feedback.submit() with explicit ratings (positive/negative) and source: "agent" to dynamically tune retrieval rankings.

5. Full 10-Agent Telemetry Scanner & Settings Overhaul

We updated our agent detection engine to check candidate paths across macOS/Linux, auto-detecting all 10 major AI coding harnesses:

Claude Code CLI (~/.claude)
Google Gemini CLI / Antigravity (~/.gemini)
Cursor IDE (~/.cursor & Application Support/Cursor)
OpenAI Codex CLI (~/.codex)
GitHub Copilot (~/.copilot)
Windsurf / Codeium (~/.codeium)
Cline / Roo Code (VS Code globalStorage)
Aider CLI (~/.aider)
Continue.dev (~/.continue)
Pi / CommandCode / Factory (~/.pi)

Conclusion: Building Software as a Living 3D Garden

Tracewood demonstrates that the future of multi-agent software development isn't flat text logs or isolated chat boxes—it's an interconnected, queryable context graph.

With HydraDB at its core, Tracewood brings transparency, supply-chain safety, and long-term memory to AI pair-programming.

Code & more: https://www.dailybuild.xyz/project/229-tracewood

Building ScopedAgent: A Production Template for Knowledge-Scoped AI Agents

Harish Kotra (he/him) — Thu, 20 Aug 2026 18:32:31 +0000

How to build production-ready customer support AI agents in Next.js 14 & LangChain.js that never hallucinate off-topic code, respect strict knowledge boundaries, and log out-of-scope user attempts across OpenAI, Anthropic, Gemini, Ollama, LM Studio, and custom endpoints.

The Problem With Generic RAG Starter Kits

Standard Retrieval-Augmented Generation (RAG) starter kits often suffer from a major flaw: unbounded scope.

When a user asks a support bot for your SaaS product to "write a Python script to scrape a website" or "explain binary search trees", traditional RAG systems still query the vector database and try their best to answer—often hallucinating or generating arbitrary code that has nothing to do with your company.

ScopedAgent addresses this problem by introducing a Pre-Retrieval Scope Guard Layer before vector search or document ingestion occurs.

Core Architecture & Flow

Every prompt follows this strict workflow:

Key Technical Implementation

1. Scope Guard Classifier (`lib/agent/scope-guard.ts`)

Before touching ChromaDB or document stores, the query is pre-classified into allowed vs. refused categories (writing_code, generic_technical, off_topic, jailbreak_attempt).

const ClassificationSchema = z.object({
  category: z.enum([
    "in_scope",
    "writing_code",
    "generic_technical",
    "competitor_analysis",
    "personal_advice",
    "off_topic",
    "jailbreak_attempt",
  ]),
  confidence: z.number().min(0).max(1),
  reasoning: z.string(),
  sanitized_query: z.string(),
});

export async function classifyQuery(
  query: string,
  context: string,
  config: Config,
  llm: BaseChatModel
): Promise<Classification> {
  const chain = SCOPE_GUARD_PROMPT.pipe(
    (llm as any).withStructuredOutput(ClassificationSchema)
  );
  return chain.invoke({
    company: config.agent.company,
    allowed_topics: config.scope.allowed_topics.map((t) => `- ${t}`).join("\n"),
    query,
    context: context || "None",
  });
}

2. Multi-Provider AI Inference Engine (`lib/providers/index.ts`)

Developers and end users can pick their preferred AI model directly in the UI header or configuration:

OpenAI: gpt-4o, gpt-4o-mini
Anthropic: claude-3-5-sonnet
Google Gemini API: gemini-1.5-flash
Local AI: Ollama (http://localhost:11434), LM Studio (http://localhost:1234), and custom OpenAI-compatible endpoints.

3. Single-File Developer Configuration (`scopedagent.config.ts`)

Developers only edit one configuration file to define their agent's scope rules, company identity, and custom refusal responses:

export default defineConfig({
  agent: {
    name: "Aria",
    company: "Northpeak Software",
    role: "Customer Support Assistant",
  },
  scope: {
    allowed_topics: [
      "product features and how-to questions",
      "pricing and plan comparisons",
      "billing and refund policy",
    ],
    refused_categories: [
      "writing_code",
      "generic_technical",
      "off_topic",
      "jailbreak_attempt",
    ],
    confidence_threshold: 0.72,
    fallback_message: "I don't have enough info on that. Contact support@northpeak.io.",
  },
});

Admin Analytics Dashboard (`/admin`)

Every out-of-scope attempt is persisted to SQLite (lib/db/queries.ts), giving developers real-time metrics on:

Top refused categories (Code Gen vs. Competitor Analysis vs. Off-topic).
Raw prompt logs and classification confidence scores.
Document re-indexing controls to update knowledge on the fly.

Try It Out & Contribute

Repository: https://github.com/harishkotra/scopedagent

DEV Community: Harish Kotra (he/him)

How I Built a Slack Bot That Flags YC Founders Before YC Even Announces Them

The core idea: two kinds of signal

Architecture

Choosing the data sources (the part that took the longest)

State management & duplicate detection (the boring part that matters)

The early-detection classifier (the fun part)

The alert model & Slack delivery

Persistence & ops

Testing

What's next / how to extend

Strata: I Got Tired of Retrieval-Based Code Intelligence, So I Built Something Else

The core idea: layers

L0: Structure without LLM

L1: Function summaries at scale

L2: Map-reduce rollups

L3: Architecture drift

L4: Dead code with confidence tiers

L5: Security attention map (not a vulnerability scanner)

L6: History and philosophy

The visualization

The cache architecture in full

What I learned

Running it yourself

PRION: Simulating 100,000 Minds Spreading a Rumor — With an LLM That Never Touches the Hot Path

1. The premise

2. Architecture: the LLM is a compiler, never a runtime

3. Tier 0: 100,000 agents with zero objects

The 20× matmul lesson

Non-consensus is structural, not tuned

4. Tier 1: text becomes parameters, once per inoculation

5. Tier 2: the text mutates as it travels

6. The wire: binary deltas at 33 KB/tick

7. War stories: what actually broke

8. What the simulation actually shows

9. What we'd build next

10. Closing thought

Building the AI Arms Race: A Reproducible Experiment to Test Whether LLMs Evolve Strategies When They Compete

The Research Question

The Game: HIDDEN SIGNAL

The Information-Isolation Boundary

Anti-Contamination: Don't Tell Them to "Adapt"

The LLM Client: The Hardest Lesson

Architecture

Module map

Deterministic Metrics

Statistical Analysis & the Arms-Race Signature

Controls That Make It Publishable

What the Smoke Test Taught Me

Reproducibility

Try It

Dead Reckoning: Resumable gRPC Server-Streaming for LLM Inference

The problem: gRPC streaming has no checkpointing

The insight: this is a checkpointing problem, not a retry problem

The protocol

Architecture

The journal: the heart of it

The Generate handler: resume vs. fresh

The bridge: surviving the "Kill Stream" moment

The CLI client: dead reckoning in the terminal

The benchmark: honest numbers from a real LLM

Prior art

What's next (and where you can help)

I built a CLI that turns a research paper into a working repo — using a persistent "implementation contract"

Self-Play Data Factory

Table of contents

Why this exists

Features

Architecture

The data flow

Technologies

Project layout

Quickstart

Point it at a real model (DeepSeek v4 flash)

CLI reference

The verifier algorithm — the heart of it

1. Letter scale A–T (20 letters), not digits

2. Logprob extraction

3. Expectation over the distribution

4. Criteria decomposition

1. Connecting Tracewood to Official HydraDB Cloud API (`@hydradb/sdk` v2)

2. Architectural Constraint Invalidation Engine (`Constraint` Nodes & `VIOLATES` Edges)

4. Dual-Store Context Fusion (`type: "all"`) & Feedback Loop (`POST /feedback`)

1. Scope Guard Classifier (`lib/agent/scope-guard.ts`)

2. Multi-Provider AI Inference Engine (`lib/providers/index.ts`)

3. Single-File Developer Configuration (`scopedagent.config.ts`)

Admin Analytics Dashboard (`/admin`)