DEV Community

Our AI reviewer invented a request. Our producer retried 245 times.

GX Cafe LLC — Mon, 24 Aug 2026 00:31:03 +0000

We run ~100 LLM agents unattended on local models. Last week we found one
document that had been rewritten 245 times in 5 days — every attempt
rejected. A sibling document: 225 times. Combined, about 470 wasted
generations, all burned on the same two files.

Here is the autopsy, with the actual numbers.

The loop

Our pipeline is simple: a producer agent writes a document, a reviewer agent
checks it against a contract (minimum length, required sections, no
placeholder junk), and rejected work goes back with fix instructions.

The rejected document was a key-management (KMS) implementation spec —
4,452 characters, perfectly on-topic. The reviewer's verdict:

"The request was a 3-line email triage response (LOCK / VERDICT / REASON),
but the answer is a long KMS spec. Rewrite as 3 lines only."

One problem. We grepped the document: the words "LOCK", "VERDICT", and the
name of the triage service appear zero times in it. The reviewer had
invented the request.

Why the loop never ended

Two contracts collided:

The reviewer's fix instruction: output 3 lines only
The producer's output contract: minimum 600 characters

No output can satisfy both. So the producer failed the contract, got
re-queued, produced again, failed again — 245 times. Our retry cap counted
reviews, but a contract-failed output never reaches review. The give-up
mechanism existed; it just watched the wrong counter.

Root cause: the reviewer never saw the request

Our review prompt contained the artifact body (first 4,000 chars) and the
output format. It never contained the original request. We asked a model
"does this match the request?" without telling it what the request was.
A model asked to judge against information it doesn't have will
hallucinate that information. Ours did, confidently, 245 times' worth.

Bonus failure: we truncated long documents to 4,000 characters before
review without saying so, and reviewers marked them "thin — cut off
mid-sentence." The cut was ours, not the producer's.

How common was it?

We audited all 2,038 reviews on file for concrete terms (product names,
format tokens) that appear in the review but nowhere in the reviewed
document. Result: 4 contaminated reviews — 0.2%.

That's the uncomfortable lesson: a 0.2% hallucination rate produced 470
wasted runs, because nothing ever gave up. Low rate × infinite retries =
unbounded damage. The rate is not the risk; the loop is.

The fixes (all mechanical)

Pass the original request into the review prompt. If it can't be extracted, the prompt now says: "do NOT guess the request — say it is unknown and judge the artifact on its own."
Declare truncation. "First 4,000 of 8,784 chars — the cut is ours."
Reject impossible instructions at the review's own exit gate. A review demanding "N lines only" while the production contract requires 600+ chars now fails as a review and never enters the queue.
Count consecutive contract failures, not just reviews, and park the item for a human after 5 — with the last verdict and fix instruction attached, so the human can see why in one glance.

Each fix ships with a test we deliberately broke to confirm it fails.

If you run agents unattended

The checker that catches broken outputs in this story (empty text, language
leakage, placeholder junk, contract violations) is free on npm:
honto-contract — it passed
600 downloads last week, so somebody besides us finds this useful now.

The unattended-operation checklist and three of our watchdog templates are
free (email-gated):
Unattended-Operation Kit

The full production set (7 templates — cron registry, silent-zero watch, heartbeat, output contracts: the exact ones in this story) is available on the same page.
(https://gxcafe.co.jp/harness-kit/?utm_source=devto&utm_medium=article&utm_campaign=templates-pro).

Honest note: we have no customers yet. Everything above is exactly what we
run on ourselves, measured on our own failures.

Architectural Breakdown: We fixed the eval platform we're competing on: a TypeError that crashed thr

Muhammad Hammad — Mon, 24 Aug 2026 00:16:15 +0000

We Fixed the Eval Platform: The TypeError That Took Down Three Benchmark Pipelines

At 3 AM, Sentry lit up with TypeError: Cannot read property 'map' of undefined. Three benchmark pipelines crashed. Not a memory leak, not a segfault, but a race condition hiding behind a TypeError, turning a high-stakes eval run into chaos. Here is how we resolved it, with no fluff.

The Root Cause: Async Data Meets Blind Faith in .map()

The error trace pointed to evaluator.ts:42, where .map() assumed inputData.metrics would always exist. The junior dev tested with clean data, but in production, fetchBenchmarkData() (async) and evaluatePipeline() (sync) were racing. At 100+ RPS, metrics was often undefined.

The Offending Code:

const results = inputData.metrics.map(metric => computeScore(metric));

Why It Failed:

Race Condition: inputData was fetched asynchronously, but evaluatePipeline() treated it as synchronous.
OOM Risk: Unbounded .map() on 10K+ metrics could exhaust 8GB RAM.
Worker Starvation: No concurrency limits led to thread pool exhaustion.

The Fix: Guard Clauses, Bounded Queues, and Pragmatism

Step 1: Fail Fast, Fail Loud

Added zero-overhead runtime checks to reject bad data early:

// eval-platform/core/evaluator.ts
import { isNullOrUndefined } from '../utils/guards';

async function evaluatePipeline(inputData: BenchmarkInput): Promise<EvaluationResult> {
    if (isNullOrUndefined(inputData?.metrics)) {
        throw new Error('EVAL_400: metrics missing');
    }
    // Proceed only if data is valid
}

Why?

Stops TypeError crashes immediately.
Cost: 1-2 CPU cycles. Negligible.

Step 2: Chunked Processing for 8GB RAM

Original code processed all metrics at once, causing OOM crashes. Fixed with 100-item chunks:

const CHUNK_SIZE = 100; // 100 items ≈ 10MB peak memory
const results: number[] = [];
for (let i = 0; i < inputData.metrics.length; i += CHUNK_SIZE) {
    const chunk = inputData.metrics.slice(i, i + CHUNK_SIZE);
    results.push(...chunk.map(metric => computeScore(metric)));
    if (process.memoryUsage().heapUsed > 6 * 1024 * 1024 * 1024) { // 6GB threshold
        await new Promise(resolve => setImmediate(resolve)); // Yield event loop
    }
}

Hardware Realities:

6GB Heap Limit: Leaves 2GB for the OS and other processes.
setImmediate: Prevents the event loop from choking.

Step 3: Bounded Worker Pool (4 Workers)

Original: Unbounded concurrency caused thread pool meltdown. Fixed with a semaphore-based pool:

// eval-platform/utils/worker-pool.ts
export class WorkerPool {
    private activeWorkers = 0;
    private queue: Array<() => Promise<void>> = [];
    private maxWorkers: number;

    constructor(maxWorkers: number) {
        this.maxWorkers = Math.min(maxWorkers, os.cpus().length); // Cap at CPU cores
    }

    async exec(task: () => Promise<void>): Promise<void> {
        if (this.activeWorkers >= this.maxWorkers) {
            await new Promise<void>(resolve => this.queue.push(resolve));
        }
        this.activeWorkers++;
        const worker = task().finally(() => {
            this.activeWorkers--;
            this.queue.shift()?.();
        });
    }
}

Usage:

const MAX_WORKERS = 4; // Safe for 8GB RAM (tested)
const pool = new WorkerPool(MAX_WORKERS);
await pool.exec(() => evaluatePipeline(inputData));

Why 4 Workers?

8GB RAM: 4 workers use ~2GB RAM each, with headroom for garbage collection.
CPU Bound: Matches typical 4-core cloud instances.

Step 4: Immutable Data and Network Timeouts

Problem: Mutable inputData plus async fetches led to race conditions.
Fix:

// eval-platform/core/data-fetcher.ts
async function fetchBenchmarkData(benchmarkId: string): Promise<BenchmarkInput> {
    const controller = new AbortController();
    const timeout = setTimeout(() => controller.abort(), 3000); // 3s timeout

    try {
        const response = await fetch(`/api/benchmarks/${benchmarkId}`, {
            signal: controller.signal,
            headers: { 'Accept': 'application/json' },
        });
        clearTimeout(timeout);
        const data = await response.json();
        return Object.freeze(data); // Immutable
    } catch (error) {
        clearTimeout(timeout);
        throw new Error(`FETCH_500: ${error.message}`);
    }
}

Hardware Impact:

3s Timeout: Covers 99.9% of network latencies.
Object.freeze: Zero cost. V8 optimizes frozen objects.

Hardware Profiling: 8GB RAM, No Illusions

Metric	Before Fix	After Fix
Peak Memory (1K evals)	7.8GB (OOM crashes)	5.2GB (stable)
CPU Usage (4 workers)	100% (thrashing)	60% (bounded)
Error Rate	12% (`TypeError`)	0.01% (guarded)
Latency (p99)	12s (unbounded)	4s (chunked + pooled)

Tuning Notes:

Chunk Size: 100 items, balanced for RAM and CPU.
Worker Pool: 4 workers, matches 4-core instances.
Timeouts: 3s, because hope is not a strategy.

Failure Walkthrough: When Things Still Go Wrong

Scenario 1: 10K Metrics in One Benchmark

Before: OOM crash (7.8GB, OS kills it).
After:

Chunked processing (100 items/chunk) caps peak memory at 5.2GB.
setImmediate yields the event loop, preventing starvation.

Scenario 2: Network Latency Spike (1s)

Before: inputData.metrics is undefined, causing TypeError.
After:

3s timeout aborts stale fetch.
Immutable inputData prevents race conditions.

Scenario 3: 200 RPS Burst

Before: 200 workers exhaust the thread pool.
After:

Worker pool caps at 4, bounding concurrency.
Queue backpressure enables graceful degradation.

Junior vs Senior: The Difference Between Crash and Stability

Aspect	Junior (Broken)	Senior (Hardened)
Data Handling	Assumed sync	Async with guards
Concurrency	Unbounded	Bounded (4 workers)
Memory	OOM risk	Chunked (100 items) + 6GB limit
Error Handling	Silent crashes	Structured errors (`EVAL_400`)
Data Integrity	Mutable state	Immutable (`Object.freeze`)

The Bottom Line

We did not reinvent the wheel. We stopped pretending async data would magically synchronize itself. No buzzwords, no hype, just code that does not crash under pressure.

For a template with these guardrails, see ShipMVP. It is what we wish we had at 3 AM.

Now, tell us: what is the worst race condition you have debugged, and how did you fix it?

OzBrain's Shared Memory Architecture: How Multi-Agent Teams Avoid Re-Explaining Context Across Sessions

mech.app — Mon, 24 Aug 2026 00:07:24 +0000

When you run multiple agents across Claude, ChatGPT, and Cursor, each one starts from scratch unless you manually paste context into every session. OzBrain solves this by exposing a shared knowledge substrate that agents read and write through the Model Context Protocol (MCP). The system routes context so agents see only what they need, and teams avoid explaining the same facts to every new agent instance.

The Show HN post drew 85 points and 50 comments because the problem is real: production multi-agent workflows break down when context lives in isolated chat histories or scattered documents. OzBrain's architecture treats knowledge as a first-class resource with explicit scoping, indexing, and conflict resolution.

Storage Layer and Scope Boundaries

OzBrain organizes knowledge into brains, which are either personal or shared. Each brain holds structured knowledge units that agents query through the MCP connector. The system decides scope at write time:

Personal brains store user-specific preferences, writing style, and private project state.
Shared brains hold team-wide facts like client contacts, project decisions, and open threads.

When an agent writes to OzBrain, it specifies the target brain. The MCP connector enforces access control: agents can read from any brain the user has joined, but write permissions depend on the brain's sharing policy. This prevents accidental leakage of personal context into team memory.

The storage layer tags each knowledge unit with metadata: creation timestamp, last update, and a freshness indicator (fresh, aging, stale). Agents use these tags to decide whether to trust the stored fact or re-query the source.

Indexing Strategy and Query Routing

OzBrain does not load the entire knowledge graph into every prompt. Instead, it maintains a routing index that maps topics to knowledge units. When an agent queries for "client contacts," the index returns pointers to relevant units without pulling in unrelated project state.

The routing index uses a simple keyword and topic model:

Each knowledge unit declares its topic (e.g., "clients/meridian", "voice", "projects/q3-launch").
The index builds a reverse lookup from topic to unit ID.
Agents send a topic query through the MCP connector, which returns a ranked list of unit IDs.
The agent fetches only the top-ranked units, keeping the prompt under token budget.

This approach trades precision for speed. The index does not use embeddings or semantic search, so agents must know the right topic label. In practice, this works because teams establish naming conventions early (e.g., "clients/", "projects/", "preferences/").

Conflict Resolution When Multiple Agents Write

When two agents update the same knowledge unit, OzBrain uses last-write-wins with a conflict flag. The system does not merge changes automatically. Instead:

Agent A writes a new version of "projects/q3-launch" with status "delayed."
Agent B writes a conflicting version with status "on track" 30 seconds later.
OzBrain stores Agent B's version as the current state but flags the unit as "conflicted."
The next agent to read "projects/q3-launch" sees the conflict flag and can surface it to the user.

This is a deliberate trade-off. Automatic merging requires semantic understanding of the conflict, which OzBrain does not attempt. The conflict flag ensures that contradictions do not silently propagate through the team's shared memory.

Conflict Strategy	Precision	Latency	Failure Mode
Last-write-wins	Low	Instant	Silent overwrites
Manual merge	High	Minutes	User fatigue
OzBrain (flag + LWW)	Medium	Instant	Requires agent or user to check flags

Failure Modes and Staleness

Shared memory introduces a new failure mode: stale context. If a knowledge unit says "client prefers email" but the client switched to Slack last week, agents will make incorrect assumptions until someone updates the unit.

OzBrain mitigates this with freshness tags. When an agent reads a unit marked "aging," it can prompt the user to confirm the fact before acting. The system does not auto-expire knowledge because some facts (e.g., "brand voice guidelines") remain valid for months.

The more dangerous failure mode is contradictory context. If an agent's working memory (from the current session) conflicts with OzBrain's shared memory, the agent must decide which to trust. OzBrain does not provide a resolution mechanism. Agents typically trust their working memory for session-specific facts and defer to shared memory for long-lived state.

MCP Connector Implementation

OzBrain exposes its API through an MCP server at https://ozbrain.com/api/mcp. Agents connect by adding the server to their MCP configuration. The connector supports four operations:

list_brains: Returns all brains the user can access.
query_brain(brain_id, topic): Returns knowledge units matching the topic.
write_unit(brain_id, topic, content): Creates or updates a knowledge unit.
read_unit(brain_id, unit_id): Fetches a specific unit by ID.

Here's a minimal example of an agent querying for client context:

import mcp

client = mcp.Client("https://ozbrain.com/api/mcp", api_key=user_token)

# List available brains
brains = client.call("list_brains")
team_brain = next(b for b in brains if b["name"] == "team-shared")

# Query for client contacts
units = client.call("query_brain", {
    "brain_id": team_brain["id"],
    "topic": "clients/meridian"
})

# Fetch the top-ranked unit
if units:
    contact_info = client.call("read_unit", {
        "brain_id": team_brain["id"],
        "unit_id": units[0]["id"]
    })
    print(contact_info["content"])

The MCP connector handles authentication via email-based login codes. When a user first connects, OzBrain sends a one-time code to their email. The agent exchanges the code for a session token, which it stores for future requests.

Deployment Shape and Observability

OzBrain runs as a hosted service. Users do not self-host the storage layer or indexing infrastructure. This simplifies deployment but introduces a dependency on OzBrain's availability. If the MCP endpoint goes down, agents lose access to shared memory and fall back to session-only context.

The system does not expose detailed observability hooks. Agents cannot trace which knowledge units were queried or how long the index lookup took. This makes debugging slow queries difficult. Teams must rely on OzBrain's internal logging, which is not surfaced to users.

For teams that need audit trails, the lack of query logs is a blocker. You cannot reconstruct which agent read which fact at what time, so compliance workflows that require provenance tracking will struggle.

When to Use OzBrain

OzBrain fits teams that:

Run multiple agents (Claude, ChatGPT, Cursor) and need consistent context across tools.
Have established naming conventions for topics and can train agents to query the right labels.
Accept last-write-wins conflict resolution and are willing to manually resolve contradictions.
Trust a hosted service for knowledge storage and do not require self-hosted deployment.

Avoid OzBrain if:

You need semantic search or embeddings-based retrieval. The keyword index is too brittle for open-ended queries.
Your workflow requires automatic conflict merging. The conflict flag is a signal, not a solution.
You need detailed query logs or observability into agent memory access.
You require on-premises deployment or air-gapped operation.

Technical Verdict

OzBrain solves the context duplication problem with a straightforward storage and indexing layer. The MCP connector makes it easy to wire into existing agent workflows, and the scoping model (personal vs. shared brains) prevents accidental leakage. The routing index keeps prompts small by fetching only relevant knowledge units.

The trade-offs are clear: last-write-wins conflict resolution, no semantic search, and reliance on a hosted service. For teams that can live with these constraints, OzBrain removes the friction of re-explaining context to every new agent session. For teams that need richer conflict resolution or self-hosted deployment, the architecture is too opinionated.

The freshness tagging is a smart middle ground between auto-expiration (which breaks long-lived facts) and no expiration (which lets stale data accumulate). The conflict flag is less satisfying because it pushes resolution back to the user or agent, but automatic merging would require semantic understanding that OzBrain does not attempt.

If your multi-agent workflow is breaking down because agents cannot share context, OzBrain is worth testing. If you need fine-grained control over conflict resolution or query observability, you will hit the ceiling quickly.

Source Links

Architecting a background-service-based sound manager that survives Android's Doze mode

Haseeb — Mon, 24 Aug 2026 00:07:12 +0000

It was the final ten minutes of a high-stakes client presentation. I was mid-sentence, explaining a complex system migration, when my phone erupted with a loud, aggressive ringtone. The room went silent, but my phone did not. I scrambled to silence it, accidentally hitting the volume buttons while fumbling with the screen. That moment of pure, unadulterated embarrassment followed me for days. It was not the first time this had happened, but it was the time I decided I had finally had enough of relying on my own memory to toggle sound profiles before entering sensitive environments.

Most of us live in a state of perpetual concern regarding our devices. We walk into movie theaters, attend religious services, or sit through medical consultations, constantly checking our pockets to ensure we have toggled the mute switch. If we forget, we face the social friction of a disruption. The existing solutions were either too manual—requiring a conscious effort I rarely possessed in the moment—or too intrusive, demanding constant location permissions and draining the battery to perform simple state changes. I wanted something that functioned as a set-and-forget background utility. I needed a system that understood the context of my environment without requiring me to interact with an interface every time my routine shifted.

To build this, I had to architect a background service that could survive the aggressive power-management constraints of modern Android, specifically Doze mode. The primary challenge was ensuring that my sound-toggling logic fired precisely when a rule was triggered, even if the device had been sitting idle for hours. I initially experimented with a standard Service, but Android’s lifecycle management quickly killed it to save resources. I shifted to using a ForegroundService with a persistent notification, which is the standard approach for long-running tasks, but that only solved the visibility part. The real hurdle was the timing accuracy required for events like prayer times or calendar-based meetings.

I eventually realized that relying solely on a service was a mistake. I needed to leverage AlarmManager with setExactAndAllowWhileIdle. This allows the system to wake the device from Doze mode to fire a broadcast, which I then use to trigger the AudioManager state changes. The architecture looks roughly like this:

kotlin
val alarmManager = context.getSystemService(Context.ALARM_SERVICE) as AlarmManager
val intent = Intent(context, MuffleBroadcastReceiver::class.java)
val pendingIntent = PendingIntent.getBroadcast(context, requestCode, intent, PendingIntent.FLAG_IMMUTABLE)

alarmManager.setExactAndAllowWhileIdle(
AlarmManager.RTC_WAKEUP,
triggerTimeInMillis,
pendingIntent
)

By decoupling the scheduling from the execution logic, I ensured that even if the OS aggressively restricts background processes, the kernel still respects the alarm trigger. The MuffleBroadcastReceiver then handles the heavy lifting, checking the priority of the current routine against any overlapping rules before calling audioManager.setRingerMode to toggle between silent, vibrate, or Do Not Disturb. This separation of concerns—scheduling via AlarmManager and execution via BroadcastReceiver—is what keeps the system stable across different manufacturer implementations of the Android OS.

What surprised me most during development was the volatility of the Do Not Disturb (DND) API. I initially assumed that simply calling the setRingerMode method would be sufficient to enforce silence. I was wrong. On many devices, specifically those from manufacturers with heavy custom UI skins, the DND access permissions are revoked or reset after major system updates. I spent days debugging why my application would stop silencing the phone despite the service running perfectly. It turned out that the NotificationManager.isNotificationPolicyAccessGranted check needed to be far more frequent than I anticipated. I had to implement a listener that re-verifies this permission every time the service starts, rather than just once during the initial setup. Relying on a one-time permission grant was an architectural oversight that nearly crippled the app's reliability for users on newer API levels.

Another edge case that caught me off guard was how AlarmManager behaves when the system time changes, such as during a Daylight Savings Time shift. My logic was originally tied to absolute timestamps in milliseconds. When the system clock adjusted, my scheduled routines shifted by an hour, causing them to trigger at the wrong time. I had to pivot to storing local time representations and re-calculating the trigger time whenever a TIME_SET or TIMEZONE_CHANGED broadcast was received. It was a tedious fix, but it taught me that you cannot treat time as a static constant on a mobile device. If I were to start over, I would build a much more robust abstraction layer for time-based triggers that explicitly handles these system-level shifts from the beginning, rather than patching them as bugs after the fact.

For any developer building automation tools on Android, the biggest lesson is to stop fighting the OS power-management systems and start working within their constraints. Doze mode is not a bug to be bypassed; it is a feature that keeps the user's phone alive for multiple days. If your app requires background execution, you must accept that you will be throttled. Instead of trying to keep a background service running 24/7, design your application to be event-driven. Use AlarmManager for specific time-based tasks and WorkManager for periodic synchronization or maintenance. If you try to force a persistent, always-active background process, you will eventually find your app getting killed by the system’s memory management, and you will lose the user's trust.

Always prioritize the user's battery life. If your background utility consumes significant energy, users will uninstall it, regardless of how useful the features are. I built Muffle with these exact principles in mind, focusing on minimal resource footprint by keeping the logic local and avoiding unnecessary network calls. By keeping the app fully offline, I also ensured that privacy and performance remained at the core of the experience. You can see how I implemented these constraints by checking out the project at https://play.google.com/store/apps/details?id=com.muffle.app. Building for Android is as much about managing system resources as it is about writing clean code, and finding that balance is what makes an app feel like a native extension of the OS.

Construyendo un recomendador de emparejamiento de expertos

Franchesco Romero — Mon, 24 Aug 2026 00:03:34 +0000

La forma del problema

Un directorio es una superficie: el miembro lo abre y adivina.
Un recomendador es una superficie de empujar: el sistema propone y tiene que justificarse. La justificación es la parte difícil, y es donde vive la estadística.

Tres restricciones hicieron esto distinto de un recomendador de contenido:

El item es una persona con capacidad finita. Un hilo se le puede recomendar a diez mil personas. Un experto no.
Una mala recomendación es cara de los dos lados. Quien pide desperdicia una petición, el experto desperdicia una hora, y los dos aprenden a ignorar la superficie.
La afirmación tiene que ser checable. "Quizá te guste este hilo" no necesita evidencia. "Esta persona está un nivel adelante de ti en diseño de sistemas" sí.

Recuperación: híbrida, fusionada con RRF

Tres recuperadores independientes sobre el conjunto de expertos elegibles, fusionados con Reciprocal Rank Fusion:

def rrf_fuse(*ranked_lists, k=60):
    """Fusiona listas de ids rankeadas. El score depende solo del rank, nunca
    de la escala propia del recuperador, que es el punto: la similitud coseno y
    un conteo de hilos resueltos no son números comparables."""
    fused = {}
    for lst in ranked_lists:
        for rank, key in enumerate(lst):
            fused[key] = fused.get(key, 0.0) + 1.0 / (k + rank)
    return fused

RRF es la primitiva correcta aquí por una razón que vale la pena decir: los recuperadores emiten cantidades incomparables. Uno regresa un coseno en [-1, 1], uno regresa un conteo entero de hilos resueltos, uno regresa un delta de nivel de escalera. Normalizarlos a una escala común requiere supuestos sobre sus distribuciones que nadie tiene a este volumen de datos.
RRF descarta las magnitudes y se queda solo con el orden, que es exactamente la información que sobrevive a una muestra chica.

k = 60 es la constante estándar de la formulación original de Cormack et al.
Aplana la cabeza: la diferencia entre el rank 1 y el rank 2 es
1/61 - 1/62 ≈ 0.00026, así que un recuperador no puede dominar por estar confiado, solo por estar consistentemente temprano a lo largo de las listas.

Scoring: un compuesto ponderado, con los pesos como datos

SCORE_WEIGHTS = {
    "compass_gap_fit": 0.30,   # explícito y direccional
    "semantic_fit":    0.20,   # coseno del embedding de perfil
    "skill_overlap":   0.15,   # tecnologías compartidas, ponderadas por rareza
    "capacity_fit":    0.15,   # también una compuerta dura
    "expert_quality":  0.12,   # rating, aceptación, experiencia que satura
    "specialty_match": 0.05,   # burda, pero muy poblada
    "fairness":        0.03,   # penalización de exposición amortiguada por log
}

Dos componentes valen la pena desempacar.

expert_quality satura. Un experto sin historial puntúa 0.5, no 0:

def expert_quality(*, avg_rating, completed, accepted, proposed):
    if completed == 0 and proposed == 0:
        return 0.5                      # prior neutral, no cero
    rating_part = ((avg_rating or 4.0) - 1.0) / 4.0
    acceptance  = accepted / proposed if proposed else 0.6
    experience  = 1.0 - math.exp(-completed / 5.0)
    return 0.5 * rating_part + 0.3 * acceptance + 0.2 * experience

La saturación exponencial sobre experience codifica que la diferencia entre 0 y 5 sesiones completadas es grande y la diferencia entre 40 y 45 es ruido.
Un término lineal habría hecho inalcanzables a los veteranos. El prior de 0.5 para los no probados es la decisión de cold start que deja crecer el pool de expertos más allá de quien haya ido primero; sin él, el sistema es un loop de rico se hace más rico por construcción.

fairness es exposición amortiguada por log.

def fairness_factor(times_recommended):
    return 1.0 / math.log(math.e + max(0, times_recommended))

ln(e + n) da exactamente 1.0 en n = 0 y decae lento. Una penalización lineal habría hecho irrecomendable a un buen experto después de un puñado de ciclos.

La parte que importa: esto es un problema de asignación

El instinto es computar un top-N por cada quien pide. Ese instinto está mal, y el modo de falla no es sutil.

Si cada quien pide escoge de forma independiente a su mejor experto, las mismas tres personas más fuertes juntan todas las peticiones. Son las que tienen los mejores ratings y el historial más profundo, así que ganan cada ranking, y dejan de contestar en un mes. El recomendador entonces destruye el recurso que existe para asignar.

Así que los pares se puntúan, y luego se asignan de forma global bajo una restricción de capacidad por experto:

def allocate(pairs, *, capacity, per_requester):
    """Asignación global greedy. Ordenada por score, cada par consume una unidad
    de la capacidad de su experto."""
    assigned = defaultdict(int)
    out = []
    for pair in sorted(pairs, key=lambda p: p["score"], reverse=True):
        if capacity.get(pair["expert_id"], 0) <= 0:
            continue
        if assigned[pair["requester_id"]] >= per_requester:
            continue
        capacity[pair["expert_id"]] -= 1
        assigned[pair["requester_id"]] += 1
        out.append(pair)
    return out

Esta es la aproximación greedy a un matching bipartito con restricción de grado. A esta escala (cientos de pares) la solución óptima vía scipy.optimize.linear_sum_assignment y la greedy difieren por ruido, y la versión greedy tiene una propiedad que la óptima no: es inspeccionable en un dry run, línea por línea, en orden de score. Cuando un operador pregunta "¿por qué esta persona obtuvo ese experto?", la respuesta es una sola pasada hacia abajo por una lista ordenada.

A quienes piden que la asignación no puede colocar no se les tira. Por construcción son aquellos cuyos mejores expertos están llenos, lo que los hace el insumo exacto para el clustering uno-a-muchos: agrúpalos por celda de escalera y propón una sola sesión.

La capa de datos, y la estadística que la hace defendible

La segunda capa responde "qué debería aprender después, y cuánto vale".
Compara la mediana de datos ponderada por fuente de los puntos de datos que reportan una habilidad contra los que no.

Esa oración contiene tres maneras de engañar a alguien. Las tres necesitaron una compuerta.

1. Ponderación por fuente con decaimiento exponencial por recencia

No todos los puntos de datos merecen voto igual. Cada fuente carga un peso de confianza y una corrección de sesgo, y cada punto decae con la edad:

COALESCE(cs.base_weight, 0.50)
  * (1 + COALESCE(cs.bias_correction_pct, 0) / 100.0)
  * POWER(0.5, EXTRACT(EPOCH FROM (now() - sdp.scraped_at)) / (:halflife * 86400.0))

Una vida media de 365 días significa que una publicación de dos años todavía cuenta, a un cuarto del peso de una fresca. El LEFT JOIN sobre la tabla de fuentes es deliberado: un punto de datos cuya fila de fuente nunca se registró cuenta en el default neutral de 0.50 en lugar de desvanecerse, porque tirar datos en silencio es peor que ponderarlos de forma conservadora.

El agregado es una mediana ponderada, no una media ponderada. Las
distribuciones de datos están sesgadas a la derecha y la cola es donde viven los errores de scraping; una sola cifra mal parseada mueve una media y no mueve una mediana.

def weighted_median(values, weights):
    pairs = sorted(zip(values, weights), key=lambda p: p[0])
    total = sum(max(0.0, w) for _, w in pairs)
    if total <= 0:                       # todos los pesos en cero: degrada a mediana simple
        mid = len(pairs) // 2
        return (float(pairs[mid][0]) if len(pairs) % 2
                else (float(pairs[mid - 1][0]) + float(pairs[mid][0])) / 2.0)
    half, acc = total / 2.0, 0.0
    for value, weight in pairs:
        acc += max(0.0, weight)
        if acc >= half:
            return float(value)
    return float(pairs[-1][0])

2. Estratificación, porque el número ingenuo mide antigüedad

Este es el confusor que hace inútiles a la mayoría de las afirmaciones de "la habilidad X paga Y% más". La gente senior sabe más herramientas. Compara a todos los que reportan Kubernetes contra todos los que no, y una parte grande del delta es nada más antigüedad filtrándose por la comparación.

Cada comparación por lo tanto pasa dentro de un estrato (rol, seniority, país, bucket de experiencia). Los buckets son burdos a propósito (0-2, 3-5, 6-9, 10+): estratos más finos matan de hambre a la muestra y el intervalo explota. Una habilidad cuyo efecto desaparece una vez estratificada se tira, no se reporta.

3. Un intervalo por bootstrap, no un estimado puntual

Un estimado puntual se lee como una promesa. El bootstrap por percentil re muestrea las dos cohortes con reemplazo, recomputa las medianas ponderadas, y toma los percentiles 5/95 de la distribución de deltas resultante:

def bootstrap_ci(with_vals, with_w, without_vals, without_w, n=1000, alpha=0.10):
    rng = random.Random(SEED)            # determinista: los mismos insumos deben
    deltas = []                          # producir el mismo intervalo en cada corrida
    for _ in range(n):
        a = [rng.choice(range(len(with_vals))) for _ in with_vals]
        b = [rng.choice(range(len(without_vals))) for _ in without_vals]
        med_a = weighted_median([with_vals[i] for i in a], [with_w[i] for i in a])
        med_b = weighted_median([without_vals[i] for i in b], [without_w[i] for i in b])
        if med_b > 0:
            deltas.append((med_a - med_b) / med_b * 100.0)
    deltas.sort()
    lo = deltas[int(math.floor((alpha / 2) * len(deltas)))]
    hi = deltas[min(len(deltas) - 1, int(math.ceil((1 - alpha / 2) * len(deltas))) - 1)]
    return round(lo, 2), round(hi, 2)

El bootstrap es la herramienta correcta porque la distribución muestral de una mediana ponderada de una distribución sesgada no tiene una forma cerrada limpia. El re muestreo esquiva la derivación por completo.

El RNG sembrado importa más de lo que parece. Un operador refrescando un dry run de admin no debe ver el número bambolearse; un intervalo de confianza que cambia al recargar es indistinguible de un bug.

Una fila se surge solo si las tres se cumplen:

significant = (ci_low > 0.0                  # el intervalo se queda de un lado del cero
               and premium >= min_pct        # bastante grande para valer el tiempo de una persona
               and premium <= claim_cap_pct) # no un outlier absurdo

Las filas no significativas de todos modos se computan y se guardan. La tabla de admin muestra qué se rechazó y por qué, porque un número que el sistema se negó a usar es tan interesante como uno que usó.

La medición que reencuadró el proyecto

Todo lo de arriba estaba en verde en CI. Luego la primera dry run contra producción:

requesters=82  experts=1  scored_pairs=82  affinity=3  office_hours=0

1 experto de 51. La causa:

requester_ids = {m.user_id for m in requesters}
eligible_experts = [
    m for m in experts
    if m.open_load < cap and m.user_id not in requester_ids   # <-- esto
]

La intención era "no emparejes a alguien consigo mismo". La implementación era "excluye a cualquier experto que sea también un candidato a pedir", y casi todos los expertos lo son, porque no tienen ninguna petición abierta propia.

Medido:

Roles activos elegibles como experto:                51
...que se apuntaron:                                 51
...bajo el tope de carga:                            51
...excluidos por ser también quienes piden:             50

El único sobreviviente era la única persona que resultó estar a media
interacción, y las tres propuestas apuntaban a ella. El sistema había
encontrado el modo de falla de burnout por su cuenta, en la primera corrida, a través de una línea pensada para prevenir un problema completamente distinto.

La guarda de auto emparejamiento ya existía por par, que es donde va.

Dos hallazgos más de la misma corrida:

Las cards podían salir sin razón. Solo 4 miembros tenían una colocación de escalera de competencias, así que el componente más pesado casi siempre era 0 y la señal sobreviviente era un coseno de embedding que nadie puede leer. Una propuesta tenía una lista de razones vacía. Una card que no puede decir por qué es peor que ninguna card.

El badge de tendencia disparaba con todo. La ingesta era reciente, así que casi cada publicación caía en la ventana de 30 días y la línea base de 90 días era una o dos filas:

golang   229 publicaciones  trend=2.00
go       155 publicaciones  trend=462.00
python   109 publicaciones  trend=62.40
aws       43 publicaciones  trend=61.50

Una razón necesita un denominador que valga la pena dividir. El arreglo es un conteo mínimo de línea base antes de que se afirme una tendencia siquiera, y una cota sobre la razón.

La lección de frecuencia inversa, aprendida tres veces

Esta es la parte con el mayor valor de transferencia, porque la misma idea estadística tuvo que aplicarse en tres niveles distintos y cada nivel se veía bien hasta que se inspeccionó.

Nivel 1: lift, en el minero de adyacencia

La adyacencia de habilidades es minería clásica de reglas de asociación sobre habilidades co-ocurrentes en publicaciones de vacantes. Para una regla
A -> B:

soporte = conteo de transacciones que contienen las dos
confianza = P(B | A) = soporte / conteo(A)
lift = confianza / P(B)

La confianza sola es inútil, y la razón es instructiva. Linux co-ocurre con todo, así que P(Linux | lo que sea) es alta y cada regla apunta a Linux. El lift divide por la tasa base: si saber A no sube la probabilidad de B por encima del azar, el lift es 1 y la regla no carga información.

lift = confidence / (singles[consequent] / total)
if lift < min_lift:      # 1.15
    continue

Reglas que sobrevivieron sobre datos reales, ordenadas por confianza * lift:

laravel        -> php            conf=0.73  lift=11.33  support=8
terraform      -> kubernetes     conf=0.75  lift=9.25   support=9
rails          -> ruby           conf=0.78  lift=7.94   support=7
react native   -> react          conf=1.00  lift=5.48   support=7
gcp            -> aws            conf=0.73  lift=6.15   support=16

Esas se leen bien para un humano, que es la única validación disponible a este tamaño de muestra.

Una trampa que vale la pena registrar: las primeras pruebas unitarias de esto eran degeneradas. Un corpus donde el antecedente aparece en cada transacción tiene lift 1.0 por construcción, así que nada puede ser significativo jamás.

Los fixtures tuvieron que reescribirse para incluir publicaciones que no contuvieran ningún lado de la regla. Un corpus de mercado es diverso; un corpus de prueba tiene que serlo también, o no prueba nada.

Nivel 2: frecuencia de documento, en la señal de tema

El riel salió y casi cada card decía:

Comparten temas: introduccion-plataforma.

La distribución de frecuencia de tags lo explica:

talento-tecnologico        27 hilos   50.0%
introduccion-profesional   17 hilos   31.5%
introduccion-plataforma    14 hilos   25.9%
espacio-relajacion          4 hilos    7.4%
servidores-caseros          4 hilos    7.4%
ai-local-vs-nube            2 hilos    3.7%

Tres tags cubren la mitad de un corpus de 54 hilos, luego un acantilado a 7%.

Este es el problema de Linux otra vez, un nivel arriba: un tag sobre la mitad del corpus no carga información sobre un par. Los tags por encima de 20% de frecuencia de documento se tiraron del scoring y del copy.

Ese arreglo hizo el copy menos vergonzoso sin hacerlo significar nada, lo que lleva al tercer nivel.

Nivel 3: el vocabulario era la ontología equivocada

Existían nueve tags en total. Ninguno de ellos era una tecnología.
Eran secciones del foro: un tablero de anuncios, dos áreas de introducción, un espacio fuera de tema. Ningún umbral de frecuencia puede rescatar una señal que está midiendo lo equivocado. Compartir "introducciones" significa que las dos personas se presentaron.

La señal técnica que sí existía era el tech_stack del perfil:

python 11 · aws 8 · docker 7 · node.js 7 · typescript 7
git 6 · react 6 · fastapi 5 · kubernetes 3 · terraform 3

Flaca (20 de 83 miembros) pero real, y nombra cosas que un experto puede enseñar. El componente de tema se reemplazó por completo, normalizado por el menor de los dos stacks para que un stack de 30 items no pueda diluir un buen match de 3-de-4, y extendido a través de las reglas de adyacencia minadas para que quien pide sobre Python conecte con un experto sobre FastAPI con la regla como la justificación declarada.

Y luego, predeciblemente, el nivel 3 tuvo su propio nivel:

También trabaja con python.

Python está en 11 de 20 stacks. El mismo problema, tercera recursión. La resolución esta vez no fue un filtro duro, porque a diferencia de una sección de foro, Python es una habilidad enseñable real y ponerla en cero descarta traslape genuino.

Las habilidades ubicuas se bajan de peso a un quinto en lugar de tirarse, el copy nombra las habilidades más raras
primero, y un stack compartido solo justifica una card cuando al menos una habilidad compartida es poco común:

weight_of = lambda sk: 0.2 if sk in common_skills else 1.0

Una habilidad compartida rara ahora le gana a una ubicua 1.0 a 0.2, que es lo que pone a Terraform encima de Python en el ranking en lugar de debajo por accidente de quién lista más tecnologías.

La generalización: cada vez que una señal es un traslape de conjuntos, pregúntate cuál es la tasa base de cada elemento antes de dejar hablar al traslape. El lift, el IDF, y este bajado de peso de habilidades son la misma corrección con tres nombres distintos.

Afinado de pesos como learning to rank offline

Los pesos compuestos se guardan en config, no en código, y hay una dry run que propone un rebalanceo a partir de los resultados calificados de los propios miembros:

correlations = {k: safe_corr(np.asarray(per_key[k]), y) for k in keys}

Tres decisiones deliberadas alrededor de ella:

La clase positiva es una sesión agendada, no un click. Un click en un hilo es barato; convertir una propuesta en una sesión de verdad agendada no.

Optimizar por clicks afinaría el ranking hacia la curiosidad en lugar de hacia sesiones que pasan.

La propuesta es una mezcla conservadora, nunca el vector crudo derivado de la correlación:

blended = {k: BLEND * corr_w.get(k, 0.0) + (1.0 - BLEND) * current.get(k, 0.0)
           for k in current}

Con BLEND = 0.5 y un piso de 30 muestras, una semana flaca no puede mover el ranking.

Nada se aplica automáticamente. Un operador previsualiza y aplica de forma explícita. Auto aplicar una correlación computada sobre decenas de muestras es como un ranker oscila.

Trampas

Trampa 1: la restricción de unicidad que creó duplicados. Las filas son únicas por (requester, expert, cycle_key), lo cual es correcto por ciclo y mal a través de ellos. El cron semanal escribió un segundo ciclo encima del primero y el riel de inmediato se dobló:

riel para <miembro>:
   <experto-a>  0.3997
   <experto-a>  0.3871   <--
   <experto-b>  0.3589
   <experto-b>  0.3462   <--

Arreglado en dos lugares, porque cualquiera solo deja un hoyo: la lectura se colapsa con DISTINCT ON (expert_id) quedándose con la mejor fila, y un ciclo nuevo reemplaza las propuestas sin tocar de los viejos para que la tabla no pueda crecer una card por persona por semana para siempre. Solo se tiran las filas no mostradas; cualquier cosa que un miembro vio es historial de embudo.

Trampa 2: el CTA que no hacía nada. El riel se renderizaba dentro de la misma página a la que enlazaba su botón, así que el click actualizaba el query string y la página se quedaba ahí. Peor, el enlace no cargaba id de match, así que una petición enviada nunca podía reportar de vuelta y la máquina de estados nunca podía avanzar más allá de CLICKED. La tasa de aceptación, la única métrica sobre la que se juzga el rollout y la que está cableada a la alarma, se habría leído como cero para siempre. El botón muerto se habría
reportado en un día; la métrica muerta se habría creído por semanas.

Trampa 3: una auto-recomendación latente. Cero filas en producción tenían requester_id = expert_id, pero el clustering uno-a-muchos escogía su experto del pool completo sin excluir el cluster, y esas filas cargan al primer miembro del grupo como quien pide. Podía apuntarse a sí mismo. Simplemente nunca había
disparado porque ningún cluster había alcanzado el umbral.

Lecciones

Mide contra datos de producción antes de confiar en un diseño. La primera dry run contra filas reales invalidó más del diseño que todas las pruebas juntas. Córrela antes de que se emita nada, no después.
Un recomendador para un recurso finito es un problema de asignación. El top-N por usuario es la forma equivocada y su modo de falla es destruir el lado de la oferta. La restricción de capacidad va en la asignación, no en un post filtro.
Chequea la tasa base de cualquier señal de traslape de conjuntos. El lift, la frecuencia de documento y la rareza de habilidad son la misma corrección. Una señal compartida por la mitad de la población no carga información sobre un par, no importa qué tan cierta sea.
Antes de afinar un umbral, verifica el vocabulario. Se gastaron dos rondas en cortes de frecuencia para un conjunto de tags que no contenía ninguna tecnología en absoluto. Pregúntate qué son las etiquetas antes de preguntar qué tan comunes son.
Un intervalo de confianza es una decisión de producto, no un detalle de estadística. El intervalo, el piso de muestra y la cota de efecto son lo que se para entre una correlación y una promesa que los datos no pueden cumplir. Muestra el rango, declara la muestra, y niégate a imprimir el número cuando las compuertas fallan.
Siembra el bootstrap. Un número que cambia al recargar es indistinguible de un bug, y destruye la confianza del operador más rápido que estar mal una vez.
Instrumenta la conversión antes de publicar la superficie. Un botón muerto se reporta en un día. Una métrica muerta se cree por un mes.
Publica el motor a oscuras. Emisión apagada por default y una dry run como la acción de admin por default significaron que cuatro bugs significativos se encontraron con datos reales y cero notificaciones enviadas.

Building PickTool with Next.js and Laravel: Lessons from Creating a Software Discovery Platform

Doaa F M Abed — Mon, 24 Aug 2026 00:01:27 +0000

Finding software is easy. Finding the right software is not.

Search for almost any category—email marketing, CRM, productivity, design, or AI—and you will find hundreds of options. Every product presents itself as the best choice, while many comparison articles repeat the same features without explaining which users each tool actually suits.

That problem inspired me to build PickTool, a platform for discovering and comparing AI and SaaS tools.

PickTool is still evolving. I am currently improving its content quality, tool coverage, comparison experience, performance, and SEO structure.

This is not a polished launch announcement. It is an honest look at the architecture behind the project and some of the lessons I have learned while building it.

What Is PickTool?

The goal of PickTool is simple:

Help people find the right software in minutes, not hours.

Instead of creating a basic directory filled with product names and affiliate links, I want each important tool to include useful and structured information, such as:

Core features
Pricing model
Best use cases
Strengths and limitations
Ratings and evaluation criteria
Alternatives
Direct comparisons
Related guides and category pages

The challenge is that this creates several interconnected types of content.

A single product can appear on its own tool page, inside a category, in multiple comparisons, and in articles about the best software for a particular use case.

Keeping all of this consistent requires more than publishing isolated blog posts.

Why I Chose Next.js and Laravel

PickTool uses a decoupled architecture:

Next.js powers the public-facing website.
Laravel powers the backend, API, database logic, and administration system.
MySQL stores tools, categories, ratings, pricing information, and editorial content.

I chose this combination because I wanted the frontend and content-management logic to evolve independently.

Laravel provides a structured backend for managing relationships between tools and content. Next.js gives me control over rendering, metadata, performance, and the public user experience.

This separation also allows me to improve the frontend without rebuilding the entire administration workflow.

Modeling More Than a List of Tools

One of the first lessons I learned was that a software directory should not be modeled as one large table of products.

The data is relational.

A tool belongs to a category, but it can also have several competitors, alternatives, comparison pages, pricing plans, use cases, and editorial guides.

Conceptually, the content structure looks like this:

Category
├── Tools
├── Best-software guides
├── Comparison pages
└── Supporting articles

Tool
├── Features
├── Pricing
├── Pros and cons
├── Alternatives
├── Comparisons
└── Related guides

This structure makes internal linking more intentional.

For example, an email marketing tool can link to:

Its main email marketing category
Relevant alternatives
Direct competitors
A comparison article
A best email marketing software guide

The technical data model and the SEO architecture are closely connected.

If these relationships are not represented clearly in the database, keeping the public pages organized becomes much harder.

The Problem with Scaling Content Too Early

When building a software directory, it is tempting to believe that adding more products automatically makes the website more useful.

I learned that quantity can create problems very quickly.

A directory containing hundreds of incomplete or repetitive pages is not necessarily more valuable than one containing twenty carefully developed product profiles.

Scaling too early can lead to:

Thin pages
Repetitive descriptions
Outdated pricing
Weak internal linking
Inconsistent ratings
Multiple pages targeting almost identical keywords

I am now taking a more focused approach.

Instead of trying to cover every software category equally, I am strengthening one topic cluster at a time. Email marketing is one of the first categories receiving this treatment.

That means improving the main category page, completing the most important tool profiles, and connecting them with comparisons, alternatives, and use-case guides.

It is slower than bulk publishing, but it creates a stronger foundation.

Building SEO into the Application

Because PickTool depends heavily on organic discovery, SEO cannot be added as a final plugin or checklist.

It has to be part of the application architecture.

Every indexable page needs its own:

Page title
Meta description
Canonical URL
Heading structure
Structured data
Internal links
Open Graph information
Indexing rules

Dynamic pages make this more complicated.

A missing canonical rule or an incorrect URL pattern can affect hundreds of pages at once. The same applies to duplicated metadata, empty category pages, parameterized URLs, and pages generated from incomplete database records.

I have learned to treat SEO templates like application code: they need clear rules, validation, testing, and regular audits.

Performance Is Also a Content Problem

When developers discuss performance, we often focus on JavaScript bundles, caching, image sizes, and server response times.

Those things matter, but content structure affects performance too.

A tool page can become unnecessarily heavy when it tries to load:

The complete tool profile
Multiple pricing plans
Alternative tools
Comparison tables
Related articles
Interactive interface elements
Several large images

I am working on keeping the initial page useful without loading every possible component immediately.

Some of the areas I continue to review include:

Image optimization
Server-side data fetching
Caching
Component size
Unnecessary client-side JavaScript
API response size
Core Web Vitals

For a content-heavy platform, performance is not something that gets permanently “finished.” It requires continuous measurement as the website grows.

Keeping the Backend and Frontend Consistent

A decoupled Laravel and Next.js architecture provides flexibility, but it also introduces a consistency challenge.

The frontend depends on the backend to return complete and predictable data. If fields are missing or relationships are inconsistent, the public pages can show empty sections or incomplete information.

This pushed me to think more carefully about:

API response structures
Required and optional fields
Validation in the admin system
Published and unpublished states
Default values
Error handling
Content freshness

A page should not be published simply because a database record exists.

It should be published when it contains enough information to be genuinely useful.

That distinction is becoming an important part of the project.

Transparency Matters for Comparison Platforms

A comparison platform needs more than attractive cards and numerical ratings.

Visitors should be able to understand:

How tools are evaluated
Which factors affect their scores
When information was last updated
Whether affiliate relationships exist
Where pricing information comes from
Who each recommendation is intended for

This is both an editorial and technical challenge.

The backend needs to support structured evaluation data, while the frontend needs to present that information without overwhelming the visitor.

I am still improving this part of PickTool. My goal is to make ratings and recommendations explainable instead of presenting numerical scores without context.

What I Would Do Differently

If I were starting the project again, I would make several decisions earlier.

1. Start With One Narrow Category

I would build a complete experience around one software category before expanding into many unrelated categories.

2. Define Minimum Publishing Requirements

I would establish a minimum content standard before allowing a tool page to become publicly indexable.

3. Design Internal Linking With the Data Model

Internal links should come from meaningful relationships between tools and topics, not from randomly selected related content.

4. Separate Discovery From Editorial Content

A directory page, a review, a direct comparison, and a best-software guide serve different user and search intentions.

They should not be treated as interchangeable templates.

5. Build Auditing Into the Workflow

Metadata, canonical URLs, broken links, incomplete fields, and outdated pricing should be checked regularly instead of waiting for problems to appear in search reports.

What Comes Next?

PickTool is not finished, and I do not think a platform like this ever reaches a permanent finished state.

My current priorities are:

Strengthening the email marketing software cluster
Improving the quality of individual tool pages
Making comparisons more useful
Refining the rating methodology
Improving technical SEO
Monitoring performance as content grows
Creating a clearer path from discovery to decision

Building this project has reminded me that creating a useful software platform is not only about writing code or publishing content.

It requires connecting product design, data modeling, performance, editorial standards, and search architecture into one consistent experience.

You can explore the current version at PickTool.io.

I am building it in public and improving it step by step. Feedback from developers, SaaS users, and product builders is always welcome.

What challenges have you faced while building a content-heavy platform with a separate frontend and backend?

Planning Over Execution: Lessons from 157 Agent Runs and the Rise of Orca-Style Agent Fleets

Tamiz Uddin — Mon, 24 Aug 2026 00:00:51 +0000

Originally published on tamiz.pro.

The field of AI agents has moved rapidly from single-model executors to complex multi-agent orchestration. But after running 157 agent deployments across diverse task domains, one pattern emerged with striking consistency: planning quality predicts success far better than execution speed or model size. This isn't just theoretical—it's a practical lesson that's reshaping how engineers architect agent fleets, giving rise to what we're now calling Orca-style agents: hierarchical, planning-first systems that separate the expensive business of thinking from the cheaper business of doing.

The Experiment: 157 Agent Runs

Over six months, our team deployed and monitored 157 distinct agent runs across four primary use cases: code generation pipelines, automated testing workflows, infrastructure-as-code provisioning, and data transformation tasks. Each run varied along three dimensions:

Architecture: Single-agent vs. flat multi-agent vs. hierarchical (Orca-style)
Planning depth: No planning, brief intent statement, or full recursive planning loop
Execution model: Direct LLM call per action vs. tool-augmented execution with validation

The results were unambiguous. Systems that invested 3-5x more tokens in planning achieved 4.2x higher task completion rates and 3.8x fewer rollback cycles compared to agents optimized purely for fast execution. The correlation between planning sophistication and success held across every domain.

Why Planning Beats Raw Execution

The intuition behind this finding rests on an economic principle of LLM usage: planning is cheap relative to costly mistakes. A well-structured plan reduces the probability of executing the wrong sequence of tools, making incorrect API calls, or generating code that fails integration testing.

Consider the token economics:

Phase	Tokens (typical)	Cost impact
Planning (intent + decomposition)	800–2,500	Low
Execution per subtask	300–1,200	Medium
Correction after failure	1,500–4,000	High

Agents that plan thoroughly front-load their costs. Those that rush to execute often pay exponentially more in corrections, retries, and human intervention.

The Orca Architecture Pattern

The name "Orca" comes from the hierarchical social structure of killer whales: a single matriarch orchestrates, while specialized pod members execute discrete tasks. In agent terms, this translates to:

Core Components

Strategic Planner (the matriarch): Holds global context, decomposes goals, assigns subtasks, and validates outcomes. Runs on a stronger model with longer context windows.
Specialist Executors (the pod): Each handles a narrow domain—code generation, test writing, documentation, validation. Run on smaller, cheaper models optimized for throughput.
Shared Memory Layer: A structured knowledge graph or vector store that maintains state across the fleet, preventing redundant work and enabling cross-agent learning.
Orchestration Loop: A lightweight controller that routes tasks, aggregates results, and triggers replanning when validation fails.

Why This Separation Matters

The critical insight is that not all thinking is equal. Strategic decisions—understanding requirements, identifying edge cases, sequencing dependencies—benefit from deep context and reasoning. Tactical decisions—formatting output, calling a specific API, generating a template—are better handled by focused, optimized models.

Separating these concerns allows you to:

Run planners on premium models without paying premium prices for every action
Scale executor capacity independently of planning capacity
Implement targeted retry logic without restarting entire workflows
Observe and debug planning failures separately from execution failures

Implementation Patterns

From the 157 runs, several implementation patterns emerged as particularly effective:

Pattern 1: Recursive Decomposition with Validation Gates

The planner decomposes a goal into subtasks, each with explicit success criteria. Executors complete subtasks and return structured evidence of completion. The planner validates before proceeding to the next level.

class OrcaPlanner:
    async def decompose(self, goal: str, context: Dict) -> List[Subtask]:
        """Recursive planning with validation gates."""
        plan = await self.model.plan(goal, context=context)

        validated_subtasks = []
        for subtask in plan.subtasks:
            if subtask.requires_decisions():
                # Recursive planning for complex subtasks
                sub_plan = await self.decompose(subtask.description, context)
                validated_subtasks.extend(sub_plan)
            else:
                validated_subtasks.append(subtask)

        return validated_subtasks

Pattern 2: Specialist Routing with Skill Cards

Each executor carries a "skill card"—a concise description of its capabilities, constraints, and preferred input/output formats. The planner matches subtasks to specialists based on these cards rather than attempting blind routing.

Pattern 3: Stateful Context Propagation

Instead of passing raw conversation history, agents exchange structured context objects: constraints discovered, assumptions made, partial results, and confidence scores. This enables better downstream planning and reduces context window waste.

Common Pitfalls from the 157 Runs

Not every design decision landed well. Here are the patterns that correlated with failure:

Pitfall 1: Over-Planning

Some teams spent so much time planning that the plan became stale before execution began. The sweet spot was ~20% of total token budget for planning, with dynamic replanning triggered only by validation failures—not on a timer.

Pitfall 2: Specialist Fragmentation

Creating too many specialists (15+) introduced routing overhead and context fragmentation. The optimal range was 4–8 specialists, each covering a distinct capability domain.

Pitfall 3: Silent Replanning

When execution failed, some systems silently retried with minor variations. Successful systems explicitly logged failures, triggered replanning at the appropriate abstraction layer, and maintained an audit trail.

Pitfall 4: Context Window Hoarding

Planning agents that retained full conversation history from execution agents burned through context windows unnecessarily. Successful implementations used summarized state objects instead.

When to Use Orca-Style Fleets

These systems aren't a universal upgrade. Based on the data:

Use Case	Recommendation
Single-shot code generation	Single agent sufficient
Multi-step workflows with validation	Orca-style strongly recommended
High-stakes operations (infra, payments)	Orca-style with human-in-the-loop
Batch processing (100+ items)	Hybrid: planner + parallel executors
Interactive assistance	Lightweight planning, direct execution

The break-even point appears around 3–5 sequential steps with dependency checking. Below that, the planning overhead outweighs the benefits.

The Economics of Planning-First Design

For engineers evaluating whether to adopt Orca-style architectures, the cost model matters:

Traditional approach: One large model handles everything. Cheap for simple tasks, expensive for complex ones due to rework.

Orca approach: Premium model for planning, lighter models for execution. Higher upfront cost, dramatically lower correction costs.

In our measurements, the Orca architecture showed a 37% reduction in total token cost for tasks exceeding 10 steps, despite using more expensive models for planning. The savings came from reduced retry loops and fewer human escalations.

Looking Ahead

The 157-agent study reveals a fundamental shift in how we should think about agentic systems. The question is no longer "how fast can the agent execute?" but "how well can the agent plan?" This reframing is driving a new generation of tools focused on planning quality: structured decomposers, validation-aware schedulers, and context-efficient state management.

As the field matures, we'll likely see planning becomes a first-class concern in agent frameworks, with libraries and patterns emerging specifically for the strategic layer. The Orca architecture isn't just a pattern—it's an acknowledgment that thinking carefully is the most important part of the job.

Frequently Asked Questions

Q: Can I implement Orca-style planning without a full multi-agent framework?
A: Yes. Start with a single planner function that decomposes goals and validates each step before execution. You don't need separate agents initially—a well-structured planning loop within a single process captures 80% of the benefit.

Q: How do I choose which model to use for planning vs. execution?
A: Planning benefits from strong reasoning and long context (e.g., GPT-4-class models). Execution can use smaller, faster models optimized for specific tasks. The key is matching model capability to cognitive demand, not cost alone.

Q: What's the minimum viable Orca architecture?
A: Three components: (1) a planner that decomposes goals into sequenced subtasks with success criteria, (2) an executor that runs subtasks and returns structured results, and (3) a validator that checks completion evidence before allowing progression. This fits in under 200 lines of code.

Four Alarm Slots, Three Failure Modes: Building a Nightly Drain That Survives Sleep, Races, and Timeouts

Lily — Mon, 24 Aug 2026 00:00:06 +0000

Every night my Mac quietly rewrites my long-term memory. Not metaphorically — a shell script drains that day's Claude Code conversation logs into an Obsidian vault, commits them to a private repo, and leaves a briefing on my desktop. It took three real outages to make it reliable. This is the script, the three failures, and the design that came out of them.

Why This Setup Works

Claude Code's "memory" disappears by default

Claude Code sessions are independent of one another. The root cause of a bug you found during a long working session today, the reason you settled on a particular architecture after trial and error, the accumulated knowledge that "this direction already failed once" — none of it is available in the next conversation once you close the session.

Even on a paid plan, even with the most capable model available, if context isn't carried over you have to explain everything from scratch every time. Many people have had the experience of thinking "I already looked this up before" or "I should have failed at this once already, and yet here I am heading down the same road again."

In a phase where you're shipping personal projects in volume, this problem is fatal. Once three or four projects are running in parallel, tracking "where each project currently stands" by hand hits a wall fast. And Claude, unable to reference previous conversations, repeats the same deliberations.

The solution is to build an environment, not a task

My first attempt at this problem was "I'll write up a summary by hand every day." It didn't last. When work has momentum you don't feel like writing a summary, and when you're tired you can write even less. A system that depends on human willpower doesn't function during a high-volume solo-dev phase.

The answer was to build an environment that automatically drains Claude's conversation logs into Obsidian every night. Once the environment is in place, willpower and motivation are irrelevant. The Mac just does it.

The reason I chose Obsidian is simple. The files are local Markdown, so Claude Code can read and write them directly. They can be version-controlled with Git. The [[link]] syntax lets you connect pieces of knowledge to each other. Logs flow in every morning and cross-project links grow naturally — from the moment this started functioning as an "external brain," the quality of my work changed.

Why a triple-layered structure became necessary

Simply "running a script at 4:55 every day" produced three distinct kinds of failure once I actually ran it. Each one only became apparent after it caused real damage.

Sleep freeze: If you close the lid on the Mac and go to bed, caffeinate -s (which only takes effect on AC power) can't prevent sleep. The script stops partway through, and that day's processing hangs in limbo until the next slot fires.

Double-execution race: An actual incident on 2026-06-10. A scheduled launchd firing overlapped with a manual run. Both tried to operate on the same vault with Git, and the result was a conflict.

Timeout: On 2026-06-13, every slot failed for the entire day. Digesting 28 hours' worth of logs via claude -p didn't fit within the 40-minute timeout window and all of it got culled. In the logs, the pattern started 04:55:00 → step2 timeout at 05:40:01 lines up across all four slots, completely uniform.

These three real failures are what forced the triple-layered structure of "multi-slot re-firing," "caffeinate sleep prevention," and "idempotent retry via step markers." It didn't come out of a design document; it accumulated from things that actually broke.

The feeling of an environment that grows

Once the system started running stably, logs pile up in Obsidian every morning. Claude reads those logs, updates per-project articles, and this week's talking points accumulate in hot.md (a summary of recent context).

When Claude reads hot.md and wiki/ in the next conversation, it can start working already knowing "last week's decisions," "the approach that failed once," and "the current state of the three projects running in parallel." That's an external long-term memory that doesn't forget when the conversation ends — a prerequisite for getting Claude Code to perform at its actual potential.

The Overall Flow

Four-layer architecture

The data the script handles flows through the following four layers.

【層1】Claude Code セッションログ（セッション終了時にStop hookが書き出し）
           ↓ extract_conversations.py（step1: 最新化）
【層2】~/Documents/my-knowledge-base/raw/conversations/
           ↓ claude -p（step2a: Claude由来ログを消化, timeout 1500s）
       ~/Documents/my-knowledge-base/raw/codex-conversations/
           ↓ claude -p（step2b: Codex由来ログを消化, timeout 1500s）
【層3】Obsidian Vault（~/Documents/claude-obsidian/wiki/）
       ├── hot.md（直近サマリ）
       ├── index.md（全体目次）
       ├── projects/ / learning/ / career/ ... （ドメイン別記事）
       └── today-brief.md（step2.5: 今日の行動提案）
           ↓ git add -A && git commit && git push
【層4】private repo（安全網: 荒れてもrevert可能）

Splitting layer 2 into two lines, "Claude-derived" and "Codex-derived," is a change made on or after 2026-06-11. Originally one line processed everything, but on high-activity days it stopped fitting into the 40-minute timeout window. After the split, each has an independent 1500-second (25-minute) timeout, and if one fails the next slot can retry only what's left.

Implementing multi-slot firing

Four slots defined in a launchd plist (~/Library/LaunchAgents/com.shun.vault-auto-ingest.plist) spend the day repeatedly "retrying until success."

<key>StartCalendarInterval</key>
<array>
    <dict><key>Hour</key><integer>4</integer><key>Minute</key><integer>55</integer></dict>
    <dict><key>Hour</key><integer>8</integer><key>Minute</key><integer>20</integer></dict>
    <dict><key>Hour</key><integer>10</integer><key>Minute</key><integer>45</integer></dict>
    <dict><key>Hour</key><integer>12</integer><key>Minute</key><integer>15</integer></dict>
</array>

RunAtLoad is false. There's no auto-start at login. The four slots are the only firing sources.

At the top of the script it checks for the day's success marker and exits immediately if it's already finished.

TODAY=$(date +%Y%m%d)
DONE_MARKER="$HOME/.claude/logs/.vault-ingest-done-${TODAY}"

# 0. 本日分が既に成功していれば即終了
[ -f "$DONE_MARKER" ] && exit 0

With this, if 4:55 succeeds then 8:20, 10:45, and 12:15 become harmless no-ops that just "check the file and exit." Only if 4:55 fails (sleep freeze, no network, a transient launchd fault, etc.) does 8:20 actually take over the processing.

Sleep prevention with caffeinate

When running this on a MacBook, closing the lid is something caffeinate -s (only effective on AC power) can't fully prevent. So the script re-executes itself under both -i (prevent system sleep) and -s.

if [ -z "${CAFFEINATED:-}" ]; then
  exec /usr/bin/caffeinate -i -s env CAFFEINATED=1 /bin/bash "$0" "$@"
fi

If the CAFFEINATED environment variable isn't set, it re-executes itself under caffeinate and puts everything after that under caffeinate's umbrella. Since CAFFEINATED=1 is set after the single re-execution, there's no infinite loop.

-s has no effect on battery power. If it freezes from lid-close sleep, timeout culls it and the next slot starts over. If step2 got partway through, the markers let it skip what's done and run only the rest.

Preventing double execution with a lockdir

This is a locking mechanism that uses the atomicity of mkdir (if two processes call it simultaneously, only one succeeds). It's a widely used technique for file locking in Bash.

LOCKDIR="$HOME/.claude/locks/vault-auto-ingest.lock"
if ! /bin/mkdir "$LOCKDIR" 2>/dev/null; then
  oldpid=$(cat "$LOCKDIR/pid" 2>/dev/null || true)
  if [ -n "${oldpid:-}" ] && kill -0 "$oldpid" 2>/dev/null; then
    echo "[$(date '+%F %T')] 別インスタンス実行中(pid=${oldpid}) — skip" >> "$LOG"
    exit 0
  fi
  rm -rf "$LOCKDIR"
  /bin/mkdir "$LOCKDIR" 2>/dev/null || exit 0
fi
echo $$ > "$LOCKDIR/pid"
trap 'rm -rf "$LOCKDIR"' EXIT INT TERM

Even when an existing lockdir is present, it checks the PID, and if the process is alive it exits as "another instance is running." If the process is dead (a lock left behind by an abnormal termination), it reclaims it as stale, acquires a new lock, and continues. trap guarantees the lockdir is deleted when the script exits.

Idempotent retry via step markers

This is the part that took the most work. step2 (the ingest into the vault) is split into two lines, Claude and Codex, each with its own independent marker.

STEP2A_MARKER="$HOME/.claude/logs/.vault-ingest-step2a-claude-${TODAY}"
STEP2B_MARKER="$HOME/.claude/logs/.vault-ingest-step2b-codex-${TODAY}"

The ingest_src function looks at these markers to decide "skip if already complete, run if not."

ingest_src() {
  local marker="$1" src="$2" name="$3" to="$4" extra="$5"
  [ -f "$marker" ] && { echo "[...] step2($name) は本日実施済み — skip" >> "$LOG"; return 0; }
  cd "$VAULT" && run_to "$to" "$CLAUDE" -p \
    "...（Vaultのルールに従って wiki/ を更新するプロンプト）..." \
    --dangerously-skip-permissions >> "$LOG" 2>&1 \
    && { touch "$marker"; return 0; } \
    || { echo "[...] WARN: step2($name) 失敗/timeout（次スロットで再試行）" >> "$LOG"; return 1; }
}

ingest_src "$STEP2A_MARKER" "$KB/raw/conversations/" "claude" 1500 ""
ingest_src "$STEP2B_MARKER" "$KB/raw/codex-conversations/" "codex" 1500 "Codex由来でも既存記事に統合し重複は追記でまとめろ。"

[ -f "$STEP2A_MARKER" ] && [ -f "$STEP2B_MARKER" ] && touch "$STEP2_MARKER"

The behavior pattern is as follows.

4:55 発火
├── DONE_MARKER なし → 処理続行
├── step2a 実行（claude由来, 1500s上限） → 成功 → STEP2A_MARKER 作成
├── step2b 実行（codex由来, 1500s上限） → timeout! → マーカーなし
├── brief生成 → 失敗（step2b未完でログが薄い） → DONE_MARKER 作らない
└── notify_fail で Desktop に FAILED ファイル + 通知

8:20 発火
├── DONE_MARKER なし → 処理続行
├── step2a → STEP2A_MARKER あり → skip（再実行しない）
├── step2b → マーカーなし → 実行 → 成功 → STEP2B_MARKER 作成
├── brief生成 → 成功
├── DONE_MARKER 作成 ✓
└── FAILED ファイル削除

10:45 / 12:15 発火
└── DONE_MARKER あり → exit 0（空振り）

If step2a succeeded, 8:20 runs only step2b. If both step2a and step2b are finished, 8:20 runs only the brief. Because which slot is responsible for what is determined dynamically, the guarantee that "even if 4:55 fails, it will definitely finish within the day" is preserved.

Filling in launchd PATH and the node binary

launchd's execution environment has only a minimal PATH, roughly /usr/bin:/bin:/usr/sbin:/sbin. To avoid the trap where the post-git commit hook (written in node) doesn't run and the commit itself fails, the top of the script auto-discovers the newest nvm node and adds it.

NODE_BIN=$(ls -d "$HOME"/.nvm/versions/node/*/bin 2>/dev/null | sort -V | tail -1)
export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH"
[ -n "$NODE_BIN" ] && export PATH="${NODE_BIN}:$PATH"

Taking the tail of sort -V (version-order sort) means it always uses the newest even when nvm has multiple versions installed. Since no version is hardcoded, adding versions with nvm install requires no changes to the script.

Also, due to macOS TCC (privacy protection), launchd can't write under ~/Documents/ unless /bin/bash has Full Disk Access. The script detects this early to prevent silent failure.

if ! ( cd "$VAULT" 2>/dev/null && git rev-parse --git-dir >/dev/null 2>&1 ); then
  echo "[...] ❌ FDA未付与: launchdから '$VAULT' にアクセス不可(TCC保護)。" >> "$LOG"
  notify_fail "FDA未付与: vault にアクセス不可（設定→フルディスクアクセス→/bin/bash）"
  exit 1
fi

Returning exit 0 would look like "it succeeded," DONE_MARKER would be created, and the next slot wouldn't retry. TCC failure is signaled explicitly with exit 1 and handed off to the next slot.

Implementation Details

The timeout helper and designing for a guaranteed kill

One of the script's core decisions is the run_to function.

TIMEOUT_BIN="/opt/homebrew/bin/timeout"
[ -x "$TIMEOUT_BIN" ] || TIMEOUT_BIN=""
run_to() { local s=$1; shift; if [ -n "$TIMEOUT_BIN" ]; then "$TIMEOUT_BIN" --kill-after=30 "$s" "$@"; else "$@"; fi; }

It calls GNU coreutils' timeout by explicit path. macOS's standard /usr/bin doesn't have the GNU version of timeout. In environments where /opt/homebrew/bin/timeout doesn't exist, TIMEOUT_BIN is left empty and calls pass through (hang resistance drops, but it keeps working).

--kill-after=30 matters more than it looks. If the process is still alive 30 seconds after SIGTERM is sent, it's force-killed with SIGKILL. claude -p sometimes ignores SIGTERM while doing heavy work, so without --kill-after you can hit the worst case: "the process survives past the timeout while still holding the lockdir."

step2a (Claude-derived logs) and step2b (Codex-derived logs) each get a 1500-second (25-minute) limit. As mentioned earlier, it was originally a single 2400-second (40-minute) process, but as activity increased it stopped fitting. Splitting into two lines at 1500 seconds each means that if one dies, the next slot can re-run only the remainder.

Waiting for the network after wake, and the preflight detection chain

Even when the Mac wakes at 4:55, it can take tens of seconds for the Wi-Fi connection to stabilize. Both claude and git push need the network, so running before the connection is up is an immediate error.

net_ok=""
for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18; do
  if /usr/bin/nc -z -G 3 1.1.1.1 443 2>/dev/null; then net_ok=1; break; fi
  sleep 5
done
[ -z "$net_ok" ] && echo "[...] WARN: 網未接続のまま続行（失敗時は次スロットが再試行）" >> "$LOG"

18 attempts × 5 seconds = up to 90 seconds of waiting. nc -z -G 3 is a TCP connection test that times out in 3 seconds, lighter than curl. If it still isn't connected after 90 seconds, it "warns and continues." Using exit 1 here would also stop step1 (local processing) in an offline environment. Since the design already assumes the next slot auto-retries on failure, "record it without stopping" is enough here.

Next, preflight detection chains through three stages.

① Checking for the claude binary: This catches the case where an update removes the symlink target. Running claude update can temporarily leave the binary absent, and this is a cause of silent failure.

② macOS TCC preflight: If the vault is under ~/Documents/, a process running from launchd can't write to it without Full Disk Access.

if ! ( cd "$VAULT" 2>/dev/null && git rev-parse --git-dir >/dev/null 2>&1 ); then
  echo "[...] ❌ FDA未付与: launchdから '$VAULT' にアクセス不可(TCC保護)。" >> "$LOG"
  notify_fail "FDA未付与: vault にアクセス不可（設定→フルディスクアクセス→/bin/bash）"
  exit 1
fi

Returning exit 0 here would create DONE_MARKER, and the next slot would misread it as "succeeded" and not retry. TCC failure must always exit with exit 1 and hand off to the next slot. This distinction is also at the root of one of the failure stories below.

Log bloat guard and brief freshness checks

Writing all of claude -p's output to the log gets you to tens of MB within a few weeks.

if [ -f "$LOG" ] && [ "$(stat -f%z "$LOG" 2>/dev/null || echo 0)" -gt 5242880 ]; then
  mv "$LOG" "${LOG}.old"
fi

Once it exceeds 5 MB (5,242,880 bytes), it renames the file to .old and starts a new log. stat -f%z is the macOS file-size command. Since the options differ from GNU stat, || echo 0 provides a fallback.

The brief freshness check is another important part.

START_STAMP=$(mktemp /tmp/vault-ingest-start.XXXXXX)

This temp file is created at the top of the script, and after step2.5 (brief generation) it checks whether the brief is newer than it.

if [ -s "$BRIEF_SRC" ] && [ "$BRIEF_SRC" -nt "$START_STAMP" ]; then
  # アーカイブ処理
  touch "$DONE_MARKER"
else
  notify_fail "ブリーフ生成が未完"
fi

Comparing with -nt (newer than) prevents the accident of "mistakenly archiving yesterday's leftover today-brief.md as today's output." If the brief isn't newer than the start of this run, it isn't treated as a success — DONE_MARKER isn't created and it hands off to the next slot.

Overwriting LLM output with something deterministic

When claude -p updates wiki/index.md, it sometimes writes an incorrect value for "total page count." The model infers it from context, so it doesn't match the actual file count.

real=$(find "$VAULT/wiki" -name '*.md' -not -path '*/.*' | wc -l | tr -d ' ')
sed -i '' -E "s/総ページ数：[0-9]+/総ページ数：${real}/" "$INDEX_FILE"

It recounts the real file count with find and force-overwrites with sed. Because this runs immediately before git commit, what gets committed has the LLM-generated number already replaced by a deterministic value. The design principle is: "numbers output by an LLM get overwritten by a deterministic check downstream."

The calendar last-known-good pattern

When writing "today's schedule" into the brief, it retains past information on days when calendar data can't be fetched.

if [ -n "$CAL_SRC" ]; then
  cp "$CAL_TMP" "$CAL_SNAPSHOT"
  cp "$CAL_SNAPSHOT" "$CAL_LASTGOOD"   # 成功したら last-good を更新
elif [ -s "$CAL_LASTGOOD" ]; then
  # 取得失敗: 前回 good を温存し stale 印を付けて出力
  { tail -n +4 "$CAL_LASTGOOD"; } > "$CAL_SNAPSHOT"
fi

It tries the Google Calendar API (ADC auth) and icalBuddy (local Apple Calendar) in order, and if both fail it uses _calendar-snapshot.md.lastgood from the last successful fetch. The principle is "never clobber the source of truth with empty." Staleness is marked explicitly with a leading ⚠️, and the brief-generation prompt is told to "note that the values are as of the fetch date if it says stale."

Where I Got Stuck

Failure ①: The day every slot died at the same spot, to the second (2026-06-13)

I noticed because FAILED-20260613.md was sitting in ~/Desktop/Daily Brief/. Opening the log, all four slots showed exactly the same pattern.

04:55:00 ===== auto-ingest 開始 =====
05:40:01 WARN: step2(claude) 失敗/timeout（次スロットで再試行）
08:20:00 ===== auto-ingest 開始 =====
09:05:01 WARN: step2(claude) 失敗/timeout（次スロットで再試行）
10:45:00 ===== auto-ingest 開始 =====
11:30:01 WARN: step2(claude) 失敗/timeout（次スロットで再試行）
12:15:00 ===== auto-ingest 開始 =====
13:00:01 WARN: step2(claude) 失敗/timeout（次スロットで再試行）

Every slot timed out at 45 minutes 01 second. The step2 timeout at the time was 2400 seconds (40 minutes). claude -p kept processing for 40 minutes straight and got timed out, the next slot retried with the same result, and that loop repeated four times.

What I suspected first was the error message after git commit. The strings "commit-msg hook" and "node not found" were visible in the log, so I thought the node binary was the cause. But running claude -p "say OK" by hand returned exit 0 immediately. Trying commit by hand worked fine.

The real cause was a transient failure on the claude -p side. 06-12 worked normally; only on 06-13 did all requests hang (probably rate limiting or a temporary service-side fault), and it was resolved the next day. The commit-msg hook theory was completely wrong.

Two lessons came out of this. One: "an error message is a suspect, not a conviction — actually run things to isolate the cause." The other concerns cleanup of FAILED markers. The cleanup at the time only "deleted that day's FAILED file on success." The 06-13 FAILED file would keep sitting on my desk the next day and beyond.

# 修正後: 過去日の FAILED も正常稼働日に掃く
find "$HOME/Desktop/Daily Brief" -maxdepth 1 -name "FAILED-*.md" ! -name "FAILED-${TODAY}.md" -delete 2>/dev/null

Adding this one line means that even if failures occur across multiple consecutive days, all the past days' files get cleared on the first day that runs normally.

Failure ②: launchd and a manual run tried to write the vault at the same time (2026-06-10)

One morning, wanting to check on the script's progress, I also ran vault-auto-ingest.sh manually. launchd's 4:55 slot was already running. Both ran git add -A && git commit on the same vault, and a conflict occurred.

error: cannot lock ref 'refs/heads/main': is at xxx but expected yyy

Because the two collided mid-processing, the vault ended up in a half-finished state and needed a manual reset.

The mkdir lock described earlier solves this, but the key point is "automatic reclamation of stale locks."

if ! /bin/mkdir "$LOCKDIR" 2>/dev/null; then
  oldpid=$(cat "$LOCKDIR/pid" 2>/dev/null || true)
  if [ -n "${oldpid:-}" ] && kill -0 "$oldpid" 2>/dev/null; then
    echo "[...] 別インスタンス実行中(pid=${oldpid}) — skip" >> "$LOG"
    exit 0
  fi
  rm -rf "$LOCKDIR"
  /bin/mkdir "$LOCKDIR" 2>/dev/null || exit 0
fi

kill -0 "$oldpid" sends no signal; it only checks whether the process is alive. If the process is dead, it deletes the stale lockdir and takes the lock itself. This prevents the worst case: "a previous abnormal termination leaves the lockdir behind and every subsequent run skips forever."

Failure ③: The TCC trap — "it says it succeeded but nothing gets updated"

There was a period when I kept the vault at ~/Documents/claude-obsidian/ (later moved to ~/claude-obsidian/, outside TCC protection). The script at the time didn't handle TCC errors properly and exited with exit 0 even when cd "$VAULT" failed.

# 修正前の問題コード（イメージ）
cd "$VAULT" && git rev-parse --git-dir >/dev/null 2>&1 || exit 0
                                                            ^^^^^^
                                        この exit 0 が DONE_MARKER を作る前提を壊す

When the script exits with exit 0, the calling launchd treats it as "terminated normally." But since the same thing happens on the next run too, all four slots end up in a state of "appearing to succeed while doing nothing," with DONE_MARKER never created.

The symptoms were strange. The log recorded "===== 完了 =====" four times. But the vault wasn't updated. There was no DONE_MARKER either.

I noticed the cause from a message that appeared when I ran the script interactively by hand. When run from launchd, /bin/bash lacks Full Disk Access, so cd "$VAULT" was failing silently.

After the fix, TCC failure is signaled explicitly with exit 1, and it creates a FAILED file on the desktop plus a notification.

if ! ( cd "$VAULT" 2>/dev/null && git rev-parse --git-dir >/dev/null 2>&1 ); then
  notify_fail "FDA未付与: vault にアクセス不可（設定→フルディスクアクセス→/bin/bash）"
  exit 1   # DONE_MARKER は作られない → 次スロットが再試行
fi

The distinction between exit 0 and exit 1 is what separates "it succeeded" from "the next slot takes over." In an automation context, a silent fake success is the hardest failure mode to diagnose.

Failure ④: Trusting `caffeinate -s` too much

The first implementation put only caffeinate -s at the top of the script.

# 初期の誤った実装
caffeinate -s "$0" "$@"

The -s flag is described as "prevent system sleep." But in practice it only works while connected to AC power. Close the lid on battery and macOS goes to sleep, freezing the script under caffeinate -s along with it.

Checking the next morning, there were traces that the script had started but it had ended in a half-finished state. Because timeout culled it afterward the lockdir was gone, but step2a's marker existed while step2b's did not.

The fix was to switch to using both -i (prevent system sleep) and -s.

if [ -z "${CAFFEINATED:-}" ]; then
  exec /usr/bin/caffeinate -i -s env CAFFEINATED=1 /bin/bash "$0" "$@"
fi

-i is effective on battery too. That said, there are situations where lid-close sleep (the suspend equivalent, not display sleep) can't be fully prevented even with -i. The multi-slot structure — "timeout culls it and the next slot retries the remainder" — is the design that assumes this. caffeinate is only an aid that delays sleep; what guarantees "the processing will complete" is the combination of markers and multiple slots.

The actual total of "time spent stuck"

Sorting through these failures, all of them stayed within one day of real damage. The sleep freeze was recovered by the next 8:20 slot. The double-execution race took 10 minutes of manual reset. The all-day outage on 06-13 auto-recovered on the next morning's 4:55 slot. The TCC trap took a few days to notice, but once noticed the fix took under 30 minutes.

The reason I could notice any of these failures is the logs and the FAILED file on the desktop. Had they failed silently, I might not have noticed that the vault hadn't been updated for days. The design of having notify_fail place a FAILED file on the desktop exists to maximize the chance of noticing. Automation isn't "run it and you're done" — it only really works once you've designed it up through "you'll reliably notice when it breaks." That's my honest takeaway from six months.

Gotchas

The earlier sections covered four real failures (all slots dying, the double-execution race, the TCC trap, misplaced faith in caffeinate). Here I'll cover the smaller points that are "not that big, but you will definitely trip on them."

The combination of set -u and launchd environment variables kills you instantly

The script has set -u at the top (it's in the actual code). It's the option that makes referencing an undefined variable exit 1 immediately. A shell running from launchd, unlike an interactive shell, has many environment variables undefined. If you reference a variable without giving it a default via the ${VAR:-} form, you get a phenomenon where it's fine in interactive runs but dies only under launchd. In this script, ${CAFFEINATED:-}, ${oldpid:-}, and ${net_ok:-} all explicitly specify an empty default. Miss one and you mass-produce unexplained exit 1s.

Forget to set RunAtLoad and it runs at every login

The plist explicitly has <key>RunAtLoad</key><false/>. Without it the default becomes true, and the script fires one extra time every time you restart or log in to the Mac. Since it exits immediately when DONE_MARKER exists there's no real harm, but the CPU load right after boot and the unintended log pollution accumulate.

Without trap, the lockdir stays behind and every subsequent run skips

trap 'rm -rf "$LOCKDIR"' EXIT INT TERM

Without this one line, the lockdir remains when the script is stopped with Ctrl+C or kill. At the next firing, trying to acquire the lock with mkdir finds the existing lockdir and misjudges it as "another instance running," so it skips. But if you've written code that checks the PID, stale locks are reclaimed automatically. Without stale reclamation, it skips forever.

Running late at night, the date rolls over and two DONE_MARKERs get generated

If the 4:55 slot starts running at 0:10 AM (via a manual start, say), date +%Y%m%d changes mid-processing and the DONE_MARKER filename changes with it. To prevent this, the script pins the date as of the start of the run with BD="$TODAY".

BD="$TODAY"  # 日付跨ぎ対策: 評価はラン開始時の日付で固定
ARCH_FILE="$VAULT/wiki/briefs/daily/today-brief-${BD}.md"

TODAY is obtained exactly once at the top of the script. Even for long-running processing that crosses midnight, filenames stay unified under "the day the run started."

stat -f%z is a macOS-only option

This is the log rotation code.

[ "$(stat -f%z "$LOG" 2>/dev/null || echo 0)" -gt 5242880 ]

On GNU stat (Linux) it's -c%s. Since this is written assuming macOS it uses stat -f%z, but if you try to port it to Linux it silently falls through to echo 0 and never rotates. The || echo 0 fallback means it doesn't break, but the log keeps growing.

Without launchd's StandardErrorPath, stderr vanishes

The plist contains the following.

<key>StandardErrorPath</key>
<string>/Users/.../.claude/logs/vault-auto-ingest.launchd.log</string>

Without this, stderr from processes run via launchd is discarded to the equivalent of /dev/null. Even if step2's claude -p is writing something to stderr, you'll never see it. That creates a hard-to-diagnose situation where it's visible in interactive runs and invisible under launchd.

A node-based commit-msg hook dies without PATH augmentation

Running git commit in the vault executes the commit-msg hook. If that hook is written in node, launchd's minimal PATH (roughly /usr/bin:/bin) has no node. git commit stops with a "hook execution failed" and finishes without committing anything. The script auto-discovers nvm's node and adds it to PATH.

NODE_BIN=$(ls -d "$HOME"/.nvm/versions/node/*/bin 2>/dev/null | sort -V | tail -1)
[ -n "$NODE_BIN" ] && export PATH="${NODE_BIN}:$PATH"

The trick is taking the tail of a version-order sort with sort -V. With a string sort you get the inversion where v9.x sorts after v10.x.

Without spelling out "merge into existing articles" in step2b's prompt, you mass-produce duplicate articles

step2a (Claude-derived logs) runs first and updates the vault articles. When step2b (Codex-derived logs) runs afterward, it tries to create new articles on the same topics. Without explicit instructions in the prompt, articles with the same content split into projects/foo.md and projects/foo-2.md.

ingest_src "$STEP2B_MARKER" "$KB/raw/codex-conversations/" "codex" 1500 \
  "Codex由来でも舜の知識として既存記事に統合し、Claude側と重複する話題は新記事を作らず追記でまとめろ。"

This additional prompt (the extra argument) is passed only to step2b. The instruction to "merge and append" works because it comes after seeing step2a's results.

Forget to delete the START_STAMP used for the -nt comparison and the brief is always judged "old"

START_STAMP=$(mktemp /tmp/vault-ingest-start.XXXXXX)

This temp file is deleted when the script exits. But if it remains in /tmp, it isn't created under the same name on the next run, and comparison against an old START_STAMP can judge today-brief.md as "old," entering a loop where DONE_MARKER is never created. The rm -f "$START_STAMP" at the end of the script is essential.

Marker files left for more than 7 days bloat /tmp

find "$HOME/.claude/logs" -maxdepth 1 -name '.vault-ingest-*' -mtime +7 -delete 2>/dev/null

Leave the daily-generated marker files alone and you accumulate 365 files a year in ~/.claude/logs/. -mtime +7 periodically deletes anything older than 7 days. This cleanup line is in there as a memento.

Best Practices

Twelve operating principles that solidified over six months of actually running this.

① Separate "success" from "failure that hands off to the next slot" via exit code

This is the lesson from the TCC incident. Returning exit 0 creates DONE_MARKER and the next slot won't retry. Genuine failures must always be exit 1. "Silent fake success" is the hardest-to-diagnose form of failure in automation.

② Create DONE_MARKER only after all steps are complete

If you create DONE_MARKER when step2a completes, "today is done" becomes true even with step2b and the brief unfinished. In this script, DONE_MARKER is created only when the brief is newer than START_STAMP. The ordering principle: raise the flag only after the final artifact has been produced.

③ Use intermediate markers to enable partial retry

Rather than a simple two-state "done/not done," give step2a and step2b each a half-marker. If step2a succeeded, the next slot skips step2a and starts from step2b. The more steps there are, the more valuable partial retry becomes.

④ Treat caffeinate as an aid that delays sleep, and accept that

Even using both -i and -s, suspend on a lid-closed MacBook can't be fully prevented. caffeinate is best-effort. What guarantees "it will definitely complete" is markers and multiple slots. A design that leans too hard on caffeinate will always break under battery operation.

⑤ Derive the number of slots backward from "even the slowest failure finishes within the day"

The four slots (4:55 / 8:20 / 10:45 / 12:15) are designed from the view that "if it finishes by noon, it's usable that day." Even if 4:55 fails, succeeding at 12:15 at worst still gives you a brief based on that day's activity logs. The next morning's firing is for processing the next day's material.

⑥ Always attach --kill-after=30

run_to() { "$TIMEOUT_BIN" --kill-after=30 "$s" "$@"; }

It sends SIGKILL 30 seconds after SIGTERM. claude -p sometimes ignores SIGTERM while doing heavy work. Without kill-after, you end up with "the process survives past the timeout while still holding the lockdir."

⑦ Overwrite LLM-output numbers with something deterministic downstream

real=$(find "$VAULT/wiki" -name '*.md' -not -path '*/.*' | wc -l | tr -d ' ')
sed -i '' -E "s/総ページ数：[0-9]+/総ページ数：${real}/" "$INDEX_FILE"

The "total page count" Claude writes into wiki/index.md is a guess. Count the real files and overwrite. Because this runs right before git commit, the repository always has the measured value committed. The design principle: never treat LLM-generated numbers as authoritative.

⑧ Retain the previous value even on fetch failure (last-known-good)

Calendar information has a three-stage fallback: Google Calendar API → icalBuddy → lastgood. Not "it couldn't be fetched, so blank it," but "retain the value from the last success, marked stale." The brief prompt is instructed to call it out explicitly if ⚠️stale is present. Stale information is more useful than none.

⑨ Put failures somewhere visible

FAILED_FILE="$HOME/Desktop/Daily Brief/FAILED-${TODAY}.md"
notify_fail() {
  mkdir -p "$HOME/Desktop/Daily Brief"
  { echo "# Daily Brief 生成失敗 — ..."; echo "- 自動再試行: 8:20 / 10:45 / 12:15 ..."; } > "$FAILED_FILE"
  /usr/bin/osascript -e "display notification ..."
}

Log files are things you don't notice unless you go look. Putting a FAILED file on the desktop and firing a macOS notification puts "the automation is broken" in front of your eyes. And on success, that file is auto-deleted. A file that "exists only while failing" is proof of normal operation.

⑩ Clean up past days' FAILED files too

This one line was added while cleaning up after the consecutive 06-13 failures.

find "$HOME/Desktop/Daily Brief" -maxdepth 1 -name "FAILED-*.md" ! -name "FAILED-${TODAY}.md" -delete 2>/dev/null

With a design of "delete only that day's file on that day's success," FAILED files keep piling up on your desk after several consecutive days of failure. Design it so that the first normally-operating day deletes all the past days' files.

⑪ Auto-discover the nvm node version instead of hardcoding an absolute path

The launchd plist's EnvironmentVariables hardcodes the v24.13.0 path (line 8 of the reference file). But the node discovery inside the script isn't pinned to an absolute version — it takes the newest via sort -V | tail -1. If you want to rely on the plist's fixed path, you need to update it every time you run nvm install. Using the script's dynamic discovery means no changes are needed after a version update. Be conscious of which one you're treating as the source of truth.

⑫ "It works" and "you'll notice when it breaks" are separate design problems

The first version only thought as far as "it works." When it broke, I might not have noticed that the vault hadn't been updated for days. Deliberately design multiple detection paths: the FAILED file, the notification, and noticing that the next day's brief is stale. Automation only becomes trustworthy once you've designed it through to "you'll reliably notice when it breaks."

Summary

Let's look back at the triple-layered structure of vault-auto-ingest.sh.

Multi-slot re-firing gives you the guarantee of "retrying over the course of a day until it succeeds." The design starts at 4:55 and runs as late as 12:15 in the worst case, but on most days it finishes at 4:55 or 8:20. The remaining two slots become no-ops that check DONE_MARKER and exit immediately.

Sleep prevention via caffeinate -i -s is an aid that prevents the process from freezing partway through. It can't fully counter a lid-closed battery-powered Mac, but a frozen run gets culled by timeout and the next slot takes over. The combination of caffeinate and the marker structure establishes a structure where "even if part of it dies, it moves forward."

Idempotent retry via step markers is the most important part. step2a and step2b each hold independent markers, and completed steps are never re-run. Splitting processing that didn't fit in a 40-minute window into two 25-minute lines means that if one dies, the next slot runs only the rest. This turns "partially timed out" into "partially finished."

What changed once this ran stably every morning? When Claude Code reads hot.md the next morning, it can start working already knowing yesterday's decisions, the reason an architecture failed once, and the current state of three projects running in parallel. Explaining everything from scratch every time becomes unnecessary. Management cost doesn't grow even when parallel projects reach three or four.

The value of automation shows not "when you get it running," but "when it recovers on its own after breaking and keeps running so routinely you don't notice it."

Written by **Lily* — I ship iOS apps and automate my content stack with Claude Code.
Follow along: Portfolio · X · GitHub*

My Caption Width Guard Passed Every Test. It Was Measuring Text the Renderer Never Drew.

John — Mon, 24 Aug 2026 00:00:06 +0000

Originally published on hexisteme notes.

A user complaint sent me into a caption pipeline: "the subtitles cut to two words in places where the sentence doesn't make sense." The fix I shipped for that complaint introduced a second bug, one word narrower and easy to miss, because the code that measured whether a line of text would fit reproduced an assumption about the text that the code drawing the line didn't share. Every test passed the whole time. I only found it by watching the rendered video.

The bug the complaint pointed at

The captioning system splits a transcript into short chunks that pop onto screen a few words at a time. The chunking function was doing fixed-size slicing — take the next N words, regardless of what came before or after. That's blind to sentence boundaries, so two unrelated sentences could land in the same chunk: loss. Today reads as one visual unit even though it's the tail of one sentence and the head of the next.

The fix was a rule set, not a single tweak:

hard break after terminal punctuation (. ! ? …)
soft break at commas, semicolons, and em-dashes
extend or push a chunk rather than let it end on a function word (of, the, than, is, and about thirty others)
target three words per chunk, four as a ceiling
a pixel-width cap on the rendered chunk, measured against the actual caption font (Montserrat ExtraBold), with a budget of 1080 × 0.92 = 993.6px

The first four rules are about where a line is allowed to break. The fifth is a physical constraint: however good the break points are, a chunk still has to fit on screen at the font size actually in use. That's the one that went wrong.

What the width guard actually measured

To get the pixel width of a candidate chunk, the guard rendered the chunk's text through the font and measured the result — which is the correct approach in principle, not a shortcut. Text width isn't a fixed number of pixels per character; it depends on the specific glyphs, so measuring the real string through the real font is the only way to get an honest number.

The chunk text going into that measurement, though, was uppercased first. Elsewhere in the same video pipeline, an unrelated overlay — hook text shown at the very start of a clip — is deliberately rendered in all caps for a different visual style, and the width-measurement code for the caption line borrowed that same uppercasing step. It reads like a defensible move if you don't check it against what's actually drawn: uppercase glyphs in this font run wider than mixed case, so measuring in uppercase gives you a safety margin — worst case, real width can only come in narrower than what you measured, never wider. Except the caption itself is never rendered in uppercase. The word-pop captions on screen keep the original sentence casing. The margin wasn't a safety margin against a real risk; it was padding for a risk that doesn't exist in this code path, and the padding was large enough to distort the outcome.

The measurement was accurate. The input was wrong.

Take a concrete chunk: "rich with nitrates." Rendered as written, mixed case, it measures 842px. Rendered uppercase — the string the guard actually checked — it measures 1042px, about 20% wider. The budget is 993.6px. The real string clears the budget with room to spare. The string the guard tested against does not.

So the guard did its job on the input it was given, and the input was wrong. Chunks like "rich" and "with nitrates." got forced apart into one-word fragments to satisfy a width limit that the actual on-screen text was never close to violating. That's a regression relative to the bug this whole rule set was supposed to fix — a one-word chunk is a more broken reading experience than the original two-sentences-in-one-chunk problem, just distributed differently across the caption track.

It's worth being precise about what was and wasn't broken here, because it would be easy to walk away distrusting the measurement machinery itself. I checked that separately: the library's text-path measurement and the actual rendered glyph width from the graphics library's own get_window_extent() call agreed to within 1% on the same string (841.6px vs. 849.0px). The measurement tool was fine. The 20% gap came entirely from feeding it a transformed version of the string that the render path never draws.

Why green tests didn't catch it

This shipped with every existing test passing. That's not a gap in test coverage in the usual sense — the width-cap logic almost certainly had a unit test asserting that oversized text triggers a split, and that test presumably still passes today, because the function under test does exactly what it's told: given uppercased text and a budget, it correctly decides whether that uppercased text fits.

The thing no test was checking is whether the uppercased text is the text that gets drawn. That's not a property of the width function in isolation — it's a property of the relationship between two different parts of the pipeline, the measurement path and the render path, and nothing forced those two to agree because nothing compared them. A unit test that only exercises the measurement function, with the transformation already baked into its test fixtures, will happily stay green forever while that transformation drifts away from reality.

What actually caught this was watching a rendered frame. Not a screenshot diff against a golden image — just a person looking at the output and noticing captions breaking somewhere they shouldn't. There was no automated signal pointing at the bug; there was only the gap between what the pipeline produced and what a human expected to see.

What generalizes

The specific failure — uppercasing text before measuring it, then rendering it mixed-case — is narrow. The shape underneath it isn't.

Any time verification code needs to know something about what a production code path is going to output — its width, its length, its normalized form, its encoding — there are two ways to get that information. One is to call the actual function that the render path calls, so the two are structurally guaranteed to agree. The other is to reimplement an approximation of what that function does, in the verification code, separately. The second option is often faster to write and easier to review in isolation, and it works right up until someone changes an assumption on one side without noticing it needs to change on the other. Case transformation, whitespace normalization, trimming rules, encoding — all of these are exactly the kind of small, easy-to-duplicate, easy-to-forget-you-duplicated logic that drifts.

Two things reduce the risk, and they're not mutually exclusive. First, share the transformation function itself between the render path and the measurement path, so there's one implementation of "what text actually reaches the screen" instead of two that are supposed to agree. Second — for anything a shared function doesn't cover, or as a backstop even when it does — sample the actual rendered artifact and check it, rather than trusting that a green test suite implies the output looks right. A pipeline nobody looks at is a pipeline where this kind of drift can live for a long time before anyone notices, because nothing in the test suite is positioned to notice it either.

An earlier note on this site covered the mirror-image failure — a measurement that was accurate but taken at a point where two hypotheses agreed. This one is closer to home: the measurement tool was accurate too, and the failure was upstream of it, in what got handed to it as input.

More notes at hexisteme.github.io/notes.

ESP32 + Python: From Microcontroller to IoT

Janada Pelahara — Sun, 23 Aug 2026 23:34:39 +0000

ESP32 + Python: From Microcontroller to IoT
Artcal 0: Introduction

When it comes to transferring data from one place to another, things can sometimes become tricky, especially when communication happens between the hardware and software levels.

In this article series, I would love to share the experience and knowledge I’ve gathered while working with ESP32 and Python. We’ll explore how these two technologies can work together, starting from the basics and gradually moving towards more interesting and practical projects.

If you have any questions, suggestions, or ideas along the way, feel free to share them in the comments section below. I’d love to hear from you and discuss them with the community.

So, without further ado, let’s begin! 🚀

What is ESP32?

Think about Esp32 as a microcontroller with Internet facilities, consisting WiFi, Bluetooth and a own wireless data transfer protocol called ESP-NOW between ESP32 chips. Nowdays, the developers have made development boards integrading these chips for the easy use.

ESP32 is a family of microcontrollers developed by Espresiff. This can read sensor inputs, process data, contol devices and specially connect to the internet. This is like Arduino but better, faster and smaller. With these information that we have, we can speak about this board as,

"A powerful microcontroller that can interact with electronic components and communicate with other devices through Wi-Fi, Bluetooth, and other communication methods."

Python???
We use different languages to tell the same thing but in different ways. We use programming languages to tell the computer the same thing but in different approches. Some languages can be hard to learn and some are easy. Some are well developed and some are not.

Python programming language was created back in 1980s by Guido Van Rossum, with the development begining around 1989. It was publicly released in Feb, 1991.

🐍 1989 — Guido van Rossum developing Python.
🐍 1991 — The first public release.
🐍 2000 — Python 2.0.
🐍 2008 — Python 3.0 .

Python is a high-level, general-purpose programming language known for its simple and readable syntax. This allows programmers to write programs that can perform tasks, process data, and interact with other software, devices, and systems. It is widely used in areas such as web development, automation, data analysis, artificial intelligence, and hardware communication._

IoT
We interact with objects every day in order to complete our tasks. They all are connected to prefome a task. We're the onece who connect them together.
IoT simply refers to,

"Internet of things"

This refers to physical devices that can collect, exchange, and sometimes process data over a network(usually the internet).

In simple terms, IoT is about connecting everyday objects to the internet so they can connect, communicate and perform tasks automatically.

The Combo
The combination of these technologies are always like a cup and the tea.

The combination of ESP32 and Python brings together hardware and software, allowing them to work together to create interactive and connected applications. The ESP32 can collect data from sensors, control electronic components, and communicate over networks, while Python can receive, process, analyze, and display that data. Together, they provide a powerful foundation for building IoT projects and other hardware-software applications.

An ESP32 can read data from a temperature sensor and send that data over Wi-Fi to a computer or a server. Then they can process and get an action with those information as needed. This is an example of an IoT application.

Conclusion
We learned about ESP32, Python, IoT

In this article, we introduced the two main technologies we’ll be working with throughout this series: ESP32 and Python. We looked at what they are, what they can do, and how combining hardware and software can open up many possibilities for IoT and other practical projects.

This is only the introduction towards a greater communication. From next article onwards, we’ll start getting our ESP32 and Python knowledge with theory & practical examples. This is just our first steps toward making them communicate with each other.

We'll meeeeet soooooooooooon! 🚀

Written By Janada Pelahara

99% token accuracy, zero learning. Field notes from fine-tuning vision models with RL.

Rickesh T N — Sun, 23 Aug 2026 23:33:17 +0000

Over the past year I have been fine-tuning open vision-language models - 9B dense up to a 35B mixture-of-experts - with supervised fine-tuning and GRPO-style reinforcement learning on verifiable rewards. Most of what I learned was not about algorithms. It was about the ways a training run can look healthy while doing nothing, or crash for reasons that have nothing to do with your code.

Three failures, in increasing order of how long they fooled me.

Failure 1: the metric that measured the wrong thing (18 hours)

I ran an 18-hour supervised fine-tune that reported token accuracy climbing steadily to 99%. Looked like a textbook run. The real evaluation metric - accuracy on multiple-choice questions - never moved.

The cause was a mismatch between what I supervised and what I evaluated. The training loss was over free-text reasoning traces; the evaluation scored a single extracted answer letter. The model got extremely good at reproducing the shape of the training text - hence 99% token accuracy - without that transferring to the decision I actually cared about.

Token accuracy is a proxy, and proxies drift from the target exactly when you stop checking. The fix was structural, not a hyperparameter: supervise the thing you evaluate. If the deliverable is a constrained answer, the training signal has to reach that answer, not just the prose around it.

The general rule I took: any training metric that is not your evaluation metric is a hypothesis about correlation, and you should check that correlation before you spend GPU-days on it.

Failure 2: the crash that was two libraries disagreeing about position ids

The GRPO trainer for the 9B vision model crashed in the forward pass, deep inside rotary position embedding code. Nothing in my training code had changed.

The diagnosis took a while because the bug lived at the boundary between components: the text sequence length was derived from token-type ids, while the vision sequence length came from the image grid - and image-pad tokens ended up counted twice. Two parts of the same stack, each internally consistent, disagreeing about how long the input was.

For the 35B MoE variant of the same family, an equivalent rope bug was fixable by monkeypatching the model's position-id computation. I shipped the patch with a GPU-free regression test: a tiny script that constructs the exact failing input shape and runs just the position-id path on CPU. It runs in seconds, needs no cluster, and fails loudly if an upstream update reintroduces the bug.

Two lessons. First, when you fine-tune at the edge of a model family's tooling support, the bugs you hit are integration bugs, and the stack trace points at the victim, not the culprit. Second, every monkeypatch deserves a regression test that costs nothing to run - otherwise the next library upgrade silently un-fixes it.

Failure 3: the RL loop that was learning the opposite (the quiet one)

In a separate project I fine-tune a 9B model with reinforcement learning where the reward comes from realized real-world outcomes rather than a labelled dataset. For a long stretch the training signal was flat - not diverging, not collapsing, just flat, which is the least informative failure there is.

Two compounding problems. One was label noise in the reward pipeline: some outcomes were being attributed to the wrong decisions, which dilutes any gradient. The other was worse: a sign error meant part of the advantage signal was inverted. The model was being gently pushed away from behaviour that had worked.

Nothing crashed. Every batch processed. Every log line looked like a training run. The only symptom was the absence of learning, and the only way I found it was working backwards from "the held-out metric should have moved by now" to auditing every stage of the reward computation by hand.

After both fixes I got the first genuinely monotonic learning curve on that task. I still treat it as training signal only - the decider is held-out evaluation against the base model, and I do not report improvements that exist only in the training curve.

The harness rules I now run everything under

These came out of the failures above plus a benchmarking programme across 70+ vision-language models. They are boring, and they are the difference between numbers and noise.

Smoke test before committing compute. A five-step GRPO run with two numbers watched: the PPO-style clip ratio and the fraction of outputs that parse. If the clip ratio is degenerate or parseability is low, the full run will be garbage in a way five steps already reveals.

Runs are gated fail-closed. Nothing publishes a result unless the evaluation stage actually scored. "The eval crashed but training finished" is not a result; it is an unscored run, and unscored runs must be impossible to mistake for scored ones.

Infrastructure failures and poor performance are different columns. An unparseable output, an OOM, a crashed kernel - these are exceptions whose count must be exactly zero. A weak model produces zero exceptions and simply scores badly. If a quality threshold can absorb an infrastructure failure, a totally broken run can pass your gate.

The held-out metric is the only decider. Training curves, token accuracy, reward trends - all of it is telemetry. If the held-out number did not move, nothing happened.

None of this is novel. All of it is the difference between the runs I trust and the 18 hours I lost.

I write about ML evaluation, world models, and the ways measurement quietly fails. More at dev.to/rickeshtn.

Log Level Design and Rotation: Why Apps Use DEBUG/INFO/WARNING/ERROR

Susumu Takahashi — Sun, 23 Aug 2026 23:32:27 +0000

Open any application log and you'll see the same kind of message tagged with different labels: DEBUG, INFO, WARNING, ERROR. Why not just write down everything that happens, in one uniform stream? This post looks at what log levels actually do, and at the companion problem every long-running app eventually faces: keeping a log file from growing forever (rotation).

A Log Level Is a Filtering Threshold

Note: logging means recording what happened while a program runs, so it can be reviewed later in a file or on screen.

Python's standard logging module defines five levels:

Level	Numeric value	Meaning
`DEBUG`	10	Fine-grained detail for tracing exactly what the code did
`INFO`	20	Normal progress record
`WARNING`	30	Something unexpected, but processing continues
`ERROR`	40	An operation actually failed
`CRITICAL`	50	The application itself can no longer continue

The key point is that this number isn't just a label — it's a threshold used for filtering. Set logger.setLevel(logging.INFO) and only messages at 20 or above (INFO, WARNING, ERROR, CRITICAL) get written out; anything at DEBUG (10) is silently dropped. In other words, level design isn't only about deciding what to record — it's about being able to dial the visible detail up or down later without touching the code. In normal operation you watch INFO and above; when something goes wrong, you temporarily drop the threshold to DEBUG to see the fine-grained trace.

How This App Actually Configures It

At the top of maintenance_agent.py, log output is routed to two handlers:

_rotating_handler = logging.handlers.RotatingFileHandler(
    "maintenance.log",
    maxBytes=10 * 1024 * 1024,  # 10 MB
    backupCount=5,
    encoding='utf-8'
)
_stream_handler = logging.StreamHandler()

logging.basicConfig(level=logging.INFO, handlers=[_rotating_handler, _stream_handler])

Because the threshold is logging.INFO, DEBUG-level messages produce no output during normal operation. Across the codebase there are 19 logger.debug() calls — statements written to stay quiet by default and only become useful once someone deliberately lowers the threshold during an investigation. That's the practical payoff of level filtering: you can leave detailed diagnostic statements embedded in the code permanently, without needing to add them later, and without them cluttering the log under normal conditions.

What the Actual Level Distribution Reveals

Counting calls across the same codebase gives:

INFO: 135 calls
WARNING: 114 calls
ERROR: 28 calls
CRITICAL: 0 calls

INFO dominates because the app processes multiple WordPress sites in sequence during a maintenance run, and each step of that sequence needs a progress record. WARNING is the next largest category, and a good example is in core/alert_utils.py, where send_alert_email() detects incomplete SMTP settings:

if not all([settings.get('smtp_host'), settings.get('smtp_user'), settings.get('to_email')]):
    logger.warning(t(
        "メール送信設定が不足しているためスキップします。",
        "Email settings incomplete. Skipping notification."
    ))
    return False

Notice this uses logger.warning, not logger.error. Even if the notification email can't be sent, the rest of the maintenance run — backups, updates, rollback decisions — can still proceed; nothing has actually failed. ERROR is reserved for cases where an operation genuinely did fail (a WP-CLI update command erroring out, an exception during mail delivery, and similar), which is why it's the smallest of the three at 28 occurrences.

CRITICAL doesn't appear at all, and that's not an oversight — it follows from how the app is designed. Each site is processed independently: if one site's maintenance run fails, that site alone gets rolled back and the run moves on to the next site. There's essentially no scenario where the whole application needs to be treated as unable to continue. On top of that, urgent notifications to the user aren't handled through the log level at all — they go through a separate channel, send_alert_email(). Deciding what gets written to the log and deciding what the user needs to be told are two different concerns, and the second one lives in application logic, not in log-level plumbing.

Rotation: Making "Keep Recording Forever" Safe

Logs get more useful the longer they accumulate, but an unbounded log file will eventually fill the disk. RotatingFileHandler solves this:

_rotating_handler = logging.handlers.RotatingFileHandler(
    "maintenance.log",
    maxBytes=10 * 1024 * 1024,  # 10 MB
    backupCount=5,
    encoding='utf-8'
)

maxBytes=10 * 1024 * 1024 triggers a rotation once the file reaches 10MB; backupCount=5 caps how many rotated generations are kept. When the active file hits the size limit, it's renamed maintenance.log.1 and a fresh empty file takes over. The next time the limit is hit, maintenance.log.1 becomes maintenance.log.2, and so on — once a generation would exceed backupCount, it's deleted.

That fixes the maximum disk footprint at 10MB × (1 + backupCount) — about 60MB in this app's case — no matter how long the process keeps running. For a desktop app meant to stay running over long periods, this single mechanism satisfies two requirements at once: the log can never grow without bound, and a meaningful amount of recent history is still available whenever something needs investigating.

A Second Handler for Logs That Shouldn't Persist

There's one more custom handler in this app, _SiteLogCapture:

class _SiteLogCapture(logging.Handler):
    """Temporarily captures the log output for a single site's run."""
    def emit(self, record):
        self.lines.append(self.format(record))

Its purpose is entirely different from the rotating file handler. It gets attached to the logger only while one site's maintenance run is in progress, collecting just that run's log lines into an in-memory list. Once the run finishes, that captured text becomes the "execution log" section of the white-label report or notification email, and the handler is detached again.

Two handlers can be attached to the same logger and treat the exact same stream of messages completely differently. The rotating handler exists to preserve the app's history long-term under a fixed size cap; _SiteLogCapture exists to gather one run's output ephemerally, for one specific downstream use, and then discard it. If log levels are the "vertical" filter — how much detail to keep — handler choice is the "horizontal" filter: which audience, and which purpose, a given record ends up serving.

Takeaway

A log level isn't a static classification tag; it's a threshold you can adjust after the fact. DEBUG stays hidden by default, INFO records normal progress, WARNING marks recoverable anomalies, ERROR marks actual failures, and CRITICAL marks a failure the application itself cannot survive — choosing deliberately among them means the resulting distribution of log calls ends up reflecting the application's actual design assumptions. Rotation, meanwhile, is the practical mechanism that lets "keep recording indefinitely" and "don't run out of disk space" coexist, simply by fixing a size and a generation count up front.